How to configure environment for Python Poetry based project.

How do I get started?

I recently switched over to Poetry as a package manager for my project CfnMason. ( CfnMason is a tool related to CloudFormation stack management, but you can read about that on the Readme as it is updated. ) The question is, how do you use Poetry when you are working on a project across multiple machines and operating systems. I guess I am going to attempt to address the issue.

So, you want to join a project, or work on a project that is using the Poetry dependency management tool. Great! But, how do you get the requirements setup for the project so that you can start working on it? How do you know which version of Python to use, which packages to install, how to build the project, or how to run the test suite?

This is an issue that I was facing, but it was not with another project, but my own as I was switching between machines. Now, I do know some of the answers to the questions above, but I was still stuck as to how to setup a project on another machine. As such, I decided to walk through the process of coming onto a new project and determining how to work with it. The project that will be used for this walk-through will be CfMason, at tool that manages some aspects of building and deploying CloudFormation stacks on AWS.

Hopefully the project that you are working on has a Readme file. Though, to be fair, documentation is hard, and is often the last thing that is added to a project. If it does, you should be able them, but if they are not provided, then the following steps are the way that I would go about working on a project that uses Poetry for dependency management. Oh, and as a note. I am making the assumption that you already have Poetry installed.

Steps to work on Poetry based Project

  1. Determine that the project is using Poetry
  2. Check the version of Python that is needed
    1. Validate local Python version
    2. Install if missing
  3. Create a virtual environment for building the application
    1. venv — for packaging and validation
    2. venvdev — for building the package
  4. Install project dependencies
  5. Build the project
  6. Run tests

Determine the project is using Poetry.

If the ReadMe does not tell you that the project is using poetry, then there is a quick way to find out.

  • Look for the file pyproject.toml in the base of the project.
  • Open the file and look for the following line
    • [tool.poetry]

Provided you are able to find this file, and line, then the project is using Poetry.

Determine the version of Python

One nice thing about Poetry is that it has a defined location to identify the version of Python. I am a big fan of this, as the difference between different versions can cause major problems. Take for example that reserved keywords changed a bunch between 3.6 and 3.7. The steps to follow are as follows.

  • Open the toml project file pyproject.toml
  • Find the supported python versions in the [tool.poetry.dependencies]
  • Find the line starting with python, and find the supported versions.
    • Ex. python = "^3.6"
  • If you don’t have that version of Python on your system, install it.

Setting up a local Virtual Env

I am a huge proponent of using a virtual environment for each application that I am working on. In some cases, I will have 2, one for including all the development modules, and one for just the ones needed for the application to run. Since Python 2 is pretty much EOL, I am not going to spend any time on how to setup a virtual environment for Python2. Instead, this is all dedicated to Python3. And I can only guarantee this on Python 3.6 or later.

foo$ python -m venv venv-dev
foo$ python -m venv venv
foo$ ls -ld venv*
drwxr-xr-x 1 foobar 197610 0 Nov 24 17:02 venv/
drwxr-xr-x 1 foobar 197610 0 Dec 25 14:19 venv-dev/

Install project dependencies

For this last part, you need to activate either of the Python Virtual Environments and then run the install code from there. This is only if you really want to install it both ways. If not, then you can just create a single virtual env directory and just install all the dependencies.

Install all the dependencies, even the ones needed for development.

foo$ poetry install
Installing dependencies from lock file


Package operations: 10 installs, 0 updates, 0 removals

  - Installing more-itertools (7.2.0)
  - Installing zipp (0.6.0)
.....
  - Installing cfnmason (0.1.0)

The other option is to just install the libraries needed to execute and run the module. I would almost prefer if it defaulted to the method below, but it works.

foo$ poetry install --no-dev
Installing dependencies from lock file

Nothing to install or update

  - Installing cfnmason (0.1.0)

Start working on the Project

That is it. You should be up and running. At least to the point where you can get started with the project. Moving forward from this point will rely a lot upon how the project is setup, and how well it is documented. But, the big factor is that you can now start working on it while using Poetry, or you have the foundation to work on a project across multiple machines.

Setting up the cfnmason project for Python

Cfnmason is yet another tool that can be used to manipulate cloudformation stacks. It is not designed to be a replacement for CloudFormation like Terraform, but as a means of making building and managing them easier. I had written a version of this ages ago in Ruby, but with most of my work now being in Python, I am creating a new version in Python. As I have never created an exportable Python package, this will track the process of building and releasing a new PyPi package.

Oh, and to keep things interesting, I am doing this on a mix of Windows 10 and Linux.

Creating the default layout using Poetry

The first thing that we need to do is to create the base package. I could do it by hand, but I want to try out the Poetry package and see if I will hate the decision later.

c:\dev> poetry new cfnmason
...
c:\dev\cfnmason> tree \f \a

|   pyproject.toml
|   README.rst
|
+---cfnmason
|       __init__.py
|
\---tests
        test_cfnmason.py
        __init__.py

Init a new Git repo and add a .gitignore file

These are more civilized times. As such, I almost always create a git repository when I am working on a project, even small ones. They may or may not be public, and it can fluctuate on which platform I use to host my code. This time I am opting for Github, and have uploaded the code.

I have been using gitignore.io to generate base .gitignore files for ages. Type in a few operating systems, the language you are coding for, any IDEs, etc, and you can have a useful .gitignore file right out of the gate. Sure you can do it by hand, but this is quick and easy, and it can always be edited later.

Modifying the Readme file

Poetry starts you with a README.rst file. I don’t know about you, but I have been working with MarkDown for ages. It is common on a number of platforms and there is support for it in a number of editors. I understand the RST files are really designed for technical documentation, but I can burn that bridge later. For now, I need a decent starting point.

We could have started by writing out the template by hand. This would have been long and tedious. Instead, I am using a template. By using a template, I am up and running quickly. I can add and remove parts that I do or do not need, and hopefully, I will not forget a major part. As this is the first pass, I am not going to update the entire thing, but at least get the project name in and the fact that I am working on it.

The license file

The last thing needed before I start working on the code is the license file. Which license you choose is up to you. Personally, I like to use the MIT license. It lets people do what they want. As this is an opensource project, I feel that people should be able to use it as much or as little as they want.

The easiest way to do this is using the Github console. Just add a new file via the web interface and name it LICENSE or LICENSE.md in all caps. Then you will have the option to choose from a list of known licenses.

That is it and next steps

Hmmm. This took longer to write than the work to get the base package up and running. And, that is going to be expected. But, it should also show some of the thoughts and considerations that are needed when creating a new project. Especially if it is going to open for the world.

With that I will leave you to it. In the next couple of days, I am going to start adding in the libraries that are needed to begin working on the project. As I have written this in Ruby before, I have a rough layout in my head of how I want the application to work. And, I know what libraries are needed to meet the core functionality.

The question that I have for myself is if I should take the time to ensure that I update the design docs. Having a design doc is a great way to ensure that you are not missing any features. I am going to have to sleep on it.