Using Python and Poetry inside a Dev Container

Every time I join a new project or I get back to one that I previously worked on, I know that I will have some issues with the development environment. I In the past, I tried to compensate with detailed setup guides but the problem is that you will always spend an unnecessary amount of time doing work that is not related to your goal — develop working software, not configuring the toolbox.

No Silver Bullet, by Fred Brooks

As Fred Brooks explained in his article “No Silver Bullet”, in any software project you have accidental and essential complexities. The former is about how you solve a problem, the latter is about the complexity of the problem itself. You want to spend your time dealing with the complexity of your problem and minimize the complexity of the tools you use.

More humbly, I need to develop some additions to one of my pet projects, something that I use for learning to code in Python. I do this in my spare time, so it may happen that several weeks pass between one change and the other.

I have started using the CookieCutter template from Hypermodern Python articles — it provides a cool starting point for developing Python projects using a modern toolchain, specifically Poetry. In order to start developing with such a project, I need some required software packages to be present on my laptop, like

1. Python 3.10
2. Poetry 1.1.13
3. Install Nox 2022.1.7 and nox-poetry 1.0.0
4. … any other dependency specific to your operating System (e.g., Linux, Windows and Mac) …

This configuration should happen before I can start working on the project itself and may be impacted by subtle changes in my local environment (and, again, you don’t want to spend time chasing an error you have no clue about).

Containers are already mainstream packaging and deployment tools for shipping your runtime environments — could we use the same approach to enforce consistency for local development environments?

You will also need to use Docker for Desktop (or any other alternative tools) — it now supports Docker Developer environments, so you may find it a good tool for your toolbox.

The alternatives

Generally speaking, when creating new environments you have the following options:

• You create the required environment from scratch, following documentation or your collaborators’ hints. Watch out for
• You use a VM configured according to the development environment requirements (e.g., the NodeJS version, the Python runtime version, Operating System, …). You then connect to the VM using a remote client — for example Remote SSH extension. Note that using WSL 2 is actually a special case for this flavor.
• You use Docker containers to virtualize your development environment, also knows as Dev Containers.

Basic principle is to threat your development environment like cattle, not like pets — something that we know well for server-side environments.

In this article we only talk about Dev Containers and Visual Studio Code provides a cool extension, called Visual Studio Code Remote — Containers, that allows for easily solving consistency across your development environment.

During this discussion I will focus on using Visual Studio Code (VSC) and not other development environment, like IntelliJ IDEA.

If you want to take a look at the sample project, please check out the GitHub repository.

Poetry in a Dev Container

From VSC you can setup a Dev Container for any existing project — for simple cases, the default configuration is enough (e.g., you only need NodeJs or Python+Pip). You get your initial setup ready, you reboot VSC to run into a container and you are done — you have your environment ready for use: you can find more details in Developing inside a Container article.

What you will end up with is a new .devcontainer/ directory inside your project that you can enhance with your own configuration:

Dev Container configuration files — note that post-install.sh is a custom optional shell script used in the sample project only.

Thus, I have changed the devcontainer.json to support my needs:

• I setup the specific Poetry version that I want
• The VSC extensions that should be installed on my Dev Container
• I mount some specific endpoints that I need — for example, I’d like to share my Git configuration between the Dev Container and my host laptop

In Dockerfile, I configured additional steps for installing more software, like Poetry and Nox:

Dockerfile configuration for the sample Dev Container

Additionally, in post-install.sh, I have collected some additional configuration files for Poetry:

• Cache and virtual environment files are to seat inside my workspace — we don’t want to pollute the container filesystem. Note that the workspace will be staying inside the hosting machine (e.g., you laptop’s file system) — so these directories are also added to .gitignore file.
• Install all project dependencies inside the virtual environment

In the end, the project’s filesystem structure will be something like this:

Sample project directory layout

After that I can run all standard commands for managing the development lifecycle:

• using Git for pushing/pulling changes
• Running nox to locally run tests and pre-commit hooks
• … whatever else …

Running nox -s tests-3.10 will allow to execute unit tests in the project

Finally, you can push your .devcontainer/ folder to version control (e.g., Git) and easily come back to your project after a few months — it should be easy. Murphy allowing, that is! :)

Known limitations

• Dev Containers vs Python Venv — at the moment this article is being written, I only leverage the dev container — at runtime I will be using the Python Virtual Environment created by Poetry. If your dependencies are pretty stable, you may try to freeze these and get rid of the virtual env altogether — up to this moment, this is fine for me :)
• You will need to fine-tune your Docker environment and allocate the amount of resources that works for you — in my case, using Docker for Desktop is leveraging WSL 2 resources and this is auto-balanced with my entire system load. No problems so far.

Conclusions and ideas for future works

This article presented a case for applying Dev Containers to Python development with Poetry in order to achieve consistency of the development environment across different hosts (e.g., old vs new laptop).

In larger projects with several people working together or joining the project, it makes sense to standardize the practices and Dev Containers are a great tool from this point of view.

Future works that I want to explore involve

• Freeze the dependencies and get rid of the Poetry-generated virtual environments altogether — how feasible is this in practice?
• Building containers from Dev Containers — you often build containerized applications so … how to do that with VSC Remote containers?
• Building and deploying AWS Lambda functions — this is another use case as well.
• In addition to setting up a single dev containers, you may also setup a multi-dev container environment leveraging Docker Compose. I personally don’t have such complex use cases for now, so I will leave it as a hint if somebody is interested.

Thanks for getting here — I hope your found this useful. Feel free to comment and suggest improvements and corrections — your feedback is valuable to me!