Python virtual environments with asdf, venv and direnv

Every language and its community seems to have its own story (and at times drama) on how to handle dependencies. We even have a conference just to talk about it. In Python, isolating dependencies and interpreter versions (together known as a "virtual environment") is notorious for tripping up beginners just getting started. The Python community understands this but is still deciding on a solution. Since Python is not my main daily language, I tend to forget about virtual environments for months at a time. This post is mostly for me to remember how I do it each time, but it's simple enough and uses familiar tooling that I hope it helps someone else.

In Python 3.3, the venv module got added natively to the default Python interpreter. This is the main workhorse between virtual environments and by learning a little about how it works, we can have a simple workflow to automatically create virtual environments when we cd into a project directory. The three main tools involved are asdf, venv (the module), and direnv.

Overview on each tool

asdf is a generic version manager for almost all of the popular language runtimes. Alongside python, I use it to manage my golang, nodejs and perl installations. One package manager to rule them all.

direnv allows you to change your shell's environment based on which current directory your shell session lives in. There are some neat features tucked into this tool so I suggest you dive into their documentation.

As you soon will see, you use the venv module to actually create the virtual environment. Once that happens, you'll have access to executables that can be used to activate these virtual environments. Since all .venv/bin/activate does is change the shell's environment variables, we skip right over using it and change the environment the same way with direnv.

My simple workflow

Change into (or create) your project directory:

$ cd /the/path/to/your/project

Start by creating the virtual environment:

$ python -m venv .venv # Directory name can be anything

Then create an .envrc (what direnv looks for) to set relevant PATH vars for the right executables to be picked up:

$ echo "export VIRTUAL_ENV=$PWD/.venv\nexport PATH=$PWD/.venv/bin:\$PATH" > .envrc

I'm pretty sure VIRTUAL_ENV isn't necessary to be set but since explicit activation does it, there's no harm. Direnv should then warn you that these new environment variables are going to be loaded. To continue,

$ direnv allow

That should be it! To confirm that python3 and pip are pointing to the right places:

$ which python3
/the/path/to/your/project/.venv/bin/python3
$ which pip
/the/path/to/your/project/.venv/bin/pip

You should have now a virtual environment activated each time to change into this directory from your shell!

$ cd /the/path/to/your/project
direnv: loading ~/path/to/your/project/.envrc
direnv: export +VIRTUAL_ENV ~PATH
$ pip install -r requirements.txt # Uses virtual environment