Use pyenv and virtual environments to manage Python complexity

In my earlier post, I wrote about how pyenv is a great tool for running multiple versions of Python on the same host. It makes it simple to install multiple versions of Python on your workstation or server and control which version executes in a shell. But as a Python developer, the Python version is only one part of the environment. Most Python developers will work on more than one project at a time and want to install a number of Python packages for use in each project. Installing modules globally is rarely a good idea, especially if you are likely to use that Python version for more than one project. What happens when one projects wants a specific version of a package that won’t work with another project? Instead, using virtualenv or anaconda is the way to go. Luckily, both work well with pyenv. In this post I’ll look at using basic virtualenv, the pyenv-virtualenv plugin, and anaconda to build an isolated virtual environment that has a package installed in it that will be isolated to that environment.

Before discussing these details, I’ll mention that this post does not talk about the complexity of maintaining package compatibility within virtual environments. That is a topic for another post (or set of posts).

virtualenv

Assuming you followed the installation steps in the first post on pyenv, you should already know how to setup your shell to use a specific version of Python. As in that post, I’ll go ahead and install a unique Python version. Then, using that version I’ll

  • install virtualenv (globally)
  • use the virtualenv command to make a virtualenv
  • activate the virtualenv
  • use pip to install packages into that virtualenv

Note that all of these examples were run on a Mac running macOS Catalina and using zsh. This can all be run using the shell of your choice on Mac, Linux, or Windows using WSL.

❯ pyenv install --list | grep 3.8  # look for the latest 3.8 version
  3.8.0
  3.8-dev
  3.8.1
  3.8.2
  3.8.3
  3.8.4
  3.8.5
  3.8.6
  miniconda-3.8.3
  miniconda3-3.8.3
❮ pyenv install 3.8.6
python-build: use [email protected] from homebrew
python-build: use readline from homebrew
Installing Python-3.8.6...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.8.6 to /Users/mcw/.pyenv/versions/3.8.6

❯ pyenv shell 3.8.6  # sets the version just for this shell
❯ pyenv which pip    # show which executable is running for pip, it's the newly installed one
/Users/mcw/.pyenv/versions/3.8.6/bin/pip
❯ pip install virtualenv
Collecting virtualenv
  Using cached virtualenv-20.1.0-py2.py3-none-any.whl (4.9 MB)
Collecting appdirs<2,>=1.4.3
  Using cached appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Collecting filelock<4,>=3.0.0
  Using cached filelock-3.0.12-py3-none-any.whl (7.6 kB)
Collecting distlib<1,>=0.3.1
  Using cached distlib-0.3.1-py2.py3-none-any.whl (335 kB)
Collecting six<2,>=1.9.0
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Installing collected packages: appdirs, filelock, distlib, six, virtualenv
Successfully installed appdirs-1.4.4 distlib-0.3.1 filelock-3.0.12 six-1.15.0 virtualenv-20.1.0

Now that virtualenv is installed in the Python environment, I can setup a virtualenv for our test project. For this example, I’m putting the virtualenv in a projects directory, but you can put it anywhere you want, including in a hidden directory in our source tree like .env.

❯ cd projects
❯ virtualenv myenv
created virtual environment CPython3.8.6.final.0-64 in 407ms
  creator CPython3Posix(dest=/Users/mcw/projects/myenv, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/mcw/Library/Application Support/virtualenv)
    added seed packages: pip==20.2.4, setuptools==50.3.2, wheel==0.35.1
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
❯ . myenv/bin/activate    # activates the 3.8.6 virtualenv to isolate our pip installs
❯ pip install requests
Collecting requests
  Using cached requests-2.24.0-py2.py3-none-any.whl (61 kB)
Collecting idna<3,>=2.5
  Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting chardet<4,>=3.0.2
  Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
  Using cached urllib3-1.25.11-py2.py3-none-any.whl (127 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Installing collected packages: idna, chardet, urllib3, certifi, requests
Successfully installed certifi-2020.6.20 chardet-3.0.4 idna-2.10 requests-2.24.0 urllib3-1.25.11
❯ python
Python 3.8.6 (default, Nov  1 2020, 17:41:10)
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.__file__
'/Users/mcw/projects/myenv/lib/python3.8/site-packages/requests/__init__.py'
>>> requests.__version__
'2.24.0'

Now just to show you quickly why virtualenvs are great, we’ll install a different version of requests in a different virtualenv. It doesn’t take much work at all.

❯ deactivate                      # deactivate our old environment
❯ virtualenv myenv2.              # same output as above, but for 2nd environment
❯ . myenv2/bin/activate           # activate our new one
❯ pip install requests==2.23.0.   # pick a different version than last time
❯ python
Python 3.8.6 (default, Nov  1 2020, 17:41:10)
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.__version__
'2.23.0'

So now there are two isolated environments, each with a different version of the requests module. We can easily switch between the two as needed.

pyenv-virtualenv plugin

It turns out that pyenv supports plugins, and the pyenv-virtualenv plugin helps you use pyenv with virtualenv (or conda, which we’ll talk about next). Take a look at their docs for the installation process. It’s very similar to the installation of pyenv itself (I used brew to install it in my environment). Once installed, you get some new commands available in pyenv.

   activate    Activate virtual environment
   deactivate   Deactivate virtual environment
   virtualenv   Create a Python virtualenv using the pyenv-virtualenv plugin
   virtualenv-delete   Uninstall a specific Python virtualenv
   virtualenv-init   Configure the shell environment for pyenv-virtualenv
   virtualenv-prefix   Display real_prefix for a Python virtualenv version
   virtualenvs   List all Python virtualenvs found in `$PYENV_ROOT/versions/*'.

If we use these commands to replicate what we did above, it would look like this.

❯ pyenv virtualenv 3.8.6 myenv3   # makes a 3rd virtualenv using the 3.8.6 version
# output as before
❯ pyenv virtualenvs
  3.8.6/envs/myenv3 (created from /Users/mcw/.pyenv/versions/3.8.6)
  myenv3 (created from /Users/mcw/.pyenv/versions/3.8.6)
❯ pyenv deactivate                # if you still have another virtualenv activated from earlier, you can deactivate this way
❯ pyenv activate myenv3
pyenv-virtualenv: prompt changing will be removed from future release. configure `export PYENV_VIRTUALENV_DISABLE_PROMPT=1' to simulate the behavior.
❯ pip install requests
# output as before
❯ python
Python 3.8.6 (default, Nov  1 2020, 17:41:10)
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.__file__
'/Users/mcw/.pyenv/versions/myenv3/lib/python3.8/site-packages/requests/__init__.py'

conda

Another nice thing about pyenv is that it makes working with Anaconda pretty straightforward. I won’t get into Anaconda in detail here, but if you plan on working on data science projects that have a large number of dependencies (like numpy, pandas, scikit-learn, tensorflow, etc.), it is usually going to be much easier to just use anaconda or miniconda to get all of your dependencies installed. Anaconda is the full install, miniconda will just install the bare necessities for you to pick and choose the packages you want quicker installs and need to save disk space.

Pyenv makes it easy to search for all versions of anaconda and miniconda without having to wade through the web site to search for an installer. Once you install a version of Anaconda using pyenv, using conda (the command for managing environments and dependencies) fits in pretty well with the environment so you don’t have to use conda commands for the basic environment creation.

❯ pyenv install --list | grep conda # to see what's available
❯ pyenv install miniconda3-4.7.12
Downloading Miniconda3-4.7.12-MacOSX-x86_64.sh...
-> https://repo.anaconda.com/miniconda/Miniconda3-4.7.12-MacOSX-x86_64.sh
Installing Miniconda3-4.7.12-MacOSX-x86_64...

# <lots of other output snipped>

❯ pyenv versions
  system
  3.6.10
  3.8.6
  3.8.6/envs/myenv3
  3.9.0
  miniconda3-4.7.12
* myenv3 (set by PYENV_VERSION environment variable)
❯ pyenv deactivate                  # if we had an environment activated already (myenv3 from above)
❯ pyenv virtualenv myconda
# this will generate lots of output as conda builds the environment

❯ pyenv activate myconda
pyenv-virtualenv: prompt changing will be removed from future release. configure `export PYENV_VIRTUALENV_DISABLE_PROMPT=1' to simulate the behavior.

Now the conda tools are available in the shell and can be used instead of pyenv to create environments. Conda is also used to install packages. I won’t get into details here, but to be complete, let’s replicate the earlier task that was done using virtualenv, but this time with conda.

❯ conda env list       # see that we have our new environment activated
# conda environments:
#
base                     /Users/mcw/.pyenv/versions/miniconda3-4.7.12
myconda               *  /Users/mcw/.pyenv/versions/miniconda3-4.7.12/envs/myconda
❯ conda install requests  # instead of pip
# this generates a lot of output showing all the dependencies being installed
❯ python
Python 3.8.5 (default, Sep  4 2020, 02:22:02)
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.__file__
'/Users/mcw/.pyenv/versions/myconda/lib/python3.8/site-packages/requests/__init__.py'

Pyenv is a useful tool for not only installing and isolating multiple versions of Python, but can easily be used to manage virtual environments, both in Anaconda and using virtualenv. Once a Python developer needs more than one version of Python on a workstation or server, it’s a great way to manage that complexity.

Don't miss any articles!

If you like this article, give me your email and I'll send you my latest articles along with other helpful links and tips with a focus on Python, pandas, and related tools.

Invalid email address
I promise not to spam you, and you can unsubscribe at any time.

6 thoughts on “Use pyenv and virtual environments to manage Python complexity”

  1. No Money for a Name

    I have a question – why is pyenv so “hidden”. After using it for a while I wonder why this is not the default tool managing python and why it does not come with every python distribution. Are there some drawbacks? Why is it not part of every python beginner tutorial or part of the official python docs?
    It seems such a great tool – but I feel that I do miss some “dark secret” why the “official” python world seems to ignore this tool – what is it?

    1. I’m not really sure why it’s not more commonly used, that’s a good question. I can’t even remember how I first discovered it, but found it useful enough that it made sense to write about it. Hopefully it will become more standard. I think I discovered it out of necessity when I needed to run 2-3 versions of Python 3 for different projects and didn’t see better ways.

  2. Hi William, thanks for reading.

    You can think of pyenv as a way to manage versions of Python available on your computer. When you use the pyenv-virtualenv plugin, you can create virtualenvs that use the versions of Python you installed with pyenv. Then if you use an IDE like PyCharm, you have to figure out how to tell it to use that virtualenv in your project. I’m not a PyCharm user, but any good IDE will have a way to point to an existing virtualenv.

  3. Thanks for your post, it is very instructive, I just have one question. Why you recommend to install Anaconda/miniconda from pyenv. I read about other people who don’t recommend it. For example the answer from Simba to this question from Stack Overflow: https://stackoverflow.com/questions/58044214/installing-anaconda-with-pyenv-unable-to-configure-virtual-environment
    Would you elaborate more why this option is better for you, from usability, maintenance, etc. Thanks!

    1. Hi David, my experience with using anaconda with pyenv is fairly limited, but I haven’t had the issues that are mentioned in the stack overflow link. In the past, I usually used anaconda for work, and would only have 1-2 environments running with limited changes. I find pyenv very helpful for managing lots of little environment for smaller projects and scripts. I’ve found the pyenv support for anaconda useful for the few times that I choose to use anaconda, but mostly build isolated environments using pyenv and pyenv-virtualenv, then manage my dependencies with pip/requirements.txt or poetry.

      I’d say if you are mostly using anaconda, then don’t bother with pyenv. If you mostly use pyenv and occasionally need anaconda, then use it with pyenv, but if it gives you trouble, just go the full anaconda route. I hope that helps, and have to admit that in my most recent jobs I’ve not been using anaconda at all, so take my advice very lightly.

Have anything to say about this topic?