Python 3.9 just came out recently, and I thought it would make sense to check out some of the new features (dict union operators, string remove prefix and suffix, etc.). Of course, doing this requires a Python 3.9 environment. Since new versions of Python may break existing code, I don’t want to update my entire system to try a new feature, rather I’d like to be able to control which version of Python I use, ideally keeping older versions around as needed. There are many ways to maintain Python environments, but one very useful tool is pyenv.
From the pyenv docs, we learn that pyenv is a set of pure shell functions that allow you to change your global Python version on a per-user basis, have per-project Python versions, or override the Python version used with an environment variable. Since pyenv depends on a shell, it’s only supported on Mac/Linux (or Windows using WSL). In this post, I’ll walk through the installation process and a few common usage scenarios.
Before we do that, it’s important to note that pyenv doesn’t handle virtualenv creation and maintenance by itself (but you can use the pyenv-virtualenv plugin for that), or just use virtualenv itself. I will plan on covering those details in a future post.
If you’re using a Mac, consider using homebrew to install pyenv. With this method, you only need one more step to finish the install (skip to step #3 below)
brew update brew install pyenv
Step #1
As an alternative, you can install using git. It’s recommended to just place this in $HOME\.pyenv, but it could be installed anywhere.
git clone https://github.com/pyenv/pyenv.git ~/.pyenv
Step #2
At this point in a git install, you need to add two variables to your environment. PYENV_ROOT needs to to point to the root of the install, and your PATH needs to include $PYENV_ROOT/bin at the front. Check out the docs for different shells, but here’s what you’d need to do for zsh, the current Mac default shell as of 10.15 Catalina.
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc
Step #3
Now if you installed with either homebrew or the git install, you need to add a pyenv init - to your environment. This is step #3 and is required for both install types (this example is for zsh, check the docs for other shells).
echo -e 'if command -v pyenv 1>/dev/null 2>&1; then\n eval "$(pyenv init -)"\nfi' >> ~/.zshrc
At this point, you’ll need to restart your shell. If you’ve never built python from scratch before, you’ll need to install the Python build dependencies. This is really just a few simple commands, and once you’ve done this you should be able to build any version of Python. You’ll want to make sure your dependencies are up to date because otherwise you may build a version of Python that will give you strange errors at startup, or it may not build at all.
OK, now that the install is over, you should be able to run pyenv (or pyenv -h) in your shell and see the usage help listing.
❯ pyenv pyenv 1.2.21 Usage: pyenv <command> [<args>] Some useful pyenv commands are: --version Display the version of pyenv commands List all available pyenv commands exec Run an executable with the selected Python version global Set or show the global Python version(s) help Display help for a command hooks List hook scripts for a given pyenv command init Configure the shell environment for pyenv install Install a Python version using python-build local Set or show the local application-specific Python version(s) prefix Display prefix for a Python version rehash Rehash pyenv shims (run this after installing executables) root Display the root directory where versions and shims are kept shell Set or show the shell-specific Python version shims List existing pyenv shims uninstall Uninstall a specific Python version version Show the current Python version(s) and its origin version-file Detect the file that sets the current pyenv version version-name Show the current Python version version-origin Explain how the current Python version is set versions List all Python versions available to pyenv whence List all Python versions that contain the given executable which Display the full path to an executable See `pyenv help <command>' for information on a specific command. For full documentation, see: https://github.com/pyenv/pyenv#readme
Now in getting back to my original motivation, I’m ready to start checking out a new version of Python, and I need to install it. To see how install works, check out the help. You can run help for any of the commands for detailed options.
❯ pyenv help install
Usage: pyenv install [-f] [-kvp] <version>
pyenv install [-f] [-kvp] <definition-file>
pyenv install -l|--list
pyenv install --version
-l/--list List all available versions
-f/--force Install even if the version appears to be installed already
-s/--skip-existing Skip if the version appears to be installed already
python-build options:
-k/--keep Keep source tree in $PYENV_BUILD_ROOT after installation
(defaults to $PYENV_ROOT/sources)
-p/--patch Apply a patch from stdin before building
-v/--verbose Verbose mode: print compilation status to stdout
--version Show version of python-build
-g/--debug Build a debug version
For detailed information on installing Python versions with
python-build, including a list of environment variables for adjusting
compilation, see: https://github.com/pyenv/pyenv#readmeIf we look for the available versions (using -l or --list), we’ll see a huge list (over 400 versions as of today).
❮ pyenv install -l Available versions: 2.1.3 2.2.3 2.3.7 2.4.0 2.4.1 ... stackless-3.5.4 stackless-3.7.5
I just want to check out the latest 3.9 version (3.9.0 as of this writing), so I’ll install it. This will take a little while.
> pyenv install 3.9.0 python-build: use [email protected] from homebrew python-build: use readline from homebrew Downloading Python-3.9.0.tar.xz... -> https://www.python.org/ftp/python/3.9.0/Python-3.9.0.tar.xz Installing Python-3.9.0... python-build: use readline from homebrew python-build: use zlib from Xcode sdk Installed Python-3.9.0 to /Users/mcw/.pyenv/versions/3.9.0
Now that it is installed, we can see it in our list. (Note I had earlier installed version 3.6.10).
❯ pyenv versions system 3.6.10 3.9.0
So now, we need to figure out how to use the different versions of Python we have installed. First, there is a global version. You can set this to just one version, or a chain of versions if you want the version specific shims to find specific versions. For example, based on the Python versions I have installed, I could set my global pyenv versions like this:
pyenv global system 3.6.10 3.9.0
This will cause the system Python to be found first, but the 3.6 and 3.9 versions can be picked up by their specific shims. Having multiple global versions can be helpful for tools that need to be able to run multiple versions of python in the same shell by invoking the specific versions (e.g. running python3.6 or python3 instead of python).
A quick side note: you may be tempted when using pyenv to use the shell builtin function which to determine which python version is in your path. That will always disappoint you, however, since it will just tell you the location of the shim. Use the pyenv which command instead. Use pyenv whence to find all the installed versions that have given Python binary commands installed.
❯ which python /Users/mcw/.pyenv/shims/python ❯ pyenv which python /usr/bin/python ❯ pyenv which python3.6 /Users/mcw/.pyenv/versions/3.6.10/bin/python3.6 ❯ pyenv which python3.9 /Users/mcw/.pyenv/versions/3.9.0/bin/python3.9 ❯ pyenv which python3.7 pyenv: python3.7: command not found ❯ pyenv whence pip 3.6.10 3.9.0
Now a global version is of some use but where pyenv is really helpful is in using local versions. You can force a specific version of Python just for a local directory. So for my motivating example of trying out Python 3.9 features, I can isolate usage of this version to a single directory. pyenv accomplishes this through a .python-version file placed in that directory, so to stop using a local version you can remove that file or use pyenv local --unset to revert to the global version.
❯ mkdir -p ~/projects/python3.9 ❯ cd ~/projects/python3.9 ❯ pyenv local 3.9.0 ❯ python --version Python 3.9.0 ❯ pyenv version 3.9.0 (set by /Users/mcw/projects/python3.9/.python-version) ❯ cat .python-version 3.9.0
Another way to set your Python version is to use a shell specific version. When this option is used, that instance of your shell will use the specified version and ignore the local and global options. The underlying implementation just sets the environment variable PYENV_VERSION, so you can just set this without using a command if you like. You can use the pyenv version command to see which version you are currently using, or pyenv versions to see all versions that are available to you.
❯ pyenv version system (set by /Users/mcw/.pyenv/version) 3.6.10 (set by /Users/mcw/.pyenv/version) 3.9.0 (set by /Users/mcw/.pyenv/version) ❯ pyenv shell 3.9.0 ❯ pyenv version 3.9.0 (set by PYENV_VERSION environment variable) ❯ export PYENV_VERSION=3.6.10 ❯ pyenv version 3.6.10 (set by PYENV_VERSION environment variable) ❯ pyenv shell --unset ❯ pyenv version system (set by /Users/mcw/.pyenv/version) 3.6.10 (set by /Users/mcw/.pyenv/version) 3.9.0 (set by /Users/mcw/.pyenv/version)
Keeping up to date
If you installed pyenv and a new version of Python came out that is not listed, you need to update your pyenv install. If you installed on a Mac with homebrew, you can update using brew:
❯ brew update && brew upgrade pyenv
If you performed a git install, you can just go to your git install and do a git pull.
And now, let me look at one of those new Python 3.9 features:
❯ python
Python 3.9.0 (default, Oct 25 2020, 16:22:53)
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> s = "a test example"
>>> s.removeprefix("a ")
'test example'And it worked!
Hopefully this has been a useful overview of how pyenv works and why you might want to use it to support multiple Python version in your environment. I find it very helpful for allowing me to have different Python versions for different projects and utilities and still play around with a new version quickly. It is especially useful for installing shell utilities that are generally useful and written in Python (think of things like linters, testing tools, or third party tools like aws tools). Now that you know how to setup pyenv, you should read about how to create virtual environments with it.