
As we get closer to
Debian Bookworm's release, I thought I'd share
one change in Python 3.11 that will surely affect many people.
Python 3.11 implements the new
PEP 668,
Marking Python base environments
as externally managed . If you use
pip
regularly on Debian, it's
likely you'll eventually hit the
externally-managed-environment
error:
error: externally-managed-environment
This environment is externally managed
> To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.
If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.
If you wish to install a non-Debian packaged Python application,
it may be easiest to use pipx install xyz, which will manage a
virtual environment for you. Make sure you have pipx installed.
See /usr/share/doc/python3.11/README.venv for more information.
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
With this PEP, Python tools can now distinguish between packages that have been
installed by the user with a tool like
pip
and ones installed using a
distribution's package manager, like
apt
.
This is generally great news: it was previously too easy to break a system by
mixing the two types of packages. This PEP will simplify our role as a
distribution, as well as improve the overall Python user experience in Debian.
Sadly, it's also likely this change will break some of your scripts, especially
CI that (legitimately) install packages via
pip
alongside system packages.
For example, I use the following
gitlab-ci
snippet to make sure my PRs don't
break my build process
:
build:flit:
stage: build
script:
- apt-get update && apt-get install -y flit python3-pip
- FLIT_ROOT_INSTALL=1 flit install
- metalfinder --help
With Python 3.11, this snippet will error out, as
pip
will refuse to install
packages alongside the system's. The fix is to tell
pip
it's OK to "break"
your system packages, either using the
--break-system-packages
parameter, or
the
PIP_BREAK_SYSTEM_PACKAGES=1
environment variable
.
This, of course, is not something you should be using in production to restore
the old behavior! The "proper" way to fix this issue, as the
externally-managed-environment
error message aptly (har har) informs you, is
to use virtual environments.
Happy hacking!