Changes/PythonMarkExternallyManaged

Mark Python base environments as “externally managed”

This is a proposed Change for Fedora Linux.
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.

Summary

PEP 668 – Marking Python base environments as “externally managed” proposes a mechanism for a Python installation to communicate to tools like pip that its global package installation context is managed by some means external to Python, such as an OS package manager. We will mark our Python installation in /usr/lib(64)/python3.X/ as managed by rpm.

Owner

• Name: Miro Hrončok
• Email: mhroncok@redhat.com

Current status

• Targeted release: Fedora Linux 38
• Last updated: 2022-10-07
• FESCo issue: <will be assigned by the Wrangler>
• Tracker bug: <will be assigned by the Wrangler>
• Release notes tracker: <will be assigned by the Wrangler>

Detailed Description

See PEP 668 – Marking Python base environments as “externally managed”.

We'll add the PEP-described marker file to the python3-libs package. The file will be installed at /usr/lib(64)/python3.X/EXTERNALLY-MANAGED, i.e. /usr/lib64/python3.11/EXTERNALLY-MANAGED on a 64bt architecture and Python 3.11.

The file will contain:

[externally-managed]
Error=To install Python packages system-wide, try dnf install
 python3-xyz, where xyz is the package you are trying to
 install.

 If you wish to install a non-RPM-packaged Python package,
 create a virtual environment using python3 -m venv path/to/venv.
 Then use path/to/venv/bin/python and path/to/venv/bin/pip.

 If you wish to install a non-RPM-packaged Python application,
 it may be easiest to use pipx install xyz, which will manage a
 virtual environment for you.
 Make sure you have pipx installed via dnf.

Following the PEPs distro recommendations, we will have 2 installation schemes. One for RPM packages and one for pip (and similar tools). Packages installed via pip will continue to be installed to /usr/local/lib(64) by default. SUers attempting to install via pip to /usr/lib(64) will see the above error unless they suppress it (the exact way of doing that is still to be determined).

Unfortunately, as of writing this change proposal, pip does not yet respect this marker file. The upstream work is tracked at https://github.com/pypa/pip/issues/11381 and the change owners plan to coordinate with pip upstream on this. Whether or not we will keep the marker even if pip still won't support PEP 668 will be decided before the contingency deadline.

Benefit to Fedora

Users will be less likely to brick their system with pip and this protection will be more aligned with the upstream recommendations.

Scope

• Proposal owners:
  • work with pip upstream to make it support the PEP
  • add the marker as described in the PEP
  • split the current sysconfig installation scheme into two and make sure the marker only affects one of them

• Other developers: N/A (not needed for this Change)

• Release engineering: N/A (not needed for this Change)

• Policies and guidelines: N/A (not needed for this Change)

• Trademark approval: N/A (not needed for this Change)

• Alignment with Objectives: N/A (not needed for this Change)

Dependencies

• pip support for the marker file
• custom sysconfig installation scheme

Contingency Plan

• Contingency mechanism: revert the change
• Contingency deadline: final freeze
• Blocks release? No

Documentation

N/A (not a System Wide Change)