Latest revision as of 11:50, 7 October 2022

Mark Python base environments as “externally managed”

This is a proposed Change for Fedora Linux.
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.

Summary

PEP 668 – Marking Python base environments as “externally managed” proposes a mechanism for a Python installation to communicate to tools like pip that its global package installation context is managed by some means external to Python, such as an OS package manager. We will mark our Python installation in /usr/lib(64)/python3.X/ as managed by rpm.

Owner

Name: Miro Hrončok
Email: mhroncok@redhat.com

Current status

Targeted release: Fedora Linux 38
Last updated: 2022-10-07
FESCo issue: <will be assigned by the Wrangler>
Tracker bug: <will be assigned by the Wrangler>
Release notes tracker: <will be assigned by the Wrangler>

Detailed Description

See PEP 668 – Marking Python base environments as “externally managed”.

We'll add the PEP-described marker file to the python3-libs package. The file will be installed at /usr/lib(64)/python3.X/EXTERNALLY-MANAGED, i.e. /usr/lib64/python3.11/EXTERNALLY-MANAGED on a 64bt architecture and Python 3.11.

The file will contain:

[externally-managed]
Error=To install Python packages system-wide, try dnf install
 python3-xyz, where xyz is the package you are trying to
 install.

 If you wish to install a non-RPM-packaged Python package,
 create a virtual environment using python3 -m venv path/to/venv.
 Then use path/to/venv/bin/python and path/to/venv/bin/pip.

 If you wish to install a non-RPM-packaged Python application,
 it may be easiest to use pipx install xyz, which will manage a
 virtual environment for you.
 Make sure you have pipx installed via dnf.

Following the PEPs distro recommendations, we will have 2 installation schemes. One for RPM packages and one for pip (and similar tools). Packages installed via pip will continue to be installed to /usr/local/lib(64) by default. SUers attempting to install via pip to /usr/lib(64) will see the above error unless they suppress it (the exact way of doing that is still to be determined).

Unfortunately, as of writing this change proposal, pip does not yet respect this marker file. The upstream work is tracked at https://github.com/pypa/pip/issues/11381 and the change owners plan to coordinate with pip upstream on this. Whether or not we will keep the marker even if pip still won't support PEP 668 will be decided before the contingency deadline.

Feedback

Benefit to Fedora

Users will be less likely to brick their system with pip and this protection will be more aligned with the upstream recommendations.

Scope

Proposal owners:
- work with pip upstream to make it support the PEP
- add the marker as described in the PEP
- split the current sysconfig installation scheme into two and make sure the marker only affects one of them

Other developers: N/A (not needed for this Change)

Release engineering: N/A (not needed for this Change)

Policies and guidelines: N/A (not needed for this Change)

Trademark approval: N/A (not needed for this Change)

Alignment with Objectives: N/A (not needed for this Change)

Upgrade/compatibility impact

How To Test

User Experience

Dependencies

Contingency Plan

Contingency mechanism: revert the change
Contingency deadline: final freeze
Blocks release? No

Documentation

N/A (not a System Wide Change)

Release Notes

@@ Line 7: / Line 7: @@
 == Summary ==
 <!-- A sentence or two summarizing what this change is and what it will do. This information is used for the overall changeset summary page for each release. Note that motivation for the change should be in the Benefit to Fedora section below, and this part should answer the question "What?" rather than "Why?". -->
+[https://peps.python.org/pep-0668/ PEP 668  – Marking Python base environments as “externally managed”] proposes a mechanism for a Python installation to communicate to tools like pip that its global package installation context is managed by some means external to Python, such as an OS package manager. We will mark our Python installation in `/usr/lib(64)/python3.X/` as managed by rpm.
 == Owner ==
@@ Line 13: / Line 14: @@
 This should link to your home wiki page so we know who you are.
 -->
-* Name: [[User:FASAcountName| Your Name]]
+* Name: [[User:Churchyard|Miro Hrončok]]
 <!-- Include you email address that you can be reached should people want to contact you about helping with your change, status is requested, or technical issues need to be resolved. If the change proposal is owned by a SIG, please also add a primary contact person. -->
-* Email: <your email address so we can contact you, invite you to meetings, etc. Please provide your Bugzilla email address if it is different from your email in FAS>
+* Email: mhroncok@redhat.com
 <!--- UNCOMMENT only for Changes with assigned Shepherd (by FESCo)
 * FESCo shepherd: [[User:FASAccountName| Shehperd name]] <email address>
 -->
 == Current status ==
@@ Line 32: / Line 32: @@
 <!-- [[Category:SystemWideChange]] -->
-* Targeted release: [[Releases/<number> | Fedora Linux <number> ]]
+* Targeted release: [[Releases/38 | Fedora Linux 38 ]]
 * Last updated: <!-- this is an automatic macro — you don't need to change this line -->  {{REVISIONYEAR}}-{{REVISIONMONTH}}-{{REVISIONDAY2}}
 <!-- After the change proposal is accepted by FESCo, tracking bug is created in Bugzilla and linked to this page
@@ Line 46: / Line 46: @@
 == Detailed Description ==
 <!-- Expand on the summary, if appropriate.  A couple sentences suffices to explain the goal, but the more details you can provide the better. -->
+See [https://peps.python.org/pep-0668/ PEP 668  – Marking Python base environments as “externally managed”].
+We'll add the PEP-described marker file to the `python3-libs` package. The file will be installed at `/usr/lib(64)/python3.X/EXTERNALLY-MANAGED`, i.e. `/usr/lib64/python3.11/EXTERNALLY-MANAGED` on a 64bt architecture and Python 3.11.
+The file will contain:
+ [externally-managed]
+ Error=To install Python packages system-wide, try `dnf install
+  python3-xyz`, where xyz is the package you are trying to
+  install.
+  If you wish to install a non-RPM-packaged Python package,
+  create a virtual environment using `python3 -m venv path/to/venv`.
+  Then use path/to/venv/bin/python and path/to/venv/bin/pip.
+  If you wish to install a non-RPM-packaged Python application,
+  it may be easiest to use `pipx install xyz`, which will manage a
+  virtual environment for you.
+  Make sure you have pipx installed via dnf.
+Following the PEPs distro recommendations, we will have 2 installation schemes. One for RPM packages and one for pip (and similar tools). Packages installed via pip will continue to be installed to `/usr/local/lib(64)` by default. SUers attempting to install via pip to `/usr/lib(64)` will see the above error unless they suppress it (the exact way of doing that is still to be determined).
+Unfortunately, as of writing this change proposal, pip does not yet respect this marker file. The upstream work is tracked at https://github.com/pypa/pip/issues/11381 and the change owners plan to coordinate with pip upstream on this. Whether or not we will keep the marker even if pip still won't support PEP 668 will be decided before the contingency deadline.
 == Feedback ==
@@ Line 78: / Line 101: @@
       https://fedoraproject.org/wiki/Changes/perl5.26 (major upgrade to a popular software stack, visible to users of that stack)
 -->
+Users will be less likely to brick their system with `pip` and this protection will be more aligned with the upstream recommendations.
 == Scope ==
 * Proposal owners:
+** work with pip upstream to make it support the PEP
+** add the marker as described in the PEP
+** split the current sysconfig installation scheme into two and make sure the marker only affects one of them
 <!-- What work do the feature owners have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
-* Other developers: <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+* Other developers: N/A (not needed for this Change)
 <!-- What work do other developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
-* Release engineering: [https://pagure.io/releng/issues #Releng issue number] <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+* Release engineering: N/A (not needed for this Change) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
 <!-- Does this feature require coordination with release engineering (e.g. changes to installer image generation or update package delivery)?  Is a mass rebuild required?  include a link to the releng issue.
 The issue is required to be filed prior to feature submission, to ensure that someone is on board to do any process development work and testing and that all changes make it into the pipeline; a bullet point in a change is not sufficient communication -->
@@ Line 96: / Line 123: @@
 <!-- If your Change may require trademark approval (for example, if it is a new Spin), file a ticket ( https://pagure.io/Fedora-Council/tickets/issues ) requesting trademark approval from the Fedora Council. This approval will be done via the Council's consensus-based process. -->
-* Alignment with Objectives:
+* Alignment with Objectives: N/A (not needed for this Change)
 <!-- Does your proposal align with the current Fedora Objectives: https://docs.fedoraproject.org/en-US/project/objectives/ ? It's okay if it doesn't, but it's something to consider -->
@@ Line 139: / Line 166: @@
 <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+* [https://github.com/pypa/pip/issues/11381 pip support for the marker file]
+* [https://discuss.python.org/t/10302/46 custom sysconfig installation scheme]
 == Contingency Plan ==
 <!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "Revert the shipped configuration".  Or it might not (e.g. rebuilding a number of dependent packages).  If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy.  -->
-* Contingency mechanism: (What to do?  Who will do it?) N/A (not a System Wide Change)  <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+* Contingency mechanism: revert the change  <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
 <!-- When is the last time the contingency mechanism can be put in place?  This will typically be the beta freeze. -->
-* Contingency deadline: N/A (not a System Wide Change)  <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+* Contingency deadline: final freeze  <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
 <!-- Does finishing this feature block the release, or can we ship with the feature in incomplete state? -->
-* Blocks release? N/A (not a System Wide Change), Yes/No <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+* Blocks release? No <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
 == Documentation ==

Search

Changes/PythonMarkExternallyManaged: Difference between revisions