Churchyard (talk | contribs) m (shebang automation is happening) |
Churchyard (talk | contribs) m (Link Changes/RetirePython2.7) |
||
(9 intermediate revisions by the same user not shown) | |||
Line 1: | Line 1: | ||
{{DISPLAYTITLE:Finalizing Fedora's Switch to Python 3}} | {{DISPLAYTITLE:Finalizing Fedora's Switch to Python 3}} | ||
{{admon/important| | {{admon/important|Mostly done|Things described on this page are mostly done. The remaining Python 2 exceptions are tracked in [https://fedora.portingdb.xyz/ Python 2 Dropping Database]. This page is no longer updated regularly. Talk to the [[SIGs/Python|Python SIG]] if you want to discuss it. A [[Changes/RetirePython2.7|proposal was made to retire Python 2 completely from Fedora 41]]. }} | ||
== Summary == | == Summary == | ||
Line 92: | Line 92: | ||
* Fedora Change: Not needed | * Fedora Change: Not needed | ||
* Description: Ensure that all <code>python-foo</code> names are virtual provides declarations, with the actual RPMs themselves being named <code>python2-foo</code>. <code>python-foo</code> names are also not to be used to declare the dependencies of a Python 2 package. Ensure that no scripts we ship in Fedora use the <code>/usr/bin/python</code> shebang and no packages require <code>/usr/bin/python</code>. All scripts shall explicitly use <code>/usr/bin/python2</code> or <code>/usr/bin/python3</code> and all packages shall explicitly require those as well. (This also applies to the <code>/usr/bin/env python</code> shebang.) This is a preparatory phase to clean up Fedora's Python packages and ensure consistency not only for new Python packages, but also for existing ones. This allows us to proceed with '''Phase 2'''. | * Description: Ensure that all <code>python-foo</code> names are virtual provides declarations, with the actual RPMs themselves being named <code>python2-foo</code>. <code>python-foo</code> names are also not to be used to declare the dependencies of a Python 2 package. Ensure that no scripts we ship in Fedora use the <code>/usr/bin/python</code> shebang and no packages require <code>/usr/bin/python</code>. All scripts shall explicitly use <code>/usr/bin/python2</code> or <code>/usr/bin/python3</code> and all packages shall explicitly require those as well. (This also applies to the <code>/usr/bin/env python</code> shebang.) This is a preparatory phase to clean up Fedora's Python packages and ensure consistency not only for new Python packages, but also for existing ones. This allows us to proceed with '''Phase 2'''. | ||
* Status: Mostly done, stray packages exist due to FTBFSes, and maintainers reverting our changes. | |||
* Steps: | * Steps: | ||
Line 99: | Line 100: | ||
** Fix ambiguous Python 2 dependency declarations, i.e using <code>python2-foo</code> instead of <code>python-foo</code> in requirements. We have started opening automatic Pull Requests with patches as announced in [https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/EJ5XP2R54V62K7FD5DIPQS3ZYKPUIULH/ Mass Pull Request filing for ambiguous Python 2 requirements]. The packages which will be targeted are tracked in [http://fedora.portingdb.xyz/namingpolicy/#require-misnamed PortingDB]. | ** Fix ambiguous Python 2 dependency declarations, i.e using <code>python2-foo</code> instead of <code>python-foo</code> in requirements. We have started opening automatic Pull Requests with patches as announced in [https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/EJ5XP2R54V62K7FD5DIPQS3ZYKPUIULH/ Mass Pull Request filing for ambiguous Python 2 requirements]. The packages which will be targeted are tracked in [http://fedora.portingdb.xyz/namingpolicy/#require-misnamed PortingDB]. | ||
** Ensure that <code>/usr/bin/python</code> or <code>/usr/bin/env python</code> shebangs are replaced with either <code>/usr/bin/python2</code> or <code>/usr/bin/python3</code>. This is currently covered by a Taskotron check. Automation is [https://pagure.io/packaging-committee/issue/738 in place as well]. | ** Ensure that <code>/usr/bin/python</code> or <code>/usr/bin/env python</code> shebangs are replaced with either <code>/usr/bin/python2</code> or <code>/usr/bin/python3</code>. This is currently covered by a Taskotron check. Automation is [https://pagure.io/packaging-committee/issue/738 in place as well]. | ||
** See also: [[Changes/ | ** See also: [[Changes/Update_comps_to_use_python3|Update comps to use python3]] ([[Releases/29 | Fedora 29]]) | ||
** | ** See also: [[Changes/Make_ambiguous_python_shebangs_error|Make ambiguous python shebangs error]] ([[Releases/30 | Fedora 30]]) | ||
==== Phase 2: Mass Python 2 Package Removal ==== | |||
* Targeted release: [[Releases/30 | Fedora 30]], continue in [[Releases/31 | Fedora 31]] | |||
* Fedora Change: [[Changes/Mass_Python_2_Package_Removal|Mass Python 2 Package Removal]] and [[Changes/F31_Mass_Python_2_Package_Removal|F31 Mass Python 2 Package Removal]] | |||
* Description: (Sub-)packages only providing python2 importable modules without additional functionality will be removed from Fedora unless some other package(s) depends on them. Python 2 will be deprecated in Fedora. Packagers can mark any other Python 2 packages as deprecated as well. See details in the change. This is done so we can focus on the remaining bits for Phase 4. | |||
==== Interlude: Update PEP 394 (not a Fedora Change) ==== | ==== Interlude: Update PEP 394 (not a Fedora Change) ==== | ||
* Targeted date: Before [[Releases/ | * Targeted date: Before [[Releases/31 | Fedora 31]] beta freeze | ||
* Formal proposal: | * Formal proposal: https://github.com/python/peps/pull/989 | ||
* Description: Work with CPython upstream to change the [https://www.python.org/dev/peps/pep-0394/ PEP 394] so that it says: '' | * Description: Work with CPython upstream to change the [https://www.python.org/dev/peps/pep-0394/ PEP 394] so that it says: ''distributions [are free to choose if] python refers to [...] python3''. | ||
* Note: This was already attempted in July 2017 but failed. However changes were made in the PEP that said: "In controlled environments aimed at expert users, where being explicit is valued over user experience (for example, in test environments and package build systems), distributions may choose to not provide the python command even if python2 is available. (All software in such a controlled environment must use python3 or python2 rather than python, which means scripts that deliberately use python need to be modified for such environments.)" | |||
* The updates from July 2017 led to [[Changes/Move_usr_bin_python_into_separate_package|Move /usr/bin/python into a separate package]] [[Releases/29 | Fedora 29]] change. | |||
==== Phase | ==== Phase 3: Switch python to refer to python3 ==== | ||
* Targeted release: | * Targeted release: [[Releases/31 | Fedora 31]] | ||
* Fedora Change: | * Fedora Change: [[Changes/Python_means_Python3]] (proposed) | ||
* Description: Make <code>python</code> mean Python 3 in Fedora. This would mean that installing a <code>python-foo</code> package will imply a Python 3 version on the package, and invoking <code>python</code> or e.g. <code>pytest</code> will run a Python 3 version of the interpreter/tool. The steps needed to achieve this will be done in combination with proposing changes to the upstream guidance in [https://www.python.org/dev/peps/pep-0394/ PEP 394]. Note that after '''Phase 1''', nothing in Fedora uses <code>/usr/bin/python</code> any more, so it should be safe for sysadmins to flip the symblink back to python2 and we should provide a documented and supported way of doing so. | * Description: Make <code>python</code> mean Python 3 in Fedora. This would mean that installing a <code>python-foo</code> package will imply a Python 3 version on the package, and invoking <code>python</code> or e.g. <code>pytest</code> will run a Python 3 version of the interpreter/tool. The steps needed to achieve this will be done in combination with proposing changes to the upstream guidance in [https://www.python.org/dev/peps/pep-0394/ PEP 394]. Note that after '''Phase 1''', nothing in Fedora uses <code>/usr/bin/python</code> any more, so it should be safe for sysadmins to flip the symblink back to python2 and we should provide a documented and supported way of doing so. | ||
Line 117: | Line 126: | ||
** Switch <code>%python_provides</code> macro to provide python-foo from python3-foo instead of python2-foo | ** Switch <code>%python_provides</code> macro to provide python-foo from python3-foo instead of python2-foo | ||
** Switch <code>/usr/bin/python</code> to point to <code>/usr/bin/python3</code> | ** Switch <code>/usr/bin/python</code> to point to <code>/usr/bin/python3</code> | ||
** Switch <code>/usr/bin/pip</code>, <code>pytest</code>, <code>nose</code>... to point tot the Python 3 version | |||
==== Phase | ==== Phase 4: Retire Python 2 for other packages, but keep it around for developers and users ==== | ||
* Targeted release: [[Releases/ | * Targeted release: [[Releases/32 | Fedora 32]] | ||
* | * Fedora Change: [[Changes/RetirePython2|Retire Python 2]] | ||
* Description: The {{package|python2}} package and all its subpackages will be removed from Fedora 32. A legacy {{package|python27}} package for developers and users will be provided. All packages in Fedora that need Python 2 to run will be removed from Fedora 32 regardless of their dependencies. All packages in Fedora that need Python 2 to build will be removed from Fedora 32 regardless of their dependencies. Exceptions can be granted by FESCo. | |||
== Benefit to Fedora == | == Benefit to Fedora == |
Latest revision as of 18:55, 3 July 2024
Summary
Currently, in Fedora package names, executables etc., python
without a version number generally means Python 2. Given that the upstream support for Python 2 will end soon, we want to prepare Fedora for a transition to Python 3 as a default. Before switching python
to refer to Python 3, a lot of preparatory work needs to be done and this page explains our plan to achieve this.
Wait, wasn't Python 3 as Default already done in Fedora 23? Yes it was, but part of that change proposal was that /usr/bin/python
and unversioned package names starting with python
will still mean Python 2, in compliance with upstream PEP 394. This change is about what happens once PEP 394 is updated (and we will drive that update if needed):
/usr/bin/python
will mean Python 3dnf install python
will mean Python 3dnf install python-foo
will mean Python 3 version of foopip
,virtualenv
,flask
,pytest
,flake8
,sphinx-build
and others like so will all run the Python 3 version, the Python 2 version will need to be run using the-2
(or2
) suffix
Owner
- Name: Iryna Shcherbina
- Name: Miro Hrončok
- Name: Petr Viktorin
- Email: python-maint at redhat.com
Current status
- Last updated: 2024-07-03
Currently (in Fedora 27):
/usr/bin/python
means Python 2dnf install python
means Python 2dnf install python-foo
means Python 2 version of foopip
,virtualenv
,flask
,pycodestyle
,flake8
,sphinx-build
and others like so (almost) all run the Python 2 version, the Python 3 version needs to be run using the-3
(or3
) suffix
/usr/bin/python
/usr/bin/python
is a symbolic link to /usr/bin/python2
and belongs to the
package. This also means that packages that require python2
/usr/bin/python
(usually due to shebangs) de facto require Python 2.
The python package
Since Fedora 26, there is no
package, only python
and python3
. python2
is virtually provided from python
.
python2
Packages with Python modules
According to the guidelines, built (binary) RPM packages with python
in their names MUST use the python2-
or python3-
prefix. python-
prefix (or -python
suffix) is forbidden. The name with the python-
prefix shall be provided from the python2-
package using the %python_provide
macro. This macro shall be used in both Python 2 and 3 subpackages, while it is currently a no-op on Python 3, but allows an easy switch in the future.
However, it wasn't always so and plenty of packages do not follow those guidelines yet.
The current situation in Fedora is as follows ("correct" means "follows current packaging guidelines"):
- Correct package supporting Python 2 and 3 builds these binary RPMs:
- python2-foo (provides python-foo as a virtual provide via %python_provides)
- python3-foo (uses %python_provides, which is currently no-op)
- "Misnamed" package supporting py2/3 builds these binary RPMs:
- python-foo (the py2 version; may provide python2-foo virtually, but usually doesn't)
- python3-foo
- Correct py2-only packages are named:
- python2-foo (provides python-foo as a virtual provide via %python_provides)
- Correct py3-only packages are named:
- python3-foo (uses %python_provides, which is currently no-op)
- "Misnamed" py2-only package might be named:
- python-foo, or
- pyfoo, or
- foo-python, etc.
- There are (close to) no "misnamed" py3-only packages
We are tracking statistics about this in the Fedora Python 3 Porting Database. There is close to a thousand packages that violate the naming guidelines by the time of Fedora 26.
Detailed Description
While the default version of Python in Fedora is Python 3, the python
command still invokes a Python 2 interpreter, and whenever one installs a python-foo
package, they get a Python 2 version of this package. Similarly, when a Python developer wants to run a familiar tool, such as virtualenv
, twine
, flake8
, pytest
and others, the Python 2 version is run. This has all been decided in long discussions around the Python 3 as Default Change, where the main argument was PEP 394 that says: for the time being, all distributions should ensure that python refers to [...] python2.
However, Python 2 is slowly approaching its end of existence and will no longer be supported upstream after 2020-01-01. Therefore we would like to start the transition to a state where python
means python3
. PEP 394 says: there will eventually come a time where the third party ecosystem surrounding Python 3 is sufficiently mature for this recommendation to be updated to suggest that the python symlink refer to python3 rather than python2. We want to be ready for this update, and furthermore we want to make this update happen as soon as possible, but not later than 2020-01-01.
This requires a lot of preparatory work before we do the actual switch. We need to make sure that nothing uses python
to imply Python 2 neither in binary RPM names nor in dependencies' names or shebangs, but rather explicitly specifies a Python version by using either python2
or python3
. The state we want to achieve in Fedora to be ready for the switch, is where python-foo
package names are only virtually provided with the %python_provide
macro, so we can switch the macro to mean no-op on Python 2 and create the virtual provide on Python 3 packages.
Other Linux Distributions and OSes
Some Linux distributions like Arch Linux (2010) or OpenMandriva (2017) have already made /usr/bin/python
pointing to Python 3,
while others like openSUSE or Debian/Ubuntu are working towards handling the transition in a more conservative way. Gentoo uses a configurable option with Python 3 as default.
Open source package manager homebrew for macOS X has renamed the python3
package to just python
on 2018-03-01.
Transition Steps
Phase 1: Eliminate the python- prefix from the Python 2 package names and /usr/bin/python from shebangs
- Targeted release for completion: Fedora 28 ~= first half of 2018
- Fedora Change: Not needed
- Description: Ensure that all
python-foo
names are virtual provides declarations, with the actual RPMs themselves being namedpython2-foo
.python-foo
names are also not to be used to declare the dependencies of a Python 2 package. Ensure that no scripts we ship in Fedora use the/usr/bin/python
shebang and no packages require/usr/bin/python
. All scripts shall explicitly use/usr/bin/python2
or/usr/bin/python3
and all packages shall explicitly require those as well. (This also applies to the/usr/bin/env python
shebang.) This is a preparatory phase to clean up Fedora's Python packages and ensure consistency not only for new Python packages, but also for existing ones. This allows us to proceed with Phase 2. - Status: Mostly done, stray packages exist due to FTBFSes, and maintainers reverting our changes.
- Steps:
- Identify the packages with naming and shebangs issues. Those include packages with misnamed binary RPMs and dependencies.
- Inform the package maintainers about the issues. Gather feedback and encourage fixing the packages.
- Rename Python 2 binary RPMs from
python-foo
topython2-foo
. This is already partially done by applying automated patches as announced in Mass package change (python2- binary package renaming). The packages which still need to be fixed are tracked in PortingDB (at this point, most of these fail to build due to unrelated problems). - Fix ambiguous Python 2 dependency declarations, i.e using
python2-foo
instead ofpython-foo
in requirements. We have started opening automatic Pull Requests with patches as announced in Mass Pull Request filing for ambiguous Python 2 requirements. The packages which will be targeted are tracked in PortingDB. - Ensure that
/usr/bin/python
or/usr/bin/env python
shebangs are replaced with either/usr/bin/python2
or/usr/bin/python3
. This is currently covered by a Taskotron check. Automation is in place as well. - See also: Update comps to use python3 ( Fedora 29)
- See also: Make ambiguous python shebangs error ( Fedora 30)
Phase 2: Mass Python 2 Package Removal
- Targeted release: Fedora 30, continue in Fedora 31
- Fedora Change: Mass Python 2 Package Removal and F31 Mass Python 2 Package Removal
- Description: (Sub-)packages only providing python2 importable modules without additional functionality will be removed from Fedora unless some other package(s) depends on them. Python 2 will be deprecated in Fedora. Packagers can mark any other Python 2 packages as deprecated as well. See details in the change. This is done so we can focus on the remaining bits for Phase 4.
Interlude: Update PEP 394 (not a Fedora Change)
- Targeted date: Before Fedora 31 beta freeze
- Formal proposal: https://github.com/python/peps/pull/989
- Description: Work with CPython upstream to change the PEP 394 so that it says: distributions [are free to choose if] python refers to [...] python3.
- Note: This was already attempted in July 2017 but failed. However changes were made in the PEP that said: "In controlled environments aimed at expert users, where being explicit is valued over user experience (for example, in test environments and package build systems), distributions may choose to not provide the python command even if python2 is available. (All software in such a controlled environment must use python3 or python2 rather than python, which means scripts that deliberately use python need to be modified for such environments.)"
- The updates from July 2017 led to Move /usr/bin/python into a separate package Fedora 29 change.
Phase 3: Switch python to refer to python3
- Targeted release: Fedora 31
- Fedora Change: Changes/Python_means_Python3 (proposed)
- Description: Make
python
mean Python 3 in Fedora. This would mean that installing apython-foo
package will imply a Python 3 version on the package, and invokingpython
or e.g.pytest
will run a Python 3 version of the interpreter/tool. The steps needed to achieve this will be done in combination with proposing changes to the upstream guidance in PEP 394. Note that after Phase 1, nothing in Fedora uses/usr/bin/python
any more, so it should be safe for sysadmins to flip the symblink back to python2 and we should provide a documented and supported way of doing so.
- Steps:
- Switch
%python_provides
macro to provide python-foo from python3-foo instead of python2-foo - Switch
/usr/bin/python
to point to/usr/bin/python3
- Switch
/usr/bin/pip
,pytest
,nose
... to point tot the Python 3 version
- Switch
Phase 4: Retire Python 2 for other packages, but keep it around for developers and users
- Targeted release: Fedora 32
- Fedora Change: Retire Python 2
- Description: The
package and all its subpackages will be removed from Fedora 32. A legacypython2
package for developers and users will be provided. All packages in Fedora that need Python 2 to run will be removed from Fedora 32 regardless of their dependencies. All packages in Fedora that need Python 2 to build will be removed from Fedora 32 regardless of their dependencies. Exceptions can be granted by FESCo.python27
Benefit to Fedora
Fedora has long been a one of the leaders of the Make Python 3 the Default Python initiative, having Python 3 as the "default Python" since Fedora 23 including a Fedora Workstation without Python 2, tracking the status of Python 3 compatibility and actively getting involved in porting upstream projects, such as (but not limited to) Samba. However, there are other distros (such as Arch) where python
means Python 3 for quite some time.
This transition plan simplifies the final transition to Python 3 when the upstream support for Python 2 ends.
How can I help?
Go trough the packages you maintain and see if they FAIL some taskotron checks. If so, please fix your packages to follow the new rules. You can also see a list of packages breaking the naming policy on PortingDB.
Scope
- Proposal owners:
Proposal owners are to communicate the idea to the packagers and rename the affected packages as well as fix the requirements via Mass_package_changes procedure.
- Other developers:
Package maintainers are expected to fix package requirements generated from shebangs by making sure that no scripts in their packages use /usr/bin/python
or /usr/bin/env python
shebangs.
Package maintainers will also be asked to review Pagure PRs with automated patches fixing the ambiguous Python 2 dependency declarations.
- Policies and guidelines:
- Using python2/python3 instead of python in binary RPM package name guidelines, fpc
- Using python2/python3 instead of python in (Build)Requires, guidelines, fpc
- Using
/usr/bin/python2
instead of/usr/bin/python
in shebangs and Requires, guidelines, fpc
How To Test
We have implemented taskotron checks for Phase 1 changes. The checks are being run each time a Python package is build in Koji, and you may view the results in taskotron resultsdb or when you submit the update in Bodhi.
Dependencies
Each phase depends on the phase before it.
Documentation
- Discussion on Fedora devel mailing list
- PEP 394
- Mass package change (python2- binary package renaming), part 1
- MASS CHANGE announcement: python2- prefix renaming, part 2
- Mass Pull Request filing for ambiguous Python 2 requirements.
Q&A
- I don't care which Python version my package runs on. What do I do? I just want to say
python
and let the build system to decide what's that.
Unfortunately that is not possible. According to the Python packaging guidelines, if the software is Python 3 compatible, it must be packaged for Python 3. Thus, you should update your package to Python 3 immediately. If you need more instructions, a guide for porting Python-based RPMs is available.
- I want to use the same specfile for both Fedora and EPEL. By forcing me to use
python2-
prefix on Fedora, you are making my life harder, because I cannot use it in EPEL. What do I do?
This can be done with the following code:
%if 0%{?rhel} %global py2_prefix python %else %global py2_prefix python2 %endif ... BuildRequires: %{py2_prefix}-requests
If this would make your specfile unbearably ugly, please contact us at the python-devel mailing list. We can add a %py2_prefix
macro to the build root for you, but we don't think it's necessary given common specfiles usually already have such if-else blocks. You can even do:
BuildRequires: python%{?fedora:2}-requests