Changes/Python means Python3

Python means Python 3

Summary

In package and command names, "Python" will mean "Python 3".

Users installing and running Python or Python packages without specifying a version will get Python 3.

Running python will run python3. Running pytest will run the Python 3 version of pytest, and similarly for pydoc, pylint, etc.

dnf install python will install python3, and similarly for other python-* provides, e.g. dnf install python-requests will install python3-requests.

This is the final preparation for the retirement of Python 2 in Fedora 32 done in line with the soon to be finalized upstream recommendations.

Owner

• Name: Miro Hrončok
• Name: Petr Viktorin
• Email: <python-maint@redhat.com>

Current status

• Targeted release: Fedora 31
• Last updated: 2019-06-25
• Tracker bug: <will be assigned by the Wrangler>
• Release notes tracker: <will be assigned by the Wrangler>

Detailed Description

Motivation

The final upstream release of Python 2 is planned for January 2020. No further fixes will be made upstream. Most of Fedora 31's lifetime is after that date. Python 2 will be maintained only by its Fedora maintainers.

In preparation for removing Python 2 from Fedora entirely, we will make the unqualified name "Python" refer to the fully supported version.

This is in line with upstream changes to PEP 394 recommendations. (At the time of this writing, these changes are still being finalized, but recommendations for Linux distributions are accepted.)

This completes the work started with Fedora 23's "Python 3 as Default" change, with only removing Python 2 left to do.

What is being changed

The following changes will be made both in the Python packaging guidelines and the actual packages.

Package provides

Before: Packages with Python 2 modules used to provide the unversioned python- name. Users could do dnf install python-requests and the python2-requests package would be installed.

After: Packages with Python 3 modules will provide the unversioned python- name. Users can do dnf install python-requests and the python3-requests package will be installed.

This applies to the python3/python2 package as well as packages with Python modules. dnf install python will install python3.

The vast majority of packages will be updated by changing the %python-provide macro, with no changes in the individual spec files. The change owners will change the macro before the mass rebuild.

Python command

Before: The /usr/bin/python command was a symbolic link to /usr/bin/python2 living in the python-unversioned-command subpackage of python2. python2 recommended python-unversioned-command, so most users got the command by default when python2 was installed.

After: The /usr/bin/python command will be a symbolic link to /usr/bin/python3 living in the python-unversioned-command subpackage of python3. python3 will recommend python-unversioned-command, so most users will get the command by default when python3 is installed (and it is installed by default).

Other commands

Similarly, the Python-specific commands will switch from Python 2 to Python 3. This includes at least the following commands:

• pip
• python-config
• wheel
• idle
• pydoc
• pytest
• nosetest
• pycodestyle
• pylint
• epylint
• pyreverse
• symilar
• unit2
• msgfmt.py
• pygettext.py
• flask
• ipython
• f2py
• ipdb
• easy_install
• ...

This list is incomplete, automation is yet to be created to discover all such commands.

Note that applications like pygmentize or cython, whose behavior doesn't depend on the Python version, are not affected. (By current guidelines, they should use Python 3 if possible.)

Changes needed by Python package maintainers

This section of the change covers action needed from Python package maintainers. Most of the packages need no change, but there are several exceptions.

Packages with ambiguous names

Tracked at: https://fedora.portingdb.xyz/namingpolicy/

If your package with a Python 2 module or plugin is named with unversioned python (such as for example claws-mail-plugins-python or PyQt4), it needs to be removed or renamed to have python2 in the name (such as, for example, claws-mail-plugins-python2 or python2-PyQt4).

If your package is an application that happens to be written in Python 2 (such as calibre), no renaming is needed (for example, to python2-calibre).

Packages with ambiguous requires

Tracked at: https://fedora.portingdb.xyz/namingpolicy/

If your package depends on (Requires, BuildRequires, Recommends, etc.) a Python package with unversioned Python name (such as for example: systemd-python, python-setuptools, PyQt5, pycairo), it will now get resolved to the Python 3 version. Such dependencies need to be updated to specific Python 2 or Python 3 names (such as for example: pythonX-systemd, pythonX-setuptools, pythonX-qt5, pythonX-cairo where X is either 2 or 3).

Packages with ambiguous provides

Most of the backwards compatibility python-* provides are handled by the %python_provide macro and the change owners will change the macro behavior, so you don't have to worry about those. However, sometimes manual provides were added. If your package has some manual backwards compatibility provides for Python 2 packages, those need to be moved to the Python 3 packages or removed.

For example, before this change, the python2-m2crypto package provided and obsoleted m2crypto. Instead, the python3-m2crypto package should do that after this change.

Packages with missing %python_provide

The %python_provide macro used to do nothing for Python 3 packages. As such, it was often forgotten and only used with the Python 2 packages. Packagers of Python 3 packages should make sure to use the %python_provide macro according to the guidelines:

%package -n python3-%{srcname}
...
%{?python_provide:%python_provide python3-%{srcname}}

Packages with Python versioned commands and tools

Some packages have two versions of the provided commands. For example, the python2-pytest package has /usr/bin/pytest-2 and /usr/bin/pytest-2.7, the python3-pytest package has /usr/bin/pytest-3 and /usr/bin/pytest-3.X. The unversioned /usr/bin/pytest command used to be a symbolic link to /usr/bin/pytest-2. Now it needs to be changed to /usr/bin/pytest-3 and moved to the python3-pytest package.

For some packages, such duplication makes no sense, because the user sees no difference. For example, there should be just one /usr/bin/pygmentize -- the user doesn't care if it runs on Python 3, Python 2 or if it is written in Rust. This is not a new rule, but if your package is not following it, now is a good time to make sure the tool uses Python 3.

Packages that need unversioned Python to be Python 2

Since Fedora 29, a specific workaround is needed to use the python command as Python 2. This workaround will now bring Python 3. If that does not work for your package, you'll need to fix it (patch it) or retire it from Fedora.

If you only need the python command to mean Python 2 during package build, you can do something like this:

mkdir tmp_path
ln -s %{__python2} tmp_path/python
PATH=$(pwd)/tmp_path:$PATH make ...

However, if your package uses the python command during runtime, this ugly workaround won't work.

Benefit to Fedora

The name "Python" will not refer to software that will be unmaintained upstream for most of Fedora 31's lifetime and retired from Fedora 32.

Scope

• Proposal owners:
  • Changes in python3, python2 packages:
    • make /usr/bin/python link to /usr/bin/python3 instead of /usr/bin/python2 (+ the same for other executables there)
    • make python-unversioned-command a subpackage of python3 instead of python2
    • make python3 (instead of python2) recommend python-unversioned-command
  • Chnages in other commands:
    • switch pytest, nosetests, ipython, pip... to python3 (see Detailed Description for the actual list)
  • Changes in python-rpm-macros:
    • make %python_provide provide python-foo for python3-foo instead of for python2-foo
  • Let the mass rebuild change the provides based on the %python_provide change
  • Examine the failures and provide help to package maintainers
  • File bugs for remaining packages that provide python-x or x-python for Python 2

• Other developers (Python package maintainers, see Detailed Description):
  • Everybody: Make sure that you require and use python2 or python3 packages explicitly
  • If your package provides python-* or *-python for a Python 2 package by other means than the %python_provide macro, move that to the Python 3 package (or remove entirely if not applicable)
  • Fix or retire your package if it requires the unversioned python command to be Python 2

• Release engineering: #8482

• Policies and guidelines:
  • Slightly adapt the %pyton_provide macro section of the Python packaging guidelines and the Python Appendix.

• Trademark approval: not needed

Upgrade/compatibility impact

Custom scripts with python shebangs will invoke Python 3 by default, whereas previosuly they invoked Python 2 by default. See the User Experience section for details.

How To Test

For your favorite Python 3 package, check that it can be installed with the ambiguous Python name. For example, check that dnf install python-pip installs pyhon3-pip instead of pyhon2-pip.

For your favorite Python command, check that it invokes the Python 3 version. For example, check that python runs Python 3.

User Experience

Users who run python directly will get Python 3. Users who run pip directly will get pip for Python 3. Users who run pytest directly will get Python 3 pytest for Python 3. ...

Scripts with ambiguous Python shebangs (#!/usr/bin/python or #!/usr/bin/env python) will be executed by Python 3 by default.

If users need the python command or the #!/usr/bin/env python shebang to run Python 2, they can easily do that by:

ln -s /usr/bin/python2 ~/.local/bin/python

Similarly, sysadmins can do that system-wide:

ln -s /usr/bin/python2 /usr/local/bin/python

If users don't want the python command at all, they can dnf remove python-unversioned-command.

Dependencies

Most packages that provide Python version-specific functionality will be affected. However, the Change owners include proven packagers and they maintain python-rpm-macros, so most will not need packager attention.

We depend on the Fedora mass rebuild to adjust macro-generated package provides.

(Also, the PEP 394 update could be delayed, or even rejected/changed. Even if that happens, this Change will not be affected: Fedora would depart from following the upstream recommendations.)

There are currently (2019-06-25) 90 packages left with ambiguous Python requires. Those need to be adapted. Lot of them unfortunately Fail to Build From Source and will be retired before the Fedora 31 branching.

Contingency Plan

• Contingency mechanism: revert the changes, mass rebuild packages with the original %python_provide macro
• Contingency deadline: beta freeze
• Blocks release? No
• Blocks product? No

Documentation

• Updated Packaging guidelines
• This Change page

Release Notes

TBD. Check User Experience section.