From Fedora Project Wiki
(Initial summary and skeleton)
 
Line 56: Line 56:
 
== Detailed Description ==
 
== Detailed Description ==
  
<!-- Expand on the summary, if appropriateA couple sentences suffices to explain the goal, but the more details you can provide the better. -->
+
=== The Saga ===
 +
 
 +
==== A Long Time Ago in the Fedora Galaxy ====
 +
 
 +
Many releases ago, when Fedora hasn't built for power and arm yet, the Python maintainers mapped the Python "platform triplet" to <code>%{_arch}-linux%{_gnu}</code>. This worked. For example, on x86_64, this is <code>x86_64-linux-gnu</code> on Fedora and this is consistent with the "platform triplet" used in filenames in upstream.
 +
 
 +
==== The Phantom Technical Debt ====
 +
 
 +
Later around the year 2015, as more architectures were added, Python build scripts were patched to use "the Fedora's architecture names":
 +
 
 +
* <code>powerpc</code> was changed to <code>ppc</code> https://src.fedoraproject.org/rpms/python3/c/0efd3d31
 +
* the arm triplet was patched https://src.fedoraproject.org/rpms/python3/c/2f6352e7
 +
* even the mips one https://src.fedoraproject.org/rpms/python3/c/4bc70e0c
 +
 
 +
At the time, that was a reasonable decision: the idea of cross-Linux builds was sci-fi, and Fedora was not trying to stay close to upstream as it is now (we had around 60 patches; today we're down to around 6).
 +
 
 +
==== Raise of the Manylinux Wheels ====
 +
 
 +
In the meantime, cross-Linux builds become a thing. The [https://www.python.org/dev/peps/pep-0513/ manylinux1 standard] was created in 2016, allowing to build Python wheels with compiled extension modules on one platform and ship them to many. The first manylinux version only supported x86_64 and i686 and hence it was not impacted by Fedora's patching decisions.
 +
 
 +
The manylinux standard arguably made the upstream Python packaging ecosystem a much nicer thing to work with. Installing packages with compiled extension modules was no longer such a pain. One could just run <code>pip install numpy</code> and not worry about a Fortran compiler. For that reason, manylinux wheels become widely adapted by the most popular projects.
 +
 
 +
==== A New Architecture ====
 +
 
 +
With the third manylinux version -- [https://www.python.org/dev/peps/pep-0599/ manylinux2014] (created in 2019, named after the oldest Linux it supports -- CentOS 7), support for more architectures was introduced: x86_64, i686, aarch64, armv7l, ppc64, ppc64le, s390x. The adaption of new architectures is somehow slow, because the [https://github.com/pypa/manylinux#manylinux2014 official manylinux2014 containers] only currently (August 2020) exist for x86_64, i686, aarch64, ppc64le and s390x.
 +
 
 +
==== Revenge of the Patches ====
 +
 
 +
We have discovered [https://github.com/pypa/manylinux/issues/687 a problem with the ppc64le manylinux2014 wheels]: The CentOS 7 manylinux2014 container images ship upstream Python without RHEL/CentOS/EPEL patches. When the extension modules it built there, it is named with an upstream named suffix: <code>.cpython-XY(m)-powerpc64le-linux-gnu.so</code>. The wheel is installable on Fedora (with Fedora's patches), but the module won't (even be considered for) import, because Fedora's Python expect the extension to be <code>.cpython-XY(m)-ppc64le-linux-gnu.so</code>.
 +
 
 +
In theory, we have the same problem on armv7hl, but there are no manylinux2014 containers available for that platform, so there are no such wheels out there (known to us).
  
 
== Feedback ==
 
== Feedback ==

Revision as of 11:03, 25 August 2020


Python Upstream Architecture Names

Summary

Use upstream architecture naming in Python (mostly in filenames) instead of previously patched Fedora names. For example, have /usr/lib64/python3.9/lib-dynload/array.cpython-39-powerpc64le-linux-gnu.so instead of /usr/lib64/python3.9/lib-dynload/array.cpython-39-ppc64le-linux-gnu.so. This makes packaging of Python itself a tad trickier, but it moves Fedora's Python closer to upstream. The difference only has impact on ppc64le and armv7hl (considering the architectures built by koji.fedoraproject.org). Packages assuming the filenames always contain %{_arch}-linux%{_gnu} will need to be adapted.

Owner

Current status

  • Targeted release: Fedora 34
  • Last updated: 2020-08-25
  • FESCo issue: <will be assigned by the Wrangler>
  • Tracker bug: <will be assigned by the Wrangler>
  • Release notes tracker: <will be assigned by the Wrangler>

Detailed Description

The Saga

A Long Time Ago in the Fedora Galaxy

Many releases ago, when Fedora hasn't built for power and arm yet, the Python maintainers mapped the Python "platform triplet" to %{_arch}-linux%{_gnu}. This worked. For example, on x86_64, this is x86_64-linux-gnu on Fedora and this is consistent with the "platform triplet" used in filenames in upstream.

The Phantom Technical Debt

Later around the year 2015, as more architectures were added, Python build scripts were patched to use "the Fedora's architecture names":

At the time, that was a reasonable decision: the idea of cross-Linux builds was sci-fi, and Fedora was not trying to stay close to upstream as it is now (we had around 60 patches; today we're down to around 6).

Raise of the Manylinux Wheels

In the meantime, cross-Linux builds become a thing. The manylinux1 standard was created in 2016, allowing to build Python wheels with compiled extension modules on one platform and ship them to many. The first manylinux version only supported x86_64 and i686 and hence it was not impacted by Fedora's patching decisions.

The manylinux standard arguably made the upstream Python packaging ecosystem a much nicer thing to work with. Installing packages with compiled extension modules was no longer such a pain. One could just run pip install numpy and not worry about a Fortran compiler. For that reason, manylinux wheels become widely adapted by the most popular projects.

A New Architecture

With the third manylinux version -- manylinux2014 (created in 2019, named after the oldest Linux it supports -- CentOS 7), support for more architectures was introduced: x86_64, i686, aarch64, armv7l, ppc64, ppc64le, s390x. The adaption of new architectures is somehow slow, because the official manylinux2014 containers only currently (August 2020) exist for x86_64, i686, aarch64, ppc64le and s390x.

Revenge of the Patches

We have discovered a problem with the ppc64le manylinux2014 wheels: The CentOS 7 manylinux2014 container images ship upstream Python without RHEL/CentOS/EPEL patches. When the extension modules it built there, it is named with an upstream named suffix: .cpython-XY(m)-powerpc64le-linux-gnu.so. The wheel is installable on Fedora (with Fedora's patches), but the module won't (even be considered for) import, because Fedora's Python expect the extension to be .cpython-XY(m)-ppc64le-linux-gnu.so.

In theory, we have the same problem on armv7hl, but there are no manylinux2014 containers available for that platform, so there are no such wheels out there (known to us).

Feedback

Benefit to Fedora

Scope

  • Proposal owners:
  • Other developers: N/A (not a System Wide Change)
  • Policies and guidelines: N/A (not a System Wide Change)
  • Trademark approval: N/A (not needed for this Change)
  • Alignment with Objectives:

Upgrade/compatibility impact

N/A (not a System Wide Change)

How To Test

N/A (not a System Wide Change)

User Experience

Dependencies

N/A (not a System Wide Change)

Contingency Plan

  • Contingency mechanism: (What to do? Who will do it?) N/A (not a System Wide Change)
  • Contingency deadline: N/A (not a System Wide Change)
  • Blocks release? N/A (not a System Wide Change), Yes/No
  • Blocks product? product

Documentation

N/A (not a System Wide Change)

Release Notes