Revision as of 23:47, 5 November 2009

Packaging Python modules for Python 3

I hope to add a parallel-installable Python 3 stack to Fedora 13.

See the feature page: https://fedoraproject.org/wiki/Features/Python3F13 and also this thread: https://www.redhat.com/archives/fedora-devel-list/2009-October/msg00054.html

This requires us to come up with a sane way to package Python 3 modules, and this requires us to generalize our python packaging rules to support more than one python runtime.

The existing Python packaging guidelines are here: Packaging/Python

Multiples Python Runtimes

There will be multiple python runtimes, one for each supported major/minor release combination.

Each runtime corresponds to a binary of the form /usr/bin/python$MAJOR.$MINOR

One of these python runtimes is the "system runtime". It can be identified by the destination of the symlink /usr/bin/python. Currently this is /usr/bin/python-2.6

The output of "rpm -q --provides" of each runtime rpm MUST contain a line of the form:

 Provides: python(abi) = $MAJOR-$MINOR

For example, a python-3.1 runtime rpm should have this output:

 Provides: python(abi) = 3.1

Similarly, python modules using these runtimes should have a corresponding "Requires" line.

Layout

Proposed rule: All files with an extension of .py/.pyo/.pyc MUST be either

within a runtime package, and below /usr/lib/python$MAJOR-$MINOR for that runtime, or
for a specific runtime package, and below /usr/lib/python$MAJOR-$MINOR/site-packages for that runtime, or
for the system python runtime.

For example, python code for the 3.1 runtime needs to be below /usr/lib/python3.1/site-packages

.pyo/.pyc files

Compiled .pyo/.pyc files embed a magic number, indicating which python version they are for; python libraries have a corresponding magic number.

Proposed rule: All .pyo/.pyc files below /usr/lib/python$MAJOR.$MINOR MUST have a magic number corresponding to that for /usr/bin/python$MAJOR.$MINOR

Thus e.g. /usr/lib/python2.6/site-packages/libxml2.pyc must have the same magic number as that of /usr/bin/python2.6

Similarly, /usr/lib/python3.1/site-packages/libxml2.pyc must have the same magic number as that of /usr/bin/python3.1

TODO: write an rpmlint test for this. See initial work here: https://www.zarb.org/pipermail/rpmlint-discuss/2009-October/000775.html and here: http://dmalcolm.fedorapeople.org/rpmlint/add-tests-for-python-bytecode-files.patch

Python modules for non-standard runtimes

(to be written)

Naming

Current python package naming guidelines are here: Packaging/NamingGuidelines#Addon_Packages_.28python_modules.29

an rpm with a "python-" prefix means a python 2 rpm, of the "default" python 2 minor version (for Fedora this will be the most recent stable upstream minor release, for EPEL it will be the minor release of 2 that came with the distro, so 2.4 for EPEL5)
an rpm with a "python3-" prefix means a python 3 rpm, of the "default" python 3 minor version (for Fedora this will be the most recent stable upstream release)

What about packages without a "python-" prefix? Proposal: If upstream has a naming convention for python2 vs python3, use it. Otherwise, add a "python3-" prefix to make things clear. I'm not sure about the details here. Examples?

What to do with things that have python in their suffix: gstreamer-python => gstreamer-python3? Or python3-gstreamer? Or python3-gstreamer-python? Most of these are subpackages of existing packages.

A cornercase is the gnome-python2 package. These are python bindings to gnome2. gnome-python2 is the upstream name. For python3, do we want python3-gnome-python2, python3-gnome2, python3-gnome-python2 ?

Common SRPM vs split SRPMs

There are two approaches I'm experimenting with to packaging modules for python 3:

create an separate specfile/srpm for the python 3 version
extend an existing specfile so that it emits a python3- subpackage as part of the build.

Split/separate SRPMs: a src.rpm for python- and another for python3-

Example: python3-setuptools https://bugzilla.redhat.com/show_bug.cgi?id=531648 (simple adaptation of python-setuptools, apparently without needing an invocation of 2to3)

Advantages:

if the python-foo maintainer doesn't care about python 3, he/she doesn't need to
the two specfiles can evolve separately; if 2 and 3 need to have different versions, they can

Disadvantages:

the two specfiles have to be maintained separately
when upstream release e.g. security fixes, they have to be tracked in two places

Single shared SRPM emitting both python- and python3- subpackages

Examples:

Emitting python3-setuptools as a subpackage from python-setuptools: https://bugzilla.redhat.com/show_bug.cgi?id=531895
Emitting python3-lxml as a subpackage from python-lxml: https://bugzilla.redhat.com/show_bug.cgi?id=533290