Python3.4GuidlinesDraft

Multiple Python Runtimes

In Fedora we have multiple python runtimes, one for each supported major release. At this point that's one for Python 2.7 and one for Python 3.4

Each runtime corresponds to a binary of the form /usr/bin/python$MAJOR.$MINOR

One of these python runtimes is the "system runtime" which is what we run when invoking /usr/bin/python. On Fedora 22 this is /usr/bin/python3.4

All python runtimes have a virtual provide for python(abi) = $MAJOR-$MINOR. For example, the python-3.4 runtime rpm has:

 $ rpm -q --provides python3 |grep -i abi
 python(abi) = 3.4

python modules using these runtimes should have a corresponding "Requires" line on the python runtime that they are used with. This is done automatically for files below /usr/lib[^/]*/python${PYVER}

Test your work
Remember to test the built RPMs and verify that they actually work! For instance, when you're packaging a python module that builds for both Python 2 and Python 3, don't test the Python 3 module but ship the Python 2 module without testing that it does what it's supposed to.

For packagers of the python interpreter
Unlike the Requires lines, the "Provides" for each runtime are manually entered into the specfile for each runtime. In theory /usr/lib/rpm/pythondeps.sh would also automatically generate "Provides" lines for the runtime, but in practice rpmbuild only invokes it for files in the rpm payload identified as "python" by the file utility, and the runtime is an ELF binary, not a python script, hence it isn't passed. It's simplest to manually supply the Provides line, rather than change these innards of rpmbuild. See Red Hat Bug 532118.

BuildRequires

To build a package containing Python 2 files, you need to have

BuildRequires: python2-devel

Similarly, when building a package which ships Python 3 files, you need

BuildRequires: python3-devel

A package that has both Python 2 and Python 3 files will need to BuildRequire both.

Macros

Note that the use of %{!? [...]} does allow this to work without the check for rhel versions but putting the conditional in documents when we can remove the entire stanza from the spec file.

In Fedora 13 and greater, the following macros are defined for you:

{

rpmbuild resets the directory at the end of each phase, so you don't need to restore the directory at the end of %prep.

%build
CFLAGS="$RPM_OPT_FLAGS" %{__python2} setup.py build

%if 0%{?with_python3}
pushd %{py3dir}
CFLAGS="$RPM_OPT_FLAGS" %{__python3} setup.py build
popd
%endif # with_python3

%install
rm -rf %{buildroot}

# Must do the python3 install first because the scripts in /usr/bin are
# overwritten with every setup.py install (and we want the python2 version
# to be the default for now).
%if 0%{?with_python3}
pushd %{py3dir}
%{__python3} setup.py install --skip-build --root $RPM_BUILD_ROOT

rm -rf %{buildroot}%{python3_sitelib}/setuptools/tests

find %{buildroot}%{python3_sitelib} -name '*.exe' | xargs rm -f
chmod +x %{buildroot}%{python3_sitelib}/setuptools/command/easy_install.py
popd
%endif # with_python3

%{__python2} setup.py install --skip-build --root $RPM_BUILD_ROOT

rm -rf ${buildroot}%{python2_sitelib}/setuptools/tests

find %{buildroot}%{python2_sitelib} -name '*.exe' | xargs rm -f
chmod +x %{buildroot}%{python2_sitelib}/setuptools/command/easy_install.py

%check
%{__python2} setup.py test

%if 0%{?with_python3}
pushd %{py3dir}
%{__python3} setup.py test
popd
%endif # with_python3

You'll notice that the %build, %install, and %check sections follow a common pattern. They do the normal steps for building the python2 module but then they switch to %{py3dir} and run the same steps for python3. Creating the new sections is generally pretty easy. First copy the existing code. Then wrap it with a pushd/popd to %{py3dir}. The usage of pushd/popd commands will ensure that the directories are logged. Finally, convert all macro references:

• %{__python2} becomes %{__python3}
• %{python2_sitelib} becomes %{python3_sitelib}
• %{python2_sitearch} becomes %{python3_sitearch}

Order can be important
As you can see in the %install section, the order in which you do the python2 versus python3 install can sometimes matter. You need to be aware of when the install is writing to the same file in both packages (in this example, a script in %{_bindir} and make sure that you're getting the version you expect.

%clean
rm -rf $RPM_BUILD_ROOT


%files
%doc psfl.txt zpl.txt docs
%{python2_sitelib}/*
%{_bindir}/easy_install
%{_bindir}/easy_install-2.6

%if 0%{?with_python3}
%files -n python3-setuptools
%doc psfl.txt zpl.txt docs
%{python3_sitelib}/*
%{_bindir}/easy_install-3.4
%endif # with_python3

%changelog

In this final section, you can see that we once again switch macros from %{python2_sitelib} to %{python3_sitelib}. Since we chose to install the python2 version of %{_bindir}/easy_install earlier we need to include that file in the python2 package rather than the python3 subpackage.

Running 2to3 from the spec file

Sometimes, upstream hasn't integrated running 2to3 on the code into their build scripts but they support making a python3 module from it if you manually run 2to3 on the source. This is the case when it's documented on the upstream's website, in a file in the tarball, or even when email with the module's author has instructions for building a python3 module from the python2 source and the authors are willing to support the result. In these cases it's usually just a matter of the upstream not having written the build script that can turn the python2 source into python3. When this happens you can run 2to3 from the spec file. Once you have it working, you can also help upstream integrate it into their build scripts which will benefit everyone in the long term.

You should usually follow upstream's directions on how to run 2to3 and build the python3 module in these cases but there's a few things you should check to make sure upstream is doing it correctly.

• Since the code is being built from a unified source, you need to copy the code to a new directory before invoking 2to3 just like the building more than once method.
• If the 2to3 program is invoked instead of using the lib2to3 library functions, make sure it's invoked with --write --nobackups. --write is needed to make 2to3 actually change the files. --nobackups avoids leaving foo.py.bak files in the module directories that then make it into the final package payload.
• Be sure to run 2to3 on the correct directory. When you run 2to3 you need to run it on the whole tree. A common mistake here for distutils packages has been to run it on the directory below setup.py, missing the setup.py file itself. This leads to errors when python3 tries to execute setup.py
• If you need to run 2to3 to fix code, use 2to3 or /usr/bin/2to3. At the moment, this program is coming from the python-tools rpm. Using 2to3 means that you'll be using a name that is supported upstream and across distros rather than /usr/bin/python3-2to3 which we have renamed in Fedora to avoid filesystem conflicts. This also makes it easier for us to test and eventually change from using the python2 2to3 to the python3 2to3. We just need to change the python3 package to provide the /usr/bin/2to3 program instead of python and all of our python packages will start using that version instead.
• If 2to3 runs into a problem, please file a Fedora bug. Please try to isolate a minimal test case that reproduces the problem when doing so.

Avoiding collisions between the Python 2 and Python 3 stacks

The Python 2 and Python 3 stacks are intended to be fully-installable in parallel. When generalizing the package for both Python 2 and Python 3, it is important to ensure that two different built packages do not attempt to place different payloads into the same path.

Executables in /usr/bin

The problem

Many existing Python packages install executables into /usr/bin.

For example if we have a console_scripts in a setup.py shared between Python 2 and Python 3 builds: these will spit out files in /usr/bin/, and these will collide.

For example python-coverage has a setup.py that contains:

    entry_points = {
        'console_scripts': [
            'coverage = coverage:main',
            ]
        },

which thus generates a /usr/bin/coverage executable (this is a Python script that runs another Python script whilst generating code-coverage information on the latter).

Similarly for the 'scripts' clause; see e.g. python-pygments: Pygments-1.1.1/setup.py has:

    scripts = ['pygmentize'],

which generates a /usr/bin/pygmentize (this is a Python script that leverages the pygments syntax-highlighting module, giving a simple command-line interface for generating syntax-highlighted files)

Guidelines

If the executables provide the same functionality independent of whether they are run on top of Python 2 or Python 3, then only the Python 3 version of the executable should be packaged.

Examples of this:

• /usr/bin/pygmentize ought to generate the same output regardless of whether it's implemented via Python 2 or Python 3, so only one version needs to be shipped.

If the executables provide different functionality for Python 2 and Python 3, then both versions should be packaged.

Examples of this:

• /usr/bin/coverage runs a python script, augmenting the interpreter with code-coverage information. Given that the interpreter itself is the thing being worked with, it's reasonable to package both versions of the executable.
• /usr/bin/bpython augments the interpreter with a "curses" interface. Again, it's reasonable to package both versions of this.
• /usr/bin/easy_install installs a module into one of the Python runtimes: we need a version for each runtime.

As an exception, for the rpms that are part of a python runtime itself, we plan to package both versions of the executables, so that e.g. both the python 2 and python 3 versions of 2to3 are packaged.

Naming

Many executables already contain a "-MAJOR.MINOR" suffix, for example /usr/bin/easy_install-3.4. These obviously can be used as-is, as they won't conflict.

For other executables, the general rule is:

• if only one executable is to be shipped, then it owns its own slot
• if executables are to be shipped for both Python 2 and Python 3, then the Python 2 version of the executable gains a Python2.7- prefix. For example, the python 3 version of "coverage" remains /usr/bin/coverage and the Python 2 version is /usr/bin/python2.7-coverage.

Packaging eggs and setuptools concerns

Eggs can mean several different things because they can be placed on disk in several formats:

• A module and a file with a .egg-info extension that contains the metadata. Created by distutils in Fedora 9 and above.
• As a module and a directory with a .egg-info extension that contains the metadata. Created using setuptools and also the invocation of setup.py in our examples below.
• As a directory with a .egg extension that contains the module and egg metadata. Created when we use easy_install -m to allow installing multiple versions of a module.
• As a single zip file with a .egg extension that contains the module and the egg metadata.

In Fedora packages, these will be installed to %{python2_sitelib} or %{python2_sitearch} directories. We do not install the single zip file version of eggs in Fedora but the three other formats are used.

How to package

The following are a summary of the guidelines for reviewers to go over when a python module is packaged. The complete policy includes examples and rationale for the way we do things.

• Must: Python eggs must be built from source. They cannot simply drop an egg from upstream into the proper directory. (See prebuilt binaries Guidelines for details)
• Must: Python eggs must not download any dependencies during the build process.
• Must: When building a compat package, it must install using easy_install -m so it won't conflict with the main package.
• Must: When building multiple versions (for a compat package) one of the packages must contain a default version that is usable via "import MODULE" with no prior setup.
• Should: A package which is used by another package via an egg interface should provide egg info.

Filtering Requires: and Provides:

RPM's dependency generator can often throw in additional dependencies and will often think packages provide functionality contrary to reality. To fix this, the dependency generator needs to be overriden so that the additional dependencies can be filtered out. See Packaging:AutoProvidesAndRequiresFiltering for details.

PyGTK2 and Numpy

This workaround is necessary until [Gnome bug #591745] is fixed.

If your package uses pygtk2, and calls the gtk.gdk.get_pixels_array() function, that package needs to explicitly Require: numpy. In the past, pygtk2 had a Requires on numpy, but since it is only used for that one function (and that function is not commonly used), the Requires has been removed to minimize the install footprint of pygtk2.