From Fedora Project Wiki

(Clean up the %{py3dir} from potential previous runs)
(Redirect Package Maintainer wiki links to docs.fp.o)
 
(64 intermediate revisions by 3 users not shown)
Line 1: Line 1:
{{Draft}}
 


A parallel-installable Python 3 stack has been added to Fedora 13.  This requires us to update the python guidelines with information on how to package both python2 and python3 modules.  This draft is meant to encompass a sane way to package Python 3 modules, generalize our python packaging rules to support more than one python runtime, and update the python2 guidelines since some things haven't been necessary since Fedora 3.
A parallel-installable Python 3 stack has been added to Fedora 13.  This requires us to update the python guidelines with information on how to package both python2 and python3 modules.  This draft is meant to encompass a sane way to package Python 3 modules, generalize our python packaging rules to support more than one python runtime, and update the python2 guidelines since some things haven't been necessary since Fedora 3.


See the feature page: https://fedoraproject.org/wiki/Features/Python3F13 <BR>
See the feature page: https://fedoraproject.org/wiki/Features/Python3F13 <BR>
and also this thread: https://www.redhat.com/archives/fedora-devel-list/2009-October/msg00054.html <BR>
also this thread: https://www.redhat.com/archives/fedora-devel-list/2009-October/msg00054.html <BR>
 
also the current Guidelines: https://fedoraproject.org/wiki/Packaging:Python <BR>
{{admon/note||Except where noted, this page will replace the existing [[Packaging:Python | Python packaging guidelines]]}}


== Addon Packages (python3 modules) ==
== Addon Packages (python3 modules) ==
Line 36: Line 35:


== Multiple Python Runtimes ==
== Multiple Python Runtimes ==
In Fedora we have multiple python runtimes, one for each supported major release.
 
{{admon/note||The rest of this page will replace the existing [[Packaging:Python | Python packaging guidelines]]}}
 
In Fedora we have multiple python runtimes, one for each supported major release. At this point that's one for python2.x and one for python3.x


Each runtime corresponds to a binary of the form <code>/usr/bin/python$MAJOR.$MINOR</code>
Each runtime corresponds to a binary of the form <code>/usr/bin/python$MAJOR.$MINOR</code>
Line 49: Line 51:


python modules using these runtimes should have a corresponding "Requires" line on the python runtime that they are used with.  This is done automatically for files below <code>/usr/lib[^/]*/python${PYVER}</code>
python modules using these runtimes should have a corresponding "Requires" line on the python runtime that they are used with.  This is done automatically for files below <code>/usr/lib[^/]*/python${PYVER}</code>
{{admon/warning||The script <code>/usr/lib/rpm/pythondeps.sh</code> is what automatically emits "Requires" lines for files below <code>/usr/lib[^/]*/python${PYVER}</code>.  The script needs reworking for python3.  I've rewritten the script, but it isn't yet in our F13 <code>rpm-build</code> rpmThis is being tracked as [[https://bugzilla.redhat.com/show_bug.cgi?id=532118 bug 532118]].}}
 
{{admon/note|For Runtime Packagers|Unlike the Requires lines, the "Provides" for each runtime are manually entered into the specfile for each runtime.  In theory <code>/usr/lib/rpm/pythondeps.sh</code> would also automatically generate "Provides" lines for the runtime, but in practice rpmbuild only invokes it for files in the rpm payload identified as "python" by the <code>file</code> utility, and the runtime is an ELF binary, not a python script, hence it isn't passed.  It's simplest to manually supply the Provides line, rather than change these innards of rpmbuild.  See [[https://bugzilla.redhat.com/show_bug.cgi?id=532118 bug 532118]].}}
{{admon/warning|Test your work| Remember to test the built RPMs and verify that they actually work!  For instance, when you're packaging a python module that builds for both python2 and python3, don't test the python2 module but ship the python3 module without testing that it does what it's supposed toIf you are requesting that an application '''switch''' from Python 2 to Python 3 for its Python implementation, please provide supporting material (e.g. a list of tests performed, and their outcome). Simply getting a package to build against Python 3 is no guarantee that the package's functionality still works.}}
 
{{admon/note|For packagers of the python interpreter|Unlike the Requires lines, the "Provides" for each runtime are manually entered into the specfile for each runtime.  In theory <code>/usr/lib/rpm/pythondeps.sh</code> would also automatically generate "Provides" lines for the runtime, but in practice rpmbuild only invokes it for files in the rpm payload identified as "python" by the <code>file</code> utility, and the runtime is an ELF binary, not a python script, hence it isn't passed.  It's simplest to manually supply the Provides line, rather than change these innards of rpmbuild.  See [[https://bugzilla.redhat.com/show_bug.cgi?id=532118 bug 532118]].}}


== BuildRequires ==
== BuildRequires ==
Line 65: Line 69:
A package that has both python2 and python3 files will need to BuildRequire both.
A package that has both python2 and python3 files will need to BuildRequire both.


=== Macros ===
== Macros ==
In Fedora less than 12 and RHEL less than 5, python2 packages that install python modules need to define macros that tell where certain directory paths are like this:
In Fedora less than 12 and RHEL less than 5, python2 packages that install python modules need to define <code>python_sitelib</code> or <code>python_sitearch</code> macros that tell where to find the python directory that modules are installed in.  This is not needed in Fedora 13 or with python3 modules as the macros are defined by <code>rpm</code> and the <code>python3-devel</code> package.  To define those conditionally you can use this:


<pre>
<pre>
%if ! (0%{?fedora} > 12 || 0%{?rhel} > 5)
%{!?python_sitelib: %global python_sitelib %(%{__python} -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())")}
%{!?python_sitelib: %global python_sitelib %(%{__python} -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())")}
%{!?python_sitearch: %global python_sitearch %(%{__python} -c "from distutils.sysconfig import get_python_lib; print(get_python_lib(1))")}
%{!?python_sitearch: %global python_sitearch %(%{__python} -c "from distutils.sysconfig import get_python_lib; print(get_python_lib(1))")}
%endif
</pre>
</pre>


This is not needed in Fedora 13 or with python3 modules as the macros are defined by <code>rpm</code> and the <code>python3-devel</code> package. Those packages define the following useful macros:
Note that the use of <code>%{!? [...]}</code> does allow this to work without the check for fedora and rhel versions but putting the conditional in documents when we can remove the entire stanza from the spec file.


In Fedora 13 and greater, the following macros are defined for you:
{|
{|
!Macro!!Normal Definition
!Macro!!Normal Definition!!Notes
|-
|-
|__python||/usr/bin/python
|__python||/usr/bin/python||Python 2 interpreter.  Also the default python interpreter
|-
|-
|__python3||/usr/bin/python3
|__python3||/usr/bin/python3||Python 3 interpreter
|-
|-
|python_sitelib||/usr/lib/python2.X/site-packages, where pure python2 modules are installed
|python_sitelib||/usr/lib/python2.X/site-packages||Where pure python2 modules are installed
|-
|-
|python_sitearch||/usr/lib64/python2.X/site-packages on x86_64<BR>/usr/lib/python2.X/site-packages on x86, where python2 extension modules that are compiled C are installed
|python_sitearch||/usr/lib64/python2.X/site-packages on x86_64<BR>/usr/lib/python2.X/site-packages on x86||Where python2 extension modules that are compiled C are installed
|-
|-
|python3_sitelib||/usr/lib/python3.X/site-packages, where pure python3 modules are installed
|python3_sitelib||/usr/lib/python3.X/site-packages||Where pure python3 modules are installed
|-
|-
|python3_sitearch||/usr/lib64/python3.X/site-packages on x86_64<BR>/usr/lib/python3.X/site-packages on x86, where python3 extension modules that are compiled C are installed
|python3_sitearch||/usr/lib64/python3.X/site-packages on x86_64<BR>/usr/lib/python3.X/site-packages on x86||Where python3 extension modules that are compiled C are installed
|-
|py3dir||%{_builddir}/python3-%{name}-%{version}-%{release}||Directory to use when building python3 modules from the same source tarball as python2 modules
|-
|py_byte_compile|| (script) ||Defined in python3-devel.  See the [#Bytecompiling_with_the_correct_python_version bytecompiling] section for usage
|}
|}


During <code>%install</code> or when listing <code>%files</code> you can use these macros to specify the path.  For instance:
During <code>%install</code> or when listing <code>%files</code> you can use the <code>python_sitearch</code> and <code>python_sitelib</code> macros to specify where the installed modules are to be found.  For instance:
 
 


<pre>
<pre>
Line 104: Line 117:
</pre>
</pre>


Using the macros have several benefits.
Using the macros has several benefits.
<ol>
<ol>
<li>It ensures that the packages are installed correctly on multilib architectures.</li>
<li>It ensures that the packages are installed correctly on multilib architectures.</li>
<li>Using these macros instead of hardcoding the directory in the specfile ensures your spec remains compatible with the installed python version even if the directory structure changes radically (for instance, if python_sitelib moves into %{_datadir})</li>
<li>Using these macros instead of hardcoding the directory in the specfile ensures your spec remains compatible with the installed python version even if the directory structure changes radically (for instance, if <code>python_sitelib</code> moves into <code>%{_datadir}</code>)</li>
</ol>
</ol>


=== Byte Compiling ===
== Byte compiling ==
{{admon/warning|brp-python-bytecompile status|The script that does automatic byte compilation is currently broken in two ways1) It embeds the wrong path in the .pyc files which can show up in tracebacks in some circumstances2) On SyntaxError, the script stops which means that not all the files may be byte compiled (or byte compiled with both .pyc and .pyos). [[https://bugzilla.redhat.com/show_bug.cgi?id=558997 bug]]}}
 
Python will automatically try to byte compile files when it runs in order to speed up startup the next time it is runThese files are saved in files with the extension of .pyc (compiled python) or .pyo (optimized compiled python).  These files are a byte code that is portable across OSes.  If you do not include them in your packages, python will try to create them when the user runs the program.  If the system administrator uses them, then the files will be successfully writtenLater, when the package is removed, the .pyc and .pyo files will be left behind on the filesystem.  To prevent that the byte compiled files need to be compiled and included in the <code>%files</code> section.  Normally, byte compilation is done for you by the <code>brp-python-bytecompile</code> script. This script runs after the <code>%install</code> section of the spec file has been processed and byte compiles any .py files that it finds (this recompilation puts the proper filesystem paths into the modules otherwise tracebacks would include the <code>%{buildroot}</code> in them). All that you need to do is include the files in the <code>%files</code> section.  The following are all acceptable ways to accomplish this:
 
<pre>
%install
install -d $RPM_BUILD_ROOT%{python_sitelib}/foo
install -pm 0644 foo.py $RPM_BUILD_ROOT%{python_sitelib}/foo/
 
Either:
 
%files
%{python_sitelib}/foo/
 
Or:
 
%files
%dir %{python_sitelib}/foo
%{python_sitelib}/foo/*
 
Or even:
 
%files
%dir %{python_sitelib}/foo
%{python_sitelib}/foo/foo.py
%{python_sitelib}/foo/foo.pyc
%{python_sitelib}/foo/foo.pyo
</pre>
 
{{admon/warning|Avoid INSTALLED_FILES|python's distutils has an <code>INSTALLED_FILES</code> feature that lists which files are installed when you run <code>python setup.py install</code>.  Do not use it for packaging as that will not list the directories which need to be specified in the <code>%files</code> section as well. Using globs in the <code>%files</code> section is simpler and safer.}}
 
{{admon/warning|Including egg info|When you run <code>%{__python} setup.py install</code> in any current Fedora, distutils generates a <code>.egg-info</code> file with metadata about the python module that is installed.  These files need to be included as well. (See [#Packaging_eggs_and_setuptools_concerns])}}
 
=== Bytecompiling with the correct python version ===
 
When byte compiling a .py file, python embeds a magic number in the byte compiled files that correspond to the runtime.  Files in <code>{%python_sitelib}</code> and <code>%{python_sitearch}</code> must correspond to the runtime for which they were built.  For instance, a pure python module compiled for the 3.1 runtime needs to be below <code>%{_usr}/lib/python3.1/site-packages</code>
When byte compiling a .py file, python embeds a magic number in the byte compiled files that correspond to the runtime.  Files in <code>{%python_sitelib}</code> and <code>%{python_sitearch}</code> must correspond to the runtime for which they were built.  For instance, a pure python module compiled for the 3.1 runtime needs to be below <code>%{_usr}/lib/python3.1/site-packages</code>


Normally, this is done for you by the <code>brp-python-bytecompile</code> script. This script runs after the <code>%install</code> section of the spec file has been processed and byte-compiles any .py files that it finds (this recompilation puts the proper filesystem paths into the modules otherwise tracebacks would include the <code>%{BUILDROOT}</code> in them).  The script determines which interpreter to byte compile the module with by following these steps:
The <code>brp-python-bytecompile</code> script tries to figure this out for you.  The script determines which interpreter to use when byte compiling the module by following these steps:


<ol>
<ol>
<li>what directory is the module installed in?  If it's /usr/lib/pythonX.Y, then pythonX.Y is used to byte compile the module.  If pythonX.Y is not installed, then an error is returned and the rpm build process will exit on an error so remember to BuildRequire the proper python package.</li>
<li>what directory is the module installed in?  If it's <code>/usr/lib{,64}/pythonX.Y</code>, then <code>pythonX.Y</code> is used to byte compile the module.  If <code>pythonX.Y</code> is not installed, then an error is returned and the rpm build process will exit on an error so remember to <code>BuildRequire</code> the proper python package.</li>


<li>the script interpreter defined in __python is used to compile the modules.  This defaults to the latest python2 version on Fedora.  If you need to compile this module for python3, set it to /usr/bin/python3 instead.  Like this:
<li>the script interpreter defined in <code>%{__python}</code> is used to compile the modules.  This defaults to the latest python2 version on Fedora.  If you need to compile this module for python3, set it to <code>/usr/bin/python3</code> instead:


<pre>
<pre>
Line 125: Line 171:
</pre>
</pre>


This step is useful when you have a python3 application that's installing a private module into its own directory.  For instance, if the foobar application installs a module for use only by the command line application in %{_datadir}/foobar.  Since these files are not in one of the python3 library paths (like /usr/lib/python3.1) you have to set <code>%{__python}</code> manually to tell brp-python-bytecompile what python interpreter to byte compile for.
Doing this is useful when you have a python3 application that's installing a private module into its own directory.  For instance, if the foobar application installs a module for use only by the command line application in <code>%{_datadir}/foobar</code>.  Since these files are not in one of the python3 library paths (ie. <code>/usr/lib/python3.1</code>) you have to override <code>%{__python}</code> to tell <code>brp-python-bytecompile</code> to use the python3 interpreter for byte compiling.
</li>
</li>
</ol>
</ol>


These settings are enough to properly byte compile any package that only builds python modules (in <code>%{python_sitelib}</code> or <code>%{python_sitearch}</code>) or builds for only a single python interpreter.  However, if the application you're packaging needs to build with both python2 and python3 and install into a private module directory (perhaps because it provides one utility written in python2 and a second utility written in python3) then you need to do this manually. Here's a sample spec file snippet that shows what to do:
These settings are enough to properly byte compile any package that builds python modules in <code>%{python_sitelib}</code> or <code>%{python_sitearch}</code> or builds for only a single python interpreter.  However, if the application you're packaging needs to build with both python2 and python3 and install into a private module directory (perhaps because it provides one utility written in python2 and a second utility written in python3) then you need to do this manually. Here's a sample spec file snippet that shows what to do:


<pre>
<pre>
# Turn off the brp-python-bytecompile script
# Turn off the brp-python-bytecompile script
%global brp_python_bytecompile %{nil}
%global __os_install_post %(echo '%{__os_install_post}' | sed -e 's!/usr/lib[^[:space:]]*/brp-python-bytecompile!!g')
# Buildrequire both python2 and python3
# Buildrequire both python2 and python3
BuildRequires: python-devel python3-devel
BuildRequires: python2-devel python3-devel
[...]
[...]


Line 145: Line 191:
# Manually invoke the python byte compile macro for each path that needs byte
# Manually invoke the python byte compile macro for each path that needs byte
# compilation.
# compilation.
%{py_byte_compile} /usr/bin/python2 %{buildroot}%{_datadir}/mypackage/foo
%{py_byte_compile} %{__python} %{buildroot}%{_datadir}/mypackage/foo
%{py_byte_compile} /usr/bin/python3 %{buildroot}%{_datadir}/mypackage/bar
%{py_byte_compile} %{__python3} %{buildroot}%{_datadir}/mypackage/bar
</pre>
</pre>


=== Common SRPM vs split SRPMs ===
The <code>%{py_byte_compile</code> macro takes two arguments.  The first is the python interpreter to use for byte compiling.  The second is a file or directory to byte compile.  If the second argument is a directory, the macro will recursively byte compile any *.py file in the directory.


Many times when you package a python module you will want to create a module for python2 and a module for python3.  To do this you can either package the modules in a single SRPM that builds a python2 and python3 subpackage or you can package them in two SRPMSIf the module is distributed as a single tarball then you must package it as a subpackage.  If it comes in multiple tarballs then package it from multiple SRPMs.
=== Including pyos ===
In the past it was common practice to %ghost .pyo files in order to save a small amount of space on the users filesystem.  However, this has two issues:
<ol>
<li>With SELinux, if a user is running python -O [APP]  it will try to write the .pyos when they don't exist.  This leads to AVC denial records in the logs.</li>
<li>If the system administrator runs python -OO [APP]  the .pyos will get created with no docstrings.  Some programs require docstrings in order to function.  On subsequent runs with python -O [APP]  python will use the cached .pyos even though a different optimization level has been requested.  The only way to fix this is to find out where the .pyos are and delete them.</li>
</ol>
 
The current method of dealing with pyo files is to '''include them as is; no %ghosting'''.
 
== Common SRPM vs split SRPMs ==
 
Many times when you package a python module you will want to create a module for python2 and a module for python3.  There are two ways of doing this: either from a single SRPM or from multipleThe rule to choose which method is simple: if the python2 and python3 modules are distributed as a single tarball (many times as a single directory of source where the <code>/usr/bin/2to3</code> program is used to transform the code at buildtime) then you must package them as subpackages built from a single SRPM.  If they come in multiple tarballs then package them from multiple SRPMs.


==== Split/separate SRPMs: a src.rpm for python- and another for python3- ====
{{admon/note|Python Bindings|python bindings are sometimes built as part of the C library's build. The ideal for these is to patch the code so it will build against both python2 and python3.  Then take a copy of the sources during the <code>%prep</code> phase, and configure one subdirectory to build against python 2, another to build against python 3.  These changes should be upstreamed.  Example: the build of <code>rpm</code> itself emits an <code>rpm-python</code> subpackage (see [[https://bugzilla.redhat.com/show_bug.cgi?id=531543 bug 531543]])}}
Given package <code>python-foo</code> in packaging CVS, there would be a separate <code>python3-foo</code> for the python 3 version. There would be no expectation that the two would need to upgrade in lock-step(The two SRPMS  could have different maintainers within Fedora: the packager of a python 2 module might not yet have any interest in python 3)


'''Example''': <code>python3-setuptools</code> https://bugzilla.redhat.com/show_bug.cgi?id=531648
=== Multiple SRPMS ===
(simple adaptation of python-setuptools, apparently without needing an invocation of 2to3)


Dave Malcolm has written a tool which generates a <code>python3-foo.spec</code> from a <code>python-foo.spec</code>; see http://dmalcolm.fedorapeople.org/python3-packaging/rpm2to3.py
When upstream ships multiple tarballs with one tarball containing python2 code and a different tarball containing python3 code, we should ship those as multiple SRPMs.  The two SRPMs could have different maintainers within Fedora and the two packages need not upgrade at the same time. Building from multiple SRPMs has some advantages and disadvantages:


'''Advantages''':
'''Advantages''':
* if the python-foo maintainer doesn't care about python 3, he/she doesn't need to
* There can be separate maintainers for python2 and python3 so each maintainer can concentrate on one stack.
* the two specfiles can evolve separately; if 2 and 3 need to have different versions, they can
* The two packages can evolve separately; if 2 and 3 need to have different versions, they can.


'''Disadvantages''':
'''Disadvantages''':
* the two specfiles have to be maintained separately
* The two specfiles have to be maintained separately
* when upstream release e.g. security fixes, they have to be tracked in two places
* When upstream releases e.g. security fixes, they have to be tracked in two places
 
The following practices are designed to help mitigate the disadvantages listed above:
 
* When packaging a module for python3 contact the maintainers for the python2 module and try to coordinate with them.
* Request at least watchbugzilla and watchcommit acls on each other's packages so you're aware of outstanding bugs.
* Complete any python 2 Merge Review when doing the python 3 version.  Doing this gets issues that apply to both packages addressed at the same time.
* Add a link to the python 2 Merge Review/Package Review to the python 3 Package Review
 
=== Subpackages ===


==== Single shared SRPM emitting both python- and python3- subpackages ====
{{admon/warning|Do not build python3 modules without upstream support|If upstream is shipping a module for python2 and does not support making that module run on python3, do not package a python3 version of it in Fedora. If running <code>2to3</code> or adding a patch enables the code to work, you can certainly tell upstream that it works to encourage them to support python3.  However, doing this on our own in Fedora is essentially creating a forkThat has a large burden for maintaining the code, fixing bugs, porting when a new version of upstream's code appears, managing a release schedule, and other tasks normally handled by upstream.  It's much better if we can cooperate with upstream to share this work than doing it all on our own.}}
'''Method'''
* Use the <code>-n</code> syntax to emit a <code>python3-foo</code> subpackage from a <code>python-foo</code> build.
* Towards the end of the <code>%prep</code> phase, copy the code to a parallel subdirectory, and invoke <code>2to3 --write --nobackups .</code> upon it
  {{admon/note|Use "--write --nobackups" when invoking <code>2to3</code>|You need the "--write" option to make 2to3 actually change the files, and "--nobackups" to avoid leaving <code>foo.py.bak</code> droppings, which otherwise would likely make it into the final package payload.|}}
  {{admon/tip|Run 2to3 on the correct directory|If your specfile runs <code>2to3</code> on the code, make sure you are running it on the full treeA common mistake here for distutils packages has been to run it on the directory below <code>setup.py</code>, missing the <code>setup.py</code> file, leading to errors when <code>python3</code> tries to execute <code>setup.py</code>}}


'''Examples''':
Sometimes upstream will ship one tarball that builds both a python2 and a python3 module.  There's several ways that upstream can structure this.  When upstream writes their build scripts to build both python2 and python3 modules in a single build this is just like building subpackages for any other package.  You expand the tarball and patch the source in <code>%prep</code>, run upstream's build scripts to build the package in <code>%build</code>, and then run upstream's build scripts to install it in <code>%install</code>.
* Emitting <code>python3-setuptools</code> as a subpackage from <code>python-setuptools</code>: https://bugzilla.redhat.com/show_bug.cgi?id=531895
* Emitting <code>python3-lxml</code> as a subpackage from <code>python-lxml</code>: https://bugzilla.redhat.com/show_bug.cgi?id=533290


'''Advantages''':
'''Advantages''':
* single src.rpm and build; avoid having to update multiple packages when things change.
* Single src.rpm to review and build
* Avoids having to update multiple packages when things change.


'''Disadvantages''':
'''Disadvantages''':
* The Fedora maintainer needs to care about python 3.  By adding python 3 to the mix, we're giving them extra work.
* The Fedora maintainer needs to care about both python 2 and python 3 modules which makes more work to maintain that package.
* 2 and 3 versions are in lockstep.  Requires upstream to case about Python 3 as well (or for Python 2, for that matter)
* The 2 and 3 versions are in lockstep.  Bugfixes need to apply to python2 while not breaking the translation into python3.
* Bugzilla components are set up by source RPM, so they would have a single shared bugzilla component.  This could be confusing to end-users, as it would be more difficult to figure out e.g. that a bug with python3-foo needs to be filed against python-foo.  There's a similar problem with checking out package sources from CVS, though this is less serious as it doesn't affect end-users so much.
* Bugzilla components are set up according to source RPM, so they will have a single shared bugzilla component.  This could be confusing to end-users, as it would be more difficult to figure out e.g. that a bug with python3-foo needs to be filed against python-foo.  There's a similar problem with checking out package sources from CVS, though this is less serious as it is less visible to end users.


==== When should we have two split SRPMs vs one shared, and vice versa? ====
Two other ways exist for the upstream to support building python3 modules from a single source:
The easy case is when upstream release separate tarballs for the python
2 and python 3 versions of code. In that case, it makes sense to follow upstream and have separate specfiles, separate source rpms, etc.


The more difficult case is when the python module is emitted as part of
==== Building more than once ====
the build of a larger module. 


One case is for an extension module giving python bindings for a library built within the larger rpmSome examples:
One way that's currently very common is for the build scripts to create either a python2 or python3 module based on which interpreter is used to run the setup.py script.  (The [http://cvs.fedoraproject.org/viewvc/rpms/python-setuptools/devel/python-setuptools.spec?revision=1.31&view=markup python-setuptools package] is currently built this way).
* the build of <code>rpm</code> itself emits an <code>rpm-python</code> subpackage (see https://bugzilla.redhat.com/show_bug.cgi?id=531543 )
* Another example is the <code>postgres</code> srpm, which emits a <code>postgresql-python</code> subpackage.
* libvirt


I believe the ideal here is to patch the code so that it will build against both python versions, then take a copy of the sources during the %prep phase, and configure one subdirectory to build against python 2, another to build against python 3.
===== Example spec file =====


=== Guidelines for adding python3 subpackages to an existing package ===
==== Provide a <code>with_python3</code> conditional ====
{{Draft}}
All parts of the build relating to python3 should be conditionalized, to make it easy to turn off the python3 build when tracking down problems.
You should add this fragment to the top of the source file:
<pre>
<pre>
%if 0%{?fedora} > 12
%if 0%{?fedora} > 12 || 0%{?rhel} > 6
%global with_python3 1
%global with_python3 1
%else
%{!?python_sitelib: %global python_sitelib %(%{__python} -c "from distutils.sysconfig import get_python_lib; print (get_python_lib())")}
%endif
%endif
%global srcname distribute
</pre>
</pre>


Rationale: we should consistently use "with_python3". The conditionals also make it easy to use the same spec for RHEL and other branches than <code>devel</code>.
At the top of our spec file we have the standard define for <code>python_sitelib</code> on older Fedora releases.  We also define <code>with_python3</code> which we'll use to conditionalize the build whenever we have a section that is only useful when building a python3 module.  Using <code>with_python3</code> allows us to do two things:
 
<ol>
<li>It makes it easy to turn off the python3 build when tracking down problems.</li>
<li>The conditionals also make it easy to use the same spec for older releases of Fedora and EPEL.</li>.
</ol>


Once python 3 support has been added to a package, you must leave it enabled.  End users could be using the python3 subpackage that is being built and turning the subpackage build on and off will cause the package to unexpectedly disappear from the repos.  You should only turn off <code>with_python3</code> as a debugging measure within scratch builds, or for releases that do not support python 3.
{{admon/warning|Leave python3 module enabled in releases|Once python 3 support has been added to a package, you must leave it enabled.  End users could be using the python3 subpackage that is being built.  If you turn the subpackage build on and off it will cause the package to unexpectedly disappear from the repos.  You should only turn off <code>with_python3</code> as a debugging measure within scratch builds, for releases that do not support python 3, or when moving a python3 module into its own, independent package.}}


All usage of this macro should look like this:
<pre>
<pre>
Name:          python-setuptools
Version:        0.6.10
Release:        2%{?dist}
Summary:        Easily build and distribute Python packages
Group:          Applications/System
License:        Python or ZPLv2.0
URL:            http://pypi.python.org/pypi/%{srcname}
Source0:        http://pypi.python.org/packages/source/d/%{srcname}/%{srcname}-%{version}.tar.gz
# Fix a failing test case
Patch0:        python-setuptools-test.patch
BuildRoot:      %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n)
BuildArch:      noarch
BuildRequires:  python2-devel
%if 0%{?with_python3}
%if 0%{?with_python3}
...
BuildRequires:  python3-devel
%endif # with_python3
%endif # if with_python3
</pre>
</pre>
This way the code will be disabled if the macro is not defined, and it is easy to visually match if/endif pairs


==== Separate python 2 and python 3 build directories ====
When we build the python3 module in addition to the python3 module we need both <code>python2-devel</code> and <code>python3-devel</code>.
The python 2 and python 3 build should be as independent as possible.


You should use the <code>%{py3dir}</code> macro to specify the location of the python 3 build directory, so that the python 2 sources (e.g. <code>%{_builddir}/Foo-1.0/</code>) are entirely independent from the python 3 sources (e.g. <code>%{_builddir}/python3-foo-1.0-1.fc13/</code>).
<pre>
%description
Setuptools is a collection of enhancements to the Python distutils that allow
you to more easily build and distribute Python packages, especially ones that
have dependencies on other packages.
 
This package contains the runtime components of setuptools, necessary to
execute the software that requires pkg_resources.py.
 
%if 0%{?with_python3}
%package -n python3-setuptools
Summary:        Easily build and distribute Python 3 packages
Group:          Applications/System


The <code>%{py3dir}</code> macro is defined for you in <code>python3-devel</code> (from 3.1.1-19.fc13 onwards) in <code>/etc/rpm/macros.python3</code> as:
%description -n python3-setuptools
<pre>
Setuptools is a collection of enhancements to the Python 3 distutils that allow
%py3dir %{_builddir}/python3-%{name}-%{version}-%{release}
you to more easily build and distribute Python 3 packages, especially ones that
have dependencies on other packages.
 
This package contains the runtime components of setuptools, necessary to
execute the software that requires pkg_resources.py.
%endif # with_python3
</pre>
</pre>
so you should not define it yourself.


==== The <code>%prep</code> phase ====
Here we define the python3 subpackage.  Note that we use <code>%package -n</code> to name the module appropriately.
The <code>%prep</code> phase of the build should prepare an entirely distinct source tree for the python3 build in the py3dir.


A recommended way to do this is to add this to the <code>%prep</code> section:
<pre>
<pre>
%prep
%prep
%setup -q -n %{srcname}-%{version}
%patch0 -p1 -b .testfix
find -name '*.txt' | xargs chmod -x


%if 0%{?with_python3}
%if 0%{?with_python3}
rm -rf %{py3dir}
rm -rf %{py3dir}
cp -a . %{py3dir}
cp -a . %{py3dir}
find %{py3dir} -name '*.py' | xargs sed -i '1s|^#!python|#!%{__python3}|'
%endif # with_python3
%endif # with_python3
find -name '*.py' | xargs sed -i '1s|^#!python|#!%{__python}|'
</pre>
</pre>
Make sure that you are copying the correct code.  The above code assumes that you are within the top of the sources directory (typically with the "Foo-1.0" within the build).  If the <code>%prep</code> has changed directory you will need to change back to the tarball location.


If your package requires you to apply some patches only to the python 2 build, and some patches only to the python 3 build, you should structure your <code>%prep</code> like this:
 
Our method in building from the same code to make the two separate modules is to keep each build as independent as possible.  To do that, we copy the source tree to <code>%{py3dir}</code> so that the python 2 sources are entirely independent from the python 3 sources.  Some things to watch out for:
 
* Be sure to clean up the <code>%{py3dir}</code> before performing the copy.  It's easy to forget that since <code>%setup</code> does that automatically for the python2 module.
* Make sure that you are copying the correct code.  The example is copying the code from within the top directory of the untarred source.  If the <code>%prep</code> has changed directory you will need to change back to the tarball location.
* Patching the source code is done before copying to <code>%{py3dir}</code>.  Since you have both a python2 and a python3 directory you might be tempted to patch each one separately.  '''Resist!'''  Upstream for your package has chosen to distribute a single source tree that builds for both python2 and python3.  For your patches to [https://docs.fedoraproject.org/en-US/package-maintainers/Staying_Close_to_Upstream_Projects/ get into upstream], you need to write patches that work with both as well.}}
 
<code>rpmbuild</code> resets the directory at the end of each phase, so you don't need to restore the directory at the end of <code>%prep</code>.
 
<pre>
<pre>
%setup
%build
CFLAGS="$RPM_OPT_FLAGS" %{__python} setup.py build
 
%if 0%{?with_python3}
pushd %{py3dir}
CFLAGS="$RPM_OPT_FLAGS" %{__python3} setup.py build
popd
%endif # with_python3
 
%install
rm -rf %{buildroot}
 
# Must do the python3 install first because the scripts in /usr/bin are
# overwritten with every setup.py install (and we want the python2 version
# to be the default for now).
%if 0%{?with_python3}
pushd %{py3dir}
%{__python3} setup.py install --skip-build --root $RPM_BUILD_ROOT
 
rm -rf %{buildroot}%{python3_sitelib}/setuptools/tests
 
find %{buildroot}%{python3_sitelib} -name '*.exe' | xargs rm -f
chmod +x %{buildroot}%{python3_sitelib}/setuptools/command/easy_install.py
popd
%endif # with_python3
 
%{__python} setup.py install --skip-build --root $RPM_BUILD_ROOT
 
rm -rf ${buildroot}%{python_sitelib}/setuptools/tests
 
find %{buildroot}%{python_sitelib} -name '*.exe' | xargs rm -f
chmod +x %{buildroot}%{python_sitelib}/setuptools/command/easy_install.py


# Apply patches relevant to both python 2 and python 3:
%check
%patch0
%{__python} setup.py test
%patch1
...


# Create source tree for python3 build:
%if 0%{?with_python3}
%if 0%{?with_python3}
cp -a . %{py3dir}
pushd %{py3dir}
%{__python3} setup.py test
popd
%endif # with_python3
%endif # with_python3
</pre>
</pre>


{{admon/note|Avoid version specific patches|Since you have both a python2 and a python3 directory you might be tempted to patch each one separately'''Resist!'''  Upstream for your package has chosen to distribute a single source tree that builds for both python2 and python3.  For your patches to [[Staying_close_to_upstream_projects| get into upstream]], you need to write patches that work with both as well.}}
You'll notice that the <code>%build</code>, <code>%install</code>, and <code>%check</code> sections follow a common patternThey do the normal steps for building the python2 module but then they switch to <code>%{py3dir}</code> and run the same steps for python3.  Creating the new sections is generally pretty easy.  First copy the existing code.  Then wrap it with a <code>pushd/popd</code> to <code>%{py3dir}</code>.  The usage of <code>pushd/popd</code> commands will ensure that the directories are logged. Finally, convert all macro references:


<code>rpmbuild</code> resets the directory at the end of each phase, so you don't need to restore the directory at the end of <code>%prep</code>.
* <code>%{__python}</code> becomes <code>%{__python3}</code>
* <code>%{python_sitelib}</code> becomes <code>%{python3_sitelib}</code>
* <code>%{python_sitearch}</code> becomes <code>%{python3_sitearch}</code>


==== Other phases ====
{{admon/warning|Order can be important|As you can see in the <code>%install</code> section, the order in which you do the python2 versus python3 install can sometimes matter.  You need to be aware of when the install is writing to the same file in both packages (in this example, a script in <code>%{_bindir}</code> and make sure that you're getting the version you expect.}}
For each of the <code>%build</code>, <code>%check</code> and <code>%install</code> phases, you should copy the existing code, wrapping it with a pushd/popd of <code>%{py3dir}</code>, and convert all macro references:
* from <code>%{__python}</code> to <code>%{__python3}</code>,  
* <code>%{python_sitelib}</code> to <code>%{python3_sitelib}</code> and  
* <code>%{python_sitearch}</code> to <code>%{python3_sitearch}</code>.


For example, this <code>%build</code> section:
<pre>
<pre>
CFLAGS="$RPM_OPT_FLAGS" %{__python} setup.py build
%clean
</pre>
rm -rf $RPM_BUILD_ROOT
 


should become:
%files
<pre>
%defattr(-,root,root,-)
# Python 2:
%doc psfl.txt zpl.txt docs
CFLAGS="$RPM_OPT_FLAGS" %{__python} setup.py build
%{python_sitelib}/*
%{_bindir}/easy_install
%{_bindir}/easy_install-2.6


# Python 3:
%if 0%{?with_python3}
%if 0%{?with_python3}
pushd %{py3dir}
%files -n python3-setuptools
CFLAGS="$RPM_OPT_FLAGS" %{__python3} setup.py build
%defattr(-,root,root,-)
popd
%doc psfl.txt zpl.txt docs
%{python3_sitelib}/*
%{_bindir}/easy_install-3.1
%endif # with_python3
%endif # with_python3
%changelog
</pre>
</pre>
so that the python 2 and python 3 versions of the code line up vertically, making it easier to see differencesThe usage of pushd/popd commands will ensure that the directories are logged.
 
In this final section, you can see that we once again switch macros from <code>%python_sitelib}</code> to <code>%{python3_sitelib}</code>.  Since we chose to install the python2 version of <code>%{_bindir}/easy_install</code> earlier we need to include that file in the python2 package rather than the python3 subpackage.
Rationale: it's not easily possible to turn this into a loop (FIXME: is it?) due to the macro differences, so we must unroll the loop and repeat ourselves.
 
==== Running 2to3 from the spec file ====
Sometimes, upstream hasn't integrated running 2to3 on the code into their build scripts but they support making a python3 module from it if you manually run 2to3 on the source.  This is the case when it's documented on the upstream's website, in a file in the tarball, or even when email with the module's author has instructions for building a python3 module from the python2 source and the authors are willing to support the resultIn these cases it's usually just a matter of the upstream not having written the build script that can turn the python2 source into python3.  When this happens you can run <code>2to3</code> from the spec file.  Once you have it working, you can also help upstream integrate it into their build scripts which will benefit everyone in the long term.
 
You should usually follow upstream's directions on how to run <code>2to3</code> and build the python3 module in these cases but there's a few things you should check to make sure upstream is doing it correctly.
 
* Since the code is being built from a unified source, you need to copy the code to a new directory before invoking 2to3 just like the [#Building_more_than_once "building more than once"] method.
* If the <code>2to3</code> program is invoked instead of using the <code>lib2to3</code> library functions, make sure it's invoked with <code>--write --nobackups</code>.  <code>--write</code> is needed to make <code>2to3</code> actually change the files.  <code>--nobackups</code> avoids leaving <code>foo.py.bak</code> files in the module directories that then make it into the final package payload.
* Be sure to run 2to3 on the correct directory.  When you run <code>2to3</code> you need to run it on the whole tree.  A common mistake here for distutils packages has been to run it on the directory below <code>setup.py</code>, missing the <code>setup.py</code> file itself.  This leads to errors when <code>python3</code> tries to execute <code>setup.py</code>
* If you need to run <code>2to3</code> to fix code, use <code>2to3</code> or <code>/usr/bin/2to3</code>.  At the moment, this program is coming from the <code>python-tools</code> rpm.  Using <code>2to3</code> means that you'll be using a name that is supported upstream and across distros rather than <code>/usr/bin/python3-2to3</code> which we have renamed in Fedora to avoid filesystem conflicts.  This also makes it easier for us to test and eventually change from using the python2 <code>2to3</code> to the python3 <code>2to3</code>.  We just need to change the python3 package to provide the <code>/usr/bin/2to3</code> program instead of python and all of our python packages will start using that version instead.
* If <code>2to3</code> runs into a problem, please [https://bugzilla.redhat.com/enter_bug.cgi?component=python&product=Fedora file a Fedora bug].  Please try to isolate a minimal test case that reproduces the problem when doing so.


== Avoiding collisions between the python 2 and python 3 stacks ==
== Avoiding collisions between the python 2 and python 3 stacks ==
Line 301: Line 440:


=== Executables in <code>/usr/bin</code> ===
=== Executables in <code>/usr/bin</code> ===
{{Draft}}
==== The problem ====
==== The problem ====
Many existing python packages install executables into <code>/usr/bin</code>.  
Many existing python packages install executables into <code>/usr/bin</code>.  
Line 330: Line 468:


==== Guidelines ====
==== Guidelines ====
If the executables provide the same functionality independent of whether they are run on top of Python 2 or Python 3, then only one version of the executable should be packaged.  Currently it will be the python 2 implementation, but once the Python 3 implementation is proven to work, the executable can be retired from the python 2 build and enabled in the python 3 package.  Be sure to test the new implementation.  FOR DISCUSSION: how do we do the transition period?
If the executables provide the same functionality independent of whether they are run on top of Python 2 or Python 3, then only one version of the executable should be packaged.  Currently it will be the python 2 implementation, but once the Python 3 implementation is proven to work, the executable can be retired from the python 2 build and enabled in the python 3 package.  Be sure to test the new implementation.  Transitioning from python2 to python3 is left to individual package maintainers except for packages in Fedora's critical path.  For these, we want to port to python3 versions in the same Fedora release if possible.


Examples of this:
Examples of this:
Line 349: Line 487:
For other executables, the general rule is:
For other executables, the general rule is:
* if only one executable is to be shipped, then it owns its own slot
* if only one executable is to be shipped, then it owns its own slot
* if executables are to be shipped for both python 2 and python 3, then the python 3 version of the executable gains a <code>python3-</code> prefix.  For example, the python 2 version of "coverage" remains <code>/usr/bin/coverage</code> and the python 3 version is <code>/usr/bin/python3-coverage</code>. FOR DISCUSSION: should the python 2 version gain a <code>python2-</code> prefix, and have the main path becoming a symlink to the <code>python2-</code> version?
* if executables are to be shipped for both python 2 and python 3, then the python 3 version of the executable gains a <code>python3-</code> prefix.  For example, the python 2 version of "coverage" remains <code>/usr/bin/coverage</code> and the python 3 version is <code>/usr/bin/python3-coverage</code>.
See [http://lists.fedoraproject.org/pipermail/devel/2010-January/129217.html this thread] for a discussion of this.


See [http://lists.fedoraproject.org/pipermail/devel/2010-January/129217.html this thread] for a discussion of this.
== Packaging eggs and setuptools concerns ==
 
{{admon/note||To be moved from the [Packaging:Python/Eggs#What_are_eggs] page}}
 
Eggs can mean several different things because they can be placed on disk in several formats:
 
* A module and a file with a .egg-info extension that contains the metadata.  Created by distutils in Fedora 9 and above.
* As a module and a directory with a .egg-info extension that contains the metadata.  Created using setuptools and also the invocation of setup.py in our examples below.
* As a directory with a .egg extension that contains the module and egg metadata.  Created when we use easy_install -m to allow installing multiple versions of a module.
* As a single zip file with a .egg extension that contains the module and the egg metadata.
 
In Fedora packages, these will be installed to %{python_sitelib} or %{python_sitearch} directories. We do not install the single zip file version of eggs in Fedora but the three other formats are used.
 
=== How to package ===
 
The following are a summary of the guidelines for reviewers to go over when a python module is packaged.  The [[Packaging/Python/Eggs|  complete policy]]  includes examples and rationale for the way we do things.
 
* '''Must''': Python eggs must be built from source.  They cannot simply drop an egg from upstream into the proper directory. (See [[Packaging:Guidelines#No_inclusion_of_pre-built_binaries_or_libraries| prebuilt binaries Guidelines]] for details)
* '''Must''': Python eggs must not download any dependencies during the build process.
* '''Must''': If egg-info files are generated by the module's build scripts they must be included in the package.
* '''Must''': When building a compat package, it must install using easy_install -m so it won't conflict with the main package.
* '''Must''': When building multiple versions (for a compat package) one of the packages must contain a default version that is usable via "import MODULE" with no prior setup.
* '''Should''': A package which is used by another package via an egg interface should provide egg info.


== Best Practices ==
== PyGTK2 and Numpy ==
{{Draft}}
{{admon/note||This is a temporary workaround which may be resolved in the future. It will no longer be necessary when [[http://bugzilla.gnome.org/show_bug.cgi?id=591745 gnome bug #591745]] is fixed.}}
Recommended best-practices for keeping python 2 and python 3 in sync:
* when packaging a module for python 3, you should approach the python 2 package owners.
* if separate maintainership for python 2 vs python 3 modules, you should request a watchbugzilla and watchcommit on each other's packages
* complete any python 2 Merge Review before doing a python 3 version
* add link to the python 2 Merge Review/Package Review to the python 3 Package Review
* if you need to run 2to3 to fix code, use <code>2to3</code> to use the <code>/usr/bin/2to3</code> from the <code>python-tools</code> rpm, rather than <code>/usr/bin/2to3-3</code> from the <code>python3-tools</code> rpm (rationale: <code>2to3</code> is the standard upstream name for this tool).
* if <code>2to3</code> runs into a problem, please [https://bugzilla.redhat.com/enter_bug.cgi?component=python&product=Fedora file a bug]. Please try to isolate a minimal test case that reproduces the problem when doing so.
* remember to test the built RPMs and verify that they actually work!


== Anti best Practices ==
If your package uses pygtk2, and calls the gtk.gdk.get_pixels_array() function, that package needs to explicitly Require: numpy. In the past, pygtk2 had a Requires on numpy, but since it is only used for that one function (and that function is not commonly used), the Requires has been removed to minimize the install footprint of pygtk2.
{{admon/warning||You shouldn't rely on INSTALLED_FILES, as that will not list directories, which will need to be specified in the %files section as well. Using globs in the %files section is safer.}}


== TODO ==
== TODO ==


These items need to be addressed before the Guidelines can be brought to the Packaging Committee
These items need to be addressed before the Guidelines can be brought to the Packaging Committee
* Should get [https://bugzilla.redhat.com/show_bug.cgi?id=532118 bug 532118] addressed so the Requires: python(abi) is automatically extracted.  Then we can remove the warning from the [[#Multiple_Runtimes]] section.  If it can't be done we'll need to tell people to manually specify "python(abi) = %{py3ver}" and define py3ver in the python3-devel package.
* '''[done]''' Should get [https://bugzilla.redhat.com/show_bug.cgi?id=532118 bug 532118] addressed so the Requires: python(abi) is automatically extracted.  Then we can remove the warning from the [[#Multiple_Runtimes]] section.  If it can't be done we'll need to tell people to manually specify "python(abi) = %{py3ver}" and define py3ver in the python3-devel package.
* Must get this bug fixed so byte compilation does not stop midway through if there's a SyntaxError [https://bugzilla.redhat.com/show_bug.cgi?id=558997 bug 558997]
* '''[done]''' Must get this bug fixed so byte compilation does not stop midway through if there's a SyntaxError [https://bugzilla.redhat.com/show_bug.cgi?id=558997 bug 558997]
* '''[done]''' brp_python_bytecompile updated
* '''[done]''' brp_python_bytecompile updated
* Write py_byte_compile macro to do manual byte compilation and get it into rpm or redhat-rpm-config so it's available to packagers
* '''[done]''' py_byte_compile macro to do manual byte compilation is in python3-devel as of python3-devel-3.1.1-21.fc13
* Approve the Naming Guidelines rewrite
* '''[done]''' it may be possible to install a common py3dir definition within <code>/etc/rpm/macros.python3</code>; we are researching whether this is techically feasible.  If it is, then that macro should simply be used.
* '''[done]''' it may be possible to install a common py3dir definition within <code>/etc/rpm/macros.python3</code>; we are researching whether this is techically feasible.  If it is, then that macro should simply be used.
* we probably need a retirement process for backing out python3 subpackages.
* '''[done]''' Script to find collisions between python 2 and python 3 builds: http://dmalcolm.fedorapeople.org/find-duplicate-provides.py
* Script to find collisions between python 2 and python 3 builds: https://bugzilla.redhat.com/attachment.cgi?id=383125
* '''[done]''' 2to3 vs 2to3-3 vs python3-2to3
* '''[done]''' 2to3 vs 2to3-3
* '''[done]''' merge current python2 guidelines
* how up-to-date is rpm2to3.py?


== Notes ==
== Notes ==
* we considered a <code>with_python2</code> macro, but for now it's simpler to omit it
* we considered a <code>with_python2</code> macro, but for now it's simpler to omit it
* we considered supplying: __python2, python2_sitelib and python2_sitearch in addition to the existing macros and gradually shift to the former, but for now it's simplest to keep using the existing macros
* we considered supplying: __python2, python2_sitelib and python2_sitearch in addition to the existing macros and gradually shift to the former, but for now it's simplest to keep using the existing macros
* re /usr/bin paths, we considered that the python 2 version could gain a <code>python2-</code> prefix, and have the main path becoming a symlink to the <code>python2-</code> version, but that's not necessary for now; we can look at that in the future
* we will probably eventually need a retirement process for backing out python3 subpackages; I don't see that as a blocker for these guidelines (dmalcolm)
* Dave Malcolm has written a tool to aid in the creation of split SRPM python3 packages.  It generates a <code>python3-foo.spec</code> from a <code>python-foo.spec</code>; see http://dmalcolm.fedorapeople.org/python3-packaging/rpm2to3.py  However, that tool was written in November 2009 and is not up-to-date with the latest version of these guidelines.  (He does not regard finishing the tool as a blocker for having these guidelines approved).
== Questions and discussion ==
*
== Additional Material ==
The following was added after the 2010-02-03 FPC meeting and probably requires further discussion:
=== BuildRequires for 2to3 ===
If your specfile manually invokes /usr/bin/2to3, add an explicit build-time requirement on the <code>/usr/bin/2to3</code>
This is currently provided by the python-tools package, but might in the future be provided by the python3-tools package.
Remember to add it within the 0%{with_python3} conditional:
<pre>
%if 0%{?with_python3}
BuildRequires:  /usr/bin/2to3
BuildRequires:  python3-setuptools, python3-devel
%endif # with_python3
</pre>
You don't need to do this if you're using lib2to3 (e.g. implicitly using python-setuptools, a.k.a Distribute: we ship lib2to3 within the core python and python3 subpackages.  Note however that in some cases the setup.py may need 2to3 to be run on it before it is valid Python 3 code).
[[Category:Archived packaging guideline drafts]]

Latest revision as of 11:33, 9 October 2021


A parallel-installable Python 3 stack has been added to Fedora 13. This requires us to update the python guidelines with information on how to package both python2 and python3 modules. This draft is meant to encompass a sane way to package Python 3 modules, generalize our python packaging rules to support more than one python runtime, and update the python2 guidelines since some things haven't been necessary since Fedora 3.

See the feature page: https://fedoraproject.org/wiki/Features/Python3F13
also this thread: https://www.redhat.com/archives/fedora-devel-list/2009-October/msg00054.html
also the current Guidelines: https://fedoraproject.org/wiki/Packaging:Python

Addon Packages (python3 modules)

An rpm with a python prefix or suffix means a python2 rpm so we need a different prefix to denote python3 packages. For this, we use python3. We have two constraints that the python2 packages don't operate under:

  1. We need to be clear about these modules being for python3 so we don't have an exception for packages that already have "py" in their names like python2 modules.
  2. Consumers of the packages need to be able to find them even if they don't know whether they're using the python2 or python3 version.

So all python3 modules MUST have python3 in their name. Other than that, the module must be in the same format as the python2 package. Some examples:

Fedora python 2 package Upstream name Proposed python 3 package name
python-lxml lxml python3-lxml
pygtk2 pygtk python3-pygtk
gstreamer-python gst-python gstreamer-python3
gnome-python2 gnome-python gnome-python3
rpm-python (part of rpm) rpm-python3


Multiple Python Runtimes

The rest of this page will replace the existing Python packaging guidelines

In Fedora we have multiple python runtimes, one for each supported major release. At this point that's one for python2.x and one for python3.x

Each runtime corresponds to a binary of the form /usr/bin/python$MAJOR.$MINOR

One of these python runtimes is the "system runtime". It can be identified by the destination of the symlink /usr/bin/python. Currently this is /usr/bin/python-2.6

Currently /usr/bin/python is actually a duplicate copy of the ELF file, rather than a symlink. This shouldn't cause any problems for packagers of python modules but we see this as [a bug] that needs fixing.

All python runtimes have a virtual provide for python(abi) = $MAJOR-$MINOR. For example, the python-3.1 runtime rpm has:

 $ rpm -q --provides python3 |grep -i abi
 python(abi) = 3.1

python modules using these runtimes should have a corresponding "Requires" line on the python runtime that they are used with. This is done automatically for files below /usr/lib[^/]*/python${PYVER}

Test your work
Remember to test the built RPMs and verify that they actually work! For instance, when you're packaging a python module that builds for both python2 and python3, don't test the python2 module but ship the python3 module without testing that it does what it's supposed to. If you are requesting that an application switch from Python 2 to Python 3 for its Python implementation, please provide supporting material (e.g. a list of tests performed, and their outcome). Simply getting a package to build against Python 3 is no guarantee that the package's functionality still works.
For packagers of the python interpreter
Unlike the Requires lines, the "Provides" for each runtime are manually entered into the specfile for each runtime. In theory /usr/lib/rpm/pythondeps.sh would also automatically generate "Provides" lines for the runtime, but in practice rpmbuild only invokes it for files in the rpm payload identified as "python" by the file utility, and the runtime is an ELF binary, not a python script, hence it isn't passed. It's simplest to manually supply the Provides line, rather than change these innards of rpmbuild. See [bug 532118].

BuildRequires

To build a package containing python2 files, you need to have

BuildRequires: python2-devel

Similarly, when building a package which ships python3 files, you need

BuildRequires: python3-devel

A package that has both python2 and python3 files will need to BuildRequire both.

Macros

In Fedora less than 12 and RHEL less than 5, python2 packages that install python modules need to define python_sitelib or python_sitearch macros that tell where to find the python directory that modules are installed in. This is not needed in Fedora 13 or with python3 modules as the macros are defined by rpm and the python3-devel package. To define those conditionally you can use this:

%if ! (0%{?fedora} > 12 || 0%{?rhel} > 5)
%{!?python_sitelib: %global python_sitelib %(%{__python} -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())")}
%{!?python_sitearch: %global python_sitearch %(%{__python} -c "from distutils.sysconfig import get_python_lib; print(get_python_lib(1))")}
%endif

Note that the use of %{!? [...]} does allow this to work without the check for fedora and rhel versions but putting the conditional in documents when we can remove the entire stanza from the spec file.

In Fedora 13 and greater, the following macros are defined for you:

Macro Normal Definition Notes
__python /usr/bin/python Python 2 interpreter. Also the default python interpreter
__python3 /usr/bin/python3 Python 3 interpreter
python_sitelib /usr/lib/python2.X/site-packages Where pure python2 modules are installed
python_sitearch /usr/lib64/python2.X/site-packages on x86_64
/usr/lib/python2.X/site-packages on x86
Where python2 extension modules that are compiled C are installed
python3_sitelib /usr/lib/python3.X/site-packages Where pure python3 modules are installed
python3_sitearch /usr/lib64/python3.X/site-packages on x86_64
/usr/lib/python3.X/site-packages on x86
Where python3 extension modules that are compiled C are installed
py3dir %{_builddir}/python3-%{name}-%{version}-%{release} Directory to use when building python3 modules from the same source tarball as python2 modules
py_byte_compile (script) Defined in python3-devel. See the [#Bytecompiling_with_the_correct_python_version bytecompiling] section for usage

During %install or when listing %files you can use the python_sitearch and python_sitelib macros to specify where the installed modules are to be found. For instance:


%files
%defattr(-,root,root,-)
# A pure python2 module
%{python_sitelib}/foomodule/
# A compiled python2 extension module
%{python_sitearch}/barmodule/
# A compiled python3 extension module
%{python3_sitearch}/bazmodule/

Using the macros has several benefits.

  1. It ensures that the packages are installed correctly on multilib architectures.
  2. Using these macros instead of hardcoding the directory in the specfile ensures your spec remains compatible with the installed python version even if the directory structure changes radically (for instance, if python_sitelib moves into %{_datadir})

Byte compiling

Python will automatically try to byte compile files when it runs in order to speed up startup the next time it is run. These files are saved in files with the extension of .pyc (compiled python) or .pyo (optimized compiled python). These files are a byte code that is portable across OSes. If you do not include them in your packages, python will try to create them when the user runs the program. If the system administrator uses them, then the files will be successfully written. Later, when the package is removed, the .pyc and .pyo files will be left behind on the filesystem. To prevent that the byte compiled files need to be compiled and included in the %files section. Normally, byte compilation is done for you by the brp-python-bytecompile script. This script runs after the %install section of the spec file has been processed and byte compiles any .py files that it finds (this recompilation puts the proper filesystem paths into the modules otherwise tracebacks would include the %{buildroot} in them). All that you need to do is include the files in the %files section. The following are all acceptable ways to accomplish this:

%install
install -d $RPM_BUILD_ROOT%{python_sitelib}/foo
install -pm 0644 foo.py $RPM_BUILD_ROOT%{python_sitelib}/foo/

Either:

%files
%{python_sitelib}/foo/

Or:

%files
%dir %{python_sitelib}/foo
%{python_sitelib}/foo/*

Or even:

%files
%dir %{python_sitelib}/foo
%{python_sitelib}/foo/foo.py
%{python_sitelib}/foo/foo.pyc
%{python_sitelib}/foo/foo.pyo
Avoid INSTALLED_FILES
python's distutils has an INSTALLED_FILES feature that lists which files are installed when you run python setup.py install. Do not use it for packaging as that will not list the directories which need to be specified in the %files section as well. Using globs in the %files section is simpler and safer.
Including egg info
When you run %{__python} setup.py install in any current Fedora, distutils generates a .egg-info file with metadata about the python module that is installed. These files need to be included as well. (See [#Packaging_eggs_and_setuptools_concerns])

Bytecompiling with the correct python version

When byte compiling a .py file, python embeds a magic number in the byte compiled files that correspond to the runtime. Files in {%python_sitelib} and %{python_sitearch} must correspond to the runtime for which they were built. For instance, a pure python module compiled for the 3.1 runtime needs to be below %{_usr}/lib/python3.1/site-packages

The brp-python-bytecompile script tries to figure this out for you. The script determines which interpreter to use when byte compiling the module by following these steps:

  1. what directory is the module installed in? If it's /usr/lib{,64}/pythonX.Y, then pythonX.Y is used to byte compile the module. If pythonX.Y is not installed, then an error is returned and the rpm build process will exit on an error so remember to BuildRequire the proper python package.
  2. the script interpreter defined in %{__python} is used to compile the modules. This defaults to the latest python2 version on Fedora. If you need to compile this module for python3, set it to /usr/bin/python3 instead:
    %global __python %{__python3}
    

    Doing this is useful when you have a python3 application that's installing a private module into its own directory. For instance, if the foobar application installs a module for use only by the command line application in %{_datadir}/foobar. Since these files are not in one of the python3 library paths (ie. /usr/lib/python3.1) you have to override %{__python} to tell brp-python-bytecompile to use the python3 interpreter for byte compiling.

These settings are enough to properly byte compile any package that builds python modules in %{python_sitelib} or %{python_sitearch} or builds for only a single python interpreter. However, if the application you're packaging needs to build with both python2 and python3 and install into a private module directory (perhaps because it provides one utility written in python2 and a second utility written in python3) then you need to do this manually. Here's a sample spec file snippet that shows what to do:

# Turn off the brp-python-bytecompile script
%global __os_install_post %(echo '%{__os_install_post}' | sed -e 's!/usr/lib[^[:space:]]*/brp-python-bytecompile!!g')
# Buildrequire both python2 and python3
BuildRequires: python2-devel python3-devel
[...]

%install
# Installs a python2 private module into %{buildroot}%{_datadir}/mypackage/foo
# and installs a python3 private module into %{buildroot}%{_datadir}/mypackage/bar
make install DESTDIR=%{buildroot}

# Manually invoke the python byte compile macro for each path that needs byte
# compilation.
%{py_byte_compile} %{__python} %{buildroot}%{_datadir}/mypackage/foo
%{py_byte_compile} %{__python3} %{buildroot}%{_datadir}/mypackage/bar

The %{py_byte_compile macro takes two arguments. The first is the python interpreter to use for byte compiling. The second is a file or directory to byte compile. If the second argument is a directory, the macro will recursively byte compile any *.py file in the directory.

Including pyos

In the past it was common practice to %ghost .pyo files in order to save a small amount of space on the users filesystem. However, this has two issues:

  1. With SELinux, if a user is running python -O [APP] it will try to write the .pyos when they don't exist. This leads to AVC denial records in the logs.
  2. If the system administrator runs python -OO [APP] the .pyos will get created with no docstrings. Some programs require docstrings in order to function. On subsequent runs with python -O [APP] python will use the cached .pyos even though a different optimization level has been requested. The only way to fix this is to find out where the .pyos are and delete them.

The current method of dealing with pyo files is to include them as is; no %ghosting.

Common SRPM vs split SRPMs

Many times when you package a python module you will want to create a module for python2 and a module for python3. There are two ways of doing this: either from a single SRPM or from multiple. The rule to choose which method is simple: if the python2 and python3 modules are distributed as a single tarball (many times as a single directory of source where the /usr/bin/2to3 program is used to transform the code at buildtime) then you must package them as subpackages built from a single SRPM. If they come in multiple tarballs then package them from multiple SRPMs.

Python Bindings
python bindings are sometimes built as part of the C library's build. The ideal for these is to patch the code so it will build against both python2 and python3. Then take a copy of the sources during the %prep phase, and configure one subdirectory to build against python 2, another to build against python 3. These changes should be upstreamed. Example: the build of rpm itself emits an rpm-python subpackage (see [bug 531543])

Multiple SRPMS

When upstream ships multiple tarballs with one tarball containing python2 code and a different tarball containing python3 code, we should ship those as multiple SRPMs. The two SRPMs could have different maintainers within Fedora and the two packages need not upgrade at the same time. Building from multiple SRPMs has some advantages and disadvantages:

Advantages:

  • There can be separate maintainers for python2 and python3 so each maintainer can concentrate on one stack.
  • The two packages can evolve separately; if 2 and 3 need to have different versions, they can.

Disadvantages:

  • The two specfiles have to be maintained separately
  • When upstream releases e.g. security fixes, they have to be tracked in two places

The following practices are designed to help mitigate the disadvantages listed above:

  • When packaging a module for python3 contact the maintainers for the python2 module and try to coordinate with them.
  • Request at least watchbugzilla and watchcommit acls on each other's packages so you're aware of outstanding bugs.
  • Complete any python 2 Merge Review when doing the python 3 version. Doing this gets issues that apply to both packages addressed at the same time.
  • Add a link to the python 2 Merge Review/Package Review to the python 3 Package Review

Subpackages

Do not build python3 modules without upstream support
If upstream is shipping a module for python2 and does not support making that module run on python3, do not package a python3 version of it in Fedora. If running 2to3 or adding a patch enables the code to work, you can certainly tell upstream that it works to encourage them to support python3. However, doing this on our own in Fedora is essentially creating a fork. That has a large burden for maintaining the code, fixing bugs, porting when a new version of upstream's code appears, managing a release schedule, and other tasks normally handled by upstream. It's much better if we can cooperate with upstream to share this work than doing it all on our own.

Sometimes upstream will ship one tarball that builds both a python2 and a python3 module. There's several ways that upstream can structure this. When upstream writes their build scripts to build both python2 and python3 modules in a single build this is just like building subpackages for any other package. You expand the tarball and patch the source in %prep, run upstream's build scripts to build the package in %build, and then run upstream's build scripts to install it in %install.

Advantages:

  • Single src.rpm to review and build
  • Avoids having to update multiple packages when things change.

Disadvantages:

  • The Fedora maintainer needs to care about both python 2 and python 3 modules which makes more work to maintain that package.
  • The 2 and 3 versions are in lockstep. Bugfixes need to apply to python2 while not breaking the translation into python3.
  • Bugzilla components are set up according to source RPM, so they will have a single shared bugzilla component. This could be confusing to end-users, as it would be more difficult to figure out e.g. that a bug with python3-foo needs to be filed against python-foo. There's a similar problem with checking out package sources from CVS, though this is less serious as it is less visible to end users.

Two other ways exist for the upstream to support building python3 modules from a single source:

Building more than once

One way that's currently very common is for the build scripts to create either a python2 or python3 module based on which interpreter is used to run the setup.py script. (The python-setuptools package is currently built this way).

Example spec file
%if 0%{?fedora} > 12 || 0%{?rhel} > 6
%global with_python3 1
%else
%{!?python_sitelib: %global python_sitelib %(%{__python} -c "from distutils.sysconfig import get_python_lib; print (get_python_lib())")}
%endif

%global srcname distribute

At the top of our spec file we have the standard define for python_sitelib on older Fedora releases. We also define with_python3 which we'll use to conditionalize the build whenever we have a section that is only useful when building a python3 module. Using with_python3 allows us to do two things:

  1. It makes it easy to turn off the python3 build when tracking down problems.
  2. The conditionals also make it easy to use the same spec for older releases of Fedora and EPEL.
  3. .
Leave python3 module enabled in releases
Once python 3 support has been added to a package, you must leave it enabled. End users could be using the python3 subpackage that is being built. If you turn the subpackage build on and off it will cause the package to unexpectedly disappear from the repos. You should only turn off with_python3 as a debugging measure within scratch builds, for releases that do not support python 3, or when moving a python3 module into its own, independent package.
Name:           python-setuptools
Version:        0.6.10
Release:        2%{?dist}
Summary:        Easily build and distribute Python packages

Group:          Applications/System
License:        Python or ZPLv2.0
URL:            http://pypi.python.org/pypi/%{srcname}
Source0:        http://pypi.python.org/packages/source/d/%{srcname}/%{srcname}-%{version}.tar.gz
# Fix a failing test case
Patch0:         python-setuptools-test.patch
BuildRoot:      %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n)

BuildArch:      noarch
BuildRequires:  python2-devel
%if 0%{?with_python3}
BuildRequires:  python3-devel
%endif # if with_python3

When we build the python3 module in addition to the python3 module we need both python2-devel and python3-devel.

%description
Setuptools is a collection of enhancements to the Python distutils that allow
you to more easily build and distribute Python packages, especially ones that
have dependencies on other packages.

This package contains the runtime components of setuptools, necessary to
execute the software that requires pkg_resources.py.

%if 0%{?with_python3}
%package -n python3-setuptools
Summary:        Easily build and distribute Python 3 packages
Group:          Applications/System

%description -n python3-setuptools
Setuptools is a collection of enhancements to the Python 3 distutils that allow
you to more easily build and distribute Python 3 packages, especially ones that
have dependencies on other packages.

This package contains the runtime components of setuptools, necessary to
execute the software that requires pkg_resources.py.
%endif # with_python3

Here we define the python3 subpackage. Note that we use %package -n to name the module appropriately.

%prep
%setup -q -n %{srcname}-%{version}

%patch0 -p1 -b .testfix

find -name '*.txt' | xargs chmod -x

%if 0%{?with_python3}
rm -rf %{py3dir}
cp -a . %{py3dir}
find %{py3dir} -name '*.py' | xargs sed -i '1s|^#!python|#!%{__python3}|'
%endif # with_python3

find -name '*.py' | xargs sed -i '1s|^#!python|#!%{__python}|'


Our method in building from the same code to make the two separate modules is to keep each build as independent as possible. To do that, we copy the source tree to %{py3dir} so that the python 2 sources are entirely independent from the python 3 sources. Some things to watch out for:

  • Be sure to clean up the %{py3dir} before performing the copy. It's easy to forget that since %setup does that automatically for the python2 module.
  • Make sure that you are copying the correct code. The example is copying the code from within the top directory of the untarred source. If the %prep has changed directory you will need to change back to the tarball location.
  • Patching the source code is done before copying to %{py3dir}. Since you have both a python2 and a python3 directory you might be tempted to patch each one separately. Resist! Upstream for your package has chosen to distribute a single source tree that builds for both python2 and python3. For your patches to get into upstream, you need to write patches that work with both as well.}}

rpmbuild resets the directory at the end of each phase, so you don't need to restore the directory at the end of %prep.

%build
CFLAGS="$RPM_OPT_FLAGS" %{__python} setup.py build

%if 0%{?with_python3}
pushd %{py3dir}
CFLAGS="$RPM_OPT_FLAGS" %{__python3} setup.py build
popd
%endif # with_python3

%install
rm -rf %{buildroot}

# Must do the python3 install first because the scripts in /usr/bin are
# overwritten with every setup.py install (and we want the python2 version
# to be the default for now).
%if 0%{?with_python3}
pushd %{py3dir}
%{__python3} setup.py install --skip-build --root $RPM_BUILD_ROOT

rm -rf %{buildroot}%{python3_sitelib}/setuptools/tests

find %{buildroot}%{python3_sitelib} -name '*.exe' | xargs rm -f
chmod +x %{buildroot}%{python3_sitelib}/setuptools/command/easy_install.py
popd
%endif # with_python3

%{__python} setup.py install --skip-build --root $RPM_BUILD_ROOT

rm -rf ${buildroot}%{python_sitelib}/setuptools/tests

find %{buildroot}%{python_sitelib} -name '*.exe' | xargs rm -f
chmod +x %{buildroot}%{python_sitelib}/setuptools/command/easy_install.py

%check
%{__python} setup.py test

%if 0%{?with_python3}
pushd %{py3dir}
%{__python3} setup.py test
popd
%endif # with_python3

You'll notice that the %build, %install, and %check sections follow a common pattern. They do the normal steps for building the python2 module but then they switch to %{py3dir} and run the same steps for python3. Creating the new sections is generally pretty easy. First copy the existing code. Then wrap it with a pushd/popd to %{py3dir}. The usage of pushd/popd commands will ensure that the directories are logged. Finally, convert all macro references:

  • %{__python} becomes %{__python3}
  • %{python_sitelib} becomes %{python3_sitelib}
  • %{python_sitearch} becomes %{python3_sitearch}
Order can be important
As you can see in the %install section, the order in which you do the python2 versus python3 install can sometimes matter. You need to be aware of when the install is writing to the same file in both packages (in this example, a script in %{_bindir} and make sure that you're getting the version you expect.
%clean
rm -rf $RPM_BUILD_ROOT


%files
%defattr(-,root,root,-)
%doc psfl.txt zpl.txt docs
%{python_sitelib}/*
%{_bindir}/easy_install
%{_bindir}/easy_install-2.6

%if 0%{?with_python3}
%files -n python3-setuptools
%defattr(-,root,root,-)
%doc psfl.txt zpl.txt docs
%{python3_sitelib}/*
%{_bindir}/easy_install-3.1
%endif # with_python3

%changelog

In this final section, you can see that we once again switch macros from %python_sitelib} to %{python3_sitelib}. Since we chose to install the python2 version of %{_bindir}/easy_install earlier we need to include that file in the python2 package rather than the python3 subpackage.

Running 2to3 from the spec file

Sometimes, upstream hasn't integrated running 2to3 on the code into their build scripts but they support making a python3 module from it if you manually run 2to3 on the source. This is the case when it's documented on the upstream's website, in a file in the tarball, or even when email with the module's author has instructions for building a python3 module from the python2 source and the authors are willing to support the result. In these cases it's usually just a matter of the upstream not having written the build script that can turn the python2 source into python3. When this happens you can run 2to3 from the spec file. Once you have it working, you can also help upstream integrate it into their build scripts which will benefit everyone in the long term.

You should usually follow upstream's directions on how to run 2to3 and build the python3 module in these cases but there's a few things you should check to make sure upstream is doing it correctly.

  • Since the code is being built from a unified source, you need to copy the code to a new directory before invoking 2to3 just like the [#Building_more_than_once "building more than once"] method.
  • If the 2to3 program is invoked instead of using the lib2to3 library functions, make sure it's invoked with --write --nobackups. --write is needed to make 2to3 actually change the files. --nobackups avoids leaving foo.py.bak files in the module directories that then make it into the final package payload.
  • Be sure to run 2to3 on the correct directory. When you run 2to3 you need to run it on the whole tree. A common mistake here for distutils packages has been to run it on the directory below setup.py, missing the setup.py file itself. This leads to errors when python3 tries to execute setup.py
  • If you need to run 2to3 to fix code, use 2to3 or /usr/bin/2to3. At the moment, this program is coming from the python-tools rpm. Using 2to3 means that you'll be using a name that is supported upstream and across distros rather than /usr/bin/python3-2to3 which we have renamed in Fedora to avoid filesystem conflicts. This also makes it easier for us to test and eventually change from using the python2 2to3 to the python3 2to3. We just need to change the python3 package to provide the /usr/bin/2to3 program instead of python and all of our python packages will start using that version instead.
  • If 2to3 runs into a problem, please file a Fedora bug. Please try to isolate a minimal test case that reproduces the problem when doing so.

Avoiding collisions between the python 2 and python 3 stacks

The python 2 and python 3 stacks are intended to be fully-installable in parallel. When generalizing the package for both python 2 and python 3, it is important to ensure that two different built packages do not attempt to place different payloads into the same path.

Executables in /usr/bin

The problem

Many existing python packages install executables into /usr/bin.

For example if we have a console_scripts in a setup.py shared between python 2 and python 3 builds: these will spit out files in /usr/bin/, and these will collide.

For example python-coverage has a setup.py that contains:

    entry_points = {
        'console_scripts': [
            'coverage = coverage:main',
            ]
        },

which thus generates a /usr/bin/coverage executable (this is a python script that runs another python script whilst generating code-coverage information on the latter).

Similarly for the 'scripts' clause; see e.g. python-pygments: Pygments-1.1.1/setup.py has:

    scripts = ['pygmentize'],

which generates a /usr/bin/pygmentize (this is a python script that leverages the pygments syntax-highlighting module, giving a simple command-line interface for generating syntax-highlighted files)

Guidelines

If the executables provide the same functionality independent of whether they are run on top of Python 2 or Python 3, then only one version of the executable should be packaged. Currently it will be the python 2 implementation, but once the Python 3 implementation is proven to work, the executable can be retired from the python 2 build and enabled in the python 3 package. Be sure to test the new implementation. Transitioning from python2 to python3 is left to individual package maintainers except for packages in Fedora's critical path. For these, we want to port to python3 versions in the same Fedora release if possible.

Examples of this:

  • /usr/bin/pygmentize ought to generate the same output regardless of whether it's implemented via Python 2 or Python 3, so only one version needs to be shipped.

If the executables provide different functionality for Python 2 and Python 3, then both versions should be packaged.

Examples of this:

  • /usr/bin/coverage runs a python script, augmenting the interpreter with code-coverage information. Given that the interpreter itself is the thing being worked with, it's reasonable to package both versions of the executable.
  • /usr/bin/bpython augments the interpreter with a "curses" interface. Again, it's reasonable to package both versions of this.
  • /usr/bin/easy_install installs a module into one of the Python runtimes: we need a version for each runtime.

As an exception, for the rpms that are part of a python runtime itself, we plan to package both versions of the executables, so that e.g. both the python 2 and python 3 versions of 2to3 are packaged.

Naming

Many executables already contain a "-MAJOR.MINOR" suffix, for example /usr/bin/easy_install-3.1. These obviously can be used as-is, as they won't conflict.

For other executables, the general rule is:

  • if only one executable is to be shipped, then it owns its own slot
  • if executables are to be shipped for both python 2 and python 3, then the python 3 version of the executable gains a python3- prefix. For example, the python 2 version of "coverage" remains /usr/bin/coverage and the python 3 version is /usr/bin/python3-coverage.

See this thread for a discussion of this.

Packaging eggs and setuptools concerns

To be moved from the [Packaging:Python/Eggs#What_are_eggs] page

Eggs can mean several different things because they can be placed on disk in several formats:

  • A module and a file with a .egg-info extension that contains the metadata. Created by distutils in Fedora 9 and above.
  • As a module and a directory with a .egg-info extension that contains the metadata. Created using setuptools and also the invocation of setup.py in our examples below.
  • As a directory with a .egg extension that contains the module and egg metadata. Created when we use easy_install -m to allow installing multiple versions of a module.
  • As a single zip file with a .egg extension that contains the module and the egg metadata.

In Fedora packages, these will be installed to %{python_sitelib} or %{python_sitearch} directories. We do not install the single zip file version of eggs in Fedora but the three other formats are used.

How to package

The following are a summary of the guidelines for reviewers to go over when a python module is packaged. The complete policy includes examples and rationale for the way we do things.

  • Must: Python eggs must be built from source. They cannot simply drop an egg from upstream into the proper directory. (See prebuilt binaries Guidelines for details)
  • Must: Python eggs must not download any dependencies during the build process.
  • Must: If egg-info files are generated by the module's build scripts they must be included in the package.
  • Must: When building a compat package, it must install using easy_install -m so it won't conflict with the main package.
  • Must: When building multiple versions (for a compat package) one of the packages must contain a default version that is usable via "import MODULE" with no prior setup.
  • Should: A package which is used by another package via an egg interface should provide egg info.

PyGTK2 and Numpy

This is a temporary workaround which may be resolved in the future. It will no longer be necessary when [gnome bug #591745] is fixed.

If your package uses pygtk2, and calls the gtk.gdk.get_pixels_array() function, that package needs to explicitly Require: numpy. In the past, pygtk2 had a Requires on numpy, but since it is only used for that one function (and that function is not commonly used), the Requires has been removed to minimize the install footprint of pygtk2.

TODO

These items need to be addressed before the Guidelines can be brought to the Packaging Committee

  • [done] Should get bug 532118 addressed so the Requires: python(abi) is automatically extracted. Then we can remove the warning from the #Multiple_Runtimes section. If it can't be done we'll need to tell people to manually specify "python(abi) = %{py3ver}" and define py3ver in the python3-devel package.
  • [done] Must get this bug fixed so byte compilation does not stop midway through if there's a SyntaxError bug 558997
  • [done] brp_python_bytecompile updated
  • [done] py_byte_compile macro to do manual byte compilation is in python3-devel as of python3-devel-3.1.1-21.fc13
  • [done] it may be possible to install a common py3dir definition within /etc/rpm/macros.python3; we are researching whether this is techically feasible. If it is, then that macro should simply be used.
  • [done] Script to find collisions between python 2 and python 3 builds: http://dmalcolm.fedorapeople.org/find-duplicate-provides.py
  • [done] 2to3 vs 2to3-3 vs python3-2to3
  • [done] merge current python2 guidelines

Notes

  • we considered a with_python2 macro, but for now it's simpler to omit it
  • we considered supplying: __python2, python2_sitelib and python2_sitearch in addition to the existing macros and gradually shift to the former, but for now it's simplest to keep using the existing macros
  • re /usr/bin paths, we considered that the python 2 version could gain a python2- prefix, and have the main path becoming a symlink to the python2- version, but that's not necessary for now; we can look at that in the future
  • we will probably eventually need a retirement process for backing out python3 subpackages; I don't see that as a blocker for these guidelines (dmalcolm)
  • Dave Malcolm has written a tool to aid in the creation of split SRPM python3 packages. It generates a python3-foo.spec from a python-foo.spec; see http://dmalcolm.fedorapeople.org/python3-packaging/rpm2to3.py However, that tool was written in November 2009 and is not up-to-date with the latest version of these guidelines. (He does not regard finishing the tool as a blocker for having these guidelines approved).

Questions and discussion

Additional Material

The following was added after the 2010-02-03 FPC meeting and probably requires further discussion:

BuildRequires for 2to3

If your specfile manually invokes /usr/bin/2to3, add an explicit build-time requirement on the /usr/bin/2to3 This is currently provided by the python-tools package, but might in the future be provided by the python3-tools package.

Remember to add it within the 0%{with_python3} conditional:

%if 0%{?with_python3}
BuildRequires:  /usr/bin/2to3
BuildRequires:  python3-setuptools, python3-devel
%endif # with_python3

You don't need to do this if you're using lib2to3 (e.g. implicitly using python-setuptools, a.k.a Distribute: we ship lib2to3 within the core python and python3 subpackages. Note however that in some cases the setup.py may need 2to3 to be run on it before it is valid Python 3 code).