From Fedora Project Wiki

(Note that the page is outdated and link to the new site.)
(82 intermediate revisions by 6 users not shown)
Line 1: Line 1:
== Multiple Python Runtimes ==
{{DISPLAYTITLE:Fedora Packaging Guidelines for Python}}
<div style="float: right; margin-left: 0.5em" class="toclimit-2">__TOC__</div>


In Fedora we have multiple python runtimes, one for each supported major releaseAt this point that's one for python2.x and one for python3.x
{{admon/warning | This is an old copy of a packaging guideline, preserved here in the wiki while we complete the transition to the Fedora documentation systemThe current version is located at https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/ .  Please update your bookmarks.}}


Each runtime corresponds to a binary of the form <code>/usr/bin/python$MAJOR.$MINOR</code>
== Python Version Support ==


One of these python runtimes is the "system runtime" which is what we run when invoking <code>/usr/bin/python</code>On Fedora 13 this is <code>/usr/bin/python-2.6</code>
In Fedora we have multiple Python runtimes, one for each supported major Python release. At this point that's one for python3.x and one for python2.7  However the python2 stack will be removed from Fedora and is [[Packaging:Deprecating_Packages|deprecated]]. Upstream support for the python2 interpreter officially ends in 2020. If a piece of software supports python3, it MUST be packaged for python3. Software using python2 MUST NOT be newly packaged without FESCo exception.


All python runtimes have a virtual provide for <code>python(abi) = $MAJOR-$MINOR</code>.  For example, the python-3.1 runtime rpm has:
For guidelines on maintaining already existing python2 packages, see the [[Packaging:Python_Appendix|appendix]].
  $ rpm -q --provides python3 |grep -i abi
  python(abi) = 3.1


python modules using these runtimes should have a corresponding "Requires" line on the python runtime that they are used with.  This is done automatically for files below <code>/usr/lib[^/]*/python${PYVER}</code>
== Multiple Python Runtimes ==


{{admon/warning|Test your work| Remember to test the built RPMs and verify that they actually work!  For instance, when you're packaging a python module that builds for both python2 and python3, don't test the python2 module but ship the python3 module without testing that it does what it's supposed to.  If you are requesting that an application '''switch''' from Python 2 to Python 3 for its Python implementation, please provide supporting material (e.g. a list of tests performed, and their outcome).  Simply getting a package to build against Python 3 is no guarantee that the package's functionality still works.}}
For backwards compatibility, <code>/usr/bin/python</code> is, if it is installed, a symbolic link to <code>/usr/bin/python2</code>.


{{admon/note|For packagers of the python interpreter|Unlike the Requires lines, the "Provides" for each runtime are manually entered into the specfile for each runtime.  In theory <code>/usr/lib/rpm/pythondeps.sh</code> would also automatically generate "Provides" lines for the runtime, but in practice rpmbuild only invokes it for files in the rpm payload identified as "python" by the <code>file</code> utility, and the runtime is an ELF binary, not a python script, hence it isn't passedIt's simplest to manually supply the Provides line, rather than change these innards of rpmbuild.  See [[rhbug:532118|Red Hat Bug 532118]].}}
Packages in Fedora MUST NOT use <code>/usr/bin/python</code>. Instead packages for Python 3 MUST use <code>/usr/bin/python3</code> (even if upstream supports both Python 2 and 3). As a result of that <code>/usr/bin/python</code> (as well as <code>/usr/bin/env python</code> and similar) MUST NOT be used in shebang lines or as a dependency of a package.  As of Fedora 30, all uses of unversioned python executables in shebang lines will fail the build.
These shebangs MUST be fixedIf it is necessary to disable the checks, please see the information in [[Packaging:Guidelines#Shebang_lines]].


== BuildRequires ==
All python runtimes have a virtual provide for <code>python(abi) = $MAJOR-$MINOR</code>.  For example, the Python 3.7 runtime package has:
To build a package containing python2 files, you need to have
  $ rpm -q --provides python3 |grep -i abi
<pre>
  python(abi) = 3.7
BuildRequires: python2-devel
</pre>


Similarly, when building a package which ships python3 files, you need
Python modules using these runtimes should have a corresponding "Requires" line on the python runtime that they are used with.  This is done automatically for files below <code>/usr/lib[^/]*/python${PYVER}</code>
<pre>
BuildRequires: python3-devel
</pre>


A package that has both python2 and python3 files will need to BuildRequire both.
Mirroring the policy for regular packages, the Python-version-specific subpackages of your package MUST NOT be removed in a release branch of Fedora.


== Macros ==
== Naming ==
In RHEL 5 and older, python2 packages that install python modules need to define <code>python_sitelib</code> or <code>python_sitearch</code> macros that tell where to find the python directory that modules are installed in.  This is not needed in current Fedora or with python3 modules as the macros are defined by <code>rpm</code> and the <code>python3-devel</code> package. To define those conditionally you can use this:
The source package for a Python library MUST be named with the <code>python-</code> prefix. A built package however must include the Python major version in the name, using the <code>python3-</code> prefix. This is accomplished by adding a subpackage. See example bellow.


<pre>
This rule does not apply to applications.
%if 0%{?rhel} && 0%{?rhel} <= 5
%{!?python_sitelib: %global python_sitelib %(%{__python} -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())")}
%{!?python_sitearch: %global python_sitearch %(%{__python} -c "from distutils.sysconfig import get_python_lib; print(get_python_lib(1))")}
%endif
</pre>


Note that the use of <code>%{!? [...]}</code> does allow this to work without the check for rhel versions but putting the conditional in documents when we can remove the entire stanza from the spec file.
== Dependencies ==
Packages building for Python 3 will need <code>BuildRequires: python3-devel</code>.


In Fedora 13 and greater, the following macros are defined for you:
Packages MUST NOT have dependencies (either build-time or runtime) on packages named with the unversioned <code>python-</code> prefix. Dependencies on Python packages instead MUST use names beginning with <code>python3-</code>.
{|
!Macro!!Normal Definition!!Notes
|-
|__python||/usr/bin/python||Python 2 interpreter.  Also the default python interpreter
|-
|__python3||/usr/bin/python3||Python 3 interpreter
|-
|python_sitelib||/usr/lib/python2.X/site-packages||Where pure python2 modules are installed
|-
|python_sitearch||/usr/lib64/python2.X/site-packages on x86_64<BR>/usr/lib/python2.X/site-packages on x86||Where python2 extension modules that are compiled C are installed
|-
|python3_sitelib||/usr/lib/python3.X/site-packages||Where pure python3 modules are installed
|-
|python3_sitearch||/usr/lib64/python3.X/site-packages on x86_64<BR>/usr/lib/python3.X/site-packages on x86||Where python3 extension modules that are compiled C are installed
|-
|py3dir||%{_builddir}/python3-%{name}-%{version}-%{release}||Directory to use when building python3 modules from the same source tarball as python2 modules
|-
|py_byte_compile|| (script) ||Defined in python3-devel. See the [#Bytecompiling_with_the_correct_python_version bytecompiling] section for usage
|}


During <code>%install</code> or when listing <code>%files</code> you can use the <code>python_sitearch</code> and <code>python_sitelib</code> macros to specify where the installed modules are to be found.  For instance:


=== Automatically generated dependencies ===


Packages MAY use the automatic Python dependency generator. This generator uses upstream egg/dist metadata (such as [https://python-packaging.readthedocs.io/en/latest/dependencies.html setuptool's install_requires]) to determine what the package should depend on. The generator parses the installed metadata from <code>/usr/lib(64)?/pythonX.Y/site-packages/<module>-<version>-pyX.Y.(egg|dist)-info/requires.txt</code>, so it will not work with software that uses plain [https://docs.python.org/3/distutils/ distutils].


<pre>
To enable this feature, add:
%files
# A pure python2 module
%{python_sitelib}/foomodule/
# A compiled python2 extension module
%{python_sitearch}/barmodule/
# A compiled python3 extension module
%{python3_sitearch}/bazmodule/
</pre>


Using the macros has several benefits.
%{?python_enable_dependency_generator}
<ol>
<li>It ensures that the packages are installed correctly on multilib architectures.</li>
<li>Using these macros instead of hardcoding the directory in the specfile ensures your spec remains compatible with the installed python version even if the directory structure changes radically (for instance, if <code>python_sitelib</code> moves into <code>%{_datadir}</code>)</li>
</ol>


== Files to include ==
Although this statement can be used anywhere in the spec, we recommend putting it just before the main package's <code>%description</code> declaration.
When installing python modules we include several different types of files.
This generates run time requires in the form of <code>pythonX.Ydist(foo)</code>. If the generated dependencies are not accurate, additional ones can still be added manuallyTo remove some, a packager MAY modify upstream-provided metadata (usually specified in the <code>setup.py</code> file) in the <code>%prep</code> section of the specfile or fall back to [[Packaging:AutoProvidesAndRequiresFiltering|filtering]] those dependencies.
<ul>
<li>*.py source files because they are used when generating tracebacks</li>
<li>*.pyc and *.pyo byte compiled files
<ul><li>python will try to create them at runtime if they don't exist which leads to spurious SELinux AVC denials in the logs</li>
<li>If the system administrator invokes python with -OO, .pyos will be created with no docstringsThis can break some programs.</li>
</ul>
<li>*.egg-info files or directories.  If these are generated by the module's build scripts they must be included in the package because they might be needed by other applications and modules at runtime.</li>  
</ul>


== Source files ==
The packager MUST inspect the generated requires for correctness. All dependencies MUST be resolvable within the targeted Fedora version.


Source files (*.py) must be included in the same packages as the byte-compiled
As an example, the upstream notebook package has (as of version 5.6.0):
versions of them.


== Byte compiling ==
install_requires = [
    'jinja2',
    'tornado>=4',
    'pyzmq>=17',
    'ipython_genutils',
    'traitlets>=4.2.1',
    'jupyter_core>=4.4.0',
    'jupyter_client>=5.2.0',
    'nbformat',
    'nbconvert',
    'ipykernel',
    'Send2Trash',
    'terminado>=0.8.1',
    'prometheus_client'
],


Python will automatically try to byte compile files when it runs in order to speed up startup the next time it is run.  These files are saved in files with the extension of .pyc (compiled python) or .pyo (optimized compiled python).  These files are a byte code that is portable across OSes.  If you do not include them in your packages, python will try to create them when the user runs the program.  If the system administrator uses them, then the files will be successfully written.  Later, when the package is removed, the .pyc and .pyo files will be left behind on the filesystem.  To prevent that the byte compiled files need to be compiled and included in the <code>%files</code> section.  Normally, byte compilation is done for you by the <code>brp-python-bytecompile</code> script. This script runs after the <code>%install</code> section of the spec file has been processed and byte compiles any .py files that it finds (this recompilation puts the proper filesystem paths into the modules otherwise tracebacks would include the <code>%{buildroot}</code> in them).  All that you need to do is include the files in the <code>%files</code> section.  The following are all acceptable ways to accomplish this:
And the resulting {{package|python3-notebook}} dependencies:


<pre>
python3.7dist(ipykernel)
%install
python3.7dist(ipython-genutils)
install -d $RPM_BUILD_ROOT%{python_sitelib}/foo
python3.7dist(jinja2)
install -pm 0644 foo.py $RPM_BUILD_ROOT%{python_sitelib}/foo/
python3.7dist(jupyter-client) >= 5.2.0
python3.7dist(jupyter-core) >= 4.4.0
python3.7dist(nbconvert)
python3.7dist(nbformat)
python3.7dist(prometheus-client)
python3.7dist(pyzmq) >= 17
python3.7dist(send2trash)
python3.7dist(terminado) >= 0.8.1
python3.7dist(tornado) >= 4
python3.7dist(traitlets) >= 4.2.1


Either:
This generator will most likely be enabled by default in the future. If a packager wishes to explicitly opt out of the generator because the upstream metadata are not applicable, a packager MUST NOT assume that the generator won't be used simply because it is not enabled explicitly.  Rather, they SHOULD opt out explicitly by adding:


%files
%{?python_disable_dependency_generator}
%{python_sitelib}/foo/


Or:
== Provides ==
Using a fictional module named "example", the subpackage containing the python3 version '''must''' provide <code>python3-example</code>.  This is of course always the case if the subpackage is named <code>python3-example</code> (as in the examples below).  If the subpackage has some other name then then <code>Provides: python3-example</code> must be added explicitly (but see the <code>%python_provide</code> macro below).


%files
%dir %{python_sitelib}/foo
%{python_sitelib}/foo/*


Or even:
=== The %python_provide macro ===


%files
In order to make the switch from Python 2 to Python 3 automatic, all packages that provide <code>python3-%{srcname}</code> (for any <code>%{srcname}</code>) SHOULD use the <code>%python_provide</code> macro with the package name, for example:
%dir %{python_sitelib}/foo
%{python_sitelib}/foo/foo.py
%{python_sitelib}/foo/foo.pyc
%{python_sitelib}/foo/foo.pyo
</pre>


{{admon/warning|Avoid INSTALLED_FILES|python's distutils has an <code>INSTALLED_FILES</code> feature that lists which files are installed when you run <code>python setup.py install</code>.  Do not use it for packaging as that will not list the directories which need to be specified in the <code>%files</code> section as well. Using globs in the <code>%files</code> section is simpler and safer.}}
  %{?python_provide:%python_provide python3-%{srcname}}


{{admon/warning|Including egg info|When you run <code>%{__python} setup.py install</code> in any current Fedora, distutils generates a <code>.egg-info</code> file with metadata about the python module that is installed.  These files need to be included as well. (See [https://fedoraproject.org/wiki/Packaging:Python#Packaging_eggs_and_setuptools_concerns Packaging eggs and setuptools concerns] )}}
This eases distribution-wide renaming of Python packages. (For example, in the future a virtual provide of <code>python-%{srcname}</code> might become appropriate for Python 3 libraries. In that case, <code>%python_provide</code> will be changed to add it.) Packages that do not include this macro would need to be adapted to such changes manually.


=== Bytecompiling with the correct python version ===
== Automatic Provides with a standardized name ==


When byte compiling a .py file, python embeds a magic number in the byte compiled files that correspond to the runtime. Files in <code>%{python_sitelib}</code> and <code>%{python_sitearch}</code> must correspond to the runtime for which they were built. For instance, a pure python module compiled for the 3.1 runtime needs to be below <code>%{_usr}/lib/python3.1/site-packages</code>
When building a Python package, RPM looks for <code>.dist-info</code> and <code>.egg-info</code> files or directories in the <code>%files</code> sections of all packages. If one or more are found, RPM parses them to find the '''standardized name''' (i.e. dist name, name on PyPI) of the packaged software, and then automatically creates two <code>Provides:</code> tags in the following format:


The <code>brp-python-bytecompile</code> script tries to figure this out for you.  The script determines which interpreter to use when byte compiling the module by following these steps:
<pre>
Provides: python3.Ydist(CANONICAL_STANDARDIZED_NAME)
Provides: python3dist(CANONICAL_STANDARDIZED_NAME)
</pre>


<ol>
The <code>3.Y</code> is the Python version used (usually 3.6 and higher), and between the parentheses is the name of the software in a '''canonical format''' used by Python tools and services such as setuptools, pip and PyPI. The canonical name is obtained by switching the standardized name to lower case and converting all runs of non-alphanumeric characters to single “-” characters. Example: “The $$$ Tree” becomes “the-tree”.
<li>what directory is the module installed in?  If it's <code>/usr/lib{,64}/pythonX.Y</code>, then <code>pythonX.Y</code> is used to byte compile the module.  If <code>pythonX.Y</code> is not installed, then an error is returned and the rpm build process will exit on an error so remember to <code>BuildRequire</code> the proper python package.</li>


<li>the script interpreter defined in <code>%{__python}</code> is used to compile the modules.  This defaults to the latest python2 version on Fedora.  If you need to compile this module for python3, set it to <code>/usr/bin/python3</code> instead:
=== Requires and BuildRequires with standardized names ===


<pre>
These Provides tags can be used to list Requires and BuildRequires of a package using the ''standardized names'' (i.e. dist name, name on PyPI) of Python modules. To make it easier, you can use the <code>%{py3_dist}</code> macro that accept one or more parameters: the ''standardized name(s)'' of the desired Python software. It will convert the name(s) to the ''canonical format'' and create the proper <code>python3dist(...)</code> tag(s).
%global __python %{__python3}
</pre>


Doing this is useful when you have a python3 application that's installing a private module into its own directory.  For instance, if the foobar application installs a module for use only by the command line application in <code>%{_datadir}/foobar</code>.  Since these files are not in one of the python3 library paths (ie. <code>/usr/lib/python3.1</code>) you have to override <code>%{__python}</code> to tell <code>brp-python-bytecompile</code> to use the python3 interpreter for byte compiling.
In addition, you can use the %{py_dist_name} macro that simply transforms any ''standardized name'' to the ''canonical format''.
</li>
</ol>


These settings are enough to properly byte compile any package that builds python modules in <code>%{python_sitelib}</code> or <code>%{python_sitearch}</code> or builds for only a single python interpreter.  However, if the application you're packaging needs to build with both python2 and python3 and install into a private module directory (perhaps because it provides one utility written in python2 and a second utility written in python3) then you need to do this manually. Here's a sample spec file snippet that shows what to do:
For example:


<pre>
<pre>
# Turn off the brp-python-bytecompile script
BuildRequires: %{py3_dist PyMySQL} >= 0.7.5
%global __os_install_post %(echo '%{__os_install_post}' | sed -e 's!/usr/lib[^[:space:]]*/brp-python-bytecompile[[:space:]].*$!!g')
# => BuildRequires: python3dist(pymysql) >= 0.7.5
# Buildrequire both python2 and python3
BuildRequires: python2-devel python3-devel
[...]


%install
Requires: %{py3_dist virtualenv pyPEG2}
# Installs a python2 private module into %{buildroot}%{_datadir}/mypackage/foo
# => Requires: python3dist(virtualenv) python3dist(pypeg2)
# and installs a python3 private module into %{buildroot}%{_datadir}/mypackage/bar
make install DESTDIR=%{buildroot}


# Manually invoke the python byte compile macro for each path that needs byte
%{py_dist_name 0-._.-._.-._.-._.-._.-._.-0}
# compilation.
# => 0-0
%py_byte_compile %{__python} %{buildroot}%{_datadir}/mypackage/foo
%py_byte_compile %{__python3} %{buildroot}%{_datadir}/mypackage/bar
</pre>
</pre>


The <code>%py_byte_compile</code> macro takes two argumentsThe first is the python interpreter to use for byte compiling.  The second is a file or directory to byte compile.  If the second argument is a directory, the macro will recursively byte compile any *.py file in the directory.
== Source Files from PyPI ==
When packaging software which is available from PyPI, you can make use of the <code>%pypi_source</code> macro.  This macro accepts from zero to three arguments and evaluates to an appropriate URL for the source file on PyPI.  The arguments are:


{{admon/warning|No %{} for py_byte_compile|RPM macros can only take arguments when they do not have curly braces around themTherefore, py_byte_compile won't work correctly if you write: <code>%{py_byte_compile} %{__python}</code>}}
# The name of the PyPI projectDefaults to <code>%srcname</code> if defined, or to <code>%pypi_name</code> if defined, or to <code>%name</code> (the package name).
# The version of the PyPI project.  Defaults to <code>%version</code> (the package version).
# The file extension to use.  Defaults to <code>tar.gz</code>.


In most cases it is not necessary to specify any arguments.


== Macros ==
The following macros are defined for you in all supported Fedora and EPEL releases:
{| class="mw-collapsible wikitable" style="width:100%"
!Macro!!Normal Definition!!Notes
|-
|__python||/usr/bin/python||Prohibited (see note below)
|-
|__python3||/usr/bin/python3||Python 3 interpreter
|-
|python_provide|| (Lua script) ||Given a package name, evaluates to either <code>Provides: python-example</code> or nothing at all depending on the Python version.  See [[Packaging:Python#The_.25python_provide_macro|here]] for an example.
|-
|python3_sitelib||/usr/lib/python3.X/site-packages||Where pure python3 modules are installed
|-
|python3_sitearch||/usr/lib64/python3.X/site-packages on x86_64<BR>/usr/lib/python3.X/site-packages on x86||Where python3 extension modules that are compiled C are installed
|-
|py3_dist|| (Lua script) ||Given a standardized name (i.e. dist name, name on PyPI) of Python software, it will convert it to a canonical format, and evaluates to <code>python3dist(CANONICAL_NAME)</code>, which is useful when listing dependencies. See [[Packaging:Python#Automatic_Provides_with_a_standardized_name|above]] for more information.
|-
|py_byte_compile|| (script) ||Defined in python3-devel.  See the [[Packaging:Python_Appendix#Manual_byte_compilation|byte compiling]] section for usage
|-
|python3_version|| 3.X ||Defined in python3-devel.  Useful when running programs with Python version in filename, such as nosetest-%{python3_version}
|-
|python3_version_nodots|| 3X ||Defined in python3-devel.  Useful when listing files explicitly in %files section , such as %{python3_sitelib}/foo/*.cpython-%{python3_version_nodots}.pyo
|-
|py3_build|| %{__python3} setup.py build ... || Various flags are added, see /usr/lib/rpm/macros.d/macros.python3 for details and similar macros. Define %py_setup_args to pass custom command line arguments to setup.py.
|-
|py3_install|| %{__python3} setup.py install --skip-build ... ||
|-
|py_dist_name|| (Lua script) ||Given a standardized name (i.e. dist name, name on PyPI) of Python software, it will convert it to a canonical format. See [[Packaging:Python#Automatic_Provides_with_a_standardized_name|above]] for more information.
|-
|pypi_source|| (Lua script) || Evaluates to the appropriate URL for the package.  See above for more information.
|-
|}


== Common SRPM vs split SRPMs ==
{{admon/note|The generic %{_python} macros|
The unversioned macros <code>%{__python}</code>, <code>%{python_sitelib}</code>, and <code>%{python_sitearch}</code> are generic macros that point to or use <code>/usr/bin/python</code>.  Since <code>/usr/bin/python</code> invocation is prohibited, you MUST NOT use these macros without manually setting <code>%{__python}</code> to <code>%{__python3}</code>.
}}


Many times when you package a python module you will want to create a module for python2 and a module for python3.  There are two ways of doing this: either from a single SRPM or from multiple.  The rule to choose which method is simple: if the python2 and python3 modules are distributed as a single tarball (many times as a single directory of source where the <code>/usr/bin/2to3</code> program is used to transform the code at buildtime) then you must package them as subpackages built from a single SRPMIf they come in multiple tarballs then package them from multiple SRPMs.
During <code>%install</code> or when listing <code>%files</code> you can use the <code>python3_sitearch</code> and <code>python3_sitelib</code> macros to specify where the installed modules are to be foundFor instance:


{{admon/note|Python Bindings|python bindings are sometimes built as part of the C library's build.  The ideal for these is to patch the code so it will build against both python2 and python3.  Then take a copy of the sources during the <code>%prep</code> phase, and configure one subdirectory to build against python 2, another to build against python 3.  These changes should be upstreamed.  Example: the build of <code>rpm</code> itself emits an <code>rpm-python</code> subpackage (see [[rhbug:531543|Red Hat Bug 531543]].)}}
<pre>
%files
# A pure python3 module
%{python3_sitelib}/foomodule/
# A compiled python3 extension module
%{python3_sitearch}/barmodule/
</pre>


=== Multiple SRPMS ===
Use of the macros has several benefits:
* It ensures that the packages are installed correctly on multilib architectures.
* Using these macros instead of hardcoding the directory in the specfile ensures your spec remains compatible with the installed python version even if the directory structure changes radically (for instance, if <code>python3_sitelib</code> moves into <code>%{_datadir}</code>).


When upstream ships multiple tarballs with one tarball containing python2 code and a different tarball containing python3 code, we should ship those as multiple SRPMs. The two SRPMs could have different maintainers within Fedora and the two packages need not upgrade at the same timeBuilding from multiple SRPMs has some advantages and disadvantages:
== Files to include ==
When packaging python modules, several types of files are included:
* *.py source files because they are used when generating tracebacks.
* *.pyc byte compiled files.
** Python will try to create them at runtime if they don't exist which leads to spurious SELinux AVC denials in the logs.
** If the system administrator invokes python with -OO, they will be created with no docstringsThis can break some programs.
* *.egg-info files or directories.  If these are generated by the module's build scripts they must be included in the package because they might be needed by other applications and modules at runtime.


'''Advantages''':
The source files MUST be included in the same package as the byte compiled versions.
* There can be separate maintainers for python2 and python3 so each maintainer can concentrate on one stack.
* The two packages can evolve separately; if 2 and 3 need to have different versions, they can.


'''Disadvantages''':
Packagers SHOULD NOT simply glob everything under the sitelib or sitearch directories.  The following SHOULD NOT be used:
* The two specfiles have to be maintained separately
* <code>%{python3_sitelib}/*</code>
* When upstream releases e.g. security fixes, they have to be tracked in two places
* <code>%{python3_sitearch}/*</code>
* <code>%{python_sitelib}/*</code>
* <code>%{python_sitearch}/*</code>


The following practices are designed to help mitigate the disadvantages listed above:
And packages MUST NOT include the top-level <code>__pycache__</code> directory (see below).


* When packaging a module for python3 contact the maintainers for the python2 module and try to coordinate with them.
== Byte compiling ==
* Request at least watchbugzilla and watchcommit acls on each other's packages so you're aware of outstanding bugs.
* Complete any python 2 Merge Review when doing the python 3 version.  Doing this gets issues that apply to both packages addressed at the same time.
* Add a link to the python 2 Merge Review/Package Review to the python 3 Package Review


=== Subpackages ===
Python will automatically try to byte compile files when it runs in order to speed up startup the next time it is run.  These files are saved in files with the extension of .pyc (compiled python). These files will be located inside a directory named <code>__pycache__</code>.


{{admon/warning|Do not build python3 modules without upstream support|If upstream is shipping a module for python2 and does not support making that module run on python3, do not package a python3 version of it in FedoraIf running <code>2to3</code> or adding a patch enables the code to work, you can certainly tell upstream that it works to encourage them to support python3However, doing this on our own in Fedora is essentially creating a fork. That has a large burden for maintaining the code, fixing bugs, porting when a new version of upstream's code appears, managing a release schedule, and other tasks normally handled by upstream. It's much better if we can cooperate with upstream to share this work than doing it all on our own.}}
The .pyc files contain byte code that is portable across OSes.  If you do not include them in your packages, python will try (and generally fail) to create them when the user runs the program.  If the system administrator runs the program, then the files will be successfully written, causing stray .pyc files which will not be removed when the package is removedTo prevent that the byte compiled files need to be compiled and included in the <code>%files</code> sectionNormally, byte compilation is done for you by the <code>brp-python-bytecompile</code> script. This script runs after the <code>%install</code> section of the spec file has been processed and byte compiles any .py files that it finds in <code>%{python3_sitelib}</code> or <code>%{python3_sitearch}</code> (this recompilation puts the proper filesystem paths into the modules otherwise tracebacks would include the <code>%{buildroot}</code> in them).


Sometimes upstream will ship one tarball that builds both a python2 and a python3 module. There's several ways that upstream can structure thisWhen upstream writes their build scripts to build both python2 and python3 modules in a single build this is just like building subpackages for any other package.  You expand the tarball and patch the source in <code>%prep</code>, run upstream's build scripts to build the package in <code>%build</code>, and then run upstream's build scripts to install it in <code>%install</code>.
You must include the .pyc files in your packageIf the build process creates a __pycache__ directory in a subdirectory of  %{python3_sitearch} or %{python3_sitelib}, you must also include all items in the __pycache__ directory.  You MUST NOT include the directories %{python3_sitearch}/__pycache__ or %{python3_sitelib}/__pycache__ because they are already owned by the python3-libs package.


'''Advantages''':
All that you need to do is include the files in the <code>%files</code> section (replacing %{python3_sitelib} with the appropriate macro for your package):
* Single src.rpm to review and build
* Avoids having to update multiple packages when things change.


'''Disadvantages''':
<pre>
* The Fedora maintainer needs to care about both python 2 and python 3 modules which makes more work to maintain that package.
%files
* The 2 and 3 versions are in lockstep.  Bugfixes need to apply to python2 while not breaking the translation into python3.
%{python3_sitelib}/foo/
* Bugzilla components are set up according to source RPM, so they will have a single shared bugzilla component.  This could be confusing to end-users, as it would be more difficult to figure out e.g. that a bug with python3-foo needs to be filed against python-foo.  There's a similar problem with checking out package sources from CVS, though this is less serious as it is less visible to end users.
</pre>


Two other ways exist for the upstream to support building python3 modules from a single source:
or, if the python code installs directly into %{python3_sitelib}:


==== Building more than once ====
<pre>
%files
%{python3_sitelib}/foo.py
%{python3_sitelib}/__pycache__/*
</pre>


One way that's currently very common is for the build scripts to create either a python2 or python3 module based on which interpreter is used to run the setup.py script.  (The [http://pkgs.fedoraproject.org/cgit/python-setuptools.git/tree/python-setuptools.spec python-setuptools package] is currently built this way).
{{admon/warning|Avoid INSTALLED_FILES|python's distutils has an <code>INSTALLED_FILES</code> feature that lists which files are installed when you run <code>%py3_install</code>.  Do not use it for packaging as that will not list the directories which need to be specified in the <code>%files</code> section as well. Using globs in the <code>%files</code> section is simpler and safer.}}


===== Example spec file =====
{{admon/warning|Including egg info|When you run <code>%py3_install</code> in any current Fedora, distutils generates a <code>.egg-info</code> file or directory with metadata about the python module that is installed.  These files need to be included as well. (See [[#Packaging_eggs_and_setuptools_concerns|Packaging:Python_Eggs]] )}}


<pre>
=== Manual byte compilation ===
%if 0%{?fedora} > 12
For more details on the internals of byte compilation, please see [[Packaging:Python_Appendix#Manual_byte_compilation|the appendix]].
%global with_python3 1
%else
%{!?python_sitelib: %global python_sitelib %(%{__python} -c "from distutils.sysconfig import get_python_lib; print (get_python_lib())")}
%endif


%global srcname distribute
== Example Python spec file ==
</pre>


At the top of our spec file we have the standard define for <code>python_sitelib</code> on older Fedora releases.  We also define <code>with_python3</code> which we'll use to conditionalize the build whenever we have a section that is only useful when building a python3 module. Using <code>with_python3</code> allows us to do two things:
The following is a very simple spec file for a Python module.


<ol>
<pre>
<li>It makes it easy to turn off the python3 build when tracking down problems.</li>
%global srcname example
<li>The conditionals also make it easy to use the same spec for older releases of Fedora and EPEL.</li>.
</ol>
 
{{admon/warning|Leave python3 module enabled in releases|Once python 3 support has been added to a package, you must leave it enabled.  End users could be using the python3 subpackage that is being built.  If you turn the subpackage build on and off it will cause the package to unexpectedly disappear from the repos.  You should only turn off <code>with_python3</code> as a debugging measure within scratch builds, for releases that do not support python 3, or when moving a python3 module into its own, independent package.}}


<pre>
Name:          python-%{srcname}
Name:          python-setuptools
Version:        1.2.3
Version:        0.6.10
Release:        1%{?dist}
Release:        2%{?dist}
Summary:        An example python module
Summary:        Easily build and distribute Python packages


Group:          Applications/System
License:        MIT
License:        Python or ZPLv2.0
URL:            https://pypi.python.org/pypi/%{srcname}
URL:            http://pypi.python.org/pypi/%{srcname}
Source0:        %pypi_source
Source0:        http://pypi.python.org/packages/source/d/%{srcname}/%{srcname}-%{version}.tar.gz
# Fix a failing test case
Patch0:        python-setuptools-test.patch
BuildRoot:      %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n)


BuildArch:      noarch
BuildArch:      noarch
BuildRequires:  python2-devel
%if 0%{?with_python3}
BuildRequires:  python3-devel
BuildRequires:  python3-devel
%endif # if with_python3
</pre>
When we build the python3 module in addition to the python3 module we need both <code>python2-devel</code> and <code>python3-devel</code>.


<pre>
%description
%description
Setuptools is a collection of enhancements to the Python distutils that allow
An python module which provides a convenient example.
you to more easily build and distribute Python packages, especially ones that
have dependencies on other packages.


This package contains the runtime components of setuptools, necessary to
execute the software that requires pkg_resources.py.


%if 0%{?with_python3}
%package -n python3-%{srcname}
%package -n python3-setuptools
Summary:        %{summary}
Summary:        Easily build and distribute Python 3 packages
%{?python_provide:%python_provide python3-%{srcname}}
Group:         Applications/System


%description -n python3-setuptools
%description -n python3-%{srcname}
Setuptools is a collection of enhancements to the Python 3 distutils that allow
An python module which provides a convenient example.
you to more easily build and distribute Python 3 packages, especially ones that
have dependencies on other packages.


This package contains the runtime components of setuptools, necessary to
execute the software that requires pkg_resources.py.
%endif # with_python3
</pre>


Here we define the python3 subpackage.  Note that we use <code>%package -n</code> to name the module appropriately.
<pre>
%prep
%prep
%setup -q -n %{srcname}-%{version}
%autosetup -n %{srcname}-%{version}


%patch0 -p1 -b .testfix
find -name '*.txt' | xargs chmod -x
%if 0%{?with_python3}
rm -rf %{py3dir}
cp -a . %{py3dir}
find %{py3dir} -name '*.py' | xargs sed -i '1s|^#!python|#!%{__python3}|'
%endif # with_python3
find -name '*.py' | xargs sed -i '1s|^#!python|#!%{__python}|'
</pre>
Our method in building from the same code to make the two separate modules is to keep each build as independent as possible.  To do that, we copy the source tree to <code>%{py3dir}</code> so that the python 2 sources are entirely independent from the python 3 sources.  Some things to watch out for:
* Be sure to clean up the <code>%{py3dir}</code> before performing the copy.  It's easy to forget that since <code>%setup</code> does that automatically for the python2 module.
* Make sure that you are copying the correct code.  The example is copying the code from within the top directory of the untarred source.  If the <code>%prep</code> has changed directory you will need to change back to the tarball location.
* Patching the source code is done before copying to <code>%{py3dir}</code>.  Since you have both a python2 and a python3 directory you might be tempted to patch each one separately.  '''Resist!'''  Upstream for your package has chosen to distribute a single source tree that builds for both python2 and python3.  For your patches to [[Staying_close_to_upstream_projects| get into upstream]], you need to write patches that work with both as well.}}
<code>rpmbuild</code> resets the directory at the end of each phase, so you don't need to restore the directory at the end of <code>%prep</code>.
<pre>
%build
%build
CFLAGS="$RPM_OPT_FLAGS" %{__python} setup.py build
%py3_build
 
%if 0%{?with_python3}
pushd %{py3dir}
CFLAGS="$RPM_OPT_FLAGS" %{__python3} setup.py build
popd
%endif # with_python3


%install
%install
rm -rf %{buildroot}
%py3_install
 
# Must do the python3 install first because the scripts in /usr/bin are
# overwritten with every setup.py install (and we want the python2 version
# to be the default for now).
%if 0%{?with_python3}
pushd %{py3dir}
%{__python3} setup.py install --skip-build --root $RPM_BUILD_ROOT
 
rm -rf %{buildroot}%{python3_sitelib}/setuptools/tests
 
find %{buildroot}%{python3_sitelib} -name '*.exe' | xargs rm -f
chmod +x %{buildroot}%{python3_sitelib}/setuptools/command/easy_install.py
popd
%endif # with_python3
 
%{__python} setup.py install --skip-build --root $RPM_BUILD_ROOT
 
rm -rf ${buildroot}%{python_sitelib}/setuptools/tests
 
find %{buildroot}%{python_sitelib} -name '*.exe' | xargs rm -f
chmod +x %{buildroot}%{python_sitelib}/setuptools/command/easy_install.py


%check
%check
%{__python} setup.py test
%if 0%{?with_python3}
pushd %{py3dir}
%{__python3} setup.py test
%{__python3} setup.py test
popd
%endif # with_python3
</pre>
You'll notice that the <code>%build</code>, <code>%install</code>, and <code>%check</code> sections follow a common pattern.  They do the normal steps for building the python2 module but then they switch to <code>%{py3dir}</code> and run the same steps for python3.  Creating the new sections is generally pretty easy.  First copy the existing code.  Then wrap it with a <code>pushd/popd</code> to <code>%{py3dir}</code>.  The usage of <code>pushd/popd</code> commands will ensure that the directories are logged.  Finally, convert all macro references:


* <code>%{__python}</code> becomes <code>%{__python3}</code>
# Note that there is no %%files section for the unversioned python module
* <code>%{python_sitelib}</code> becomes <code>%{python3_sitelib}</code>
%files -n python3-%{srcname}
* <code>%{python_sitearch}</code> becomes <code>%{python3_sitearch}</code>
%license COPYING
 
%doc README.rst
{{admon/warning|Order can be important|As you can see in the <code>%install</code> section, the order in which you do the python2 versus python3 install can sometimes matter.  You need to be aware of when the install is writing to the same file in both packages (in this example, a script in <code>%{_bindir}</code> and make sure that you're getting the version you expect.}}
%{python3_sitelib}/%{srcname}/
 
%{python3_sitelib}/%{srcname}-*.egg-info/
<pre>
%{_bindir}/sample-exec
%clean
rm -rf $RPM_BUILD_ROOT
 
 
%files
%doc psfl.txt zpl.txt docs
%{python_sitelib}/*
%{_bindir}/easy_install
%{_bindir}/easy_install-2.6
 
%if 0%{?with_python3}
%files -n python3-setuptools
%doc psfl.txt zpl.txt docs
%{python3_sitelib}/*
%{_bindir}/easy_install-3.1
%endif # with_python3


%changelog
%changelog
</pre>
</pre>


In this final section, you can see that we once again switch macros from <code>%{python_sitelib}</code> to <code>%{python3_sitelib}</code>.  Since we chose to install the python2 version of <code>%{_bindir}/easy_install</code> earlier we need to include that file in the python2 package rather than the python3 subpackage.
== Packaging eggs ==
 
Please see the Python eggs [[Packaging:Python_Eggs|guidelines]] for information specific to Python eggs.
==== Running 2to3 from the spec file ====
Sometimes, upstream hasn't integrated running 2to3 on the code into their build scripts but they support making a python3 module from it if you manually run 2to3 on the source.  This is the case when it's documented on the upstream's website, in a file in the tarball, or even when email with the module's author has instructions for building a python3 module from the python2 source and the authors are willing to support the result.  In these cases it's usually just a matter of the upstream not having written the build script that can turn the python2 source into python3.  When this happens you can run <code>2to3</code> from the spec file.  Once you have it working, you can also help upstream integrate it into their build scripts which will benefit everyone in the long term.
 
You should usually follow upstream's directions on how to run <code>2to3</code> and build the python3 module in these cases but there's a few things you should check to make sure upstream is doing it correctly.
 
* Since the code is being built from a unified source, you need to copy the code to a new directory before invoking 2to3 just like the [[#Building_more_than_once| building more than once]] method.
* If the <code>2to3</code> program is invoked instead of using the <code>lib2to3</code> library functions, make sure it's invoked with <code>--write --nobackups</code>.  <code>--write</code> is needed to make <code>2to3</code> actually change the files.  <code>--nobackups</code> avoids leaving <code>foo.py.bak</code> files in the module directories that then make it into the final package payload.
* Be sure to run 2to3 on the correct directory.  When you run <code>2to3</code> you need to run it on the whole tree.  A common mistake here for distutils packages has been to run it on the directory below <code>setup.py</code>, missing the <code>setup.py</code> file itself.  This leads to errors when <code>python3</code> tries to execute <code>setup.py</code>
* If you need to run <code>2to3</code> to fix code, use <code>2to3</code> or <code>/usr/bin/2to3</code>.  At the moment, this program is coming from the <code>python-tools</code> rpm.  Using <code>2to3</code> means that you'll be using a name that is supported upstream and across distros rather than <code>/usr/bin/python3-2to3</code> which we have renamed in Fedora to avoid filesystem conflicts.  This also makes it easier for us to test and eventually change from using the python2 <code>2to3</code> to the python3 <code>2to3</code>.  We just need to change the python3 package to provide the <code>/usr/bin/2to3</code> program instead of python and all of our python packages will start using that version instead.
* If <code>2to3</code> runs into a problem, please [https://bugzilla.redhat.com/enter_bug.cgi?component=python&product=Fedora file a Fedora bug].  Please try to isolate a minimal test case that reproduces the problem when doing so.
 
== Avoiding collisions between the python 2 and python 3 stacks ==
The python 2 and python 3 stacks are intended to be fully-installable in parallel. When generalizing the package for both python 2 and python 3, it is important to ensure that two different built packages do not attempt to place different payloads into the same path.
 
=== Executables in <code>/usr/bin</code> ===
==== The problem ====
Many existing python packages install executables into <code>/usr/bin</code>.
 
For example if we have a <code>console_scripts</code> in a <code>setup.py</code> shared between
python 2 and python 3 builds: these will spit out files in <code>/usr/bin/</code>,
and these will collide.
 
For example <code>python-coverage</code> has a <code>setup.py</code> that contains:
<pre>
    entry_points = {
        'console_scripts': [
            'coverage = coverage:main',
            ]
        },
</pre>
 
which thus generates a <code>/usr/bin/coverage</code> executable (this is a python
script that runs another python script whilst generating code-coverage
information on the latter).
 
Similarly for the 'scripts' clause; see e.g. <code>python-pygments</code>:
<code>Pygments-1.1.1/setup.py</code> has:
<pre>
    scripts = ['pygmentize'],
</pre>
which generates a <code>/usr/bin/pygmentize</code> (this is a python script that leverages the pygments syntax-highlighting module, giving a simple command-line interface for generating syntax-highlighted files)
 
==== Guidelines ====
If the executables provide the same functionality independent of whether they are run on top of Python 2 or Python 3, then only one version of the executable should be packaged.  Currently it will be the python 2 implementation, but once the Python 3 implementation is proven to work, the executable can be retired from the python 2 build and enabled in the python 3 package.  Be sure to test the new implementation.  Transitioning from python2 to python3 is left to individual package maintainers except for packages in Fedora's critical path.  For these, we want to port to python3 versions in the same Fedora release if possible.


Examples of this:
== Reviewer checklist ==
* <code>/usr/bin/pygmentize</code> ought to generate the same output regardless of whether it's implemented via Python 2 or Python 3, so only one version needs to be shipped.
The following briefly summarizes the guidelines for reviewers to go over:
 
* '''Must''': Python modules must be built from source.  They cannot simply drop an egg or whl from upstream into the proper directory. (See [[Packaging:Guidelines#No_inclusion_of_pre-built_binaries_or_libraries| prebuilt binaries Guidelines]] for details)
If the executables provide different functionality for Python 2 and Python 3, then both versions should be packaged.
* '''Must''': Python modules must not download any dependencies during the build process.
 
Examples of this:
* <code>/usr/bin/coverage</code> runs a python script, augmenting the interpreter with code-coverage information.  Given that the interpreter itself is the thing being worked with, it's reasonable to package both versions of the executable.
* <code>/usr/bin/bpython</code> augments the interpreter with a "curses" interface.  Again, it's reasonable to package both versions of this.
* <code>/usr/bin/easy_install</code> installs a module into one of the Python runtimes: we need a version for each runtime.
 
As an exception, for the rpms that are part of a python runtime itself, we plan to package both versions of the executables, so that e.g. both the python 2 and python 3 versions of <code>2to3</code> are packaged.
 
==== Naming ====
Many executables already contain a "-MAJOR.MINOR" suffix, for example <code>/usr/bin/easy_install-3.1</code>.  These obviously can be used as-is, as they won't conflict.
 
For other executables, the general rule is:
* if only one executable is to be shipped, then it owns its own slot
* if executables are to be shipped for both python 2 and python 3, then the python 3 version of the executable gains a <code>python3-</code> prefix.  For example, the python 2 version of "coverage" remains <code>/usr/bin/coverage</code> and the python 3 version is <code>/usr/bin/python3-coverage</code>.
See [http://lists.fedoraproject.org/pipermail/devel/2010-January/129217.html this thread] for a discussion of this.
 
== Packaging eggs and setuptools concerns ==
 
Eggs can mean several different things because they can be placed on disk in several formats:
 
* A module and a file with a .egg-info extension that contains the metadata.  Created by distutils in Fedora 9 and above.
* As a module and a directory with a .egg-info extension that contains the metadata.  Created using setuptools and also the invocation of setup.py in our examples below.
* As a directory with a .egg extension that contains the module and egg metadata.  Created when we use easy_install -m to allow installing multiple versions of a module.
* As a single zip file with a .egg extension that contains the module and the egg metadata.
 
In Fedora packages, these will be installed to %{python_sitelib} or %{python_sitearch} directories.  We do not install the single zip file version of eggs in Fedora but the three other formats are used.
 
=== How to package ===
 
The following are a summary of the guidelines for reviewers to go over when a python module is packaged.  The [[Packaging:Python Eggs|  complete policy]]  includes examples and rationale for the way we do things.
 
* '''Must''': Python eggs must be built from source.  They cannot simply drop an egg from upstream into the proper directory. (See [[Packaging:Guidelines#No_inclusion_of_pre-built_binaries_or_libraries| prebuilt binaries Guidelines]] for details)
* '''Must''': Python eggs must not download any dependencies during the build process.
* '''Must''': When building a compat package, it must install using easy_install -m so it won't conflict with the main package.
* '''Must''': When building a compat package, it must install using easy_install -m so it won't conflict with the main package.
* '''Must''': When building multiple versions (for a compat package) one of the packages must contain a default version that is usable via "import MODULE" with no prior setup.
* '''Must''': When building multiple versions (for a compat package) one of the packages must contain a default version that is usable via "import MODULE" with no prior setup.
* '''Should''': If you build a python module you should use the <code>%python_provide</code> macro.
* '''Should''': A package which is used by another package via an egg interface should provide egg info.
* '''Should''': A package which is used by another package via an egg interface should provide egg info.


== Filtering Requires: and Provides: ==
[[Category:Packaging guidelines]]
RPM's dependency generator can often throw in additional dependencies and will often think packages provide functionality contrary to reality. To fix this, the dependency generator needs to be overriden so that the additional dependencies can be filtered out.  See [[Packaging:AutoProvidesAndRequiresFiltering]] for details.
[[Category:Python]]
 
== PyGTK2 and Numpy ==
{{admon/note||This is a temporary workaround which may be resolved in the future. It will no longer be necessary when [[http://bugzilla.gnome.org/show_bug.cgi?id=591745 gnome bug #591745]] is fixed.}}
 
If your package uses pygtk2, and calls the gtk.gdk.get_pixels_array() function, that package needs to explicitly Require: numpy. In the past, pygtk2 had a Requires on numpy, but since it is only used for that one function (and that function is not commonly used), the Requires has been removed to minimize the install footprint of pygtk2.
 
[[Category:Packaging guidelines]] [[Category:Python]]

Revision as of 20:50, 8 November 2018

Warning.png
This is an old copy of a packaging guideline, preserved here in the wiki while we complete the transition to the Fedora documentation system. The current version is located at https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/ . Please update your bookmarks.

Python Version Support

In Fedora we have multiple Python runtimes, one for each supported major Python release. At this point that's one for python3.x and one for python2.7 However the python2 stack will be removed from Fedora and is deprecated. Upstream support for the python2 interpreter officially ends in 2020. If a piece of software supports python3, it MUST be packaged for python3. Software using python2 MUST NOT be newly packaged without FESCo exception.

For guidelines on maintaining already existing python2 packages, see the appendix.

Multiple Python Runtimes

For backwards compatibility, /usr/bin/python is, if it is installed, a symbolic link to /usr/bin/python2.

Packages in Fedora MUST NOT use /usr/bin/python. Instead packages for Python 3 MUST use /usr/bin/python3 (even if upstream supports both Python 2 and 3). As a result of that /usr/bin/python (as well as /usr/bin/env python and similar) MUST NOT be used in shebang lines or as a dependency of a package. As of Fedora 30, all uses of unversioned python executables in shebang lines will fail the build. These shebangs MUST be fixed. If it is necessary to disable the checks, please see the information in Packaging:Guidelines#Shebang_lines.

All python runtimes have a virtual provide for python(abi) = $MAJOR-$MINOR. For example, the Python 3.7 runtime package has:

 $ rpm -q --provides python3 |grep -i abi
 python(abi) = 3.7

Python modules using these runtimes should have a corresponding "Requires" line on the python runtime that they are used with. This is done automatically for files below /usr/lib[^/]*/python${PYVER}

Mirroring the policy for regular packages, the Python-version-specific subpackages of your package MUST NOT be removed in a release branch of Fedora.

Naming

The source package for a Python library MUST be named with the python- prefix. A built package however must include the Python major version in the name, using the python3- prefix. This is accomplished by adding a subpackage. See example bellow.

This rule does not apply to applications.

Dependencies

Packages building for Python 3 will need BuildRequires: python3-devel.

Packages MUST NOT have dependencies (either build-time or runtime) on packages named with the unversioned python- prefix. Dependencies on Python packages instead MUST use names beginning with python3-.


Automatically generated dependencies

Packages MAY use the automatic Python dependency generator. This generator uses upstream egg/dist metadata (such as setuptool's install_requires) to determine what the package should depend on. The generator parses the installed metadata from /usr/lib(64)?/pythonX.Y/site-packages/<module>-<version>-pyX.Y.(egg|dist)-info/requires.txt, so it will not work with software that uses plain distutils.

To enable this feature, add:

%{?python_enable_dependency_generator}

Although this statement can be used anywhere in the spec, we recommend putting it just before the main package's %description declaration. This generates run time requires in the form of pythonX.Ydist(foo). If the generated dependencies are not accurate, additional ones can still be added manually. To remove some, a packager MAY modify upstream-provided metadata (usually specified in the setup.py file) in the %prep section of the specfile or fall back to filtering those dependencies.

The packager MUST inspect the generated requires for correctness. All dependencies MUST be resolvable within the targeted Fedora version.

As an example, the upstream notebook package has (as of version 5.6.0):

install_requires = [
    'jinja2',
    'tornado>=4',
    'pyzmq>=17',
    'ipython_genutils',
    'traitlets>=4.2.1',
    'jupyter_core>=4.4.0',
    'jupyter_client>=5.2.0',
    'nbformat',
    'nbconvert',
    'ipykernel',
    'Send2Trash',
    'terminado>=0.8.1',
    'prometheus_client'
],

And the resulting Package-x-generic-16.pngpython3-notebook dependencies:

python3.7dist(ipykernel)
python3.7dist(ipython-genutils)
python3.7dist(jinja2)
python3.7dist(jupyter-client) >= 5.2.0
python3.7dist(jupyter-core) >= 4.4.0
python3.7dist(nbconvert)
python3.7dist(nbformat)
python3.7dist(prometheus-client)
python3.7dist(pyzmq) >= 17
python3.7dist(send2trash)
python3.7dist(terminado) >= 0.8.1
python3.7dist(tornado) >= 4
python3.7dist(traitlets) >= 4.2.1

This generator will most likely be enabled by default in the future. If a packager wishes to explicitly opt out of the generator because the upstream metadata are not applicable, a packager MUST NOT assume that the generator won't be used simply because it is not enabled explicitly. Rather, they SHOULD opt out explicitly by adding:

%{?python_disable_dependency_generator}

Provides

Using a fictional module named "example", the subpackage containing the python3 version must provide python3-example. This is of course always the case if the subpackage is named python3-example (as in the examples below). If the subpackage has some other name then then Provides: python3-example must be added explicitly (but see the %python_provide macro below).


The %python_provide macro

In order to make the switch from Python 2 to Python 3 automatic, all packages that provide python3-%{srcname} (for any %{srcname}) SHOULD use the %python_provide macro with the package name, for example:

  %{?python_provide:%python_provide python3-%{srcname}}

This eases distribution-wide renaming of Python packages. (For example, in the future a virtual provide of python-%{srcname} might become appropriate for Python 3 libraries. In that case, %python_provide will be changed to add it.) Packages that do not include this macro would need to be adapted to such changes manually.

Automatic Provides with a standardized name

When building a Python package, RPM looks for .dist-info and .egg-info files or directories in the %files sections of all packages. If one or more are found, RPM parses them to find the standardized name (i.e. dist name, name on PyPI) of the packaged software, and then automatically creates two Provides: tags in the following format:

Provides: python3.Ydist(CANONICAL_STANDARDIZED_NAME)
Provides: python3dist(CANONICAL_STANDARDIZED_NAME)

The 3.Y is the Python version used (usually 3.6 and higher), and between the parentheses is the name of the software in a canonical format used by Python tools and services such as setuptools, pip and PyPI. The canonical name is obtained by switching the standardized name to lower case and converting all runs of non-alphanumeric characters to single “-” characters. Example: “The $$$ Tree” becomes “the-tree”.

Requires and BuildRequires with standardized names

These Provides tags can be used to list Requires and BuildRequires of a package using the standardized names (i.e. dist name, name on PyPI) of Python modules. To make it easier, you can use the %{py3_dist} macro that accept one or more parameters: the standardized name(s) of the desired Python software. It will convert the name(s) to the canonical format and create the proper python3dist(...) tag(s).

In addition, you can use the %{py_dist_name} macro that simply transforms any standardized name to the canonical format.

For example:

BuildRequires: %{py3_dist PyMySQL} >= 0.7.5
# => BuildRequires: python3dist(pymysql) >= 0.7.5

Requires: %{py3_dist virtualenv pyPEG2}
# => Requires: python3dist(virtualenv) python3dist(pypeg2)

%{py_dist_name 0-._.-._.-._.-._.-._.-._.-0}
# => 0-0

Source Files from PyPI

When packaging software which is available from PyPI, you can make use of the %pypi_source macro. This macro accepts from zero to three arguments and evaluates to an appropriate URL for the source file on PyPI. The arguments are:

  1. The name of the PyPI project. Defaults to %srcname if defined, or to %pypi_name if defined, or to %name (the package name).
  2. The version of the PyPI project. Defaults to %version (the package version).
  3. The file extension to use. Defaults to tar.gz.

In most cases it is not necessary to specify any arguments.

Macros

The following macros are defined for you in all supported Fedora and EPEL releases:

Macro Normal Definition Notes
__python /usr/bin/python Prohibited (see note below)
__python3 /usr/bin/python3 Python 3 interpreter
python_provide (Lua script) Given a package name, evaluates to either Provides: python-example or nothing at all depending on the Python version. See here for an example.
python3_sitelib /usr/lib/python3.X/site-packages Where pure python3 modules are installed
python3_sitearch /usr/lib64/python3.X/site-packages on x86_64
/usr/lib/python3.X/site-packages on x86
Where python3 extension modules that are compiled C are installed
py3_dist (Lua script) Given a standardized name (i.e. dist name, name on PyPI) of Python software, it will convert it to a canonical format, and evaluates to python3dist(CANONICAL_NAME), which is useful when listing dependencies. See above for more information.
py_byte_compile (script) Defined in python3-devel. See the byte compiling section for usage
python3_version 3.X Defined in python3-devel. Useful when running programs with Python version in filename, such as nosetest-%{python3_version}
python3_version_nodots 3X Defined in python3-devel. Useful when listing files explicitly in %files section , such as %{python3_sitelib}/foo/*.cpython-%{python3_version_nodots}.pyo
py3_build %{__python3} setup.py build ... Various flags are added, see /usr/lib/rpm/macros.d/macros.python3 for details and similar macros. Define %py_setup_args to pass custom command line arguments to setup.py.
py3_install %{__python3} setup.py install --skip-build ...
py_dist_name (Lua script) Given a standardized name (i.e. dist name, name on PyPI) of Python software, it will convert it to a canonical format. See above for more information.
pypi_source (Lua script) Evaluates to the appropriate URL for the package. See above for more information.
Note.png
The generic %{_python} macros
The unversioned macros %{__python}, %{python_sitelib}, and %{python_sitearch} are generic macros that point to or use /usr/bin/python. Since /usr/bin/python invocation is prohibited, you MUST NOT use these macros without manually setting %{__python} to %{__python3}.

During %install or when listing %files you can use the python3_sitearch and python3_sitelib macros to specify where the installed modules are to be found. For instance:

%files
# A pure python3 module
%{python3_sitelib}/foomodule/
# A compiled python3 extension module
%{python3_sitearch}/barmodule/

Use of the macros has several benefits:

  • It ensures that the packages are installed correctly on multilib architectures.
  • Using these macros instead of hardcoding the directory in the specfile ensures your spec remains compatible with the installed python version even if the directory structure changes radically (for instance, if python3_sitelib moves into %{_datadir}).

Files to include

When packaging python modules, several types of files are included:

  • *.py source files because they are used when generating tracebacks.
  • *.pyc byte compiled files.
    • Python will try to create them at runtime if they don't exist which leads to spurious SELinux AVC denials in the logs.
    • If the system administrator invokes python with -OO, they will be created with no docstrings. This can break some programs.
  • *.egg-info files or directories. If these are generated by the module's build scripts they must be included in the package because they might be needed by other applications and modules at runtime.

The source files MUST be included in the same package as the byte compiled versions.

Packagers SHOULD NOT simply glob everything under the sitelib or sitearch directories. The following SHOULD NOT be used:

  • %{python3_sitelib}/*
  • %{python3_sitearch}/*
  • %{python_sitelib}/*
  • %{python_sitearch}/*

And packages MUST NOT include the top-level __pycache__ directory (see below).

Byte compiling

Python will automatically try to byte compile files when it runs in order to speed up startup the next time it is run. These files are saved in files with the extension of .pyc (compiled python). These files will be located inside a directory named __pycache__.

The .pyc files contain byte code that is portable across OSes. If you do not include them in your packages, python will try (and generally fail) to create them when the user runs the program. If the system administrator runs the program, then the files will be successfully written, causing stray .pyc files which will not be removed when the package is removed. To prevent that the byte compiled files need to be compiled and included in the %files section. Normally, byte compilation is done for you by the brp-python-bytecompile script. This script runs after the %install section of the spec file has been processed and byte compiles any .py files that it finds in %{python3_sitelib} or %{python3_sitearch} (this recompilation puts the proper filesystem paths into the modules otherwise tracebacks would include the %{buildroot} in them).

You must include the .pyc files in your package. If the build process creates a __pycache__ directory in a subdirectory of  %{python3_sitearch} or %{python3_sitelib}, you must also include all items in the __pycache__ directory. You MUST NOT include the directories %{python3_sitearch}/__pycache__ or %{python3_sitelib}/__pycache__ because they are already owned by the python3-libs package.

All that you need to do is include the files in the %files section (replacing %{python3_sitelib} with the appropriate macro for your package):

%files
%{python3_sitelib}/foo/

or, if the python code installs directly into %{python3_sitelib}:

%files
%{python3_sitelib}/foo.py
%{python3_sitelib}/__pycache__/*
Warning.png
Avoid INSTALLED_FILES
python's distutils has an INSTALLED_FILES feature that lists which files are installed when you run %py3_install. Do not use it for packaging as that will not list the directories which need to be specified in the %files section as well. Using globs in the %files section is simpler and safer.
Warning.png
Including egg info
When you run %py3_install in any current Fedora, distutils generates a .egg-info file or directory with metadata about the python module that is installed. These files need to be included as well. (See Packaging:Python_Eggs )

Manual byte compilation

For more details on the internals of byte compilation, please see the appendix.

Example Python spec file

The following is a very simple spec file for a Python module.

%global srcname example

Name:           python-%{srcname}
Version:        1.2.3
Release:        1%{?dist}
Summary:        An example python module

License:        MIT
URL:            https://pypi.python.org/pypi/%{srcname}
Source0:        %pypi_source

BuildArch:      noarch
BuildRequires:  python3-devel

%description
An python module which provides a convenient example.


%package -n python3-%{srcname}
Summary:        %{summary}
%{?python_provide:%python_provide python3-%{srcname}}

%description -n python3-%{srcname}
An python module which provides a convenient example.


%prep
%autosetup -n %{srcname}-%{version}

%build
%py3_build

%install
%py3_install

%check
%{__python3} setup.py test

# Note that there is no %%files section for the unversioned python module
%files -n python3-%{srcname}
%license COPYING
%doc README.rst
%{python3_sitelib}/%{srcname}/
%{python3_sitelib}/%{srcname}-*.egg-info/
%{_bindir}/sample-exec

%changelog

Packaging eggs

Please see the Python eggs guidelines for information specific to Python eggs.

Reviewer checklist

The following briefly summarizes the guidelines for reviewers to go over:

  • Must: Python modules must be built from source. They cannot simply drop an egg or whl from upstream into the proper directory. (See prebuilt binaries Guidelines for details)
  • Must: Python modules must not download any dependencies during the build process.
  • Must: When building a compat package, it must install using easy_install -m so it won't conflict with the main package.
  • Must: When building multiple versions (for a compat package) one of the packages must contain a default version that is usable via "import MODULE" with no prior setup.
  • Should: If you build a python module you should use the %python_provide macro.
  • Should: A package which is used by another package via an egg interface should provide egg info.