From Fedora Project Wiki

m (Writeup of https://pagure.io/packaging-committee/issue/755)
(Writeup of https://pagure.io/packaging-committee/issue/772)
Line 84: Line 84:
 
== Manual byte compilation ==
 
== Manual byte compilation ==
  
When byte compiling a .py file, python embeds a magic number in the byte compiled files that correspond to the runtime.  Files in <code>%{python?_sitelib}</code> and <code>%{python?_sitearch}</code> must correspond to the runtime for which they were built.  For instance, a pure python module compiled for the 3.4 runtime needs to be below <code>%{_usr}/lib/python3.4/site-packages</code>
+
When byte compiling a .py file, python embeds a magic number in the byte compiled files that correspond to the runtime.  Files in <code>%{python?_sitelib}</code> and <code>%{python?_sitearch}</code> MUST correspond to the runtime for which they were built.  For instance, a pure python module compiled for the 3.4 runtime MUST be below <code>%{_usr}/lib/python3.4/site-packages</code>
 +
 
 +
{{admon/warning|Fedora 29+ only|The following guidelines have been changed in Fedora 29 and won't work in older Fedoras (or EPELs). See bellow for  previous version of the guideline  for older releases.}}
  
 
The <code>brp-python-bytecompile</code> script tries to figure this out for you.  The script determines which interpreter to use when byte compiling the module by following these steps:
 
The <code>brp-python-bytecompile</code> script tries to figure this out for you.  The script determines which interpreter to use when byte compiling the module by following these steps:
  
<ol>
+
# What directory is the module installed in?  If it's <code>/usr/lib{,64}/pythonX.Y</code>, then <code>pythonX.Y</code> is used to byte compile the module.  If <code>pythonX.Y</code> is not installed, then an error is returned and the rpm build process will exit on an error so remember to <code>BuildRequire</code> the proper python package.
<li>what directory is the module installed in?  If it's <code>/usr/lib{,64}/pythonX.Y</code>, then <code>pythonX.Y</code> is used to byte compile the module.  If <code>pythonX.Y</code> is not installed, then an error is returned and the rpm build process will exit on an error so remember to <code>BuildRequire</code> the proper python package.</li>
+
# By default the script interpreter defined in <code>%{__python}</code> is used to compile the modules outside of those directories.  This defaults to <code>/usr/bin/python</code> (that's Python 2.7 on Fedora). This only happens for backwards compatibility reasons.
 +
 
 +
If you have <code>*.py</code> files outside of the <code>/usr/lib(64)?/pythonX.Y/</code> directories, you '''MUST''' disable their automatic compilation. If you require those files to be byte compiled  (e.g. it's an importable Python module) you can then compile them explicitly using the <code>%py_byte_compile</code> macro.
 +
 
 +
Example for package that has both Python versions:
 +
 
 +
# Disable automatic compilation of Python files in extra directories
 +
%global _python_bytecompile_extra 0
 +
 +
# Buildrequire both python2 and python3
 +
BuildRequires: python2-devel python3-devel
 +
 +
%install
 +
# Installs a python2 private module into %{buildroot}%{_datadir}/mypackage/foo
 +
# and installs a python3 private module into %{buildroot}%{_datadir}/mypackage/bar
 +
make install DESTDIR=%{buildroot}
 +
 +
# Manually invoke the python byte compile macro for each path that needs byte
 +
# compilation.
 +
%py_byte_compile %{__python2} %{buildroot}%{_datadir}/mypackage/foo
 +
%py_byte_compile %{__python3} %{buildroot}%{_datadir}/mypackage/bar
 +
 
 +
Note that this '''does not''' disable the compilation of files in <code>/usr/lib(64)?/pythonX.Y/</code>.
 +
 
 +
The <code>%py_byte_compile</code> macro takes two arguments.  The first is the python interpreter to use for byte compiling.  The second is a file or directory to byte compile.  If the second argument is a directory, the macro will recursively byte compile any *.py file in the directory.
  
<li>the script interpreter defined in <code>%{__python}</code> is used to compile the modules.  This defaults to the latest python2 version on Fedora. If you need to compile this module for python3, set it to <code>/usr/bin/python3</code> instead:
+
{{admon/warning|No %{} for py_byte_compile|RPM macros can only take arguments when they do not have curly braces around them.  Therefore, py_byte_compile won't work correctly if you write: <code>%{py_byte_compile} %{__python2}</code>}}
 +
 
 +
 
 +
== Manual byte compilation for Fedora 28 or earlier (including EPEL) ==
 +
 
 +
 
 +
The script interpreter defined in <code>%{__python}</code> is used to compile the modules outside of <code>/usr/lib(64)?/pythonX.Y/</code> directories.  This defaults to <code>/usr/bin/python</code> (that's Python 2.7 on Fedora). If you need to compile the modules for python3, set it to <code>/usr/bin/python3</code> instead:
  
 
<pre>
 
<pre>
Line 120: Line 152:
 
%py_byte_compile %{__python3} %{buildroot}%{_datadir}/mypackage/bar
 
%py_byte_compile %{__python3} %{buildroot}%{_datadir}/mypackage/bar
 
</pre>
 
</pre>
 +
 +
Note that this '''does disable''' the compilation of files in <code>/usr/lib(64)?/pythonX.Y/</code>.
  
 
In Fedora 27 or earlier, use <code>%global __os_install_post %(echo '%{__os_install_post}' | sed -e 's!/usr/lib[^[:space:]]*/brp-python-bytecompile[[:space:]].*$!!g')</code>
 
In Fedora 27 or earlier, use <code>%global __os_install_post %(echo '%{__os_install_post}' | sed -e 's!/usr/lib[^[:space:]]*/brp-python-bytecompile[[:space:]].*$!!g')</code>
 
instead of the <code>%undefine</code> line above.
 
instead of the <code>%undefine</code> line above.
  
The <code>%py_byte_compile</code> macro takes two arguments.  The first is the python interpreter to use for byte compiling.  The second is a file or directory to byte compile.  If the second argument is a directory, the macro will recursively byte compile any *.py file in the directory.
 
  
{{admon/warning|No %{} for py_byte_compile|RPM macros can only take arguments when they do not have curly braces around them.  Therefore, py_byte_compile won't work correctly if you write: <code>%{py_byte_compile} %{__python2}</code>}}
 
  
  
 
[[Category:Packaging guidelines]]
 
[[Category:Packaging guidelines]]
 
[[Category:Python]]
 
[[Category:Python]]

Revision as of 17:40, 15 June 2018

Additional Python Guidelines

Here are some additional Python-related guidelines, moved here in order to keep the main page manageable.

Using separate build directories

Sometimes is it impossible to build both versions from the same source directory. Most often this happens when sources are "translated" to python3 in the source directory and made incompatible with python2 in the process. This used to be fairly common, but is fortunately much rarer now. Some things to look for are:

  • Sources are not Python 3 compatible (print without parentheses is used, old module names like ConfigParser are imported),
  • six module is not used,
  • 2to3 is run in setup.py without creating a separate build directory.

Our method in building from the same code to make the two separate modules is to keep each build as independent as possible. To do that, we copy the source tree to python3 so that the python 2 sources are entirely independent from the python 3 sources.

Some things to watch out for:

  • Make sure that you are copying the correct code. The example is copying the code from within the top directory of the untarred source. If the %prep has changed directory you will need to change back to the tarball location.
  • Patching the source code is done before copying to python3. Since you have both a python2 and a python3 directory you might be tempted to patch each one separately. Resist! Upstream for your package has chosen to distribute a single source tree that builds for both python2 and python3. For your patches to get into upstream, you need to write patches that work with both as well.

rpmbuild resets the directory at the end of each phase, so you don't need to restore the directory at the end of %prep.

%prep
%setup -qc
mv %{srcname}-%{version} python2
pushd python2
%patch0 -p1 -b .testfix

find -name '*.txt' | xargs chmod -x
# copy common doc files to top dir
cp -pr docs psfl.txt zpl.txt ../
popd

cp -a python2 python3
find python3 -name '*.py' | xargs sed -i '1s|^#!.*|#!%{__python3}|'
find python2 -name '*.py' | xargs sed -i '1s|^#!.*|#!%{__python2}|'
%build
pushd python2
%py2_build
popd

pushd python3
%py3_build
popd

%install
# Must do the python2 install first because the scripts in /usr/bin are
# overwritten with every setup.py install (and we want the python3 version
# to be the default).
pushd python2
%py2_install
popd

pushd python3
%py3_install
popd

%check
pushd python2
%{__python2} setup.py test
popd

pushd python3
%{__python3} setup.py test
popd

You'll notice that the %build, %install, and %check sections again follow a pattern similar to the previous example. They switch to the python2 directory and do the normal steps for building the python2 module, and then switch to the python3 directory and run the same steps for python3. The usage of pushd/popd commands will ensure that the directories are logged.

Running 2to3 from the spec file

Sometimes, upstream hasn't integrated running 2to3 on the code into their build scripts but they support making a python3 module from it if you manually run 2to3 on the source. This is the case when it's documented on the upstream's website, in a file in the tarball, or even when email with the module's author has instructions for building a python3 module from the python2 source and the authors are willing to support the result. In these cases it's usually just a matter of the upstream not having written the build script that can turn the python2 source into python3. When this happens you can run 2to3 from the spec file. Once you have it working, you can also help upstream integrate it into their build scripts which will benefit everyone in the long term.

You should usually follow upstream's directions on how to run 2to3 and build the python3 module in these cases but there's a few things you should check to make sure upstream is doing it correctly.

  • Since the code is being built from a unified source, you need to copy the code to a new directory before invoking 2to3 just like the building more than once method.
  • If the 2to3 program is invoked instead of using the lib2to3 library functions, make sure it's invoked with --write --nobackups. --write is needed to make 2to3 actually change the files. --nobackups avoids leaving foo.py.bak files in the module directories that then make it into the final package payload.
  • Be sure to run 2to3 on the correct directory. When you run 2to3 you need to run it on the whole tree. A common mistake here for distutils packages has been to run it on the directory below setup.py, missing the setup.py file itself. This leads to errors when python3 tries to execute setup.py
  • If you need to run 2to3 to fix code, use 2to3 or /usr/bin/2to3. At the moment, this program is coming from the python-tools rpm. Using 2to3 means that you'll be using a name that is supported upstream and across distros rather than /usr/bin/python3-2to3 which we have renamed in Fedora to avoid filesystem conflicts. This also makes it easier for us to test and eventually change from using the python2 2to3 to the python3 2to3. We just need to change the python3 package to provide the /usr/bin/2to3 program instead of python and all of our python packages will start using that version instead.
  • If 2to3 runs into a problem, please file a Fedora bug. Please try to isolate a minimal test case that reproduces the problem when doing so.

Manual byte compilation

When byte compiling a .py file, python embeds a magic number in the byte compiled files that correspond to the runtime. Files in %{python?_sitelib} and %{python?_sitearch} MUST correspond to the runtime for which they were built. For instance, a pure python module compiled for the 3.4 runtime MUST be below %{_usr}/lib/python3.4/site-packages

Warning.png
Fedora 29+ only
The following guidelines have been changed in Fedora 29 and won't work in older Fedoras (or EPELs). See bellow for previous version of the guideline for older releases.

The brp-python-bytecompile script tries to figure this out for you. The script determines which interpreter to use when byte compiling the module by following these steps:

  1. What directory is the module installed in? If it's /usr/lib{,64}/pythonX.Y, then pythonX.Y is used to byte compile the module. If pythonX.Y is not installed, then an error is returned and the rpm build process will exit on an error so remember to BuildRequire the proper python package.
  2. By default the script interpreter defined in %{__python} is used to compile the modules outside of those directories. This defaults to /usr/bin/python (that's Python 2.7 on Fedora). This only happens for backwards compatibility reasons.

If you have *.py files outside of the /usr/lib(64)?/pythonX.Y/ directories, you MUST disable their automatic compilation. If you require those files to be byte compiled (e.g. it's an importable Python module) you can then compile them explicitly using the %py_byte_compile macro.

Example for package that has both Python versions:

# Disable automatic compilation of Python files in extra directories
%global _python_bytecompile_extra 0

# Buildrequire both python2 and python3
BuildRequires: python2-devel python3-devel

%install
# Installs a python2 private module into %{buildroot}%{_datadir}/mypackage/foo
# and installs a python3 private module into %{buildroot}%{_datadir}/mypackage/bar
make install DESTDIR=%{buildroot}

# Manually invoke the python byte compile macro for each path that needs byte
# compilation.
%py_byte_compile %{__python2} %{buildroot}%{_datadir}/mypackage/foo
%py_byte_compile %{__python3} %{buildroot}%{_datadir}/mypackage/bar

Note that this does not disable the compilation of files in /usr/lib(64)?/pythonX.Y/.

The %py_byte_compile macro takes two arguments. The first is the python interpreter to use for byte compiling. The second is a file or directory to byte compile. If the second argument is a directory, the macro will recursively byte compile any *.py file in the directory.

Warning.png
No %{} for py_byte_compile
RPM macros can only take arguments when they do not have curly braces around them. Therefore, py_byte_compile won't work correctly if you write: %{py_byte_compile} %{__python2}


Manual byte compilation for Fedora 28 or earlier (including EPEL)

The script interpreter defined in %{__python} is used to compile the modules outside of /usr/lib(64)?/pythonX.Y/ directories. This defaults to /usr/bin/python (that's Python 2.7 on Fedora). If you need to compile the modules for python3, set it to /usr/bin/python3 instead:

%global __python %{__python3}

Doing this is useful when you have a python3 application that's installing a private module into its own directory. For instance, if the foobar application installs a module for use only by the command line application in %{_datadir}/foobar. Since these files are not in one of the python3 library paths (ie. /usr/lib/python3.1) you have to override %{__python} to tell brp-python-bytecompile to use the python3 interpreter for byte compiling. </li> </ol>

These settings are enough to properly byte compile any package that builds python modules in %{python?_sitelib} or %{python?_sitearch} or builds for only a single python interpreter. However, if the application you're packaging needs to build with both python2 and python3 and install into a private module directory (perhaps because it provides one utility written in python2 and a second utility written in python3) then you need to do this manually. Here's a sample spec file snippet that shows what to do:

# Turn off the brp-python-bytecompile script
%undefine __brp_python_bytecompile
# Buildrequire both python2 and python3
BuildRequires: python2-devel python3-devel
[...]

%install
# Installs a python2 private module into %{buildroot}%{_datadir}/mypackage/foo
# and installs a python3 private module into %{buildroot}%{_datadir}/mypackage/bar
make install DESTDIR=%{buildroot}

# Manually invoke the python byte compile macro for each path that needs byte
# compilation.
%py_byte_compile %{__python2} %{buildroot}%{_datadir}/mypackage/foo
%py_byte_compile %{__python3} %{buildroot}%{_datadir}/mypackage/bar

Note that this does disable the compilation of files in /usr/lib(64)?/pythonX.Y/.

In Fedora 27 or earlier, use %global __os_install_post %(echo '%{__os_install_post}' | sed -e 's!/usr/lib[^[:space:]]*/brp-python-bytecompilespace:.*$!!g') instead of the %undefine line above.