From Fedora Project Wiki
mNo edit summary
(Changed technical details based on upstream approved PR)
 
(35 intermediate revisions by 4 users not shown)
Line 1: Line 1:
{{admon/important | Incomplete | This page is incomplete! }}


<!-- Self Contained or System Wide Change Proposal?
<!-- Self Contained or System Wide Change Proposal?
Line 27: Line 26:
== Summary ==
== Summary ==
<!-- A sentence or two summarizing what this change is and what it will do. This information is used for the overall changeset summary page for each release. -->
<!-- A sentence or two summarizing what this change is and what it will do. This information is used for the overall changeset summary page for each release. -->
The current way of automatic Python byte-compiling of files outside Python-specific directories is too magical and error-prone. It is built on heuristics that are increasingly wrong.
We will provide a way to opt-out of it and adjust the guidelines to prefer explicit bytecompilation of such files. Later, the old behavior will be opt-in only or will cease to exist.
Note that bytecompilation in Python-specific directories (e.g. <code>/usr/lib/python3.6/</code>) is not affected.


== Owner ==
== Owner ==
Line 34: Line 38:
-->
-->
* Name: [[User:Churchyard|Miro Hrončok]]
* Name: [[User:Churchyard|Miro Hrončok]]
* Name: [[User:pviktori|Petr Viktorin]]
<!-- Include you email address that you can be reached should people want to contact you about helping with your change, status is requested, or technical issues need to be resolved. If the change proposal is owned by a SIG, please also add a primary contact person. -->
<!-- Include you email address that you can be reached should people want to contact you about helping with your change, status is requested, or technical issues need to be resolved. If the change proposal is owned by a SIG, please also add a primary contact person. -->
* Email: mhroncok@redhat.com
* Email: mhroncok@redhat.com, pviktori@redhat.com
* Release notes owner: <!--- To be assigned by docs team [[User:FASAccountName| Release notes owner name]] <email address> -->
* Release notes owner: <!--- To be assigned by docs team [[User:FASAccountName| Release notes owner name]] <email address> -->
<!--- UNCOMMENT only for Changes with assigned Shepherd (by FESCo)
<!--- UNCOMMENT only for Changes with assigned Shepherd (by FESCo)
Line 56: Line 61:
CLOSED as NEXTRELEASE -> change is completed and verified and will be delivered in next release under development
CLOSED as NEXTRELEASE -> change is completed and verified and will be delivered in next release under development
-->
-->
* Tracker bug: <will be assigned by the Wrangler>
* Tracker bug: [https://bugzilla.redhat.com/show_bug.cgi?id=1551326 #1551326]
* Release Notes tracking: [https://pagure.io/fedora-docs/release-notes/issue/125 #125]


== Detailed Description ==
== Detailed Description ==
Line 62: Line 68:
<!-- Expand on the summary, if appropriate.  A couple sentences suffices to explain the goal, but the more details you can provide the better. -->
<!-- Expand on the summary, if appropriate.  A couple sentences suffices to explain the goal, but the more details you can provide the better. -->


As for Fedora 28, all <code>*.py</code> files outside of the <code>/usr/lib(64)?/pythonX.Y/</code> directories are bytecompiled by <code>/usr/bin/python</code>.
=== Background ===
This is overly magical and assumes several things (not always right):
 
When a Python modules is imported, the source file (<code>*.py</code>) is automatically compiled to bytecode.
The bytecode is automatically cached in “pyc files” next to the source (e.g. <code><i>moldulename</i>.pyc</code> and <code><i>moldulename</i>.pyo</code> in Python 2.7; <code>__pycache__/<i>moldulename</i>.cpython-36*.pyc</code> in Python 3.6).
 
For RPM packages installed system-wide, creating cache files would generally require root priviliges. So they are included in RPMs rather than generated on import.
 
=== Status quo ===
 
For packagers' convenience, rpmbuild's <code>brp-python-bytecompile</code> script generates <code>.pyc</code>/<code>.pyo</code> cache files automatically:
 
* In interpreter-specific directories, such as <code>/usr/lib/python3.6/</code>, these are compiled using the appropriate interpreter.
* Some directories, such as <code>/bin</code>, <code>/sbin</code> and <code>/usr/share/doc</code>, are excluded.
* Outside these directories, modules are compiled using <code>%{__python}</code>, which is <code>/usr/bin/python</code>, which is currently Python 2. This is only done if the <code>%{__python}</code> binary is available.
 
The first two points are good, straightforward, and the automatism is limited to Python-specific directories.
The last point, however, relies on several assumptions which are not always right:
 
* All files named <code>*.py</code> are Python modules that need to be bytecompiled. (Quite an accurate heuristic, but has very bad behavior on false positives: it can affect packages that don't have anything to do with Python.)
* Any package that has <code>.py</code> files BuildRequires <code>/usr/bin/python</code>. (If it does not, the package will build differently depending on whether <code>/usr/bin/python</code> happens to be available during rpmbuild.)
* When a Python module is not in <code>/usr/lib(64)?/pythonX.Y/</code> it is intended for the <code>%{__python}</code> interpreter – currently Python 2. (But the module could also be, for example, for Python 3, or PyPy, or several of those.)
 
Bytecompilation outside Python-specific directories may be changed by redefining <code>%__python</code> to:
 
* python3: This is documented in the guidelines as a way to enable the automatic bytecompilation for python3. Around 70 packages use it.
* python2: This *should* be done to ensure consistency when <code>%{__python}</code> is switched. Only about 2 packages does it, because the magic "just works" for the time being.
 
Automatic bytecompilation oudside Python-specific directories cannot be disabled without disabling ''all'' bytecompilation. Also, it can not be done for more than one interpreter.
 
See [https://fedoraproject.org/w/index.php?title=Packaging:Python_Appendix&oldid=419140#Manual_byte_compilation Packaging:Python Appendix] for more information (this links to a specific revision so the link makes sense once this change is implemented and the guidelines are changed).
 
The current behavior is surprising. [https://pagure.io/copr/copr/c/8fa8fe2f1583088a0162d89a13e0dee70a0db801?branch=master Mistakes are made.] Things are done or not done based on the presence of <code>/usr/bin/python</code>.
 
Here is a simple example of a package that builds fine without <code>/usr/bin/python</code> in the buildroot but fails when it's there.
The same issue was [https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/67GW7UF4HMIFVCMLHVSNIGJ37GVVTMXQ/#67GW7UF4HMIFVCMLHVSNIGJ37GVVTMXQ reported on fedora-devel] for a "real" package.
 
Name:          reproducer
Version:        0.1
Release:        1%{?dist}
Summary:        Reproducer for a bytecompile script issue
License:        MIT
BuildArch:      noarch
%description
This package will build fine if /usr/bin/python is *not* in the buildroot.
%prep
echo "Poland" > country-name.pl
echo "Paraguay" > country-name.py
echo "Saint Helena" > country-name.sh
echo "Serbia" > country-name.rs
%build
%install
mkdir -p %{buildroot}%{_datadir}/%{name}
cp country-name.* %{buildroot}%{_datadir}/%{name}
%files
%dir %{_datadir}/%{name}/
%{_datadir}/%{name}/country-name.??
 
=== How we are changing it ===
 
For the time being, we keep the old behavior working, so the hundreds of packages that implicitly rely on it do not break all at once.
However, we will not (automatically) apply it to Python 3, so it will be phased out as packages switch to Python 3.
 
A opt-out mechanism for automatic compilation of files outside of <code>/usr/lib(64)?/pythonX.Y/</code> will be provided (setting <code>%_python_bytecompile_extra</code> to 0 or undefining it).
Speaking code, this will disable [https://github.com/rpm-software-management/rpm/blob/ab3fab29de51c7e68c9911d3b7809109da92fa6d/scripts/brp-python-bytecompile#L82 the final part of the brp-python-bytecompile script].
 
Guidelines will be adjusted to say the following:
 
* if you have <code>*.py</code> files outside of the <code>/usr/lib(64)?/pythonX.Y/</code>, you '''MUST''' disable their automatic compilation, and compile them explicitly by the <code>%py_byte_compile</code> macro if and only if the bytecompilation is desired (i.e. it's an importable Python module).
 
Example for package that has both Python versions:
 
# Turn off the brp-python-bytecompile automagic
%global _python_bytecompile_extra 0
# Buildrequire both python2 and python3
BuildRequires: python2-devel python3-devel
%install
# Installs a python2 private module into %{buildroot}%{_datadir}/mypackage/foo
# and installs a python3 private module into %{buildroot}%{_datadir}/mypackage/bar
make install DESTDIR=%{buildroot}
# Manually invoke the python byte compile macro for each path that needs byte
# compilation.
%py_byte_compile %{__python2} %{buildroot}%{_datadir}/mypackage/foo
%py_byte_compile %{__python3} %{buildroot}%{_datadir}/mypackage/bar
 
Note that unlike the current example in the guidelines linked above, this does not disable the compilation of files in <code>/usr/lib(64)?/pythonX.Y/</code>.


* all files named <code>*.py</code> are Python modules that need to be bytecompiled
Example for Python 3 only:
* when a file is not in <code>/usr/lib(64)?/pythonX.Y/</code> it is intended for the <code>/usr/bin/python</code> interpreter
** That is currently Python 2, but may be removed or changed to Python 3 at any point in the future.
** This is only the default behavior, it can be changed by redefining <code>%__python</code> to:
*** python3: that currently happens in various packages and is documented in the guidelines as a way to do it
*** python2: nobody does that, because the magic "just works" for this use case - relying on the fact that <code>/usr/bin/python</code> is currently python2
**** that assumption is forbidden by the Python packaging guidelines, yet here it is heavily used


See [https://fedoraproject.org/w/index.php?title=Packaging:Python_Appendix&oldid=419140#Manual_byte_compilation Packaging:Python Appendix] for more information (this links to specific revision because this change will change it).
# Turn off the brp-python-bytecompile automagic
%global _python_bytecompile_extra 0
BuildRequires: python3-devel
%install
# Installs a python3 private module into %{buildroot}%{_datadir}/mypackage/bar
make install DESTDIR=%{buildroot}
# Manually invoke the python byte compile macro for each path that needs byte
# compilation.
%py_byte_compile %{__python3} %{buildroot}%{_datadir}/mypackage/bar
 
The Python 2 only example is analogical.
 
The notion of redefining <code>%{__python}</code> will be removed from the guidelines.
 
Currently, <code>%py_byte_compile</code> lives in {{package|python3-devel}}. We'll move it to some generic package that all Python devel packages require (such as {{package|python-rpm-macros}} ) (this has been done already).
 
Analogically, packagers can <code>%global _python_bytecompile_extra 1</code> to explicitly say they rely on the current behavior.
Later (i.e. not in Fedora 29, but approximately when <code>%{__python}</code> stops being python2), we'll make the old behavior opt-in (or disable it entirely if no package uses  <code>%_python_bytecompile_extra</code>).
 
All <code>brp-python-bytecompile</code> changes will be [https://github.com/rpm-software-management/rpm/pull/434 shared and discussed in upstream RPM].
 
We will provide pull requests for the ~50 packages that redefine <code>__python</code> to <code>%{__python3}</code> (the currently recommended way to enable automatic byte-compilation on Python 3).


== Benefit to Fedora ==
== Benefit to Fedora ==


More explicit specfiles when it comes to Python byte compilation. This will ease the change once we decide <code>/usr/bin/python</code> is no longer python2.
The new guidelines will be less error prone.
Note that we'd prefer to switch to the new behavior right now, but we keep it opt-in to not break the ~500 packages that use it.
   
   
 
<!-- What is the benefit to the platform?  If this is a major capability update, what has changed?  If this is a new functionality, what capabilities does it bring? Why will Fedora become a better distribution or project because of this proposal?-->
<!-- What is the benefit to the platform?  If this is a major capability update, what has changed?  If this is a new functionality, what capabilities does it bring? Why will Fedora become a better distribution or project because of this proposal?-->


== Scope ==
== Scope ==
* Proposal owners:
* Proposal owners: Make it work technically, propose the new guidelines, file pull requests for python3 modules that follow the current guidelines.
<!-- What work do the feature owners have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
<!-- What work do the feature owners have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->


* Other developers: N/A (not a System Wide Change) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Other developers: Maintainers of python3 packages that redefine <code>%__python</code> should merge provided pull requests. Others may opt-in for the new behavior or explicitly stick with the old one (not a System Wide Change, they don't have to do anything). <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- What work do other developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
<!-- What work do other developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->


* Release engineering: [https://pagure.io/releng/issues #Releng issue number] (a check of an impact with Release Engineering is needed) <!-- REQUIRED FOR SYSTEM WIDE AS WELL AS FOR SELF CONTAINED CHANGES -->
* Release engineering: [https://pagure.io/releng/issue/7315 #7315] (a check of an impact with Release Engineering is needed) <!-- REQUIRED FOR SYSTEM WIDE AS WELL AS FOR SELF CONTAINED CHANGES -->
<!-- Does this feature require coordination with release engineering (e.g. changes to installer image generation or update package delivery)?  Is a mass rebuild required?  include a link to the releng issue.  
<!-- Does this feature require coordination with release engineering (e.g. changes to installer image generation or update package delivery)?  Is a mass rebuild required?  include a link to the releng issue.  
The issue is required to be filed prior to feature submission, to ensure that someone is on board to do any process development work and testing, and that all changes make it into the pipeline; a bullet point in a change is not sufficient communication -->
The issue is required to be filed prior to feature submission, to ensure that someone is on board to do any process development work and testing, and that all changes make it into the pipeline; a bullet point in a change is not sufficient communication -->
** [[Fedora_Program_Management/ReleaseBlocking/Fedora{{FedoraVersionNumber|next}}|List of deliverables]]: N/A (not a System Wide Change) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- ** [[Fedora_Program_Management/ReleaseBlocking/Fedora{{FedoraVersionNumber|next}}|List of deliverables]]: N/A (not a System Wide Change) REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- Please check the list of Fedora release deliverables and list all the differences the feature brings -->
<!-- Please check the list of Fedora release deliverables and list all the differences the feature brings -->


* Policies and guidelines: N/A (not a System Wide Change) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Policies and guidelines: will be changed as described in description <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- Do the packaging guidelines or other documents need to be updated for this feature?  If so, does it need to happen before or after the implementation is done?  If a FPC ticket exists, add a link here. -->
<!-- Do the packaging guidelines or other documents need to be updated for this feature?  If so, does it need to happen before or after the implementation is done?  If a FPC ticket exists, add a link here. -->


* Trademark approval: N/A (not needed for this Change)
* Trademark approval: not needed
<!-- If your Change may require trademark approval (for example, if it is a new Spin), file a ticket ( https://fedorahosted.org/council/ ) requesting trademark approval from the Fedora Council. This approval will be done via the Council's consensus-based process. -->
<!-- If your Change may require trademark approval (for example, if it is a new Spin), file a ticket ( https://fedorahosted.org/council/ ) requesting trademark approval from the Fedora Council. This approval will be done via the Council's consensus-based process. -->


Line 104: Line 222:


<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
N/A (not a System Wide Change)
None expected.


== How To Test ==
== How To Test ==
Line 122: Line 240:


<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
N/A (not a System Wide Change)
More specific instructions based on the examples in description will be here once ready. In the meantime, feel free to test the examples in the description as you see fit.


== User Experience ==
== User Experience ==
<!-- If this change proposal is noticeable by its target audience, how will their experiences change as a result?  Describe what they will see or notice. -->
<!-- If this change proposal is noticeable by its target audience, how will their experiences change as a result?  Describe what they will see or notice. -->
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
N/A (not a System Wide Change)
The users of this change are packagers. The new behavior should make byte-compilation more obvious, explicit, and discoverable.
 
Users of Fedora should not feel this (except if this change uncovers a packaging bug).
== Dependencies ==
<!-- What other packages (RPMs) depend on this package?  Are there changes outside the developers' control on which completion of this change depends?  In other words, completion of another change owned by someone else and might cause you to not be able to finish on time or that you would need to coordinate?  Other upstream projects like the kernel (if this is not a kernel change)? -->
 
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
N/A (not a System Wide Change)  


== Contingency Plan ==
== Contingency Plan ==


<!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "Revert the shipped configuration".  Or it might not (e.g. rebuilding a number of dependent packages).  If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy.  -->
<!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "Revert the shipped configuration".  Or it might not (e.g. rebuilding a number of dependent packages).  If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy.  -->
* Contingency mechanism: (What to do?  Who will do it?) N/A (not a System Wide Change)  <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Contingency mechanism: we'll finish the change later (not a System Wide Change)  <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- When is the last time the contingency mechanism can be put in place?  This will typically be the beta freeze. -->
<!-- When is the last time the contingency mechanism can be put in place?  This will typically be the beta freeze. -->
* Contingency deadline: N/A (not a System Wide Change)  <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Contingency deadline: none (not a System Wide Change)  <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- Does finishing this feature block the release, or can we ship with the feature in incomplete state? -->
<!-- Does finishing this feature block the release, or can we ship with the feature in incomplete state? -->
* Blocks release? N/A (not a System Wide Change), Yes/No <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Blocks release? no (not a System Wide Change) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Blocks product? product <!-- Applicable for Changes that blocks specific product release/Fedora.next -->
* Blocks product? no <!-- Applicable for Changes that blocks specific product release/Fedora.next -->


== Documentation ==
== Documentation ==
Line 149: Line 262:


<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
N/A (not a System Wide Change)
The guidelines will be the documentation.


== Release Notes ==
== Release Notes ==
Line 158: Line 271:
-->
-->


[[Category:ChangePageIncomplete]]
This change does not deserve Release Notes, it is not user facing.
 
[[Category:ChangeAcceptedF29]]
<!-- When your change proposal page is completed and ready for review and announcement -->
<!-- When your change proposal page is completed and ready for review and announcement -->
<!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler -->
<!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler -->

Latest revision as of 12:17, 28 May 2018


No more automagic Python bytecompilation

Summary

The current way of automatic Python byte-compiling of files outside Python-specific directories is too magical and error-prone. It is built on heuristics that are increasingly wrong. We will provide a way to opt-out of it and adjust the guidelines to prefer explicit bytecompilation of such files. Later, the old behavior will be opt-in only or will cease to exist.

Note that bytecompilation in Python-specific directories (e.g. /usr/lib/python3.6/) is not affected.

Owner

Current status

Detailed Description

Background

When a Python modules is imported, the source file (*.py) is automatically compiled to bytecode. The bytecode is automatically cached in “pyc files” next to the source (e.g. moldulename.pyc and moldulename.pyo in Python 2.7; __pycache__/moldulename.cpython-36*.pyc in Python 3.6).

For RPM packages installed system-wide, creating cache files would generally require root priviliges. So they are included in RPMs rather than generated on import.

Status quo

For packagers' convenience, rpmbuild's brp-python-bytecompile script generates .pyc/.pyo cache files automatically:

  • In interpreter-specific directories, such as /usr/lib/python3.6/, these are compiled using the appropriate interpreter.
  • Some directories, such as /bin, /sbin and /usr/share/doc, are excluded.
  • Outside these directories, modules are compiled using %{__python}, which is /usr/bin/python, which is currently Python 2. This is only done if the %{__python} binary is available.

The first two points are good, straightforward, and the automatism is limited to Python-specific directories. The last point, however, relies on several assumptions which are not always right:

  • All files named *.py are Python modules that need to be bytecompiled. (Quite an accurate heuristic, but has very bad behavior on false positives: it can affect packages that don't have anything to do with Python.)
  • Any package that has .py files BuildRequires /usr/bin/python. (If it does not, the package will build differently depending on whether /usr/bin/python happens to be available during rpmbuild.)
  • When a Python module is not in /usr/lib(64)?/pythonX.Y/ it is intended for the %{__python} interpreter – currently Python 2. (But the module could also be, for example, for Python 3, or PyPy, or several of those.)

Bytecompilation outside Python-specific directories may be changed by redefining %__python to:

  • python3: This is documented in the guidelines as a way to enable the automatic bytecompilation for python3. Around 70 packages use it.
  • python2: This *should* be done to ensure consistency when %{__python} is switched. Only about 2 packages does it, because the magic "just works" for the time being.

Automatic bytecompilation oudside Python-specific directories cannot be disabled without disabling all bytecompilation. Also, it can not be done for more than one interpreter.

See Packaging:Python Appendix for more information (this links to a specific revision so the link makes sense once this change is implemented and the guidelines are changed).

The current behavior is surprising. Mistakes are made. Things are done or not done based on the presence of /usr/bin/python.

Here is a simple example of a package that builds fine without /usr/bin/python in the buildroot but fails when it's there. The same issue was reported on fedora-devel for a "real" package.

Name:           reproducer
Version:        0.1
Release:        1%{?dist}
Summary:        Reproducer for a bytecompile script issue
License:        MIT
BuildArch:      noarch

%description
This package will build fine if /usr/bin/python is *not* in the buildroot.

%prep
echo "Poland" > country-name.pl
echo "Paraguay" > country-name.py
echo "Saint Helena" > country-name.sh
echo "Serbia" > country-name.rs

%build

%install
mkdir -p %{buildroot}%{_datadir}/%{name}
cp country-name.* %{buildroot}%{_datadir}/%{name}

%files
%dir %{_datadir}/%{name}/
%{_datadir}/%{name}/country-name.??

How we are changing it

For the time being, we keep the old behavior working, so the hundreds of packages that implicitly rely on it do not break all at once. However, we will not (automatically) apply it to Python 3, so it will be phased out as packages switch to Python 3.

A opt-out mechanism for automatic compilation of files outside of /usr/lib(64)?/pythonX.Y/ will be provided (setting %_python_bytecompile_extra to 0 or undefining it). Speaking code, this will disable the final part of the brp-python-bytecompile script.

Guidelines will be adjusted to say the following:

  • if you have *.py files outside of the /usr/lib(64)?/pythonX.Y/, you MUST disable their automatic compilation, and compile them explicitly by the %py_byte_compile macro if and only if the bytecompilation is desired (i.e. it's an importable Python module).

Example for package that has both Python versions:

# Turn off the brp-python-bytecompile automagic
%global _python_bytecompile_extra 0

# Buildrequire both python2 and python3
BuildRequires: python2-devel python3-devel

%install
# Installs a python2 private module into %{buildroot}%{_datadir}/mypackage/foo
# and installs a python3 private module into %{buildroot}%{_datadir}/mypackage/bar
make install DESTDIR=%{buildroot}

# Manually invoke the python byte compile macro for each path that needs byte
# compilation.
%py_byte_compile %{__python2} %{buildroot}%{_datadir}/mypackage/foo
%py_byte_compile %{__python3} %{buildroot}%{_datadir}/mypackage/bar

Note that unlike the current example in the guidelines linked above, this does not disable the compilation of files in /usr/lib(64)?/pythonX.Y/.

Example for Python 3 only:

# Turn off the brp-python-bytecompile automagic
%global _python_bytecompile_extra 0

BuildRequires: python3-devel

%install
# Installs a python3 private module into %{buildroot}%{_datadir}/mypackage/bar
make install DESTDIR=%{buildroot}

# Manually invoke the python byte compile macro for each path that needs byte
# compilation.
%py_byte_compile %{__python3} %{buildroot}%{_datadir}/mypackage/bar

The Python 2 only example is analogical.

The notion of redefining %{__python} will be removed from the guidelines.

Currently, %py_byte_compile lives in python3-devel. We'll move it to some generic package that all Python devel packages require (such as python-rpm-macros ) (this has been done already).

Analogically, packagers can %global _python_bytecompile_extra 1 to explicitly say they rely on the current behavior. Later (i.e. not in Fedora 29, but approximately when %{__python} stops being python2), we'll make the old behavior opt-in (or disable it entirely if no package uses %_python_bytecompile_extra).

All brp-python-bytecompile changes will be shared and discussed in upstream RPM.

We will provide pull requests for the ~50 packages that redefine __python to %{__python3} (the currently recommended way to enable automatic byte-compilation on Python 3).

Benefit to Fedora

More explicit specfiles when it comes to Python byte compilation. This will ease the change once we decide /usr/bin/python is no longer python2. The new guidelines will be less error prone.

Note that we'd prefer to switch to the new behavior right now, but we keep it opt-in to not break the ~500 packages that use it.


Scope

  • Proposal owners: Make it work technically, propose the new guidelines, file pull requests for python3 modules that follow the current guidelines.
  • Other developers: Maintainers of python3 packages that redefine %__python should merge provided pull requests. Others may opt-in for the new behavior or explicitly stick with the old one (not a System Wide Change, they don't have to do anything).
  • Release engineering: #7315 (a check of an impact with Release Engineering is needed)
  • Policies and guidelines: will be changed as described in description
  • Trademark approval: not needed

Upgrade/compatibility impact

None expected.

How To Test

More specific instructions based on the examples in description will be here once ready. In the meantime, feel free to test the examples in the description as you see fit.

User Experience

The users of this change are packagers. The new behavior should make byte-compilation more obvious, explicit, and discoverable. Users of Fedora should not feel this (except if this change uncovers a packaging bug).

Contingency Plan

  • Contingency mechanism: we'll finish the change later (not a System Wide Change)
  • Contingency deadline: none (not a System Wide Change)
  • Blocks release? no (not a System Wide Change)
  • Blocks product? no

Documentation

The guidelines will be the documentation.

Release Notes

This change does not deserve Release Notes, it is not user facing.