Churchyard (talk | contribs) |
Churchyard (talk | contribs) |
||
Line 118: | Line 118: | ||
Furthermore, container images and other minimal systems maintainers may choose to exclude the {{package|python3-libs-bytecode-opt-0}} package to save more disk space if desired. | Furthermore, container images and other minimal systems maintainers may choose to exclude the {{package|python3-libs-bytecode-opt-0}} package to save more disk space if desired. | ||
Note that by splitting the three optimization levels to different RPM packages, files can no longer be hardlinked between each other. This results in a slight size increase when all three optimization levels are installed. The change owners consider the need for all three subpackages to be present simultaneously on one size-sensitive system unlikely and hence consider this a fair trade. | |||
¹ No real data was collected to support this claim. This hypothesis is made by the Python maintainers based on their own experience. | ¹ No real data was collected to support this claim. This hypothesis is made by the Python maintainers based on their own experience. |
Revision as of 17:28, 7 September 2020
Python: Optional Bytecode Cache
Summary
The Python standard library bytecode cache files (e.g. /usr/lib64/python3.9/.../__pycache__/*.pyc
) will be moved from the
package to three new optional subpackages (split by optimization level). The non-optimized bytecode cache will be recommended by python3-libs
and installed by default but removable. The bytecode cache optimization level 1 and 2 will not be recommended (and hence will not be installed by default) but will be installable. The default SELinux policy will be adapted not to audit AVC denials when the bytecode cache is created by Python on runtime. This will save 8.89 MiB disk space on default installations or 17.12 MiB on minimal installations (by opting-out from the recommended subpackage with non-optimized bytecode cache). When all three new packages are installed, the size will increase slightly over the status quo (by 4.5 MiB).
python3-libs
Owner
- Name: Miro Hrončok
- Name: Lumír Balhar
- Name: Tomáš Orsava
- Name: Lukáš Vrabec (selinux-policy maintainer)
- Email: python-maint@redhat.com
Current status
- Targeted release: Fedora 34
- Last updated: 2020-09-07
- FESCo issue: <will be assigned by the Wrangler>
- Tracker bug: <will be assigned by the Wrangler>
- Release notes tracker: <will be assigned by the Wrangler>
Detailed Description
What is the Python bytecode cache
When Python code is interpreted, it is compiled to Python bytecode. When a pure Python module is imported for the first time, the compiled bytecode is serialized and cached to a .pyc
file located in the __pycache__
directory next to the .py
source. Subsequent imports use the cache directly, until it is invalidated (for example when the .py
source is edited and its mtime
stamp is bumped) -- at that point, the cache is updated. This behavior is explained in detail in PEP 3147. The invalidation is described in PEP 552.
Python can operate in 3 different optimization levels: 0, 1 and 2. By default, the optimization level is 0. When invoked with the -O
command line option optimization is set to 1, similarly with -OO
it is 2. Bytecode cache for different optimization levels is saved with different filenames as described in PEP 488.
As an example, a Python module located at /path/to/basename.py
will have bytecode cache files for CPython 3.9 stored as:
/path/to/__pycache__/basename.cpython-39.pyc
for the non-optimized bytecode/path/to/__pycache__/basename.cpython-39.opt-1.pyc
for optimization level 1/path/to/__pycache__/basename.cpython-39.opt-2.pyc
for optimization level 2
Python bytecode cache in RPM packages (status quo)
Pure Python modules shipped in RPM packages (and namely the ones shipped trough the
package) are located at paths not writable by regular user, under python3-libs
/usr/lib(64)/python3.9/
, hence the bytecode cache is also located in such locations. To work around this problem, the bytecode cache is pre-compiled when RPM packages are built and
ships and owns the sources as well as the bytecode cache:
python3-libs
$ rpm -ql python3-libs ... /usr/lib64/python3.9/__pycache__/ast.cpython-39.opt-1.pyc /usr/lib64/python3.9/__pycache__/ast.cpython-39.opt-2.pyc /usr/lib64/python3.9/__pycache__/ast.cpython-39.pyc ... /usr/lib64/python3.9/ast.py ...
As a result, the package is quite big, essentially shipping all pure Python modules 4 times.
Depending of the module content, its bytecode cache files might be identical across optimization levels. For such cases, the files are hardlinked to reduce the bloat:
$ ls -1i /usr/lib64/python3.9/collections/__pycache__/abc.*pyc 8634 /usr/lib64/python3.9/collections/__pycache__/abc.cpython-39.opt-1.pyc 8634 /usr/lib64/python3.9/collections/__pycache__/abc.cpython-39.opt-2.pyc 8634 /usr/lib64/python3.9/collections/__pycache__/abc.cpython-39.pyc
This is however not possible for all the modules from
:
python3-libs
$ ls -1i /usr/lib64/python3.9/__pycache__/ast.*pyc 8438 /usr/lib64/python3.9/__pycache__/ast.cpython-39.opt-1.pyc 8440 /usr/lib64/python3.9/__pycache__/ast.cpython-39.opt-2.pyc 8441 /usr/lib64/python3.9/__pycache__/ast.cpython-39.pyc
What if the bytecode cache would not be packaged
When the bytecode cache is not packaged, several things happen:
- When non-root users run Python, the imported modules are never cached. As a result, the startup time of Python apps might be slightly larger than necessary until root runs them.
- When root runs Python, the imported modules are cached. As a result untracked
.pyc
files start to pop up in/usr/lib(64)/python3.9/
. When the system is updated to a newer Python version, the untracked files remain on the filesystem until manually cleaned up. - When root runs Python in SELinux restricted context, the imported modules are attempted to be cached but SELinux does not allow that. The result is same as (1) with a lot of noise from SELinux.
Packaging the bytecode cache into optional subpackages
To be able to save quite some disk space without disrupting the user experience, we propose to ship the pre-compiled bytecode cache previously included in
as follows:
python3-libs
- Pre-compiled non-optimized bytecode cache files (
*.cpython-39.pyc
) will be packaged in
.python3-libs-bytecode-opt-0
- Pre-compiled level 1 optimized bytecode cache files (
*.cpython-39.opt-1.pyc
) will be packaged in
.python3-libs-bytecode-opt-1
- Pre-compiled level 2 optimized bytecode cache files (
*.cpython-39.opt-2.pyc
) will be packaged in
.python3-libs-bytecode-opt-2
In order to properly own any runtime-generated bytecode-cache files, the
will list all files listed in the three abovementioned packages as python3-libs
%ghost
.
will not Require any of the bytecode cache packages, hence the packages will be (un)installable and fully optional.
python3-libs
Given that almost all Fedora Python packages invoke Python in the non-optimized mode¹,
will Recommend python3-libs
and hence the package will be installed by default together with Python; the user experience will remain the same for the vast majority of users and use cases.
python3-libs-bytecode-opt-0
Furthermore, container images and other minimal systems maintainers may choose to exclude the
package to save more disk space if desired.
python3-libs-bytecode-opt-0
Note that by splitting the three optimization levels to different RPM packages, files can no longer be hardlinked between each other. This results in a slight size increase when all three optimization levels are installed. The change owners consider the need for all three subpackages to be present simultaneously on one size-sensitive system unlikely and hence consider this a fair trade.
¹ No real data was collected to support this claim. This hypothesis is made by the Python maintainers based on their own experience.
SELinux policy changes
In order to suppress the otherwise omnipresent AVC denial messages about Python failing to write the bytecode cache, the Python maintainers have teamed up with the Fedora's selinux-policy maintainers to suppress those. The implementation details about this are available at:
When Python runs under the root user in SELinux restricted context, SELinux will still prevent it from writing the bytecode cache, but it will not clutter the audit log.
Size impact
Sizes calculated in mock
on x86_64.
Situation | Size of /usr/lib(64)/python3.9 in MiB |
Difference in MiB |
---|---|---|
Status quo (before this change) | 31.84 | |
Default (non-optimized cache only) | 22.96 | -8.89 |
No cache | 14.72 | -17.12 |
Levels 0 and 1 | 29.71 | -2.13 |
All optimization levels (like before) | 36.35 | +4.50 |
Speed impact
Rejected ideas
In this section, we briefly describe ideas that were presented by others or considered by the change owners, but rejected.
Stop shipping mandatory .py
sources, ship only .pyc
cache
Make Python not attempt to write bytecode cache into /usr/lib(64)/python3.9
Not realized ideas
In this section, we briefly describe ideas that were presented by others or considered by the change owners, but were not realized (e.g. for capacity reasons). Such ideas may be realized later.
Store bytecode cache in /var/cache
and/or ~/.cache
Apply this change to all Python RPM packages
Feedback
Benefit to Fedora
Scope
- Proposal owners:
- Update the Python package, see https://src.fedoraproject.org/rpms/python3.9/pull-request/29
- Update selinux-policy, see https://github.com/fedora-selinux/selinux-policy/pull/404
- Other developers: N/A (not a System Wide Change)
- Release engineering: N/A (not a System Wide Change)
- Policies and guidelines: N/A (not a System Wide Change)
- Trademark approval: N/A (not needed for this Change)
- Alignment with Objectives: Fedora Minimization Objective
Upgrade/compatibility impact
N/A (not a System Wide Change)
How To Test
N/A (not a System Wide Change)
User Experience
Dependencies
N/A (not a System Wide Change)
Contingency Plan
- Contingency mechanism: (What to do? Who will do it?) N/A (not a System Wide Change)
- Contingency deadline: N/A (not a System Wide Change)
- Blocks release? N/A (not a System Wide Change), Yes/No
- Blocks product? product
Documentation
N/A (not a System Wide Change)