From Fedora Project Wiki
Line 115: Line 115:


<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
N/A (not a System Wide Change)  
The following script can be used to verify that the change is in effect:
 
<pre>
import ctypes
import sys
EMPTY_TUPLE_SINGLETON = ()
 
def get_empty_tuple(lib):
    # Call PyTuple_New(0)</nowiki>
    func = lib.PyTuple_New
    func.argtypes = (ctypes.c_ssize_t,)
    func.restype = ctypes.py_object
    return func(0)
def test_lib(libname, lib):
    obj = get_empty_tuple(lib)
    if obj is EMPTY_TUPLE_SINGLETON:
        print("%s: SAME namespace" % libname)
    else:
        print("%s: DIFFERENT namespace" % libname)
 
def test():
    program = ctypes.pythonapi
 
    if hasattr(sys, 'abiflags'):
        abiflags = sys.abiflags
    else:
        # Python 2
        abiflags = ''
    ver = sys.version_info
    filename = ('libpython%s.%s%s.so.1.0'
                % (ver.major, ver.minor, abiflags))
    libpython = ctypes.cdll.LoadLibrary(filename)
 
    test_lib('program', program)
    test_lib('libpython', libpython)
 
test()
</pre>


== User Experience ==
== User Experience ==

Revision as of 13:02, 22 October 2019

Build Python 3 to statically link with libpython3.8.a for better performance

Summary

Python 3 traditionally in Fedora was built with a shared library libpython3.?.so and the final binary was dynamically linked against that shared library. This change is about creating the static library and linking the final python3 binary against it, as it provides significant performance improvement, up to 15% depending on the workload. The static library will not be shipped. The shared library will continue to exist in a separate subpackage. In essence, python3 will no longer depend on libpython.

Owner

Current status

  • Targeted release: Fedora 32
  • Last updated: 2019-10-22
  • Tracker bug: <will be assigned by the Wrangler>
  • Release notes tracker: <will be assigned by the Wrangler>

Detailed Description

When we compile the python3 package on Fedora (prior to this change), we create the libpython3.?.so shared library and the final python3 binary (/usr/bin/python3) is dynamically linked against it. However by building the libpython3.?.a static library and statically linking the final binary against it, we can achieve a performance gain of approximately 15% depending on the workload. Link time optimizations and profile guided optimizations also have a greater impact when python3 is linked statically.

Since Python 3.8, C extensions are no longer linked to libpython by default (for example, they need to utilize the --embed flag for python3-config to do so). During the Python 3.8 upgrade and rebuilds we've uncovered various cases of packages linking to libpython implicitly through various hacks within their buildsystems and fixed as many as possible. However, there are legitimate reasons to link to libpython and for those cases libpython should be provided so applications that embed Python can continue to do so.

This mirrors the Debian/Ubuntu way of building python, where they offer a statically linked binary and an additional libpython subpackage. The libpython subpackage will be created and python3-devel will depend on it, so packages that embed Python will keep working.

By applying this change, libpython's namespace will be separated from python's, so python C extension which are linked to libpython might experience side effects or break.

C extensions that are not linked to libpython and are loaded in Python or an application embedding Python, as well as C extensions linked to libpython in an application embedding Python will not be affected

Currently there is no upstream option to build the static library, as well as the shared one and statically link the final binary to it, so we have to rely on a downstream patch to achieve it. We plan to work with upstream to incorporate the changes there as well.

Benefit to Fedora

Python's performance will increase significantly depending on the workload. Since many core components of the OS also depend on python this could lead to an increase in their performance as well.

Scope

  • Proposal owners:
    • Review and merge the pull request with the implementation.
    • Go through the python C extension packages that are linked to libpython and test if things work correctly. Will provide a copr repository for testing.
  • Other developers: Other developers are encouraged to test the new statically linked python3 to see if their package works as expected
  • Policies and guidelines: The packaging guidelines will need to be updated to explicitly mention that C extensions should not be linked to libpython, and that the python3 binary is statically linked.
  • Trademark approval: N/A (not needed for this Change)

Upgrade/compatibility impact

N/A (not a System Wide Change)

How To Test

The following script can be used to verify that the change is in effect:

import ctypes
import sys
 
EMPTY_TUPLE_SINGLETON = ()

def get_empty_tuple(lib):
    # Call PyTuple_New(0)</nowiki>
    func = lib.PyTuple_New
    func.argtypes = (ctypes.c_ssize_t,)
    func.restype = ctypes.py_object
    return func(0)
 
def test_lib(libname, lib):
    obj = get_empty_tuple(lib)
    if obj is EMPTY_TUPLE_SINGLETON:
        print("%s: SAME namespace" % libname)
    else:
        print("%s: DIFFERENT namespace" % libname)

def test():
    program = ctypes.pythonapi

    if hasattr(sys, 'abiflags'):
        abiflags = sys.abiflags
    else:
        # Python 2
        abiflags = ''
    ver = sys.version_info
    filename = ('libpython%s.%s%s.so.1.0'
                % (ver.major, ver.minor, abiflags))
    libpython = ctypes.cdll.LoadLibrary(filename)

    test_lib('program', program)
    test_lib('libpython', libpython)

test()

User Experience

Dependencies

N/A (not a System Wide Change)

Contingency Plan

  • Contingency mechanism: (What to do? Who will do it?) N/A (not a System Wide Change)
  • Contingency deadline: N/A (not a System Wide Change)
  • Blocks release? N/A (not a System Wide Change), Yes/No
  • Blocks product? product

Documentation

N/A (not a System Wide Change)

Release Notes