Churchyard (talk | contribs) (Contingency Plan) |
Churchyard (talk | contribs) (Documentation) |
||
Line 275: | Line 275: | ||
<!-- REQUIRED FOR SYSTEM WIDE CHANGES --> | <!-- REQUIRED FOR SYSTEM WIDE CHANGES --> | ||
This page is the documentation. | |||
== Release Notes == | == Release Notes == |
Revision as of 11:28, 4 January 2019
Python distutils to use build flags for extension builders (XXX better name?)
Summary
The build flags (CFLAGS
, CXXFLAGS
and LDFLAGS
) saved in the Python's distutils module for building extension modules are switched from %{build_cflags}
, %{build_cxxflags}
and %{build_ldflags}
to %{extension_cflags}
, %{extension_cxxflags}
and %{extension_ldflags}
.
This currently means no GCC plugins (such as annobin) are activated and no GCC spec files (-specs=
arguments) are used by default when building Python extension modules (e.g. with python3 setup.py build
). That also means the
package can loose a runtime dependency on python3-devel
.
redhat-rpm-config
The change mostly affects building extension modules by users, outside of RPM environment. The Python standard library and Fedora's Python 3 RPM packages are still built with the "traditional" set of flags (%{build_cflags}
and friends), unless the package uses nonstandard methods to build the extensions.
Only Python 3.7 (
) and 3.6 (python3
) will be changed.
python36
Owner
- Name: Miro Hrončok, Charalampos Stratakis
- Email: python-maint@redhat.com
- Release notes owner:
Current status
- Targeted release: Fedora 30
- Last updated: 2019-01-04
- Tracker bug: <will be assigned by the Wrangler>
Detailed Description
The Problem
When Python is built, it saves the flags (CFLAGS
, CXXFLAGS
and LDFLAGS
) for further use when building extension modules into a designated Makefile
. The distutils module (a component responsible for building Python packages and extension modules) then reads the file and applies the flags. You can see the file at /usr/lib64/python3.7/config-3.7m-x86_64-linux-gnu/Makefile
in
. This is mostly done to make user-built extension modules binary compatible with the Python interpreter they are being built for.
python3-libs
Traditionally (=before this change), the
package was created in a way that it simply saved the same set of flags used for building itself.
This proved problematic as the flags used to build Fedora packages grew specific things (not actually needed for binary compatibility of the extension modules) and several workarounds needed to be made, most specifically the python3
package got a runtime dependency on python3-devel
:
redhat-rpm-config
- https://bugzilla.redhat.com/show_bug.cgi?id=1217376
- https://bugzilla.redhat.com/show_bug.cgi?id=1496757
- https://bugzilla.redhat.com/show_bug.cgi?id=1218294
The problematic flags are GCC plugins (such as annobin) and GCC spec files (-specs=
arguments).
Example: Any Python developer using Fedora automatically builds Python extension modules with annobin and hardening flags by default even if they don't need that. They might build the extension on Fedora, test it and later ship it and build it on a CI that is not based on Fedora and get a different results.
The solution
The solution is not to save the problematic flags, but only the flags needed. Until recently, this would be hackish, but a designated set of flags was created in Fedora 30, supposed to be used by extension builders (such as the Python's distutil module): %{extension_cflags}
, %{extension_cxxflags}
and %{extension_ldflags}
.
The documentation for the flags currently reads:
The current set of differences are:
- No GCC plugins (such as annobin) are activated.
- No GCC spec files (
-specs=
arguments) are used.Additional flags may be removed in the future if they prove to be incompatible with alternative toolchains.
Python already had (an incomplete) ability to specify a different set of flags for itself and for the extensions. In Python 3.7.2 and 3.6.8 the ability was completed by providing a way to do this for the LDFAGS
(this was actually driven by Fedora's needs). Currently we are able to set a different set of flags for building Python and for building user extension modules. The code to do this can be inspected at the Fedora pull request implementing this change.
Impact on users building extension modules
When an user compiles a Python extension module and does not specify any flags explicitly, this is what they used to had:
$ python3 setup.py build running build running build_ext building 'demo' extension creating build creating build/temp.linux-x86_64-3.7 gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python3.7m -c demo.c -o build/temp.linux-x86_64-3.7/demo.o creating build/lib.linux-x86_64-3.7 gcc -pthread -shared -Wl,-z,relro -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -g build/temp.linux-x86_64-3.7/demo.o -L/usr/lib64 -lpython3.7m -o build/lib.linux-x86_64-3.7/demo.cpython-37m-x86_64-linux-gnu.so
After the change, they will get:
$ python3 setup.py build running build running build_ext building 'demo' extension creating build creating build/temp.linux-x86_64-3.7 gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python3.7m -c demo.c -o build/temp.linux-x86_64-3.7/demo.o creating build/lib.linux-x86_64-3.7 gcc -pthread -shared -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -g build/temp.linux-x86_64-3.7/demo.o -L/usr/lib64 -lpython3.7m -o build/lib.linux-x86_64-3.7/demo.cpython-37m-x86_64-linux-gnu.so
Users are able to provide their own flags by setting the CFLAGS
/CXXFLAGS
and LDFLAGS
environment variables. They can for example opt-in for annobin if they wish to do so.
Impact on RPM packages building extension modules
When the spec file of the package sets the CFLAGS
/CXXFLAGS
and LDFLAGS
variables to the expected Fedora values (e.g. CFLAGS=${RPM_OPT_FLAGS} LDFLAGS=${RPM_LD_FLAGS}
or %{build_cflags}
, %{build_cxxflags}
, %{build_ldflags}
, %{optflags}
etc.) everything works as expected. The currently documented way of building Python packages (%py3_build
and %py3_install
) does this for you, we recommend using it.
There might be some differences in the log, because the flags concatenate. Here is the example from the
build.log.
python-psycopg2
Before this change:
gcc -pthread -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fPIC -DPSYCOPG_DEFAULT_PYDATETIME=1 -DPSYCOPG_VERSION=2.7.5 (dt dec pq3 ext lo64) -DPG_VERSION_NUM=110000 -DHAVE_LO64=1 -I/usr/include/python3.7m -I. -I/usr/include -I/usr/include/pgsql/server -c psycopg/psycopgmodule.c -o build/temp.linux-x86_64-3.7/psycopg/psycopgmodule.o -Wdeclaration-after-statement
After this change:
gcc -pthread -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fPIC -DPSYCOPG_DEFAULT_PYDATETIME=1 -DPSYCOPG_VERSION=2.7.5 (dt dec pq3 ext lo64) -DPG_VERSION_NUM=110000 -DHAVE_LO64=1 -I/usr/include/python3.7m -I. -I/usr/include -I/usr/include/pgsql/server -c psycopg/psycopgmodule.c -o build/temp.linux-x86_64-3.7/psycopg/psycopgmodule.o -Wdeclaration-after-statement
If the package does python3 setup.py build
to build the extension modules manually without setting the flags, the flags are not set properly. This is probably very rare. Even before the %py3_build
and %py3_install
macros, the preferred way was to call:
CFLAGS="$RPM_OPT_FLAGS" %{__python} setup.py build # from EL4
However, LDFLAGS
are not set in that example.
There are 485 packages that have the %{python3_sitearch}
macro in them and hence are most likely to build Python extension modules.
Out of those, 173 have setup.py
directly in the spec. 65 have setup.py build
. Change owners will go trough the list and provide fixes. Packages are encouraged to switch to the %py3_build
macro themselves.
Worst case: Some package won't have annobin.
Benefit to Fedora
Python developers will get more upstream-like experience when building Python extension modules.
Python developers won't need
(and dozens of other languages specific packages that go with that) installed just to do so.
New decisions made about the distro packages won't necessarily affect Python developers building their extension modules in a negative way.
redhat-rpm-macros
Scope
- Proposal owners:
- Review, merge and build the pull request with the implementation.
- Go through the packages manually invoking
setup.py
to build extension modules and provide fixes. Update to%py3_build
where possible, set the flags manually otherwise.
- Other developers: Are encouraged to switch to
%py3_build
(but they don't need to do anything, this is not a System Wide Change).
- Release engineering: XXX TODO #Releng issue number (a check of an impact with Release Engineering is needed).
- Policies and guidelines: already in place.
Upgrade/compatibility impact
Not anticipated. Extension modules (built for the same Python version) are compatible with the interpreter with or without the removed flags back and forth.
How To Test
For users (Python developers)
- build your favorite Python extension module in venv or outside venv with
setup.py build
orsetup.py build_ext
- observe the used flags and check annobin and
--specs
are not there, report bugs for
otherwise (and block our tracking bug)python3
- check if the extension works as expected
For packagers (Fedora contributors)
- build your favorite RPM package with Python extension module
- observe the used flags and check annobin and
--specs
are there, report bugs for that package otherwise (and block our tracking bug) - check if the package works as expected
User Experience
See Benefit to Fedora above.
Dependencies
Changes in
are already done.
redhat-rpm-config
We will check dependent Fedora packages.
Contingency Plan
- Contingency mechanism: change owners can revert the change at any point
- Contingency deadline: final freeze
- Blocks release? No
- Blocks product? None
Documentation
This page is the documentation.