Revision as of 14:55, 19 March 2021

reduce dependencies on python3-setuptools

Summary

We'll do two things to reduce the number of Python packages unnecessarily Requiring python3-setuptools:

1. python3-devel will no longer Require python3-setuptools 2. packages with console_scripts/gui_scripts entry points will no longer automatically Require python3-setuptools

Owner

Name: Tomáš Hrnčiar
Name: Miro Hrončok
Email: python-maint@redhat.com

Current status

Targeted release: Fedora Linux 35
Last updated: 2021-03-19
FESCo issue: <will be assigned by the Wrangler>
Tracker bug: <will be assigned by the Wrangler>
Release notes tracker: <will be assigned by the Wrangler>

Detailed Description

Historically, for over a decade, setuptools was omnipresent in Python packaging. For that reason, the python3-devel package required python3-setuptools. This is however changing, with PEP517 and PEP518 based upstream packaging, setuptools changed to a very popular yet optional way of building packages, many upstreams switched to poetry or flit instead. Since we received multiple user reports about an unnecessary dependency, we wanted to get rid of the dependency, but we have encountered a problem, which is best summarized in the Python spec file:

# This is not "API" (packages that need setuptools should still BuildRequire it)
# However some packages apparently can build both with and without setuptools
# producing egg-info as file or directory (depending on setuptools presence).
# Directory-to-file updates are problematic in RPM, so we ensure setuptools is
# installed when -devel is required.
# See https://bugzilla.redhat.com/show_bug.cgi?id=1623914
# See https://fedoraproject.org/wiki/Packaging:Directory_Replacement
Requires: (python3-setuptools if rpm-build)

To avoid this problem, python3-devel currently requires (python3-setuptools if rpmbuild) for backwards compatibility. Other than that, python3-setuptools was always brought into the buildroot transitively with python3-rpm-generators. Since the generators no longer use setuptools but packaging we can finally remove the dependency.

Packages that don't BuildRequire python3-setuptools but use setuptools during the build will fall into one of the following three categories:

They will fail to build from source with errors like: ModuleNotFoundError: No module named 'setuptools'.
They will successfully build from source because they will fallback to the standard library distutils module - their .egg-info directory will become a text file instead, causing errors when RPM tries to update the package (and they will possibly miss some runtime dependencies because the metadata generated by distutils is not complete).
They will continue to build unchanged because one or more packages they BuildRequire pulls in python3-setuptools transitively - at any point in the future, this can change and either (1) or (2) will happen.

We tested this change in the past when we asked other maintainers to explicitly BuildRequire python-3setuptools. Some of them responded many did not. Our analysis was based on grepping sources for any mention of setuptools, such as import setuptools or from setuptools import . Nowadays, the same grep-based query showed us roughly the same number (about 335 + 72 that failed to be analyzed) of packages relying on python3-setuptools without BuildRequiring it. To see the direct impact, we have also briefly removed the Requires from python3-devel package used in testing python3.10 in COPR. It revealed only 140 packages that fail to build from the source because of the change, however many packages bring in python3-setuptools transitively. Some of the packages might build successfully even without python3-setuptools, because they fallback to distutils if dependency on setuptools is not satisfied.

To avoid such problems we will:

Mass update packages we analyzed as impacted to add the missing BuildRequires on python3-setuptools.
Do the change while doing the update from Python 3.9 to Python 3.10 so the path of the .egg-info changes from /usr/lib(64)/python3.9/site-packages/....egg-info to /usr/lib(64)/python3.10/site-packages/....egg-info, hence no longer causing troubles on upgrades if some packages actually do change it from directory to file (and changing it the other way around in the future is safe).

Important dates and plan

See Python3.10 change - we plan to mass update the spec files the week before *2021-05-25: Python 3.10.0 beta 2*, if you wish to opt-out from the mass update, please let us know prior to 18th of May 2021.

Below is the list of packages we plan to update with mass spec rebuild. If you wish to opt-out of this update, please edit this page and move your package to the list of packages to be ignored.

Packages to be updated:

GeographicLib rmattes smani
R2spec pingou
barman slaanesh tsao
btrfs-progs ignatenkobrain josef ngompa sandeen
bumpversion duriantang jdornak
cranc lenkaseg
electrum tredaell
gajim michich
gns3-gui kwizart
gns3-net-converter kwizart
kismon fab
langtable mfabian
lecm spredzy
legofy lkf williamjmorenor
mapserver devrim jujens oliver pali smani
mysql-connector-python hhorak hubbitus hvad mschorm
percol hubbitus
porcupine kushal
protonvpn-cli calexandru2018 jflory7
pyhunspell mfabian
pyjokes pbrobinson pwhalen
pyscard orion sjenning
python-Pyped uggla
python-ROPGadget mikep
python-acoustid terjeros
python-aiodns fantom
python-arviz sergiopr
python-b2sdk jonny
python-btchip jonny xenithorb
python-cerealizer spot
python-cmigemo hubbitus
python-construct moezroy terjeros
python-css-parser zbyszek
python-debrepo ktdreyer
python-dialog mjakubicek noodles raphgro sundaram zbyszek
python-django-contact-form mrunge
python-django-registration orphan
python-django-reversion mrunge
python-django-tagging jdornak mrunge piotrp
python-dtfabric fab
python-flake8-docstrings cottsay
python-flask-sphinx-themes devrim orphan
python-formats uggla
python-fypp rathann
python-google-i18n-address pwouters
python-googletrans lyessaadi
python-heapdict qulogic
python-hgdistver fab
python-html5-parser kevin
python-humblewx rickardlindberg
python-i3ipc msimacek
python-kerberos rcritten simo
python-lark-parser totol
python-logfury jonny
python-mailmerge bex
python-meld3 kevin stevetraylen tsao
python-mnemonic jonny
python-nbxmpp michich
python-oauth2 ignatenkobrain pjp spot sundaram
python-poyo chedi wakko666
python-precis_i18n michich
python-proteus sharkcz
python-publicsuffix2 rathann
python-pulsectl pfrields
python-pycha potty sharkcz
python-pylons-sphinx-themes abompard
python-pymc3 sergiopr
python-pyotp icon
python-pyroute2 jirka pavlix psavelye
python-pytest-astropy-header sergiopr
python-pytest-watch jujens
python-relatorio sharkcz
python-rmtest lberk mgoodwin nathans
python-shamir-mnemonic jonny
python-smbc twaugh zdohnal
python-spdx jbertozzi
python-spdx-lookup jbertozzi
python-tempdir rathann
python-tortilla uggla
python-trezor jonny
python-unidecode pjp sundaram
python-unidiff dcallagh
python-upt-cpan jbertozzi
python-upt-fedora jbertozzi
python-upt-pypi jbertozzi
python-upt-rubygems jbertozzi
python-xlwt leamas moezroy rathann
qr-code-generator xvitaly
rpmspectool nphilipp
sepolicy_analysis vmojzis
stomppy stevetraylen
trytond-account sharkcz
trytond-account-be sharkcz
trytond-account-de-skr03 sharkcz
trytond-account-invoice sharkcz
trytond-account-invoice-history sharkcz
trytond-account-invoice-line-standalone sharkcz
trytond-account-product sharkcz
trytond-account-statement sharkcz
trytond-account-stock-anglo-saxon sharkcz
trytond-account-stock-continental sharkcz
trytond-analytic-account sharkcz
trytond-analytic-invoice sharkcz
trytond-analytic-purchase sharkcz
trytond-analytic-sale sharkcz
trytond-company sharkcz
trytond-company-work-time sharkcz
trytond-country sharkcz
trytond-currency sharkcz
trytond-dashboard sharkcz
trytond-google-maps sharkcz
trytond-ldap-authentication sharkcz
trytond-party sharkcz
trytond-party-siret sharkcz
trytond-product sharkcz
trytond-product-cost-fifo sharkcz
trytond-product-cost-history sharkcz
trytond-product-price-list sharkcz
trytond-project sharkcz
trytond-project-plan sharkcz
trytond-project-revenue sharkcz
trytond-purchase sharkcz
trytond-purchase-invoice-line-standalone sharkcz
trytond-sale sharkcz
trytond-sale-opportunity sharkcz
trytond-sale-price-list sharkcz
trytond-stock sharkcz
trytond-stock-forecast sharkcz
trytond-stock-inventory-location sharkcz
trytond-stock-location-sequence sharkcz
trytond-stock-product-location sharkcz
trytond-stock-supply sharkcz
trytond-stock-supply-day sharkcz
trytond-timesheet sharkcz
txt2tags junghans
ubertooth avsej
unicorn fab mikep
upt jbertozzi
winpdb spot
yawn jsafrane miminar vcrhonek

Packages to be ignored:

fontforge - false positive
qemu - false positive
xrootd - false positive

Packages that failed to be analysed:

0ad ignatenkobrain pcpa pwalter
GConf2 alexl buc caillon caolanm mbarnes rhughes ssp walters
TeXmacs jnovy orion
anaconda anaconda-maint jkonecny m4rtink rvykydal sbueno vpodzime vponcova
astrometry lupinix
bamf jspaleta salimma
boost denisarnaud jwakely trodgers
colobot suve
cozy suve
cppcheck c72578 jussilehtola sgrubb
criu adrian avagin
cxxtest mgieseki
dnf dmach jmracek jrohel mblaha pkratoch
eclipse-pydev jjohnstn orphan
edk2 bonzini crobinso kraxel
etcd cypret eparis gscrivano jchaloup lsm5 strigazi walters
fonts-tweak-tool jamesni tagoh
gdl orion
gimp-resynthesizer luya
global cheeselee landgraf
google-noto-emoji-fonts mfabian pwu
gst-devtools limb
guitarix bsjones tartina verdurin
hfg-gmuend-openmoji-fonts mavit
kernel acaringi airlied ajax bskeggs eparis glisse jcline jforbes jwboyer jwilson jwrdegoede kernel-maint lgoncalv linville myoung nhorman patrickt quintela sandeen steved
kernel-tools acaringi jcline jforbes jwboyer lgoncalv patrickt pbrobinson
kicad avigne coremodule lkundrak stevenfalco tnorth
lammps ellio167 junghans
libabigail dodji sinnykumari
libguestfs mdbooth rjones
libiptcdata dcm hobbes1069 jchaloup
libkkc ueno
libproxy amigadave cicku danw npmccallum
libreoffice caolanm dtardon erack sbergmann
libsvm besser82 jjames
link-grammar devos fabiand limb
nest ankursinha
nispor cathay4t ffmancera
opal-prd hegdevasant
openbabel alexpl jussilehtola rathann
openexr hobbes1069
proxyfuzz psklenar
python-OBD rathann
python-caja monnerat raveit65
python-flask-rstpages rmarko
python-graph-tool ankursinha
python-matrix-nio ankursinha
python-pivy hobbes1069 zultron
python-pycha potty sharkcz
python-pyqtgraph swt2c
python-pytest-spec fab
python-requests-cache codeblock hobbes1069
python-uranium-lulzbot orphan
python-webpy mrunge
python-wsaccel jujens
root ellert
rust-pyo3 decathorpe
rust-python3-sys decathorpe
seqan2 sagitter
sugar-browse aperezbios callkalpa chimosky manuq pbrobinson
sugar-calculator callkalpa chimosky pbrobinson tuxbrewr
suricata jtaylor sgrubb
texlive jnovy spot than
tiled ablu suve
tsung cicku tartare
unity-gtk-module besser82
vdrift limb rmattes
voikko-fi vpv
weechat asrob gchamoul hguemar niveusluna salimma stingray
xemacs-packages-extra jjames stevetraylen
zinnia liangsuilong pwu

Feedback

There was some discussion on various mailing list threads but no other alternatives were proposed.

Benefit to Fedora

From Fedora Packaging Guidelines: It is important that your package list all necessary build dependencies using the BuildRequires: tag. As our testing showed, this is not true for many packages. Some of them can successfully build either with or without setuptools (they use try-except import and fallback to distutils from the standard library). Such packages are especially dangerous when not BuildRequiring setuptools -- they can produce different results depending on the presence of setuptools: either an .egg-info metadata directory (w/setuptools) or .egg-info text file (w/distutils). RPM has troubles when upgrading directories to files.

Scope

Some packages use distutils as a fallback in case setuptools is unavailable. The problem is that setuptools generates egg.info directory and distutils creates egg.info file. This would lead to failures caused by the known issue of RPM that cannot change directory to file. To avoid this we plan to do a mass spec update before we upgrade Python to version 3.10. This way we ensure that path will be correct after the mass rebuild.

Proposal owners:
1. Test affected packages in Copr.
2. Do the mass spec update to introduce missing BuildRequire to them.

Other developers: There should be no additional work for other developers. There is a possibility that some packages will slip through our grep based query. This might cause some of them to FTBFS.

Release engineering: #Releng issue number

Policies and guidelines: N/A (not needed for this Change)

Trademark approval: N/A (not needed for this Change)

Alignment with Objectives: N/A

Upgrade/compatibility impact

All affected packages needs rebuild after the mass spec update but since we plan to do it with the upgrade to Python 3.10 we will rebuild it altogether.

How To Test

Interested testers can push explicit BuildRequire on setuptools to the distgit and see in our Python 3.10 COPR whether the package starts to build again. Webhook should trigger rebuild automatically on new commit in rawhide branch.

User Experience

Regular distro users shouldn't notice any change in system behaviour.

Dependencies

300+ packages will need to BuildRequire setuptools explicitly. This will be done by the mass spec update.

Contingency Plan

Contingency mechanism: Commit removing python3-devel Requires on (python3-setuptools if rpmbuild) will be reverted.
Contingency deadline: mass rebuild
Blocks release? No

Documentation

This page is a documentation, this is not a user facing change.

Release Notes

@@ Line 2: / Line 2: @@
 <!-- The actual name of your proposed change page should look something like: Changes/Your_Change_Proposal_Name.  This keeps all change proposals in the same namespace -->
-= python3-devel won't pull in setuptools by default in rpmbuild =
+= reduce dependencies on python3-setuptools =
 <!-- The name of your change proposal -->
 == Summary ==
-python3-devel currently requires python3-setuptools if rpmbuild is used. Due to a change in RPM generators, this is not needed anymore hence we are removing it.
+We'll do two things to reduce the number of Python packages unnecessarily Requiring python3-setuptools:
+. python3-devel will no longer Require python3-setuptools
+. packages with console_scripts/gui_scripts entry points will no longer automatically Require python3-setuptools
 <!-- A sentence or two summarizing what this change is and what it will do. This information is used for the overall changeset summary page for each release. Note that motivation for the change should be in the Benefit to Fedora section below, and this part should answer the question "What?" rather than "Why?". -->

Search

Changes/Reduce dependencies on python3-setuptools: Difference between revisions