Revision as of 09:31, 15 March 2021

python3-devel won't pull in setuptools by default in rpmbuild

Summary

python3-devel currently requires python-setuptools if rpmbuild is used. Due to a change in RPM generators, this is not needed anymore hence we are removing it.

Owner

Name: Tomáš Hrnčiar
Name: Miro Hrončok
Email: python-maint@redhat.com

Current status

Targeted release: Fedora Linux 35
Last updated: 2021-03-15
FESCo issue: <will be assigned by the Wrangler>
Tracker bug: <will be assigned by the Wrangler>
Release notes tracker: <will be assigned by the Wrangler>

Detailed Description

python3-devel currently requires python3-setuptools if rpmbuild is used. This was needed for RPM generators but since it adapted packaging, this is not needed anymore hence we are removing it. We tested this change in the past when we asked other maintainers to explicitly BuildRequire python-3setuptools. Some of them responded, some not. Our analysis was based on greping sources for any mention of setuptools, such as import setuptools or from setuptools import. Nowadays, the same grep based query showed us roughly the same number (about 335 + 72 that failed to be analysed) of packages relying on python3-setuptools. To see direct impact we removed this Require from python-devel package used in testing python3.10 COPR. It showed us 140 packages affected by this change. Even though it is fewer than expected 300+ packages, to avoid any breakages we plan to do a mass spec update on all of them to explicitly BuildRequire python3-setuptools. Some of them might build because they fallback to distutils if dependency on python3-setuptools is not satisfied.

Important dates and plan

2021-05-25: Python 3.10.0 beta 2
- The ideal point when we could start with mass spec update
2021-05-31: Expected side tag-merge (optimistic)
2021-06-17: Python 3.10.0 beta 3
2021-06-25: Expected side tag-merge (realistic)
2021-07-10: Python 3.10.0 beta 4
2021-07-17: Expected side tag-merge (pessimistic)
- Latest point to opt out from mass spec update
2021-07-21: Fedora 35 Mass Rebuild

Below is the list of packages we plan to update with mass spec rebuild. If you wish to opt out of this update, please edit this page and move your package to the list of packages to be ignored.

Packages to be updated:

GeographicLib rmattes smani
R2spec pingou
barman slaanesh tsao
btrfs-progs ignatenkobrain josef ngompa sandeen
bumpversion duriantang jdornak
cranc lenkaseg
electrum tredaell
gajim michich
gns3-gui kwizart
gns3-net-converter kwizart
kismon fab
langtable mfabian
lecm spredzy
legofy lkf williamjmorenor
mapserver devrim jujens oliver pali smani
mysql-connector-python hhorak hubbitus hvad mschorm
percol hubbitus
porcupine kushal
protonvpn-cli calexandru2018 jflory7
pyhunspell mfabian
pyjokes pbrobinson pwhalen
pyscard orion sjenning
python-Pyped uggla
python-ROPGadget mikep
python-acoustid terjeros
python-aiodns fantom
python-arviz sergiopr
python-b2sdk jonny
python-btchip jonny xenithorb
python-cerealizer spot
python-cmigemo hubbitus
python-construct moezroy terjeros
python-css-parser zbyszek
python-debrepo ktdreyer
python-dialog mjakubicek noodles raphgro sundaram zbyszek
python-django-contact-form mrunge
python-django-registration orphan
python-django-reversion mrunge
python-django-tagging jdornak mrunge piotrp
python-dtfabric fab
python-flake8-docstrings cottsay
python-flask-sphinx-themes devrim orphan
python-formats uggla
python-fypp rathann
python-google-i18n-address pwouters
python-googletrans lyessaadi
python-heapdict qulogic
python-hgdistver fab
python-html5-parser kevin
python-humblewx rickardlindberg
python-i3ipc msimacek
python-kerberos rcritten simo
python-lark-parser totol
python-logfury jonny
python-mailmerge bex
python-meld3 kevin stevetraylen tsao
python-mnemonic jonny
python-nbxmpp michich
python-oauth2 ignatenkobrain pjp spot sundaram
python-poyo chedi wakko666
python-precis_i18n michich
python-proteus sharkcz
python-publicsuffix2 rathann
python-pulsectl pfrields
python-pycha potty sharkcz
python-pylons-sphinx-themes abompard
python-pymc3 sergiopr
python-pyotp icon
python-pyroute2 jirka pavlix psavelye
python-pytest-astropy-header sergiopr
python-pytest-watch jujens
python-relatorio sharkcz
python-rmtest lberk mgoodwin nathans
python-shamir-mnemonic jonny
python-smbc twaugh zdohnal
python-spdx jbertozzi
python-spdx-lookup jbertozzi
python-tempdir rathann
python-tortilla uggla
python-trezor jonny
python-unidecode pjp sundaram
python-unidiff dcallagh
python-upt-cpan jbertozzi
python-upt-fedora jbertozzi
python-upt-pypi jbertozzi
python-upt-rubygems jbertozzi
python-xlwt leamas moezroy rathann
qr-code-generator xvitaly
rpmspectool nphilipp
sepolicy_analysis vmojzis
stomppy stevetraylen
trytond-account sharkcz
trytond-account-be sharkcz
trytond-account-de-skr03 sharkcz
trytond-account-invoice sharkcz
trytond-account-invoice-history sharkcz
trytond-account-invoice-line-standalone sharkcz
trytond-account-product sharkcz
trytond-account-statement sharkcz
trytond-account-stock-anglo-saxon sharkcz
trytond-account-stock-continental sharkcz
trytond-analytic-account sharkcz
trytond-analytic-invoice sharkcz
trytond-analytic-purchase sharkcz
trytond-analytic-sale sharkcz
trytond-company sharkcz
trytond-company-work-time sharkcz
trytond-country sharkcz
trytond-currency sharkcz
trytond-dashboard sharkcz
trytond-google-maps sharkcz
trytond-ldap-authentication sharkcz
trytond-party sharkcz
trytond-party-siret sharkcz
trytond-product sharkcz
trytond-product-cost-fifo sharkcz
trytond-product-cost-history sharkcz
trytond-product-price-list sharkcz
trytond-project sharkcz
trytond-project-plan sharkcz
trytond-project-revenue sharkcz
trytond-purchase sharkcz
trytond-purchase-invoice-line-standalone sharkcz
trytond-sale sharkcz
trytond-sale-opportunity sharkcz
trytond-sale-price-list sharkcz
trytond-stock sharkcz
trytond-stock-forecast sharkcz
trytond-stock-inventory-location sharkcz
trytond-stock-location-sequence sharkcz
trytond-stock-product-location sharkcz
trytond-stock-supply sharkcz
trytond-stock-supply-day sharkcz
trytond-timesheet sharkcz
txt2tags junghans
ubertooth avsej
unicorn fab mikep
upt jbertozzi
winpdb spot
yawn jsafrane miminar vcrhonek

Packages to be ignored:

fontforge - false positive
xrootd - false positive

Packages that failed to be analysed:

0ad ignatenkobrain pcpa pwalter
GConf2 alexl buc caillon caolanm mbarnes rhughes ssp walters
TeXmacs jnovy orion
anaconda anaconda-maint jkonecny m4rtink rvykydal sbueno vpodzime vponcova
astrometry lupinix
bamf jspaleta salimma
boost denisarnaud jwakely trodgers
colobot suve
cozy suve
cppcheck c72578 jussilehtola sgrubb
criu adrian avagin
cxxtest mgieseki
dnf dmach jmracek jrohel mblaha pkratoch
eclipse-pydev jjohnstn orphan
edk2 bonzini crobinso kraxel
etcd cypret eparis gscrivano jchaloup lsm5 strigazi walters
fonts-tweak-tool jamesni tagoh
gdl orion
gimp-resynthesizer luya
global cheeselee landgraf
google-noto-emoji-fonts mfabian pwu
gst-devtools limb
guitarix bsjones tartina verdurin
hfg-gmuend-openmoji-fonts mavit
kernel acaringi airlied ajax bskeggs eparis glisse jcline jforbes jwboyer jwilson jwrdegoede kernel-maint lgoncalv linville myoung nhorman patrickt quintela sandeen steved
kernel-tools acaringi jcline jforbes jwboyer lgoncalv patrickt pbrobinson
kicad avigne coremodule lkundrak stevenfalco tnorth
lammps ellio167 junghans
libabigail dodji sinnykumari
libguestfs mdbooth rjones
libiptcdata dcm hobbes1069 jchaloup
libkkc ueno
libproxy amigadave cicku danw npmccallum
libreoffice caolanm dtardon erack sbergmann
libsvm besser82 jjames
link-grammar devos fabiand limb
nest ankursinha
nispor cathay4t ffmancera
opal-prd hegdevasant
openbabel alexpl jussilehtola rathann
openexr hobbes1069
proxyfuzz psklenar
python-OBD rathann
python-caja monnerat raveit65
python-flask-rstpages rmarko
python-graph-tool ankursinha
python-matrix-nio ankursinha
python-pivy hobbes1069 zultron
python-pycha potty sharkcz
python-pyqtgraph swt2c
python-pytest-spec fab
python-requests-cache codeblock hobbes1069
python-uranium-lulzbot orphan
python-webpy mrunge
python-wsaccel jujens
qemu berrange bonzini crobinso dwmw2 ehabkost jforbes lkundrak quintela rjones
root ellert
rust-pyo3 decathorpe
rust-python3-sys decathorpe
seqan2 sagitter
sugar-browse aperezbios callkalpa chimosky manuq pbrobinson
sugar-calculator callkalpa chimosky pbrobinson tuxbrewr
suricata jtaylor sgrubb
texlive jnovy spot than
tiled ablu suve
tsung cicku tartare
unity-gtk-module besser82
vdrift limb rmattes
voikko-fi vpv
weechat asrob gchamoul hguemar niveusluna salimma stingray
xemacs-packages-extra jjames stevetraylen
zinnia liangsuilong pwu

Feedback

There was some discussion on various mailing list threads but no other alternatives were proposed.

Benefit to Fedora

From Fedora Packaging Guidelines: It is important that your package list all necessary build dependencies using the BuildRequires: tag. As our testing showed, this is not true for many packages. Some of them can successfully build either with or without setuptools (they use try-except import and fallback to distutils from the standard library). Such packages are especially dangerous when not BuildRequiring setuptools -- they can produce different results depending on the presence of setuptools: either an .egg-info metadata directory (w/setuptools) or .egg-info text file (w/distutils). RPM has troubles when upgrading directories to files.

Scope

Some packages use distutils as a fallback in case setuptools is unavailable. The problem is that setuptools generates egg.info directory and distutils creates egg.info file. This would lead to failures caused by the known issue of RPM that cannot change directory to file. To avoid this we plan to do a mass spec update before we upgrade Python to version 3.10. This way we ensure that path will be correct after the mass rebuild.

Proposal owners:
1. Test affected packages in Copr.
2. Do the mass spec update to introduce missing BuildRequire to them.

Other developers: There should be no additional work for other developers. There is a possibility that some packages will slip through our grep based query. This might cause some of them to FTBFS.

Release engineering: #Releng issue number

Policies and guidelines: N/A (not needed for this Change)

Trademark approval: N/A (not needed for this Change)

Alignment with Objectives: N/A

Upgrade/compatibility impact

All affected packages needs rebuild after the mass spec update.

How To Test

Interested testers can push explicit BuildRequire on setuptools to the distgit and see in our Python 3.10 COPR whether the package starts to build again. Webhook should trigger rebuild automatically on new commit in rawhide branch.

User Experience

Regular distro users shouldn't notice any change in system behaviour.

Dependencies

140 packages will need to BuildRequire setuptools explicitly. This will be done by the mass spec update.

Contingency Plan

Contingency mechanism: Commit removing Requires on python-setuptools will be reverted.
Contingency deadline: mass rebuild
Blocks release? No

Documentation

This page is a documentation, this is not a user facing change.

Release Notes

@@ Line 47: / Line 47: @@
 <!-- Expand on the summary, if appropriate.  A couple sentences suffices to explain the goal, but the more details you can provide the better. -->
 python3-devel currently requires python3-setuptools if rpmbuild is used. This was needed for RPM generators but since it adapted packaging, this is not needed anymore hence we are removing it. We tested this change in the past when we [https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/GCPGM34ZGEOVUHSBGZTRYR5XKHTIJ3T7/ asked] other maintainers to explicitly BuildRequire python-3setuptools. Some of them responded, some not. Our analysis was based on greping sources for any mention of setuptools, such as ''import setuptools'' or ''from setuptools import''. Nowadays, the same grep based query showed us roughly the same number (about 335 + 72 that failed to be analysed) of packages relying on python3-setuptools. To see direct impact we removed this Require from python-devel package used in testing [https://copr.fedorainfracloud.org/coprs/g/python/python3.10/packages/ python3.10] COPR. It showed us 140 packages affected by this change. Even though it is fewer than expected 300+ packages, to avoid any breakages we plan to do a mass spec update on all of them to explicitly BuildRequire python3-setuptools. Some of them might build because they fallback to distutils if dependency on python3-setuptools is not satisfied.
+=== Important dates and plan ===
+* 2021-05-25: Python 3.10.0 beta 2
+** The ideal point when we could start with mass spec update
+* 2021-05-31: Expected side tag-merge (optimistic)
+* 2021-06-17: Python 3.10.0 beta 3
+* 2021-06-25: Expected side tag-merge (realistic)
+* 2021-07-10: Python 3.10.0 beta 4
+* 2021-07-17: Expected side tag-merge (pessimistic)
+** Latest point to opt out from mass spec update
+* 2021-07-21: Fedora 35 Mass Rebuild
 Below is the list of packages we plan to update with mass spec rebuild. If you wish to opt out of this update, please edit this page and move your package to the list of packages to be ignored.

Search

Changes/Reduce dependencies on python3-setuptools: Difference between revisions