(Some typo fixes) |
Churchyard (talk | contribs) m (→Request a side tag: Ask for autosigning) |
||
(47 intermediate revisions by 4 users not shown) | |||
Line 1: | Line 1: | ||
= Upgrading Python from 3.N-1 to 3.N = | = Upgrading Python from 3.N-1 to 3.N = | ||
Here are the notes from [[Changes/Python3.7|upgrading Python | Here are the notes from [[Changes/Python3.7|upgrading Python from 3.6 to 3.7]]. They are created to save us time next time when upgrading to 3.8. It is not a comprehensive guide, but if it once gets to it, it'd be great. | ||
For Python 3.9, see section [[#Ideas for Python 3.9|Ideas for Python 3.9]] at the bottom. | |||
== Create a Change Proposal == | == Create a Change Proposal == | ||
Before anything can happen, we need to create a Change Proposal. Follow the [[Changes/Policy|normal procedure]], get inspired by previous changes, but make sure the process | Before anything can happen, we need to create a Change Proposal. Follow the [[Changes/Policy|normal procedure]], get inspired by previous changes, but make sure the process hasn't changed (copypasta is good, but check if the template hasn't been updated). | ||
== Create python3N package == | == Create python3N package == | ||
Line 14: | Line 16: | ||
== Make a copr repo, bootstrap "important stuff" in it == | == Make a copr repo, bootstrap "important stuff" in it == | ||
Bootstrapping Python packages is nontrivial. Chances are high that | Bootstrapping Python packages is nontrivial. Chances are high that from the upgrade to 3.N-1, the bootstrap sequence has changed, as more packages switched from python2-sphinx to python3-sphinx or from python3-pytoml to python3-toml etc. Start as early as possible. | ||
For 3.7, I've used [https://github.com/sclorg/rpm-list-builder rpm-list-builder]. The [https://github.com/hroncok/rpm-list-builder/blob/python37/python37.yaml yaml recipe] is archived on my GitHub. It has some comments. Note that rpm-list-builder has limitations: | For 3.7, I've used [https://github.com/sclorg/rpm-list-builder rpm-list-builder]. The [https://github.com/hroncok/rpm-list-builder/blob/python37/python37.yaml yaml recipe] is archived on my GitHub. It has some comments. Note that rpm-list-builder has limitations: | ||
* it doesn't bump releases, so you need to do that manually if you build one package once and the new build has new functionality (aka not just for tests) | * it doesn't bump releases, so you need to do that manually if you build one package once and the new build has new functionality (aka not just for tests) | ||
* the "continue | * the "continue from here" future isn't very useful for us and was designed for different purpose, so what I did is that most of the yaml was commented out and I was uncommenting sections that I needed keeping finished stuff above, TODO stuff at the bottom | ||
* everything is sequential. go do something different | * everything is sequential. go do something different from time to time | ||
What wasn't done for 3.7 but might have been a good idea is to restart the procedure once in a while. Packages early in the bootstrap sequence were built once with alphas and never tested with betas and rc. I suggest restarting the procedure later in the process. | |||
Preserving the yaml recipe is good, you can use it for Fedora later. | |||
* The Python 3.8 recipe: https://github.com/hroncok/rpm-list-builder/blob/python38/python38.yaml | |||
* The Python 3.9 recipe: https://github.com/hroncok/rpm-list-builder/blob/python39/python39.yaml | |||
In the Copr settings, add this external repo: | |||
http://kojipkgs.fedoraproject.org/repos/rawhide/latest/$basearch/ | |||
And '''disable mock bootstrap'''. | |||
This is the command I've used: | This is the command I've used: | ||
Line 33: | Line 42: | ||
I had local fedpkg clones, so I could create quick hotfixes in them. Also a must for {{package|python3}} package - you have it updated to 3.N, but rawhide still has 3.N-1. | I had local fedpkg clones, so I could create quick hotfixes in them. Also a must for {{package|python3}} package - you have it updated to 3.N, but rawhide still has 3.N-1. | ||
{{admon/important|Some packages will need a release bump|It is unfortunately required that you manually bump some packages, especially those with bootstrap bconds. For example when Fedora has python3-setuptools-41.6.0-1.fc32 and you rebuild it with bootstrap enabled without bumping, it is python3-setuptools-41.6.0-1.fc32~bootstrap and that sorts lower. This feature of the dist tag requires you to run <code>rpmdev-bumpspec *.spec</code> on such packages in the source directory prior to proceeding.}} | |||
=== Rebuild everything else in copr === | |||
This shall give you the packages that need rebuild for Python 3.9: | |||
$ (repoquery --refresh --repo=koji --source --whatrequires 'libpython3.8.so.1.0()(64bit)' --whatrequires 'python(abi) = 3.8' --whatrequires '/usr/bin/python3.8' --whatrequires 'python3.8dist(*)') | pkgname | sort | uniq | egrep -v '^python3$' > python38.pkgs && (repoquery --refresh --repo=python39 --source --whatrequires 'libpython3.9.so.1.0()(64bit)' --whatrequires 'python(abi) = 3.9' --whatrequires '/usr/bin/python3.9' --whatrequires 'python3.9dist(*)') | pkgname | sort | uniq | egrep -v '^python3$' > python39.pkgs && comm -23 python38.pkgs python39.pkgs > todo.pkgs && wc -l *.pkgs | |||
<code>pkgname</code> command is from {{package|fedora-packager}} 0.6.0.5+. | |||
The koji repo is more up to date than rawhide. Use the following repo file if you don't already have it (save as <code>/etc/yum.repos.d/koji.repo</code>): | |||
[koji] | |||
name=koji | |||
baseurl=http://kojipkgs.fedoraproject.org/repos/rawhide/latest/x86_64/ | |||
enabled=0 | |||
[koji-source] | |||
name=koji-source | |||
baseurl=http://kojipkgs.fedoraproject.org/repos/rawhide/latest/src/ | |||
enabled=0 | |||
Inspect the resulting files and manually remove <code>python3.X</code> from all of them. Add all packages to copr: | |||
$ for pkg in $(cat python38.pkgs); do echo $pkg; copr add-package-distgit --name $pkg --webhook-rebuild on @python/python3.9 || copr edit-package-distgit --name $pkg --webhook-rebuild on @python/python3.9; done | |||
And build the ones that are still needed: | |||
$ for pkg in $(cat todo.pkgs); do echo $pkg; copr build-package --nowait --name $pkg @python/python3.9; done | |||
Or, use moreutils-parallel (note that there are two packages in Fedora that provide the parallel command; you want moreutils-parallel): | |||
$ parallel -j 12 copr build-package --nowait @python/python3.9 --name -- $(cat todo.pkgs) | |||
Use [https://github.com/hroncok/mini-mass-rebuild/blob/master/monitor_check.py monitor_check.py] to see and report the failures. | |||
== Once ready, move into Fedora proper == | == Once ready, move into Fedora proper == | ||
Line 51: | Line 96: | ||
koji wait-repo <name> --build=<nvr> | koji wait-repo <name> --build=<nvr> | ||
If [https://pagure.io/releng/issue/9308 Koji uses the mock's bootstrap option], you need to request it to be disabled for the side tag in order to prevent broken dependencies of dnf in the bootstrap chroot. A releng member can do that via: | |||
koji edit-tag <name> -x 'mock.use_bootstrap=0' | |||
Once {{package|dnf}} and {{package|dnf-plugins-core}} (and/or possibly other Python packages needed in the bootstrap chroot in the future) are rebuilt and installable in the side tag, the mock's bootstrap option can be enabled again (possibly not worth the trouble unless not using it prevents some other builds). | |||
Ask for autosigning to be enabled. That will significantly speed up the merging procedure. | |||
=== Reproduce the bootstrap sequence === | === Reproduce the bootstrap sequence === | ||
Just get the recipe and rebuild in the side tag. [https://github.com/sclorg/rpm-list-builder/issues/108 rpm-list-builder could not do it with 3.7] so I've used a [https://github.com/hroncok/rpm-list-builder/blob/python37/dirty-rebuild-script.py dirty script]. Be prepared for failures, packages got updated in the meantime and new deps were added. Tough life. | |||
=== Rebuild everything else === | === Rebuild everything else === | ||
Now this gets a bit magical but what I did is that I've used [https://github.com/hroncok/mini-mass-rebuild mini-mass-rebuild] to rebuild everything. Remember to bump the release and use the side tag (the fedpkg-build.sh script needs to adjusted). | Now, this gets a bit magical but what I did is that I've used [https://github.com/hroncok/mini-mass-rebuild mini-mass-rebuild] to rebuild everything. Remember to bump the release and use the side tag (the fedpkg-build.sh script needs to be adjusted). | ||
What is everything? First, i get the packages that buildrequire {{package|python3-devel}}: | What is everything? First, i get the packages that buildrequire {{package|python3-devel}}: | ||
Line 70: | Line 123: | ||
Also a good idea what actually needs a rebuild is: | Also a good idea what actually needs a rebuild is: | ||
dnf repoquery --whatrequires 'libpython3.6m.so.1.0()(64bit)' | dnf repoquery --whatrequires 'libpython3.6m.so.1.0()(64bit)' --whatrequires 'python(abi) == 3.6' --whatrequires '/usr/bin/python3.6' --whatrequires 'python3.6dist(*)' --source | ||
Use 3.N-1 instead of 3.6. | Use 3.N-1 instead of 3.6. | ||
Remember that you already built stuff that doesn't need an extra rebuild. To query the side tag repo, use: | |||
baseurl=https://kojipkgs.fedoraproject.org/repos/<target_name>/latest/$basearch/ | |||
To remove a list of packages from the list of packages, I use comm (requires sorted files): | |||
comm -23 all_rawhide already_built > remains | |||
There are also [https://github.com/hroncok/mini-mass-rebuild/blob/master/build-status.sh build-status.sh] and [https://github.com/hroncok/mini-mass-rebuild/blob/master/build-failure-reason.sh build-failure-reason.sh] in mini-mass-rebuild. | There are also [https://github.com/hroncok/mini-mass-rebuild/blob/master/build-status.sh build-status.sh] and [https://github.com/hroncok/mini-mass-rebuild/blob/master/build-failure-reason.sh build-failure-reason.sh] in mini-mass-rebuild. | ||
Line 93: | Line 140: | ||
for i in free open closed failed canceled; do echo -e $i\\t$(grep " $i" ../status | wc -l); done | for i in free open closed failed canceled; do echo -e $i\\t$(grep " $i" ../status | wc -l); done | ||
cat ../status | grep " closed" | cut -d" " -f1 | tee ../closed | cat ../status | grep " closed" | cut -d" " -f1 | tee ../closed | ||
comm -23 ../remains ../closed > ../remains | |||
And: | And: | ||
Line 99: | Line 146: | ||
parallel -j 32 bash ../build-failure-reason.sh -- $(cat ../remains) | tee ../reason | parallel -j 32 bash ../build-failure-reason.sh -- $(cat ../remains) | tee ../reason | ||
cat ../reason | grep ' build' # FTBFS | cat ../reason | grep ' build' # FTBFS | ||
cat ../reason | grep ' root' # missing dependency | |||
It's a good idea to mass rebuild everything that remains until the number of closed (built) packages stops at 0. Don't | It's a good idea to mass rebuild everything that remains until the number of closed (built) packages stops at 0. Don't accidentally bump the release over and over again. Later, start investigating the build failure reasons. Untangle any circular deps if found. Add them to yaml recipe for next time. File bugs (block the tracking bug, create an alias (PYTHON3N)). [https://pagure.io/fedora-misc-package-utilities/blob/master/f/find-package-maintainers E-mail maintainers.] Cry. | ||
=== Request a side tag merge === | === Request a side tag merge === | ||
Line 110: | Line 157: | ||
When the side tag is being merged, be available. After the merge, stuff thet has higher nevr in rawhide will "win" over the side tag. | When the side tag is being merged, be available. After the merge, stuff thet has higher nevr in rawhide will "win" over the side tag. | ||
Double check if | Double check if everything that was bootstrapped (not everything from the recipe, but where actual bootstrapping was involved) has higher epoch:version-release in the side tag - rebuild it in the side tag again if it was bumped in regular rawhide, or you will need to bootstrap it again after the merge. | ||
Once merged, releng will give you list of packages that had higher nevr in rawhide. Bump-rebuild them in rawhide once you can koji-wait for python3-3.N.0... build. | Once merged, releng will give you list of packages that had higher nevr in rawhide. Bump-rebuild them in rawhide once you can koji-wait for python3-3.N.0... build. | ||
Line 118: | Line 165: | ||
If there are still FTBFSes (they are), [https://pagure.io/fedora-misc-package-utilities/blob/master/f/find-package-maintainers e-mail the maintainers] again. | If there are still FTBFSes (they are), [https://pagure.io/fedora-misc-package-utilities/blob/master/f/find-package-maintainers e-mail the maintainers] again. | ||
Monitor the state. | Monitor the state. Get the initial list of broken deps: | ||
$ (dnf repoquery --disablerepo='*' --enablerepo='koji' --whatrequires 'libpython3.6m.so.1.0()(64bit) | $ (dnf repoquery --disablerepo='*' --enablerepo='koji' --whatrequires 'libpython3.6m.so.1.0()(64bit)' --whatrequires 'python(abi) == 3.6' --source) | pkgname | sort | uniq > rawhide | ||
Diff and update it regularly: | Diff and update it regularly: | ||
dnf clean all && (dnf repoquery --disablerepo='*' --enablerepo='koji' --whatrequires 'libpython3.6m.so.1.0()(64bit) | $ dnf clean all && (dnf repoquery --disablerepo='*' --enablerepo='koji' --whatrequires 'libpython3.6m.so.1.0()(64bit)' --whatrequires 'python(abi) == 3.6' --source) | pkgname | sort | uniq | diff -u rawhide - | ||
When packages are fixed, check dependent packages. Rebuild those (they might get unblocked). | When packages are fixed, check dependent packages. Rebuild those (they might get unblocked). | ||
== After the compose is built == | === After the compose is built === | ||
The packages are in rawhide repos, yay! Mass rebuild will happen soon, bugs will be open automatically, your job is almost done here. | The packages are in rawhide repos, yay! Mass rebuild will happen soon, bugs will be open automatically, your job is almost done here. Once the FTBFS bugs are opened, make the change tracking bug depend on them and mark any previously filled bugs as duplicates. You can bump the bugs in bulks (go to ''show dependency tree'' and ''change several''). | ||
Keep monitoring stuff. Also monitor devel mialing list for questions. | Keep monitoring stuff. Also monitor devel mialing list for questions. | ||
Rebuild everything in (the 3rd party repo that shall not be named), you can use mini-mass-rebuild with little tweaks. If you are a provenpackager in Fedora, you have the rights here as well, but if you never built anything in there, you need to ask. No nonfree stuff needed rebuilding during 3.7, only free (that might change in the future). | Rebuild everything in (the 3rd party repo that shall not be named), you can use mini-mass-rebuild with little tweaks. If you are a provenpackager in Fedora, you have the rights here as well, but if you never built anything in there, you need to ask. No nonfree stuff needed rebuilding during 3.7, only free (that might change in the future). | ||
Update the [https://pagure.io/fedora-kickstarts/blob/main/f/fedora-python-classroom-common.ks fedora-python-classroom kickstart] to not filter Python 3.N, but 3.N-1. | |||
== Rebuilds with new betas == | |||
When a new Python beta comes out, there can be two kinds of complications: | |||
* The ABI changed, requiring rebuilds of all extencion modules | |||
* The PYC magic number changed, requiring rebuilds of all Python modules | |||
To find out if we have one of these situations, rebuild everything in COPR with the new Python beta and analyze the failures. | |||
First, build the new Python beta release in COPR. | |||
Then, get fresh lists copies of python38.pkgs and python39.pkgs. Combine them and filter out Python itself: | |||
(repoquery --refresh --repo=koji --source --whatrequires 'libpython3.8.so.1.0()(64bit)' --whatrequires 'python(abi) = 3.8') | pkgname | sort | uniq > python38.pkgs | |||
(repoquery --refresh --repo=koji --source --whatrequires 'libpython3.9.so.1.0()(64bit)' --whatrequires 'python(abi) = 3.9') | pkgname | sort | uniq > python39.pkgs | |||
cat python39.pkgs python38.pkgs | sort | uniq | egrep -v '^python3.9$' > torebuild.pkgs | |||
Then, rebuild in COPR: | |||
for pkg in $(torebuild.pkgs); do echo $pkg; copr build-package --nowait --background --name $pkg @python/python3.9; done | |||
Analyze the failures and see what needs to be done when updating to the new beta in Fedora. | |||
=== Before the beta freeze === | |||
Once the Fedora beta freeze is approaching, it's a good idea to ask FESCo to give permission to retire all remaining leaf packages with bugs in NEW. See the [https://pagure.io/fesco/issue/1965 Mass retiring packages that still require Python 3.6 in Fedora 29] ticket for example. This removes cruft and it is apparently also the only way to scare some maintainers to fix their packages. | |||
Also, the [https://developer.fedoraproject.org/tech/languages/python/python-installation.html Developer Portal] docs might need updating. This doesn't block anything, so file a bug about it (on a triaged component in Bugzilla), so someone with free cycles can do it someday. | |||
==== Broken upgrade path ==== | |||
It is likely that some packages that used to require Python 3.N-1 are removed at this point (not only by the abovementioned FESCo ticket). It is crucial to obsolete them, or users would be unable to upgrade to the new Fedora version, having problems like <code>nothing provides python(abi) = 3.N-1 needed by coolpkg-0.1.2-3.fc31.noarch</code>. | |||
Such packages need to be obsoleted via the special {{package|fedora-obsolete-packages}} package. There is [https://github.com/hroncok/mini-mass-rebuild/blob/master/obsolete_packages.py a script] that can figure out all the needed lines to be added and a [https://bugzilla.redhat.com/show_bug.cgi?id=1754151 Bugzilla about this from Fedora 32 (Python 3.8)] that explains the procedure. | |||
It is likely that the list of obsoletes packages will be changing. Repeat this step to update it before the final freeze and also during the Fedora release lifetime. | |||
=== After the beta freeze === | |||
All remaining FTBFS bugs block upgrades to the new Fedora version. Request [https://qa.fedoraproject.org/blockerbugs/propose_bug freeze exceptions] for all of them. | |||
== Also, create pythonN-1 package == | |||
This needs a [https://bugzilla.redhat.com/show_bug.cgi?id=1592231 new package review request]. Ask releng to [https://pagure.io/releng/issue/7580 unretire it]. If you will create a nice git merge commit and a PR, note that releng will push a revert commit to the retired repo, so do it afterwards to save yourself doing it twice. | |||
= Ideas for Python 3.9 = | |||
== Inspiration from Perl == | |||
Perl has their own build dependency resolver tailored to Perl. It deals only with Perl build/run-time dependencies, all others are ignored. | |||
* https://repo.or.cz/Fedora-Rebuild.git | |||
Perl has a common bootstrap macro throughout the Perl ecosystem. It turns the build dependency graph into a tree, that can be easily built. | |||
* e.g. https://src.fedoraproject.org/rpms/perl-Alien-Build/blob/rawhide/f/perl-Alien-Build.spec | |||
== Applied to Python == | |||
We could use the [https://docs.fedoraproject.org/en-US/packaging-guidelines/#bootstrapping FPC-blessed "bootstrap" bcond], but we would have to set it for every package we're building. However, if we have a Python-specific bootstrap macro (e.g. <code>python_global_bootstrap</code>), we can set it centrally (e.g. in the `python3` package) and then every Python package that uses this macro will be built using a bootstrap mode, without us needing to change its specfile. |
Latest revision as of 21:17, 19 June 2022
Upgrading Python from 3.N-1 to 3.N
Here are the notes from upgrading Python from 3.6 to 3.7. They are created to save us time next time when upgrading to 3.8. It is not a comprehensive guide, but if it once gets to it, it'd be great.
For Python 3.9, see section Ideas for Python 3.9 at the bottom.
Create a Change Proposal
Before anything can happen, we need to create a Change Proposal. Follow the normal procedure, get inspired by previous changes, but make sure the process hasn't changed (copypasta is good, but check if the template hasn't been updated).
Create python3N package
This can happen as soon as the first alpha is released. If you have enough resources, try to do it even before.
The python3X packages are designed to have flat/nonflat switches since
, so you will use this package once you want to update python36
.
python3
Make a copr repo, bootstrap "important stuff" in it
Bootstrapping Python packages is nontrivial. Chances are high that from the upgrade to 3.N-1, the bootstrap sequence has changed, as more packages switched from python2-sphinx to python3-sphinx or from python3-pytoml to python3-toml etc. Start as early as possible.
For 3.7, I've used rpm-list-builder. The yaml recipe is archived on my GitHub. It has some comments. Note that rpm-list-builder has limitations:
- it doesn't bump releases, so you need to do that manually if you build one package once and the new build has new functionality (aka not just for tests)
- the "continue from here" future isn't very useful for us and was designed for different purpose, so what I did is that most of the yaml was commented out and I was uncommenting sections that I needed keeping finished stuff above, TODO stuff at the bottom
- everything is sequential. go do something different from time to time
What wasn't done for 3.7 but might have been a good idea is to restart the procedure once in a while. Packages early in the bootstrap sequence were built once with alphas and never tested with betas and rc. I suggest restarting the procedure later in the process.
Preserving the yaml recipe is good, you can use it for Fedora later.
- The Python 3.8 recipe: https://github.com/hroncok/rpm-list-builder/blob/python38/python38.yaml
- The Python 3.9 recipe: https://github.com/hroncok/rpm-list-builder/blob/python39/python39.yaml
In the Copr settings, add this external repo:
http://kojipkgs.fedoraproject.org/repos/rawhide/latest/$basearch/
And disable mock bootstrap.
This is the command I've used:
rm work-directory/* -rf && rpmlb --download local --source-directory source-directory python37.yaml python37 --work-directory work-directory --build copr --copr-repo @python/python3.7
I had local fedpkg clones, so I could create quick hotfixes in them. Also a must for
package - you have it updated to 3.N, but rawhide still has 3.N-1.
python3
Rebuild everything else in copr
This shall give you the packages that need rebuild for Python 3.9:
$ (repoquery --refresh --repo=koji --source --whatrequires 'libpython3.8.so.1.0()(64bit)' --whatrequires 'python(abi) = 3.8' --whatrequires '/usr/bin/python3.8' --whatrequires 'python3.8dist(*)') | pkgname | sort | uniq | egrep -v '^python3$' > python38.pkgs && (repoquery --refresh --repo=python39 --source --whatrequires 'libpython3.9.so.1.0()(64bit)' --whatrequires 'python(abi) = 3.9' --whatrequires '/usr/bin/python3.9' --whatrequires 'python3.9dist(*)') | pkgname | sort | uniq | egrep -v '^python3$' > python39.pkgs && comm -23 python38.pkgs python39.pkgs > todo.pkgs && wc -l *.pkgs
pkgname
command is from
0.6.0.5+.
fedora-packager
The koji repo is more up to date than rawhide. Use the following repo file if you don't already have it (save as /etc/yum.repos.d/koji.repo
):
[koji] name=koji baseurl=http://kojipkgs.fedoraproject.org/repos/rawhide/latest/x86_64/ enabled=0 [koji-source] name=koji-source baseurl=http://kojipkgs.fedoraproject.org/repos/rawhide/latest/src/ enabled=0
Inspect the resulting files and manually remove python3.X
from all of them. Add all packages to copr:
$ for pkg in $(cat python38.pkgs); do echo $pkg; copr add-package-distgit --name $pkg --webhook-rebuild on @python/python3.9 || copr edit-package-distgit --name $pkg --webhook-rebuild on @python/python3.9; done
And build the ones that are still needed:
$ for pkg in $(cat todo.pkgs); do echo $pkg; copr build-package --nowait --name $pkg @python/python3.9; done
Or, use moreutils-parallel (note that there are two packages in Fedora that provide the parallel command; you want moreutils-parallel):
$ parallel -j 12 copr build-package --nowait @python/python3.9 --name -- $(cat todo.pkgs)
Use monitor_check.py to see and report the failures.
Once ready, move into Fedora proper
Ready can differ. For 3.7 it was when rc1 was released and all important stuff from the yaml recipe was built in copr (in fact, couple of packages were not, but the fix was known).
Request a side tag
Example: Side tag for Python 3.7 releng issue
You'll get a side tag name (target). It can differ. For 3.7 it was f29-python
.
To build in the side tag, you do:
fedpkg build --target=<name>
To wait for builds to get into the repos, do:
koji wait-repo <name> --build=<nvr>
If Koji uses the mock's bootstrap option, you need to request it to be disabled for the side tag in order to prevent broken dependencies of dnf in the bootstrap chroot. A releng member can do that via:
koji edit-tag <name> -x 'mock.use_bootstrap=0'
Once
and dnf
(and/or possibly other Python packages needed in the bootstrap chroot in the future) are rebuilt and installable in the side tag, the mock's bootstrap option can be enabled again (possibly not worth the trouble unless not using it prevents some other builds).
dnf-plugins-core
Ask for autosigning to be enabled. That will significantly speed up the merging procedure.
Reproduce the bootstrap sequence
Just get the recipe and rebuild in the side tag. rpm-list-builder could not do it with 3.7 so I've used a dirty script. Be prepared for failures, packages got updated in the meantime and new deps were added. Tough life.
Rebuild everything else
Now, this gets a bit magical but what I did is that I've used mini-mass-rebuild to rebuild everything. Remember to bump the release and use the side tag (the fedpkg-build.sh script needs to be adjusted).
What is everything? First, i get the packages that buildrequire
:
python3-devel
dnf repoquery --disablerepo='*' --enablerepo='rawhide-source' --whatrequires python3-devel
But I forgot the virtual provides:
dnf repoquery --disablerepo='*' --enablerepo='rawhide' --provides python3-devel
Also a good idea what actually needs a rebuild is:
dnf repoquery --whatrequires 'libpython3.6m.so.1.0()(64bit)' --whatrequires 'python(abi) == 3.6' --whatrequires '/usr/bin/python3.6' --whatrequires 'python3.6dist(*)' --source
Use 3.N-1 instead of 3.6.
Remember that you already built stuff that doesn't need an extra rebuild. To query the side tag repo, use:
baseurl=https://kojipkgs.fedoraproject.org/repos/<target_name>/latest/$basearch/
To remove a list of packages from the list of packages, I use comm (requires sorted files):
comm -23 all_rawhide already_built > remains
There are also build-status.sh and build-failure-reason.sh in mini-mass-rebuild.
parallel -j 32 bash ../build-status.sh -- $(cat ../remains) | tee ../status for i in free open closed failed canceled; do echo -e $i\\t$(grep " $i" ../status | wc -l); done cat ../status | grep " closed" | cut -d" " -f1 | tee ../closed comm -23 ../remains ../closed > ../remains
And:
parallel -j 32 bash ../build-failure-reason.sh -- $(cat ../remains) | tee ../reason cat ../reason | grep ' build' # FTBFS cat ../reason | grep ' root' # missing dependency
It's a good idea to mass rebuild everything that remains until the number of closed (built) packages stops at 0. Don't accidentally bump the release over and over again. Later, start investigating the build failure reasons. Untangle any circular deps if found. Add them to yaml recipe for next time. File bugs (block the tracking bug, create an alias (PYTHON3N)). E-mail maintainers. Cry.
Request a side tag merge
Once you feel things look good (ask QA to confirm) and the number of FTBFS is reasonably low (for 3.7 it was 300 (half waiting for deps)), ask releng to merge the side tag. Do this at least a week before the mass rebuild. Releng will most likely not respond within a day, so better plan a future date (a ~week in advance would work best). E-mail devel-announce.
It was not yet possible during 3.7, but check with nirik if creating a test compose from side tag is possible, it could prevent issues like this one (anaconda tracebacks, no compose). Without a proper compose, the rawhide repos are not updated.
When the side tag is being merged, be available. After the merge, stuff thet has higher nevr in rawhide will "win" over the side tag. Double check if everything that was bootstrapped (not everything from the recipe, but where actual bootstrapping was involved) has higher epoch:version-release in the side tag - rebuild it in the side tag again if it was bumped in regular rawhide, or you will need to bootstrap it again after the merge.
Once merged, releng will give you list of packages that had higher nevr in rawhide. Bump-rebuild them in rawhide once you can koji-wait for python3-3.N.0... build.
After a side tag merge
If there are still FTBFSes (they are), e-mail the maintainers again.
Monitor the state. Get the initial list of broken deps:
$ (dnf repoquery --disablerepo='*' --enablerepo='koji' --whatrequires 'libpython3.6m.so.1.0()(64bit)' --whatrequires 'python(abi) == 3.6' --source) | pkgname | sort | uniq > rawhide
Diff and update it regularly:
$ dnf clean all && (dnf repoquery --disablerepo='*' --enablerepo='koji' --whatrequires 'libpython3.6m.so.1.0()(64bit)' --whatrequires 'python(abi) == 3.6' --source) | pkgname | sort | uniq | diff -u rawhide -
When packages are fixed, check dependent packages. Rebuild those (they might get unblocked).
After the compose is built
The packages are in rawhide repos, yay! Mass rebuild will happen soon, bugs will be open automatically, your job is almost done here. Once the FTBFS bugs are opened, make the change tracking bug depend on them and mark any previously filled bugs as duplicates. You can bump the bugs in bulks (go to show dependency tree and change several).
Keep monitoring stuff. Also monitor devel mialing list for questions.
Rebuild everything in (the 3rd party repo that shall not be named), you can use mini-mass-rebuild with little tweaks. If you are a provenpackager in Fedora, you have the rights here as well, but if you never built anything in there, you need to ask. No nonfree stuff needed rebuilding during 3.7, only free (that might change in the future).
Update the fedora-python-classroom kickstart to not filter Python 3.N, but 3.N-1.
Rebuilds with new betas
When a new Python beta comes out, there can be two kinds of complications:
- The ABI changed, requiring rebuilds of all extencion modules
- The PYC magic number changed, requiring rebuilds of all Python modules
To find out if we have one of these situations, rebuild everything in COPR with the new Python beta and analyze the failures.
First, build the new Python beta release in COPR.
Then, get fresh lists copies of python38.pkgs and python39.pkgs. Combine them and filter out Python itself:
(repoquery --refresh --repo=koji --source --whatrequires 'libpython3.8.so.1.0()(64bit)' --whatrequires 'python(abi) = 3.8') | pkgname | sort | uniq > python38.pkgs (repoquery --refresh --repo=koji --source --whatrequires 'libpython3.9.so.1.0()(64bit)' --whatrequires 'python(abi) = 3.9') | pkgname | sort | uniq > python39.pkgs cat python39.pkgs python38.pkgs | sort | uniq | egrep -v '^python3.9$' > torebuild.pkgs
Then, rebuild in COPR:
for pkg in $(torebuild.pkgs); do echo $pkg; copr build-package --nowait --background --name $pkg @python/python3.9; done
Analyze the failures and see what needs to be done when updating to the new beta in Fedora.
Before the beta freeze
Once the Fedora beta freeze is approaching, it's a good idea to ask FESCo to give permission to retire all remaining leaf packages with bugs in NEW. See the Mass retiring packages that still require Python 3.6 in Fedora 29 ticket for example. This removes cruft and it is apparently also the only way to scare some maintainers to fix their packages.
Also, the Developer Portal docs might need updating. This doesn't block anything, so file a bug about it (on a triaged component in Bugzilla), so someone with free cycles can do it someday.
Broken upgrade path
It is likely that some packages that used to require Python 3.N-1 are removed at this point (not only by the abovementioned FESCo ticket). It is crucial to obsolete them, or users would be unable to upgrade to the new Fedora version, having problems like nothing provides python(abi) = 3.N-1 needed by coolpkg-0.1.2-3.fc31.noarch
.
Such packages need to be obsoleted via the special
package. There is a script that can figure out all the needed lines to be added and a Bugzilla about this from Fedora 32 (Python 3.8) that explains the procedure.
fedora-obsolete-packages
It is likely that the list of obsoletes packages will be changing. Repeat this step to update it before the final freeze and also during the Fedora release lifetime.
After the beta freeze
All remaining FTBFS bugs block upgrades to the new Fedora version. Request freeze exceptions for all of them.
Also, create pythonN-1 package
This needs a new package review request. Ask releng to unretire it. If you will create a nice git merge commit and a PR, note that releng will push a revert commit to the retired repo, so do it afterwards to save yourself doing it twice.
Ideas for Python 3.9
Inspiration from Perl
Perl has their own build dependency resolver tailored to Perl. It deals only with Perl build/run-time dependencies, all others are ignored.
Perl has a common bootstrap macro throughout the Perl ecosystem. It turns the build dependency graph into a tree, that can be easily built.
Applied to Python
We could use the FPC-blessed "bootstrap" bcond, but we would have to set it for every package we're building. However, if we have a Python-specific bootstrap macro (e.g. python_global_bootstrap
), we can set it centrally (e.g. in the python3
package) and then every Python package that uses this macro will be built using a bootstrap mode, without us needing to change its specfile.