From Fedora Project Wiki

< SIGs‎ | Python

mNo edit summary
m (→‎Request a side tag: Ask for autosigning)
 
(54 intermediate revisions by 4 users not shown)
Line 1: Line 1:
= Upgrading Python from 3.N-1 to 3.N  =
= Upgrading Python from 3.N-1 to 3.N  =


Here are the notes from [[Changes/Python3.7|upgrading Python form 3.6 to 3.7]]. Those are created to save us time next time when upgrading to 3.8. It is not a comprehensive guide, but if it once get to it, it'd be great.
Here are the notes from [[Changes/Python3.7|upgrading Python from 3.6 to 3.7]]. They are created to save us time next time when upgrading to 3.8. It is not a comprehensive guide, but if it once gets to it, it'd be great.
 
For Python 3.9, see section [[#Ideas for Python 3.9|Ideas for Python 3.9]] at the bottom.


== Create a Change Proposal ==
== Create a Change Proposal ==


Before anything can happen, we need to create a Change Proposal. Follow the [[Changes/Policy|normal procedure]], get inspired by previous changes, but make sure the process haven't changed (copypasta is good, but check if the template hasn't been updated).
Before anything can happen, we need to create a Change Proposal. Follow the [[Changes/Policy|normal procedure]], get inspired by previous changes, but make sure the process hasn't changed (copypasta is good, but check if the template hasn't been updated).


== Create python3N package ==
== Create python3N package ==
Line 14: Line 16:
== Make a copr repo, bootstrap "important stuff" in it ==
== Make a copr repo, bootstrap "important stuff" in it ==


Bootstrapping Python packages is nontrivial. Chances are high that form the upgrade to 3.N-1, the bootstrap sequence has changed, as more packages switched to python3-sphinx etc. Start as early as possible.
Bootstrapping Python packages is nontrivial. Chances are high that from the upgrade to 3.N-1, the bootstrap sequence has changed, as more packages switched from python2-sphinx to python3-sphinx or from python3-pytoml to python3-toml etc. Start as early as possible.


For 3.7, I've used [https://github.com/sclorg/rpm-list-builder rpm-list-builder]. The [https://github.com/hroncok/rpm-list-builder/blob/python37/python37.yaml yaml recipe] is archived on my GitHub. It has some comments. Note that rpm-list-builder has limitations:
For 3.7, I've used [https://github.com/sclorg/rpm-list-builder rpm-list-builder]. The [https://github.com/hroncok/rpm-list-builder/blob/python37/python37.yaml yaml recipe] is archived on my GitHub. It has some comments. Note that rpm-list-builder has limitations:


* it doesn't bump releases, so you need to do that manually if you build one package once and the new build has new functionality (aka not just for tests)
* it doesn't bump releases, so you need to do that manually if you build one package once and the new build has new functionality (aka not just for tests)
* the "continue form here" future isn't very useful for us and was designed for different purpose, so what i did is that most of the yaml was commented out and I was uncommenting sections that i needed keeping finished stuff above, TODO stuff at the botttom
* the "continue from here" future isn't very useful for us and was designed for different purpose, so what I did is that most of the yaml was commented out and I was uncommenting sections that I needed keeping finished stuff above, TODO stuff at the bottom
* everything is sequential. go do soemthing different form time to time
* everything is sequential. go do something different from time to time
 
What wasn't done for 3.7 but might have been a good idea is to restart the procedure once in a while. Packages early in the bootstrap sequence were built once with alphas and never tested with betas and rc. I suggest restarting the procedure later in the process.
 
Preserving the yaml recipe is good, you can use it for Fedora later.


When {{package|gdb}} is updated in Fedora, you need to delete all {{package|python3}} builds from the copr repo, and bootsrap {{package|gdb}} - {{package|python3}} - {{package|gdb}} once more.
* The Python 3.8 recipe: https://github.com/hroncok/rpm-list-builder/blob/python38/python38.yaml
* The Python 3.9 recipe: https://github.com/hroncok/rpm-list-builder/blob/python39/python39.yaml


What wasn't done for 3.7 but might have been a good idea is to restart the procedure once in a while. Packages early in the bootstrap sequence were build once with alphas and never tested with betas and rc. I suggest to restart the procedure on beta1 and on rc1.
In the Copr settings, add this external repo:


Preserving the yaml recipe is good, you can use it for Fedora later.
http://kojipkgs.fedoraproject.org/repos/rawhide/latest/$basearch/
 
And '''disable mock bootstrap'''.


This is the command I've used:
This is the command I've used:
Line 33: Line 42:


I had local fedpkg clones, so I could create quick hotfixes in them. Also a must for {{package|python3}} package - you have it updated to 3.N, but rawhide still has 3.N-1.
I had local fedpkg clones, so I could create quick hotfixes in them. Also a must for {{package|python3}} package - you have it updated to 3.N, but rawhide still has 3.N-1.
{{admon/important|Some packages will need a release bump|It is unfortunately required that you manually bump some packages, especially those with bootstrap bconds. For example when Fedora has python3-setuptools-41.6.0-1.fc32 and you rebuild it with bootstrap enabled without bumping, it is python3-setuptools-41.6.0-1.fc32~bootstrap and that sorts lower. This feature of the dist tag requires you to run <code>rpmdev-bumpspec *.spec</code> on such packages in the source directory prior to proceeding.}}
=== Rebuild everything else in copr ===
This shall give you the packages that need rebuild for Python 3.9:
$ (repoquery --refresh --repo=koji --source --whatrequires 'libpython3.8.so.1.0()(64bit)' --whatrequires 'python(abi) = 3.8' --whatrequires '/usr/bin/python3.8' --whatrequires 'python3.8dist(*)') | pkgname | sort | uniq | egrep -v '^python3$' > python38.pkgs && (repoquery --refresh --repo=python39 --source --whatrequires 'libpython3.9.so.1.0()(64bit)' --whatrequires 'python(abi) = 3.9' --whatrequires '/usr/bin/python3.9' --whatrequires 'python3.9dist(*)') | pkgname | sort | uniq | egrep -v '^python3$' > python39.pkgs && comm -23 python38.pkgs python39.pkgs > todo.pkgs && wc -l *.pkgs
<code>pkgname</code> command is from {{package|fedora-packager}} 0.6.0.5+.
The koji repo is more up to date than rawhide. Use the following repo file if you don't already have it (save as <code>/etc/yum.repos.d/koji.repo</code>):
[koji]
name=koji
baseurl=http://kojipkgs.fedoraproject.org/repos/rawhide/latest/x86_64/
enabled=0
[koji-source]
name=koji-source
baseurl=http://kojipkgs.fedoraproject.org/repos/rawhide/latest/src/
enabled=0
Inspect the resulting files and manually remove <code>python3.X</code> from all of them. Add all packages to copr:
$ for pkg in $(cat python38.pkgs); do echo $pkg; copr add-package-distgit --name $pkg --webhook-rebuild on @python/python3.9  || copr edit-package-distgit --name $pkg --webhook-rebuild on @python/python3.9; done
And build the ones that are still needed:
$ for pkg in $(cat todo.pkgs); do echo $pkg; copr build-package --nowait --name $pkg @python/python3.9; done
Or, use moreutils-parallel (note that there are two packages in Fedora that provide the parallel command; you want moreutils-parallel):
$ parallel -j 12 copr build-package --nowait @python/python3.9 --name -- $(cat todo.pkgs)
Use [https://github.com/hroncok/mini-mass-rebuild/blob/master/monitor_check.py monitor_check.py] to see and report the failures.
== Once ready, move into Fedora proper ==
Ready can differ. For 3.7 it was when rc1 was released and all important stuff from the yaml recipe was built in copr (in fact, couple of packages were not, but the fix was known).
=== Request a side tag ===
Example: [https://pagure.io/releng/issue/7564 Side tag for Python 3.7 releng issue]
You'll get a side tag name (target). It can differ. For 3.7 it was <code>f29-python</code>.
To build in the side tag, you do:
fedpkg build --target=<name>
To wait for builds to get into the repos, do:
koji wait-repo <name> --build=<nvr>
If [https://pagure.io/releng/issue/9308 Koji uses the mock's bootstrap option], you need to request it to be disabled for the side tag in order to prevent broken dependencies of dnf in the bootstrap chroot. A releng member can do that via:
koji edit-tag <name> -x 'mock.use_bootstrap=0'
Once {{package|dnf}} and {{package|dnf-plugins-core}} (and/or possibly other Python packages needed in the bootstrap chroot in the future) are rebuilt and installable in the side tag, the mock's bootstrap option can be enabled again (possibly not worth the trouble unless not using it prevents some other builds).
Ask for autosigning to be enabled. That will significantly speed up the merging procedure.
=== Reproduce the bootstrap sequence ===
Just get the recipe and rebuild in the side tag. [https://github.com/sclorg/rpm-list-builder/issues/108 rpm-list-builder could not do it with 3.7] so I've used a [https://github.com/hroncok/rpm-list-builder/blob/python37/dirty-rebuild-script.py dirty script]. Be prepared for failures, packages got updated in the meantime and new deps were added. Tough life.
=== Rebuild everything else ===
Now, this gets a bit magical but what I did is that I've used [https://github.com/hroncok/mini-mass-rebuild mini-mass-rebuild] to rebuild everything. Remember to bump the release and use the side tag (the fedpkg-build.sh script needs to be adjusted).
What is everything? First, i get the packages that buildrequire {{package|python3-devel}}:
  dnf repoquery  --disablerepo='*' --enablerepo='rawhide-source' --whatrequires python3-devel
But I forgot the virtual provides:
dnf repoquery  --disablerepo='*' --enablerepo='rawhide' --provides python3-devel
Also a good idea what actually needs a rebuild is:
dnf repoquery --whatrequires 'libpython3.6m.so.1.0()(64bit)' --whatrequires 'python(abi) == 3.6' --whatrequires '/usr/bin/python3.6' --whatrequires 'python3.6dist(*)' --source
Use 3.N-1 instead of 3.6.
Remember that you already built stuff that doesn't need an extra rebuild. To query the side tag repo, use:
baseurl=https://kojipkgs.fedoraproject.org/repos/<target_name>/latest/$basearch/
To remove a list of packages from the list of packages, I use comm (requires sorted files):
comm -23 all_rawhide already_built > remains
There are also [https://github.com/hroncok/mini-mass-rebuild/blob/master/build-status.sh build-status.sh] and [https://github.com/hroncok/mini-mass-rebuild/blob/master/build-failure-reason.sh build-failure-reason.sh] in mini-mass-rebuild.
parallel -j 32 bash ../build-status.sh -- $(cat ../remains) | tee ../status
for i in free open closed failed canceled; do echo -e $i\\t$(grep " $i" ../status | wc -l); done
cat ../status | grep " closed" | cut -d" " -f1 | tee ../closed
comm -23 ../remains ../closed > ../remains
And:
parallel -j 32 bash ../build-failure-reason.sh -- $(cat ../remains) | tee ../reason
cat ../reason | grep ' build'  # FTBFS
cat ../reason | grep ' root'  # missing dependency
It's a good idea to mass rebuild everything that remains until the number of closed (built) packages stops at 0. Don't accidentally bump the release over and over again. Later, start investigating the build failure reasons. Untangle any circular deps if found. Add them to yaml recipe for next time. File bugs (block the tracking bug, create an alias (PYTHON3N)). [https://pagure.io/fedora-misc-package-utilities/blob/master/f/find-package-maintainers E-mail maintainers.] Cry.
=== Request a side tag merge ===
Once you feel things look good (ask QA to confirm) and the number of FTBFS is reasonably low (for 3.7 it was 300 (half waiting for deps)), ask releng to merge the side tag. Do this at least a week before the mass rebuild. Releng will most likely not respond within a day, so better plan a future date (a ~week in advance would work best). E-mail devel-announce.
It was not yet possible during 3.7, but check with [[User:Kevin|nirik]] if creating a test compose from side tag is possible, it could prevent issues like [https://bugzilla.redhat.com/show_bug.cgi?id=1598574 this one] (anaconda tracebacks, no compose). Without a proper compose, the rawhide repos are not updated.
When the side tag is being merged, be available. After the merge, stuff thet has higher nevr in rawhide will "win" over the side tag.
Double check if everything that was bootstrapped (not everything from the recipe, but where actual bootstrapping was involved) has higher epoch:version-release in the side tag - rebuild it in the side tag again if it was bumped in regular rawhide, or you will need to bootstrap it again after the merge.
Once merged, releng will give you list of packages that had higher nevr in rawhide. Bump-rebuild them in rawhide once you can koji-wait for python3-3.N.0... build.
=== After a side tag merge ===
If there are still FTBFSes (they are), [https://pagure.io/fedora-misc-package-utilities/blob/master/f/find-package-maintainers e-mail the maintainers] again.
Monitor the state. Get the initial list of broken deps:
$ (dnf repoquery  --disablerepo='*' --enablerepo='koji' --whatrequires 'libpython3.6m.so.1.0()(64bit)' --whatrequires 'python(abi) == 3.6' --source) | pkgname | sort | uniq > rawhide
Diff and update it regularly:
$ dnf clean all && (dnf repoquery  --disablerepo='*' --enablerepo='koji' --whatrequires 'libpython3.6m.so.1.0()(64bit)' --whatrequires 'python(abi) == 3.6' --source) | pkgname | sort | uniq | diff -u rawhide -
When packages are fixed, check dependent packages. Rebuild those (they might get unblocked).
=== After the compose is built ===
The packages are in rawhide repos, yay! Mass rebuild will happen soon, bugs will be open automatically, your job is almost done here. Once the FTBFS bugs are opened, make the change tracking bug depend on them and mark any previously filled bugs as duplicates. You can bump the bugs in bulks (go to ''show dependency tree'' and ''change several'').
Keep monitoring stuff. Also monitor devel mialing list for questions.
Rebuild everything in (the 3rd party repo that shall not be named), you can use mini-mass-rebuild with little tweaks. If you are a provenpackager in Fedora, you have the rights here as well, but if you never built anything in there, you need to ask.  No nonfree stuff needed rebuilding during 3.7, only free (that might change in the future).
Update the [https://pagure.io/fedora-kickstarts/blob/main/f/fedora-python-classroom-common.ks fedora-python-classroom kickstart] to not filter Python 3.N, but 3.N-1.
== Rebuilds with new betas ==
When a new Python beta comes out, there can be two kinds of complications:
* The ABI changed, requiring rebuilds of all extencion modules
* The PYC magic number changed, requiring rebuilds of all Python modules
To find out if we have one of these situations, rebuild everything in COPR with the new Python beta and analyze the failures.
First, build the new Python beta release in COPR.
Then, get fresh lists copies of python38.pkgs and python39.pkgs. Combine them and filter out Python itself:
(repoquery --refresh --repo=koji --source --whatrequires 'libpython3.8.so.1.0()(64bit)' --whatrequires 'python(abi) = 3.8') | pkgname | sort | uniq > python38.pkgs
(repoquery --refresh --repo=koji --source --whatrequires 'libpython3.9.so.1.0()(64bit)' --whatrequires 'python(abi) = 3.9') | pkgname | sort | uniq > python39.pkgs
cat python39.pkgs python38.pkgs | sort | uniq | egrep -v '^python3.9$' > torebuild.pkgs
Then, rebuild in COPR:
for pkg in $(torebuild.pkgs); do echo $pkg; copr build-package --nowait --background --name $pkg @python/python3.9; done
Analyze the failures and see what needs to be done when updating to the new beta in Fedora.
=== Before the beta freeze ===
Once the Fedora beta freeze is approaching, it's a good idea to ask FESCo to give permission to retire all remaining leaf packages with bugs in NEW. See the [https://pagure.io/fesco/issue/1965 Mass retiring packages that still require Python 3.6 in Fedora 29] ticket for example. This removes cruft and it is apparently also the only way to scare some maintainers to fix their packages.
Also, the [https://developer.fedoraproject.org/tech/languages/python/python-installation.html Developer Portal] docs might need updating. This doesn't block anything, so file a bug about it (on a triaged component in Bugzilla), so someone with free cycles can do it someday.
==== Broken upgrade path ====
It is likely that some packages that used to require Python 3.N-1 are removed at this point (not only by the abovementioned FESCo ticket). It is crucial to obsolete them, or users would be unable to upgrade to the new Fedora version, having problems like <code>nothing provides python(abi) = 3.N-1 needed by coolpkg-0.1.2-3.fc31.noarch</code>.
Such packages need to be obsoleted via the special {{package|fedora-obsolete-packages}} package. There is [https://github.com/hroncok/mini-mass-rebuild/blob/master/obsolete_packages.py a script] that can figure out all the needed lines to be added and a [https://bugzilla.redhat.com/show_bug.cgi?id=1754151 Bugzilla about this from Fedora 32 (Python 3.8)] that explains the procedure.
It is likely that the list of obsoletes packages will be changing. Repeat this step to update it before the final freeze and also during the Fedora release lifetime.
=== After the beta freeze ===
All remaining FTBFS bugs block upgrades to the new Fedora version. Request [https://qa.fedoraproject.org/blockerbugs/propose_bug freeze exceptions] for all of them.
== Also, create pythonN-1 package ==
This needs a [https://bugzilla.redhat.com/show_bug.cgi?id=1592231 new package review request]. Ask releng to [https://pagure.io/releng/issue/7580 unretire it]. If you will create a nice git merge commit and a PR, note that releng will push a revert commit to the retired repo, so do it afterwards to save yourself doing it twice.
= Ideas for Python 3.9 =
== Inspiration from Perl ==
Perl has their own build dependency resolver tailored to Perl. It deals only with Perl build/run-time dependencies, all others are ignored.
* https://repo.or.cz/Fedora-Rebuild.git
Perl has a common bootstrap macro throughout the Perl ecosystem. It turns the build dependency graph into a tree, that can be easily built.
* e.g. https://src.fedoraproject.org/rpms/perl-Alien-Build/blob/rawhide/f/perl-Alien-Build.spec
== Applied to Python ==
We could use the [https://docs.fedoraproject.org/en-US/packaging-guidelines/#bootstrapping FPC-blessed "bootstrap" bcond], but we would have to set it for every package we're building. However, if we have a Python-specific bootstrap macro (e.g. <code>python_global_bootstrap</code>), we can set it centrally (e.g. in the `python3` package) and then every Python package that uses this macro will be built using a bootstrap mode, without us needing to change its specfile.

Latest revision as of 21:17, 19 June 2022

Upgrading Python from 3.N-1 to 3.N

Here are the notes from upgrading Python from 3.6 to 3.7. They are created to save us time next time when upgrading to 3.8. It is not a comprehensive guide, but if it once gets to it, it'd be great.

For Python 3.9, see section Ideas for Python 3.9 at the bottom.

Create a Change Proposal

Before anything can happen, we need to create a Change Proposal. Follow the normal procedure, get inspired by previous changes, but make sure the process hasn't changed (copypasta is good, but check if the template hasn't been updated).

Create python3N package

This can happen as soon as the first alpha is released. If you have enough resources, try to do it even before. The python3X packages are designed to have flat/nonflat switches since Package-x-generic-16.pngpython36, so you will use this package once you want to update Package-x-generic-16.pngpython3.

Make a copr repo, bootstrap "important stuff" in it

Bootstrapping Python packages is nontrivial. Chances are high that from the upgrade to 3.N-1, the bootstrap sequence has changed, as more packages switched from python2-sphinx to python3-sphinx or from python3-pytoml to python3-toml etc. Start as early as possible.

For 3.7, I've used rpm-list-builder. The yaml recipe is archived on my GitHub. It has some comments. Note that rpm-list-builder has limitations:

  • it doesn't bump releases, so you need to do that manually if you build one package once and the new build has new functionality (aka not just for tests)
  • the "continue from here" future isn't very useful for us and was designed for different purpose, so what I did is that most of the yaml was commented out and I was uncommenting sections that I needed keeping finished stuff above, TODO stuff at the bottom
  • everything is sequential. go do something different from time to time

What wasn't done for 3.7 but might have been a good idea is to restart the procedure once in a while. Packages early in the bootstrap sequence were built once with alphas and never tested with betas and rc. I suggest restarting the procedure later in the process.

Preserving the yaml recipe is good, you can use it for Fedora later.

In the Copr settings, add this external repo:

http://kojipkgs.fedoraproject.org/repos/rawhide/latest/$basearch/

And disable mock bootstrap.

This is the command I've used:

rm work-directory/* -rf && rpmlb --download local --source-directory source-directory python37.yaml python37 --work-directory work-directory --build copr --copr-repo @python/python3.7

I had local fedpkg clones, so I could create quick hotfixes in them. Also a must for Package-x-generic-16.pngpython3 package - you have it updated to 3.N, but rawhide still has 3.N-1.

Important.png
Some packages will need a release bump
It is unfortunately required that you manually bump some packages, especially those with bootstrap bconds. For example when Fedora has python3-setuptools-41.6.0-1.fc32 and you rebuild it with bootstrap enabled without bumping, it is python3-setuptools-41.6.0-1.fc32~bootstrap and that sorts lower. This feature of the dist tag requires you to run rpmdev-bumpspec *.spec on such packages in the source directory prior to proceeding.

Rebuild everything else in copr

This shall give you the packages that need rebuild for Python 3.9:

$ (repoquery --refresh --repo=koji --source --whatrequires 'libpython3.8.so.1.0()(64bit)' --whatrequires 'python(abi) = 3.8' --whatrequires '/usr/bin/python3.8' --whatrequires 'python3.8dist(*)') | pkgname | sort | uniq | egrep -v '^python3$' > python38.pkgs && (repoquery --refresh --repo=python39 --source --whatrequires 'libpython3.9.so.1.0()(64bit)' --whatrequires 'python(abi) = 3.9' --whatrequires '/usr/bin/python3.9' --whatrequires 'python3.9dist(*)') | pkgname | sort | uniq | egrep -v '^python3$' > python39.pkgs && comm -23 python38.pkgs python39.pkgs > todo.pkgs && wc -l *.pkgs

pkgname command is from Package-x-generic-16.pngfedora-packager 0.6.0.5+.

The koji repo is more up to date than rawhide. Use the following repo file if you don't already have it (save as /etc/yum.repos.d/koji.repo):

[koji]
name=koji
baseurl=http://kojipkgs.fedoraproject.org/repos/rawhide/latest/x86_64/
enabled=0

[koji-source]
name=koji-source
baseurl=http://kojipkgs.fedoraproject.org/repos/rawhide/latest/src/
enabled=0

Inspect the resulting files and manually remove python3.X from all of them. Add all packages to copr:

$ for pkg in $(cat python38.pkgs); do echo $pkg; copr add-package-distgit --name $pkg --webhook-rebuild on @python/python3.9  || copr edit-package-distgit --name $pkg --webhook-rebuild on @python/python3.9; done

And build the ones that are still needed:

$ for pkg in $(cat todo.pkgs); do echo $pkg; copr build-package --nowait --name $pkg @python/python3.9; done

Or, use moreutils-parallel (note that there are two packages in Fedora that provide the parallel command; you want moreutils-parallel):

$ parallel -j 12 copr build-package --nowait @python/python3.9 --name -- $(cat todo.pkgs)

Use monitor_check.py to see and report the failures.

Once ready, move into Fedora proper

Ready can differ. For 3.7 it was when rc1 was released and all important stuff from the yaml recipe was built in copr (in fact, couple of packages were not, but the fix was known).

Request a side tag

Example: Side tag for Python 3.7 releng issue

You'll get a side tag name (target). It can differ. For 3.7 it was f29-python.

To build in the side tag, you do:

fedpkg build --target=<name>

To wait for builds to get into the repos, do:

koji wait-repo <name> --build=<nvr>

If Koji uses the mock's bootstrap option, you need to request it to be disabled for the side tag in order to prevent broken dependencies of dnf in the bootstrap chroot. A releng member can do that via:

koji edit-tag <name> -x 'mock.use_bootstrap=0'

Once Package-x-generic-16.pngdnf and Package-x-generic-16.pngdnf-plugins-core (and/or possibly other Python packages needed in the bootstrap chroot in the future) are rebuilt and installable in the side tag, the mock's bootstrap option can be enabled again (possibly not worth the trouble unless not using it prevents some other builds).

Ask for autosigning to be enabled. That will significantly speed up the merging procedure.

Reproduce the bootstrap sequence

Just get the recipe and rebuild in the side tag. rpm-list-builder could not do it with 3.7 so I've used a dirty script. Be prepared for failures, packages got updated in the meantime and new deps were added. Tough life.

Rebuild everything else

Now, this gets a bit magical but what I did is that I've used mini-mass-rebuild to rebuild everything. Remember to bump the release and use the side tag (the fedpkg-build.sh script needs to be adjusted).

What is everything? First, i get the packages that buildrequire Package-x-generic-16.pngpython3-devel:

 dnf repoquery  --disablerepo='*' --enablerepo='rawhide-source' --whatrequires python3-devel

But I forgot the virtual provides:

dnf repoquery  --disablerepo='*' --enablerepo='rawhide' --provides python3-devel

Also a good idea what actually needs a rebuild is:

dnf repoquery --whatrequires 'libpython3.6m.so.1.0()(64bit)' --whatrequires 'python(abi) == 3.6' --whatrequires '/usr/bin/python3.6' --whatrequires 'python3.6dist(*)' --source

Use 3.N-1 instead of 3.6.

Remember that you already built stuff that doesn't need an extra rebuild. To query the side tag repo, use:

baseurl=https://kojipkgs.fedoraproject.org/repos/<target_name>/latest/$basearch/

To remove a list of packages from the list of packages, I use comm (requires sorted files):

comm -23 all_rawhide already_built > remains

There are also build-status.sh and build-failure-reason.sh in mini-mass-rebuild.

parallel -j 32 bash ../build-status.sh -- $(cat ../remains) | tee ../status
for i in free open closed failed canceled; do echo -e $i\\t$(grep " $i" ../status | wc -l); done
cat ../status | grep " closed" | cut -d" " -f1 | tee ../closed
comm -23 ../remains ../closed > ../remains

And:

parallel -j 32 bash ../build-failure-reason.sh -- $(cat ../remains) | tee ../reason
cat ../reason | grep ' build'  # FTBFS
cat ../reason | grep ' root'   # missing dependency

It's a good idea to mass rebuild everything that remains until the number of closed (built) packages stops at 0. Don't accidentally bump the release over and over again. Later, start investigating the build failure reasons. Untangle any circular deps if found. Add them to yaml recipe for next time. File bugs (block the tracking bug, create an alias (PYTHON3N)). E-mail maintainers. Cry.

Request a side tag merge

Once you feel things look good (ask QA to confirm) and the number of FTBFS is reasonably low (for 3.7 it was 300 (half waiting for deps)), ask releng to merge the side tag. Do this at least a week before the mass rebuild. Releng will most likely not respond within a day, so better plan a future date (a ~week in advance would work best). E-mail devel-announce.

It was not yet possible during 3.7, but check with nirik if creating a test compose from side tag is possible, it could prevent issues like this one (anaconda tracebacks, no compose). Without a proper compose, the rawhide repos are not updated.

When the side tag is being merged, be available. After the merge, stuff thet has higher nevr in rawhide will "win" over the side tag. Double check if everything that was bootstrapped (not everything from the recipe, but where actual bootstrapping was involved) has higher epoch:version-release in the side tag - rebuild it in the side tag again if it was bumped in regular rawhide, or you will need to bootstrap it again after the merge.

Once merged, releng will give you list of packages that had higher nevr in rawhide. Bump-rebuild them in rawhide once you can koji-wait for python3-3.N.0... build.

After a side tag merge

If there are still FTBFSes (they are), e-mail the maintainers again.

Monitor the state. Get the initial list of broken deps:

$ (dnf repoquery  --disablerepo='*' --enablerepo='koji' --whatrequires 'libpython3.6m.so.1.0()(64bit)' --whatrequires 'python(abi) == 3.6' --source) | pkgname | sort | uniq > rawhide

Diff and update it regularly:

$ dnf clean all && (dnf repoquery  --disablerepo='*' --enablerepo='koji' --whatrequires 'libpython3.6m.so.1.0()(64bit)' --whatrequires 'python(abi) == 3.6' --source) | pkgname | sort | uniq | diff -u rawhide -

When packages are fixed, check dependent packages. Rebuild those (they might get unblocked).

After the compose is built

The packages are in rawhide repos, yay! Mass rebuild will happen soon, bugs will be open automatically, your job is almost done here. Once the FTBFS bugs are opened, make the change tracking bug depend on them and mark any previously filled bugs as duplicates. You can bump the bugs in bulks (go to show dependency tree and change several).

Keep monitoring stuff. Also monitor devel mialing list for questions.

Rebuild everything in (the 3rd party repo that shall not be named), you can use mini-mass-rebuild with little tweaks. If you are a provenpackager in Fedora, you have the rights here as well, but if you never built anything in there, you need to ask. No nonfree stuff needed rebuilding during 3.7, only free (that might change in the future).

Update the fedora-python-classroom kickstart to not filter Python 3.N, but 3.N-1.

Rebuilds with new betas

When a new Python beta comes out, there can be two kinds of complications:

  • The ABI changed, requiring rebuilds of all extencion modules
  • The PYC magic number changed, requiring rebuilds of all Python modules

To find out if we have one of these situations, rebuild everything in COPR with the new Python beta and analyze the failures.

First, build the new Python beta release in COPR.

Then, get fresh lists copies of python38.pkgs and python39.pkgs. Combine them and filter out Python itself:

(repoquery --refresh --repo=koji --source --whatrequires 'libpython3.8.so.1.0()(64bit)' --whatrequires 'python(abi) = 3.8') | pkgname | sort | uniq > python38.pkgs
(repoquery --refresh --repo=koji --source --whatrequires 'libpython3.9.so.1.0()(64bit)' --whatrequires 'python(abi) = 3.9') | pkgname | sort | uniq > python39.pkgs
cat python39.pkgs python38.pkgs | sort | uniq | egrep -v '^python3.9$' > torebuild.pkgs

Then, rebuild in COPR:

for pkg in $(torebuild.pkgs); do echo $pkg; copr build-package --nowait --background --name $pkg @python/python3.9; done

Analyze the failures and see what needs to be done when updating to the new beta in Fedora.


Before the beta freeze

Once the Fedora beta freeze is approaching, it's a good idea to ask FESCo to give permission to retire all remaining leaf packages with bugs in NEW. See the Mass retiring packages that still require Python 3.6 in Fedora 29 ticket for example. This removes cruft and it is apparently also the only way to scare some maintainers to fix their packages.

Also, the Developer Portal docs might need updating. This doesn't block anything, so file a bug about it (on a triaged component in Bugzilla), so someone with free cycles can do it someday.

Broken upgrade path

It is likely that some packages that used to require Python 3.N-1 are removed at this point (not only by the abovementioned FESCo ticket). It is crucial to obsolete them, or users would be unable to upgrade to the new Fedora version, having problems like nothing provides python(abi) = 3.N-1 needed by coolpkg-0.1.2-3.fc31.noarch.

Such packages need to be obsoleted via the special Package-x-generic-16.pngfedora-obsolete-packages package. There is a script that can figure out all the needed lines to be added and a Bugzilla about this from Fedora 32 (Python 3.8) that explains the procedure.

It is likely that the list of obsoletes packages will be changing. Repeat this step to update it before the final freeze and also during the Fedora release lifetime.

After the beta freeze

All remaining FTBFS bugs block upgrades to the new Fedora version. Request freeze exceptions for all of them.

Also, create pythonN-1 package

This needs a new package review request. Ask releng to unretire it. If you will create a nice git merge commit and a PR, note that releng will push a revert commit to the retired repo, so do it afterwards to save yourself doing it twice.

Ideas for Python 3.9

Inspiration from Perl

Perl has their own build dependency resolver tailored to Perl. It deals only with Perl build/run-time dependencies, all others are ignored.

Perl has a common bootstrap macro throughout the Perl ecosystem. It turns the build dependency graph into a tree, that can be easily built.

Applied to Python

We could use the FPC-blessed "bootstrap" bcond, but we would have to set it for every package we're building. However, if we have a Python-specific bootstrap macro (e.g. python_global_bootstrap), we can set it centrally (e.g. in the python3 package) and then every Python package that uses this macro will be built using a bootstrap mode, without us needing to change its specfile.