Pushing updates SOP

Description

Fedora updates for released er releases are typically pushed once a work day. This SOP covers the steps involved.

Note that the process is slightly different between EPEL and Fedora releases. Fedora Rawhide and EPEL-6 do not need or use this process.

Action

Login to a machine that is configured for sigul client support and has the bodhi client installed. If you have not configured your workstation, you can use:

For EPEL-4, EPEL-5: relepel01.
For F12, F13, F14: releng02.

Decide what releases you're going to push.

Don't mix types
Please do not push Fedora and EPEL in the same push. Also, please do not push updates for a branched fedora pre-release at the same time as updates for released Fedora.

Get a list of packages to push

$ bodhi -P --push-release F12 --push-release F13

Blank list?
If the list is blank, that means the prior push failed, and manual intervention may be required. See 'Common Issues' below.

WAIT!
Do NOT press "Yes" yet. Simply view the list and say 'n'.

List the releases above you wish to push from: F12, F13, F14, EL4, EL5.

The list of updates should be in your homedirectory named 'Stable-$Branch' or 'Testing-$Branch' for each of the Branches you wished to push.

Sign the packages.

For Fedora releases and EPEL 6: Sign builds using scripts/sigulsign_unsigned.py from releng git repo

$ sigulsign_unsigned.py -vv --write-all fedora-13 $(cat *F13)

(Make sure you sign each release with the right key... ie, 'fedora-13' key with F13 packages)

For EPEL4 and 5 releases: Sign builds using scripts/sign_unsigned.py from releng git repo

$ sign_unsigned.py --level=EPEL --write-rpms --builds $(cat *EL5 *EL4)

Run this process from screen
It is strongly suggested that the above procedure be run from screen. Doing so requires a host configured as sigul client. See the sigul client SOP NEED LINK

Check for policy violations

For EPEL releases, check the web interface to make sure all packages have followed the guidelines. Two weeks in testing or +3 karma or security update before allowing to go to stable.

Perform the bodhi push

Re-run the earlier bodhi command from step 2 and say 'y' to push.

Verification

1. Tail the bodhi server log to watch progress on releng02 or relepel01.

$ tail -f /var/log/bodhi/server.log

2. Wait for the bodhi masher report (sent to bodhi-adminmember@fp.o) for success/failure or errors in the logs. Alternatively, wait for the update reports to be sent to the EPEL or test lists.

Consider Before Running

Pushes often fall over due to tagging issues or unsigned packages. Be prepared to work through the failures and restart pushes from time to time

$ bodhi -P --resume-push

Common issues / problems with pushes

When the push fails due to new unsigned packages that were added after you started the process. re-run step 4a or 4b with just the package names that need to be signed, then resume.

When the push fails due to an old package that has no signature, run: koji write-signed-rpm <gpgkeyid> <n-v-r> and resume.

When the push fails due to a package moving from testing to stable and leaving an old version of the same package in testing: koji untag-pkg <tag> <n-v-r> to untag the package and resume.

When the push fails due to a package not being tagged with updates-testing when being moved stable: koji tag-pkg dist-<tag>-updates-testing <n-v-r>

When the push fails and you see in the server.log "Identity shutting down", bodhi has quit for some reason. Do "sudo service httpd restart" and resume the push.

Other issues should be addressed by releng or bodhi developers in #fedora-admin.

Search