From Fedora Project Wiki

m (Kevin moved page Pushing fedora updates to Pushing updates SOP: This applies to epel too now.)
(Update SOP for bodhi2)
Line 1: Line 1:
 
== Description ==
 
== Description ==
Fedora updates for released releases are typically pushed once a day. This SOP covers the steps involved.
+
Fedora updates are typically pushed once a day. This SOP covers the steps involved.
  
 
== Action ==
 
== Action ==
  
=== Login to a machine that is configured for sigul client support and has the bodhi client installed. If you have not configured your workstation, you can use: ===
+
=== Login to a machine that is configured for sigul client support and has the bodhi client installed. If you have not configured your workstation, you can use: bodhi-backend01.phx2.fedoraproject.org ===
* For EPEL: relepel01
 
* For Fedora: releng04
 
  
 
=== Decide what releases you're going to push. ===
 
=== Decide what releases you're going to push. ===
{{admon/warning|Don't mix types| Please do not push Fedora and EPEL in the same push. }}
 
  
* Only one Fedora push and one EPEL push may be running (separately) at the same time.
+
* If there is a Freeze ongoing, you SHOULD NOT push all stable requests for a branched release, only specific blocker or freeze exception requests that QA will request in a releng ticket.
* During freezes, you must push updates-testing for branched separately from everything else. '''DO NOT push stable updates for branched during freezes.'''
+
 
For example:
+
* If there is no Freeze ongoing you can push all Fedora and EPEL releases at the same time if you wish.  
<pre>
 
bodhi -P --push-release=F21 --push-request=testing --bodhi-url=http://localhost/updates
 
bodhi -P --push-release=F20 --push-release=F19    --bodhi-url=http://localhost/updates
 
</pre>
 
* Outside of freezes, you may push all Fedora releases in the same push.
 
For example:
 
<pre>
 
bodhi -P --push-release=F21 --push-release=F20 --push-release=F19 --bodhi-url=http://localhost/updates
 
</pre>
 
  
 
=== Get a list of packages to push ===
 
=== Get a list of packages to push ===
 
<pre>
 
<pre>
$ bodhi -P --push-release F18 --push-release F19
+
$ cd ~masher
 +
$ sudo -u masher bodhi-push --releases '23 22 21 5 6 7' --username <yourusername>
 +
<enter your password+2factorauth, then your fas password>
 +
$ cp ~masher/* ~/
 
</pre>
 
</pre>
 
{{admon/warning|Blank list?| If the list is blank, that means the prior push failed, and manual intervention may be required. See 'Common Issues' below.}}
 
{{admon/warning|Blank list?| If the list is blank, that means the prior push failed, and manual intervention may be required. See 'Common Issues' below.}}
 
{{admon/warning|WAIT!| Do NOT press "Yes" yet. Simply view the list and say 'n'.}}
 
{{admon/warning|WAIT!| Do NOT press "Yes" yet. Simply view the list and say 'n'.}}
  
List the releases above you wish to push from: F18, F19, F20, EL-5, EL-6.
+
List the releases above you wish to push from: 23 22 21 5 6 7, etc
  
You can also specify '--push-request=testing' to limit pushes. Valid types are 'testing', 'stable', and 'security'.  
+
You can also specify '--push-request=testing' to limit pushes. Valid types are 'testing' or 'stable'.
  
 
The list of updates should be in your home directory named 'Stable-$Branch' or 'Testing-$Branch' for each of the Branches you wished to push.
 
The list of updates should be in your home directory named 'Stable-$Branch' or 'Testing-$Branch' for each of the Branches you wished to push.
Line 50: Line 41:
 
</pre>
 
</pre>
  
{{admon/caution|Run this process from <code>screen</code>| It is strongly suggested that the above procedure be run from screen.  Doing so requires a host configured as sigul client.  See the sigul client SOP '''http://fedoraproject.org/wiki/Sigul_Client_Setup_SOP'''}}
+
* You may need to add 'NSS_HASH_ALG_SUPPORT=+MD5' before sigulsign_unsigned (or add it to your ~/.bashrc).
 
 
* If your host is running RHEL6, add 'NSS_HASH_ALG_SUPPORT=+MD5' before sigulsign_unsigned (or add it to your ~/.bashrc).
 
  
 
=== Repeat gathering updates and signing steps ===
 
=== Repeat gathering updates and signing steps ===
Line 63: Line 52:
  
 
== Verification ==
 
== Verification ==
1. Tail the bodhi server log to watch progress on releng04 or relepel01.
+
1. Monitor the sysemd journal
 
<pre>
 
<pre>
$ tail -f /var/log/bodhi/server.log | egrep -v 'nagios-plugins|Starting new HTTPS'
+
$ journalctl -u fedmsg-hub -l -f
 
</pre>
 
</pre>
(The above regex improves signal:noise ratio caused by monitoring scripts.)
 
 
2. Monitor https://admin.fedoraproject.org/updates/admin/masher
 
  
3. Wait for the bodhi masher report (sent to bodhi-adminmember@fp.o) for success/failure or errors in the logs. Alternatively, wait for the update reports to be sent to the EPEL or test lists.
+
2. Watch for fedmsgs through the process. It will indicate what releases it's working on, etc.  
  
 
== Consider Before Running ==
 
== Consider Before Running ==
 
Pushes often fall over due to tagging issues or unsigned packages.  Be prepared to work through the failures and restart pushes from time to time
 
Pushes often fall over due to tagging issues or unsigned packages.  Be prepared to work through the failures and restart pushes from time to time
 
<pre>
 
<pre>
$ bodhi -P --resume-push
+
$ sudo -u masher bodhi-push --resume
 
</pre>
 
</pre>
 +
 +
Bodhi will ask you which push(es) you want to resume.
  
 
Consider testing if the mash lock file exists.
 
Consider testing if the mash lock file exists.
 
May indicate a previous push has not completed, or somehow failed:
 
May indicate a previous push has not completed, or somehow failed:
 
<pre>
 
<pre>
$ ls /mnt/koji/mash/updates/MASHING-FEDORA
+
$ ls /mnt/koji/mash/updates/MASHING-*
 
</pre>
 
</pre>
  
Line 101: Line 89:
 
* When an update was deleted from bodhi, it may still be in the updates-testing tag. You may need to: koji untag-pkg <tag> <n-v-r> to untag the package and resume. You may need to add --force.  
 
* When an update was deleted from bodhi, it may still be in the updates-testing tag. You may need to: koji untag-pkg <tag> <n-v-r> to untag the package and resume. You may need to add --force.  
  
Other issues should be addressed by releng or bodhi developers in #fedora-admin.  
+
Other issues should be addressed by releng or bodhi developers in #fedora-releng.  
  
 
[[Category:Release Engineering SOPs]]
 
[[Category:Release Engineering SOPs]]

Revision as of 16:13, 23 August 2015

Description

Fedora updates are typically pushed once a day. This SOP covers the steps involved.

Action

Login to a machine that is configured for sigul client support and has the bodhi client installed. If you have not configured your workstation, you can use: bodhi-backend01.phx2.fedoraproject.org

Decide what releases you're going to push.

  • If there is a Freeze ongoing, you SHOULD NOT push all stable requests for a branched release, only specific blocker or freeze exception requests that QA will request in a releng ticket.
  • If there is no Freeze ongoing you can push all Fedora and EPEL releases at the same time if you wish.

Get a list of packages to push

$ cd ~masher
$ sudo -u masher bodhi-push --releases '23 22 21 5 6 7' --username <yourusername>
<enter your password+2factorauth, then your fas password>
$ cp ~masher/* ~/
Warning.png
Blank list?
If the list is blank, that means the prior push failed, and manual intervention may be required. See 'Common Issues' below.
Warning.png
WAIT!
Do NOT press "Yes" yet. Simply view the list and say 'n'.

List the releases above you wish to push from: 23 22 21 5 6 7, etc

You can also specify '--push-request=testing' to limit pushes. Valid types are 'testing' or 'stable'.

The list of updates should be in your home directory named 'Stable-$Branch' or 'Testing-$Branch' for each of the Branches you wished to push.

Sign the packages.

  • Sign builds using scripts/sigulsign_unsigned.py from releng git repo
$ sigulsign_unsigned.py -vv --write-all fedora-19 $(cat Stable-F19 Testing-F19)

(Make sure you sign each release with the right key... ie, 'fedora-19' key with F19 packages, or 'epel-5' with EL-5 packages)

Here is another example, inside a loop:

for i in 20 19 18 ; do NSS_HASH_ALG_SUPPORT=+md5 sigulsign_unsigned.py fedora-$i -v --write-all $(cat {Stable,Testing}-F${i}) ; done
  • You may need to add 'NSS_HASH_ALG_SUPPORT=+MD5' before sigulsign_unsigned (or add it to your ~/.bashrc).

Repeat gathering updates and signing steps

After gathering the list of updates and signing them, repeat the process until there are no new updates to be signed. You want to do this because as you are signing updates, maintainers are submitting new ones. There is a window while you are signing that a new update will be added and if you just push then, the push will fail with an unsigned package.

Perform the bodhi push

Re-run the earlier bodhi command from step 2 and say 'y' to push.

Verification

1. Monitor the sysemd journal

$ journalctl -u fedmsg-hub -l -f

2. Watch for fedmsgs through the process. It will indicate what releases it's working on, etc.

Consider Before Running

Pushes often fall over due to tagging issues or unsigned packages. Be prepared to work through the failures and restart pushes from time to time

$ sudo -u masher bodhi-push --resume

Bodhi will ask you which push(es) you want to resume.

Consider testing if the mash lock file exists. May indicate a previous push has not completed, or somehow failed:

$ ls /mnt/koji/mash/updates/MASHING-*

Common issues / problems with pushes

  • When the push fails due to new unsigned packages that were added after you started the process. re-run step 4a or 4b with just the package names that need to be signed, then resume.
  • When the push fails due to an old package that has no signature, run: koji write-signed-rpm <gpgkeyid> <n-v-r> and resume.
  • When the push fails due to a package moving from testing to stable and leaving an old version of the same package in testing: koji untag-pkg <tag> <n-v-r> to untag the package and resume. You may need to add --force.
  • When the push fails due to a package not being tagged with updates-testing when being moved stable: koji tag-pkg dist-<tag>-updates-testing <n-v-r>
  • When the push fails and you see in the server.log "Identity shutting down", bodhi has quit for some reason. Do "sudo service httpd restart" and resume the push.
  • When signing fails, you may need to ask that the sigul bridge or server be restarted.
  • When an update was deleted from bodhi, it may still be in the updates-testing tag. You may need to: koji untag-pkg <tag> <n-v-r> to untag the package and resume. You may need to add --force.

Other issues should be addressed by releng or bodhi developers in #fedora-releng.