User:Dwa

David Aquilina

I'm a release engineer at Red Hat, working on Fedora for POWER.

Contact Info

E-mail: dwa (at) redhat.com

IRC: 'dwa' on freenode, in #fedora-ppc

Secondary Arch Releng

Builds

Principles

• Never build an NVR before it's built on primary (koji.fedoraproject.org). The code used for a given NVR must match exactly between primary and secondary. koji-shadow uses source RPMs pulled from koji.fp.o, when building manually you should use a git url from pkgs.fedoraproject.org (e.g. git://pkgs.fedoraproject.org/kernel?#5394ba72a9d27667d10801685f71c003a7e205bc for kernel-3.9.6-301.fc19 )

• Never build 'real' (i.e. not scratch) packages with code that hasn't been committed to pkgs.fp.o.
  • Milestone composes (TCs & RCs for alpha, beta and GA) should only ever be done with real packages. Including scratch builds in test composes should be kept to an absolute minimum.

• Buildroots and dependencies should match primary as closely as possible. koji-shadow enforces this automatically, the only time you should build a package manually is when something is broken in a way which prevents koji-shadow from building it.
  • e.g. if eclipse-4.3.0-0.60.git7bf397.fc19 had a previous, broken-on-secondary version of eclipse in its buildroot, you should build eclipse-4.3.0-0.60.git7bf397.fc19 manually.
  • 'build a package manually' means submitting the build to ppc.koji.fp.o with e.g. ppc-koji build f19 git://blah
  • If not broken, build a single package with koji-shadow -c <config file> --build <nvr> <release>-build
    • This assumes <config file> specifies prefer_new and import_noarch, and you're using a koji-shadow from git which supports those options in conjunction with building a single NVR (previous versions of koji-shadow, using '--build <nvr>' caused it to match the buildroot exactly).
    • e.g.: /home/dwa/shadow/koji-shadow -c /home/dwa/shadow/f19.conf --build gnome-initial-setup-0.12-1.fc19 f19-build

Methodology

1. koji-stalk.py monitors fedmsg for completed build notifications from koji.fp.o.
   • koji-stalk.py is run on the hub out of a screen session by dwa
   • Use /home/dwa/shadow/f19.conf as the koji-shadow configuration file.
   • Use /home/dwa/shadow/koji-shadow as the koji-shadow binary.
   • ./koji-stalk.py -c /home/dwa/shadow/f19.conf --shadow /home/dwa/shadow/koji-shadow
2. When it receives a build notification, it kicks off koji-shadow for that NVR, and tags it into the -updates-candidate tag.
3. Tags are automatically synced with primary
   • Tags are synced out of cron on the hub by ppc-sync-tagged-primary.py
   • dgilmore also runs sync-tagged-primary against all arches after every stable push.
4. Signing the tags and mashing/pushing updates are all manually done.

• koji-stalk.py should be the primary way that builds happen.
• The koji-stalk logs should be monitored for failed builds to know where to investigate problems.
  • If an NVR fails to build, the output from koji-shadow is saved in http://ppc.koji.fedoraproject.org/reports/koji-stalk/.
  • koji-stalk.py will print any backlog for each distro, if any, every 10 minutes.
• koji-shadow should be occasionally manually run against the stable tags (and possibly updates-testing?) to catch any builds that might have been missed by the script (e.g. it crashed, or was stopped for system maintenance or to update the script).
  • stable tags are the release tag pre-GA (e.g. 'f19'), and the updates tags (e.g. 'f18-updates') post-GA.
  • koji-stalk.py builds ahead of updates-candidate, so if unknown builds are missed the occasional runs against stable or updates-testing will eventually, but not immediately, catch them.
  • koji-stalk prints its queues (if any) every 10 minutes in the log, should you need to stop the script to update it.

Compose & Build scripts

• Scripts for mashing & composing milestones, bleed repos, and nightly images are on my people page.
  • mash-milestone.sh mashes a repo for a particular release while enforcing strict signing, unlike buildbranched. Edit the DATE variable before running it.
  • compose.sh composes test releases. It uses the fedora-install-fedora.ks which points to the branched & bleed repos, and does not create source or debuginfo trees. Edit the DATE variable before running it.
  • compose-milestone.sh composes milestone releases. Ensure there's a $milestone-fedora-install-fedora.ks which points to a repo which was mashed with strict signing (such as what mash-milestone.sh creates). Edit the DATE and MILESTONE variables before running it.
  • make-bleed.sh creates a bleed repo of packages from the PKGS variable. Edit the PKGS variable before running it.
  • All of the above should be run from ppc-composer.

• koji-stalk.py is a script for monitoring fedmsg and kicking off builds as soon as they finish on primary.

• All build scripts (koji-stalk.py and koji-shadow) are currently run from the hub.

• All compose and mash scripts, pushing updates, and signing are done from the composer.

• koji-stalk.py and sigulsign_unsigned.py are available from the releng git repo.

TC & RC Composes

• TCs are Test Composes. They will never be formal releases so the only thing that makes them different from daily ISOs is the package list for the bleed repo is taken from primary's releng ticket for that release/milestone, e.g. https://fedorahosted.org/rel-eng/ticket/5623
  1. Grab the list of NVRs from the ticket, then edit make-bleed.sh and put that (space separated) list in the PKGS variable.
  2. To make life easier for you down the road, it's best to sign those builds now. Run sigulsign_unsigned.py --arch ppc -v fedora-19-secondary <space separated list of NVRs>, and take note of any builds which don't exist yet (then build them, and sign).
  3. Run make-bleed.sh -s ba094068 to build the bleed repos.
     • ba094068 is the key ID for fedora-19-secondary. As an argument, it is case-sensitive.
  4. Edit compose.sh to reflect the correct DATE label (e.g. f19-20130618-GA-TC5) and run it.

• RCs are Release Candidates. They may become formal releases, so we need to ensure that all packages in an RC are signed. RCs also have full source and debuginfo trees generated as well, so they take a little longer to make.
  1. Sign the entire tag for the release.
     • sigulsign_unsigned.py --arch ppc -v --write --tag f19 fedora-19-secondary
  2. Ensure the bleed repo only contains signed packages.
     • The following command should not return any output: rpm -q --qf="%{RSAHEADER}\n" -p /mnt/koji/mash/bleed-repo/*/*.rpm | grep -v ba094068
  3. Edit mash-milestone.sh to have the appropriate date label, then run it. Have a cup of coffee.
     • If mash-milestone.sh complains of unsigned packages, you may need to re-run sigulsign_unsigned.py with the --write-all option then try again.
  4. Edit compose-milestone.sh for the appropriate milestone & date labels. the OLD variable will affect which tree is used in repodiff.
  5. Ensure that /mnt/data/kickstarts/f19-kickstarts/$MILESTONE-fedora-install-fedora.ks exists and has an appropriate package list & exclude list.
  6. Run compose-milestone.sh. The install tree will automatically be placed in /mnt/koji/stage. Be sure to check the repoclosure output before asking people to test.

Staging Releases

Sign the CHECKSUM file:

cat Fedora-18-ppc64-CHECKSUM > /tmp/sum
sigul sign-text -o /tmp/signed fedora-18-secondary /tmp/sum
sudo mv /tmp/signed Fedora-18-ppc64-CHECKSUM

Copy the ppc64 and source directories to:

• /mnt/koji/tree/releases/(version)/Fedora/ for GA releases
• /mnt/koji/tree/releases/test/(version)-(milestone)/Fedora for alpha & beta releases

Mashing & Pushing Updates

• The following tags are mashed and pushed regularly (1-3x/week):
  • f19-updates-testing
  • f19-updates
  • f18-updates-testing
  • f18-updates
• Due to ppc64p7, you must use a mock chroot of at least f18 otherwise ppc64p7 packages won't be included in the mash.
• All steps are performed on ppc-composer.

Step by step

Using f19-updates-testing as an example:

1. Sign the tag
   • sigulsign_unsigned.py --arch ppc -v --tag f19-updates-testing fedora-19-secondary
   • Use fedora-18-secondary for f18 tags, and fedora-17-secondary for f17 tags.
2. Mash the tag.
   • With the aliases below in place, simply run mash-19-updates-testing on ppc-composer.
   • If mashing complains of unsigned packages, run sigulsign_unsigned.py --arch ppc -v --tag f19-updates-testing --write-all fedora-19-secondary
3. Copy the mash to /mnt/koji/tree/updates/testing/19.
   • Again with the below aliases, switch to the directory the mash completed in (e.g. /mnt/koji/mash/updates-testing-f19-20130612/19-updates-testing) and run push-updates /mnt/koji/tree/updates/testing/19
   • For stable tags (i.e. f17-updates, f18-updates, and eventually f19-updates) they are pushed to /mnt/koji/tree/updates/(release).

alias mash-17-updates="mash -o /mnt/koji/mash/updates-f17-date +%Y%m%d 17-updates -f ~/comps/comps-f17.xml -c /etc/mash/mash.ppc.conf -p /mnt/koji/tree/updates/17"

alias mash-18-updates='mock -r fedora-18-ppc64 --chroot "mash -o /mnt/koji/mash/updates-f18-date +%Y%m%d 18-updates -f /tmp/comps-f18.xml -c /etc/mash/mash.ppc.conf -p /mnt/koji/tree/updates/18/"'

alias mash-18-updates-testing='mock -r fedora-18-ppc64 --chroot "mash -o /mnt/koji/mash/updates-testing-f18-date +%Y%m%d 18-updates-testing -f /tmp/comps-f18.xml -c /etc/mash/mash.ppc.conf -p /mnt/koji/tree/updates/testing/18/"'

alias mash-19-updates-testing='mock -r fedora-19-ppc64 --uniqueext=mash --chroot "mash -o /mnt/koji/mash/updates-testing-f19-date +%Y%m%d 19-updates-testing -f /tmp/comps-f19.xml -c /etc/mash/mash.ppc.conf -p /mnt/koji/tree/updates/testing/19/"'

alias mash-19-updates='mock -r fedora-19-ppc64 --uniqueext=mash --chroot "mash -o /mnt/koji/mash/updates-f19-date +%Y%m%d 19-updates -f /tmp/comps-f19.xml -c /etc/mash/mash.ppc.conf -p /mnt/koji/tree/updates/19/"'

push-updates() { sudo rsync -rlptDHhv --delay-updates ppc ppc64 source "$@" ;}

Creating New Mash Configs

• Your mash host (or chroot) needs to be at least Fedora 18 to properly mash ppc64p7 RPMs.

• Existing mash configurations are available from http://dwa.fedorapeople.org/secondary-releng/mash-configs/

• Using 19-updates.ppc.mash as an example, replace the configuration name (i.e. '19-updates'), the tag name ('f19-updates'), the key ID ('ba094068'), the repoviewurl, and repoviewtitle with the new information and save it in a new file in /etc/mash.

• You will also need to have mash.ppc.conf present and reference it in your mash command line (-c /etc/mash/mash.ppc.conf) to fetch packages from ppc.koji.

Branching

Draft Information
This section should be considered a draft and is not final or necessarily correct!

The examples in this section are for f20 branching from rawhide.

• Create empty updates & updates-testing repositories (in /mnt/koji/tree/updates{/testing}/20/{ppc,ppc64,source/SRPMS})
• Set the /mnt/koji/mash/branched link to point to the last pre-branch rawhide
• Update the mock configs on composer
  • for branched mocks, change 'fc19' to 'fc20', any repos that point to /mnt/koji/repos/f19-build to /mnt/koji/repos/f20-build
  • for rawhide mocks, change 'fc20' to 'fc21', etc.
• Update serverstatus to reflect F20 status
  • edit ppc-hub:/usr/local/bin/get-no-of-unsigned.py to reflect new key
  • edit ppc-hub:/usr/local/bin/get-no-of-unsigned.sh to include f20 & f20-updates-testing
  • edit ppc-hub:/usr/share/koji-web/scripts/serverstatus.chtml to include new tags/releases
  • edit ppc-hub:/usr/share/koji-web/scripts/index.py to include new tags/releases
    • backup index.py in case someone accidentally updates koji and overwrites it [fixme - can we break out the serverstatus stuff into a separate file?]
• Update /home/dwa/bin/ppc-sync-tagged-primary.py to remove EOL releases and add new ones.
• Update /home/dwa/bin/sync-override-tags.py to remove EOL releases and add new ones.
• Update nightly-compose.sh, compose-milestone.sh, compose.sh, mash-milestone.sh for new release number.
• Grab the latest fedora-install-fedora.ks (from the spin-kickstarts git repo) and make any necessary changes to it for our local copy in ppc-composer:/mnt/data/kickstarts/f20-kickstarts

Miscellaneous

mis-tagged shadowbuild packages

• Find packages that shouldn't be there:

  ppc-koji latest-pkg SHADOWBUILD-f16-build --all --quiet | grep fc18 | awk {'print $1'} > /tmp/mistagged-pkgs 

• Find the latest versions that should be.:

  for i in `awk -F- '{if (NF > 4) {print $1"-"$2"-"$3} else if (NF == 4) {print $1"-"$2} else {print $1}}' /tmp/mistagged-pkgs`; do ppc-koji latest-pkg f16-updates $i --quiet | awk {'print $1'}; done | grep -v Warning > /tmp/shouldbetagged-pkgs

  • 'grep -v Warning' is needed depending on your koji configuration, to supress the pkgurl deprecated option warning.
• Untag the offenders:

  ppc-koji untag-pkg SHADOWBUILD-f16-build `cat /tmp/mistagged-pkgs`

• Tag the right ones:

  ppc-koji tag-pkg SHADOWBUILD-f16-build `cat /tmp/shouldbetagged-pkgs`

• Regen the repo:

  ppc-koji regen-repo SHADOWBUILD-f16-build

sigul client setup

in ~/.sigul/client.conf:

[client]
bridge-hostname: secondary-signer
server-hostname: secondary-signer-server

in /etc/hosts (internal to phx):

10.5.124.145  sigul-bridge.ausil.us sigul-bridge secondary-signer

If your client is RHEL 6.4, in ~/.bashrc:

export NSS_HASH_ALG_SUPPORT=+MD5

Whenever your Fedora certificate expires, after putting the new one into place, run the following commands:

certutil -d ~/.sigul -D -n sigul-client-cert
sigul_setup_client

Infrastructure Maintenance

/mnt/koji

If /mnt/koji needs to be remounted, e.g. to migrate to a new filer, some downtime is unavoidable. The following should be done:

1. Disable all builders (ppc-koji disable-host ppc-builder{1..7}) and wait until their existing tasks have finished.
   • Any tasks that haven't yet been picked up by a builder will be queued and preserved when you restart everything.
2. Stop koji-stalk (since it logs into /mnt/koji/reports)
3. Stop the kojira and httpd services on the hub
4. Unmount /mnt/koji/repos from the createrepo hosts (from the hub, pssh -h ~dwa/hosts/createrepo umount /mnt/koji/repos)
5. Unmount /mnt/koji from all hosts (pssh -h ~dwa/hosts/all umount /mnt/koji)
   • If unmount fails for any reason, use lsof /mnt/koji as root to figure out what still has /mnt/koji open
6. Do the needful things & stuff for the specific maintenance / migration you're doing.
7. Remount /mnt/koji
8. Remount /mnt/koji/repos on createrepo hosts
9. Start httpd on the hub
10. Start kojira on the hub
11. Restart koji-stalk
12. Re-enable the builders
13. profit!