User:Dwa

David Aquilina

I'm a release engineer at Red Hat, working on Fedora for POWER.

Contact Info

E-mail: dwa (at) redhat.com

IRC: 'dwa' on freenode, in #fedora-ppc

Secondary Arch Releng

Builds

Principles

• Never build an NVR before it's built on primary (koji.fedoraproject.org). The code used for a given NVR must match exactly between primary and secondary. koji-shadow uses source RPMs pulled from koji.fp.o, when building manually you should use a git url from pkgs.fedoraproject.org (e.g. git://pkgs.fedoraproject.org/kernel?#5394ba72a9d27667d10801685f71c003a7e205bc for kernel-3.9.6-301.fc19 )

• Never build 'real' (i.e. not scratch) packages with code that hasn't been committed to pkgs.fp.o.
  • Milestone composes (TCs & RCs for alpha, beta and GA) should only ever be done with real packages. Including scratch builds in test composes should be kept to an absolute minimum.

• Buildroots and dependencies should match primary as closely as possible. koji-shadow enforces this automatically, the only time you should build a package manually is when something is broken in a way which prevents koji-shadow from building it.
  • e.g. if eclipse-4.3.0-0.60.git7bf397.fc19 had a previous, broken-on-secondary version of eclipse in its buildroot, you should build eclipse-4.3.0-0.60.git7bf397.fc19 manually.
  • 'build a package manually' means submitting the build to ppc.koji.fp.o with e.g. ppc-koji build f19 git://blah
  • If not broken, build a single package with koji-shadow -c <config file> --build <nvr> <release>-build
    • This assumes <config file> specifies prefer_new and import_noarch, and you're using a koji-shadow from git which supports those options in conjunction with building a single NVR (previous versions of koji-shadow, using '--build <nvr>' caused it to match the buildroot exactly.

Methodology

1. koji-stalk.py monitors fedmsg for completed build notifications from koji.fp.o.
   • koji-stalk.py is run on the hub out of a screen session by dwa
   • Use /home/dwa/shadow/f19.conf as the koji-shadow configuration file.
   • Use /home/dwa/shadow/koji-shadow as the koji-shadow binary.
2. When it receives a build notification, it kicks off koji-shadow for that NVR, and tags it into the -updates-candidate tag.
3. Tags are automatically synced with primary
   • Tags are synced out of cron on the hub by ppc-sync-tagged-primary.py
   • dgilmore also runs sync-tagged-primary against all arches after every stable push.
4. Signing the tags and mashing/pushing updates are all manually done.

• koji-stalk.py should be the primary way that builds happen.
• The koji-stalk logs should be monitored for failed builds to know where to investigate problems.
  • If an NVR fails to build, the output from koji-shadow is saved in http://ppc.koji.fedoraproject.org/reports/koji-stalk/.
  • koji-stalk.py will print any backlog for each distro, if any, every 10 minutes.
• koji-shadow should be occasionally manually run against the stable tags (and possibly updates-testing?) to catch any builds that might have been missed by the script (e.g. it crashed, or was stopped for system maintenance or to update the script).
  • stable tags are the release tag pre-GA (e.g. 'f19'), and the updates tags (e.g. 'f18-updates') post-GA.
  • koji-stalk.py builds ahead of updates-candidate, so if unknown builds are missed the occasional runs against stable or updates-testing will eventually, but not immediately, catch them.
  • koji-stalk prints its queues (if any) every 10 minutes in the log, should you need to stop the script to update it.

Compose & Build scripts

• Scripts for mashing & composing milestones, bleed repos, and nightly images are on my people page.
  • mash-milestone.sh mashes a repo for a particular release while enforcing strict signing, unlike buildbranched. Edit the DATE variable before running it.
  • compose.sh composes test releases. It uses the fedora-install-fedora.ks which points to the branched & bleed repos, and does not create source or debuginfo trees. Edit the DATE variable before running it.
  • compose-milestone.sh composes milestone releases. Ensure there's a $milestone-fedora-install-fedora.ks which points to a repo which was mashed with strict signing (such as what mash-milestone.sh creates). Edit the DATE and MILESTONE variables before running it.
  • make-bleed.sh creates a bleed repo of packages from the PKGS variable. Edit the PKGS variable before running it.

• koji-stalk.py is a script for monitoring fedmsg and kicking off builds as soon as they finish on primary.

• All build scripts (koji-stalk.py and koji-shadow) are currently run from the hub.

• All compose and mash scripts, and pushing updates, are done from the composer.

TC & RC Composes

• TCs are Test Composes. They will never be formal releases so the only thing that makes them different from daily ISOs is the package list for the bleed repo is taken from primary's releng ticket for that release/milestone, e.g. https://fedorahosted.org/rel-eng/ticket/5623
  1. Grab the list of NVRs from the ticket, then edit make-bleed.sh and put that (space separated) list in the PKGS variable.
  2. To make life easier for you down the road, it's best to sign those builds now. Run sigulsign_unsigned.py --arch ppc -v fedora-19-secondary <space separated list of NVRs>, and take note of any builds which don't exist yet (then build them, and sign).
  3. Run make-bleed.sh -s ba094068 to build the bleed repos.
     • ba094068 is the key ID for fedora-19-secondary. As an argument, it is case-sensitive.
  4. Edit compose.sh to reflect the correct DATE label (e.g. f19-20130618-GA-TC5) and run it.

• RCs are Release Candidates. They may become formal releases, so we need to ensure that all packages in an RC are signed. RCs also have full source and debuginfo trees generated as well, so they take a little longer to make.
  1. Sign the entire tag for the release.
     • sigulsign_unsigned.py --arch ppc -v --write --tag f19 fedora-19-secondary
  2. Ensure the bleed repo only contains signed packages.
     • The following command should not return any output: rpm -q --qf="%{RSAHEADER}\n" -p /mnt/koji/mash/bleed-repo/*/*.rpm | grep -v ba094068
  3. Edit mash-milestone.sh to have the appropriate date label, then run it. Have a cup of coffee.
     • If mash-milestone.sh complains of unsigned packages, you may need to re-run sigulsign_unsigned.py with the --write-all option then try again.
  4. Edit compose-milestone.sh for the appropriate milestone & date labels. the OLD variable will affect which tree is used in repodiff.
  5. Ensure that /mnt/data/kickstarts/f19-kickstarts/$MILESTONE-fedora-install-fedora.ks exists and has an appropriate package list & exclude list.
  6. Run compose-milestone.sh. The install tree will automatically be placed in /mnt/koji/stage. Be sure to check the repoclosure output before asking people to test.

Staging Releases

Sign the CHECKSUM file:

cat Fedora-18-ppc64-CHECKSUM > /tmp/sum
sigul sign-text -o /tmp/signed fedora-18-secondary /tmp/sum
sudo mv /tmp/signed Fedora-18-ppc64-CHECKSUM

Copy the ppc64 and source directories to:

• /mnt/koji/tree/releases/(version)/Fedora/ for GA releases
• /mnt/koji/tree/releases/test/(version)-(milestone)/Fedora for alpha & beta releases

Mashing & Pushing Updates

• After signing the -updates or -updates-testing tag, they should be mashed & pushed. Use the following aliases to mash updates. Note that due to ppc64p7 support you must use a mock chroot of at least f18 for f18+ updates.

alias mash-17-updates="mash -o /mnt/koji/mash/updates-f17-date +%Y%m%d 17-updates -f ~/comps/comps-f17.xml -c /etc/mash/mash.ppc.conf -p /mnt/koji/tree/updates/17"

alias mash-18-updates='mock -r fedora-18-ppc64 --chroot "mash -o /mnt/koji/mash/updates-f18-date +%Y%m%d 18-updates -f /tmp/comps-f18.xml -c /etc/mash/mash.ppc.conf -p /mnt/koji/tree/updates/18/"'

alias mash-18-updates-testing='mock -r fedora-18-ppc64 --chroot "mash -o /mnt/koji/mash/updates-testing-f18-date +%Y%m%d 18-updates-testing -f /tmp/comps-f18.xml -c /etc/mash/mash.ppc.conf -p /mnt/koji/tree/updates/testing/18/"'

alias mash-19-updates-testing='mock -r fedora-branched-releng-ppc64 --uniqueext=mash --chroot "mash -o /mnt/koji/mash/updates-testing-f19-date +%Y%m%d 19-updates-testing -f /tmp/comps-f19.xml -c /etc/mash/mash.ppc.conf -p /mnt/koji/tree/updates/testing/19/"'

• Use the following alias for pushing updates to /mnt/koji/tree/updates/(release) (or /mnt/koji/tree/updates/testing/(release) as appropriate).

push-updates() { sudo rsync -rlptDHhv --delay-updates ppc ppc64 source "$@" ;}

mis-tagged shadowbuild packages

• Find packages that shouldn't be there:

  ppc-koji latest-pkg SHADOWBUILD-f16-build --all --quiet | grep fc18 | awk {'print $1'} > /tmp/mistagged-pkgs 

• Find the latest versions that should be.:

  for i in `awk -F- '{if (NF > 4) {print $1"-"$2"-"$3} else if (NF == 4) {print $1"-"$2} else {print $1}}' /tmp/mistagged-pkgs`; do ppc-koji latest-pkg f16-updates $i --quiet | awk {'print $1'}; done | grep -v Warning > /tmp/shouldbetagged-pkgs

  • 'grep -v Warning' is needed depending on your koji configuration, to supress the pkgurl deprecated option warning.
• Untag the offenders:

  ppc-koji untag-pkg SHADOWBUILD-f16-build `cat /tmp/mistagged-pkgs`

• Tag the right ones:

  ppc-koji tag-pkg SHADOWBUILD-f16-build `cat /tmp/shouldbetagged-pkgs`

• Regen the repo:

  ppc-koji regen-repo SHADOWBUILD-f16-build

sigul client setup

in ~/.sigul/client.conf:

[client]
bridge-hostname: secondary-signer
server-hostname: secondary-signer-server

in /etc/hosts (internal to phx):

10.5.124.145  sigul-bridge.ausil.us sigul-bridge secondary-signer

If your client is RHEL 6.4, in ~/.bashrc:

export NSS_HASH_ALG_SUPPORT=+MD5