From Fedora Project Wiki

Revision as of 18:23, 17 June 2013 by Dwa (talk | contribs) (Methodology)

David Aquilina

I'm a release engineer at Red Hat, working on Fedora for POWER.

Contact Info

E-mail: dwa (at)

IRC: 'dwa' on freenode, in #fedora-ppc

Secondary Arch Releng



  • Never build an NVR before it's built on primary ( The code used for a given NVR must match exactly between primary and secondary. koji-shadow uses source RPMs pulled from koji.fp.o, when building manually you should use a git url from (e.g. git:// for kernel-3.9.6-301.fc19 )
  • Never build 'real' (i.e. not scratch) packages with code that hasn't been committed to pkgs.fp.o.
    • Milestone composes (TCs & RCs for alpha, beta and GA) should only ever be done with real packages. Including scratch builds in test composes should be kept to an absolute minimum.
  • Buildroots and dependencies should match primary as closely as possible. koji-shadow enforces this automatically, the only time you should build a package manually is when something is broken in a way which prevents koji-shadow from building it.
    • e.g. if eclipse-4.3.0-0.60.git7bf397.fc19 had a previous, broken-on-secondary version of eclipse in its buildroot, you should build eclipse-4.3.0-0.60.git7bf397.fc19 manually.
    • 'build a package manually' means submitting the build to ppc.koji.fp.o with e.g. 'ppc-koji build f19 git://blah'


  1. monitors fedmsg for completed build notifications from koji.fp.o.
    • is currently run by dwa on the hub out of a screen session. it (technically the koji-shadow processes it spawns) must be run by someone who has admin privs on ppc.koji.fp.o
  2. When it receives a build notification, it kicks off koji-shadow for that NVR, and tags it into the -updates-candidate tag.
  3. Tags are synced with primary by the script, which is run out of cron on the hub by dwa just for ppc, and also manually by dgilmore for all arches after he does bodhi pushes.
  4. Signing the tags and mashing/pushing updates are all manually done.
  • should be the primary way that builds happen.
  • The koji-stalk logs should be monitored for failed builds to know where to investigate problems.
  • koji-shadow should be occasionally manually run against the stable tags (and possibly updates-testing?) to catch any builds that might have been missed by the script (e.g. it crashed, or was stopped for system maintenance or to update the script).
    • stable tags are the release tag pre-GA (e.g. 'f19'), and the updates tags (e.g. 'f18-updates') post-GA.
    • builds ahead of updates-candidate, so if unknown builds are missed the occasional runs against stable or updates-testing will eventually, but not immediately, catch them.
    • koji-stalk prints its queues (if any) every 10 minutes in the log, should you need to stop the script to update it.

Compose & Build scripts

  • Scripts for mashing & composing milestones, bleed repos, and nightly images are on my people page.
  • is a script for monitoring fedmsg and kicking off builds as soon as they finish on primary.

TC & RC Composes

  • TCs are Test Composes. They will never be formal releases so the only thing that makes them different from daily ISOs is the package list for the bleed repo is taken from primary's releng ticket for that release/milestone, e.g.
    • Grab the list of NVRs from the ticket, then edit and put that (space separated) list in the PKGS variable. Run to build the bleed repos, and take note of any builds which don't exist yet. Then edit so it has the appropriate labels and run it.
  • RCs are Release Candidates. They may become formal releases, so we need to ensure that all packages in an RC are signed. RCs also have full source and debuginfo trees generated as well, so they take a little longer to make.
    1. Sign the entire tag for the release.
    2. Edit to have the appropriate date label, then run it. Have a cup of coffee.
      • If complains of unsigned packages, you may need to re-run with the --write-all option then try again.
    3. Edit for the appropriate milestone & date labels. the OLD variable will affect which tree is used in repodiff.
    4. Ensure that /mnt/data/kickstarts/f19-kickstarts/$MILESTONE-fedora-install-fedora.ks exists and has an appropriate package list & exclude list.
    5. Run The install tree will automatically be placed in /mnt/koji/stage. Be sure to check the repoclosure output before asking people to test.

Staging Releases

Sign the CHECKSUM file:

cat Fedora-18-ppc64-CHECKSUM > /tmp/sum
sigul sign-text -o /tmp/signed fedora-18-secondary /tmp/sum
sudo mv /tmp/signed Fedora-18-ppc64-CHECKSUM

Copy the ppc64 and source directories to:

  • /mnt/koji/tree/releases/(version)/Fedora/ for GA releases
  • /mnt/koji/tree/releases/test/(version)-(milestone)/Fedora for alpha & beta releases

Handy Aliases

alias mash-17-updates="mash -o /mnt/koji/mash/updates-f17-`date +%Y%m%d` 17-updates -f ~/comps/comps-f17.xml -c /etc/mash/mash.ppc.conf -p /mnt/koji/tree/updates/17"
alias mash-18-updates='mock -r fedora-18-ppc64 --chroot "mash -o /mnt/koji/mash/updates-f18-`date +%Y%m%d` 18-updates -f /tmp/comps-f18.xml -c /etc/mash/mash.ppc.conf -p /mnt/koji/tree/updates/18/"'
alias mash-18-updates-testing='mock -r fedora-18-ppc64 --chroot "mash -o /mnt/koji/mash/updates-testing-f18-`date +%Y%m%d` 18-updates-testing -f /tmp/comps-f18.xml -c /etc/mash/mash.ppc.conf -p /mnt/koji/tree/updates/testing/18/"'
alias mash-19-updates-testing='mock -r fedora-branched-releng-ppc64 --uniqueext=mash --chroot "mash -o /mnt/koji/mash/updates-testing-f19-`date +%Y%m%d` 19-updates-testing -f /tmp/comps-f19.xml -c /etc/mash/mash.ppc.conf -p /mnt/koji/tree/updates/testing/19/"'
push-updates() { sudo rsync -rlptDHhv --delay-updates ppc ppc64 source "$@" ;}

mis-tagged shadowbuild packages

  • Find packages that shouldn't be there:
    ppc-koji latest-pkg SHADOWBUILD-f16-build --all --quiet | grep fc18 | awk {'print $1'} > /tmp/mistagged-pkgs 
  • Find the latest versions that should be.:
    for i in `awk -F- '{if (NF > 4) {print $1"-"$2"-"$3} else if (NF == 4) {print $1"-"$2} else {print $1}}' /tmp/mistagged-pkgs`; do ppc-koji latest-pkg f16-updates $i --quiet | awk {'print $1'}; done | grep -v Warning > /tmp/shouldbetagged-pkgs
    • 'grep -v Warning' is needed depending on your koji configuration, to supress the pkgurl deprecated option warning.
  • Untag the offenders:
    ppc-koji untag-pkg SHADOWBUILD-f16-build `cat /tmp/mistagged-pkgs`
  • Tag the right ones:
    ppc-koji tag-pkg SHADOWBUILD-f16-build `cat /tmp/shouldbetagged-pkgs`
  • Regen the repo:
    ppc-koji regen-repo SHADOWBUILD-f16-build

sigul client setup

in ~/.sigul/client.conf:

bridge-hostname: secondary-signer
server-hostname: secondary-signer-server

in /etc/hosts (internal to phx): sigul-bridge secondary-signer

If your client is RHEL 6.4, in ~/.bashrc: