From Fedora Project Wiki

Functional Requirements

  • Automated build/compose pipeline
    • Atomic Host build artifacts should only be refreshed/rebuilt when something in their package manifest changed/updated in a new compose
  • Automated test pipeline
    • All Atomic Host build artifacts should be automatically tested with an acceptable level of test coverage that we trust the resulting "successful" marked artifacts are ready for release
  • Automated Release pipeline
    • Once all of the artifacts pass the automated tests, they are then automatically signed and released.
  • Websites
  • Updates
    • Updates for the components of Atomic Host's OSTrees should have automated tests in taskotron
    • Updates for the OSTree of Atomic Host should be refreshed as soon as one of the components of it's manifest hit's the Fedora or Fedora Updates stable repository.
    • updates-testing OSTrees should be visible in Bodhi. This means that the automated test results will be shown, along with manual tests that can be run by community testers, as well as karma/test case/bug gating.

Impacted Tools

"Releng-o-tron" (for lack of a better name)

A fedmsg consumer that listens for changes, queries PDC for products that are affected, and triggers composes & automatic tests accordingly.

Updates/updates-testing Workflow

  • [Incomplete] Workflow diagram:
  • Package maintainer commits package changes to git, builds it in koji, and submits an update to bodhi
  • Taskotron automatically kicks off the RPM-based tests
  • Updates are signed (currently by hand, which should eventually be automated)
  • Bodhi "pushes" the update in a batch with others, which currently involves mashing a repository and composing the Atomic OSTrees.
    • Ideally, we want to create koji tasks for both mash and ostree composes, and have bodhi simply send a single fedmsg with the batch of updates listed, and wait for releng-o-tron to do the coordination.
  • Releng-o-tron picks up the fedmsg with all of the updates for the push
    • Triggers mashing updates/updates-testing tasks in koji
    • For each update in the batch, determine which artifacts are affected by querying PDC
    • Upon completion of the repositories, trigger artifact composes against them
    • Send a fedmsg upon completion. Really, it will store the results in a releng-o-tron "resultsdb" and the resultsdb is already instrumented to send a fedmsg.
  • Task-o-tron picks up the fedmsg and tests the artifacts, and kicks off tests accordingly
    • Tests OSTree, ISO, AMI, etc
  • After Taskotron gives everything the "thumbs-up" (via fedmsg), releng-o-tron symlinks/rsyncs everything live, and then sends a fedmsg
  • Bodhi picks up the fedmsg and "finishes" the push (sends emails, updates/closes bugs, comments on updates, etc)
    • A question: do we then only "finish" a push after compose artifacts are completed? That's new territory if so. We currently only finish a push when both the mash and rpm-ostree are completed anyway. So it's already an "all or nothing" model.

Rawhide Gating

  • Packagers build packages in a 'rawhide-pending' tag, or something like that. We disallow building in the traditional rawhide tag directly to enforce this.
    • We could potentially streamline the workflow of getting new upstream releases into rawhide and tested with anitya->
  • koji publishes a message when a new rawhide-pending build is built.
  • qa taskotron notices this and runs some series of integrity checks on rawhide-pending. It publishes a message about this.
  • If good, releng-o-tron notices qa-taskotron's result and it moves all the rawhide-pending builds into rawhide-proper (or whatever its called).
  • If bad, it moves suspect builds from rawhide-pending into rawhide-badnewsbears.
  • qa-taskotron can be set up to notice this as well, and start the integrity-check again.
  • We can configure FMN to send emails to packagers when their packages go to rawhide-badnewsbears.


  • releng-o-tron should know how to compose everything, ideally with a simple plugin framework that can allows us to easily add new artifacts.
  • releng-o-tron should be the final gate-keeper of all of the bits
  • It should perform/trigger the syncing of bits to the master mirror, instead of relying on a cron job to do so.
  • We should ensure no fedmsgs ever get dropped (via gilmsg)
  • Along with knowing how to compose OSTrees, it should also be able to handle merging, tagging, and archival/garbage collection of branches.
  • Automated cloning/creation of new releases. Handles interacting with PDC, pkgdb, bz, bodhi, etc.
  • Streamline the End Of Life process. With a single command/click of a button it should update the pkgdb, update bugs, carve out OSTrees, and archive content.


  • We may need to rethink how we push bits to the users. With our current mirroring setup, there are some mirrors that only pull twice a day. If we're hoping to asynchronously push bits out all day, we might want to investigate a proper CDN setup.

First steps for implementing this workflow for Atomic

Goals of the first 2 Phases

  • Lay the groundwork for the new Product Definition Center, and building automated workflows around it
  • Automate the composition of Atomic OSTrees (and other artfacts?) when dependent packages are updated (in rawhide? or just updates/testing?)
  • Integrate automated testing of the OSTrees into Task-o-tron
  • Initial implementation of "releng-o-tron" with a basic Atomic plugin
  • Farm out our Atomic composes to the Koji buildsystem
  • Break the Bodhi push process out into releng-o-tron plugins


* PDC needs to become Atomic-aware
  • pdc-updater plugin to listen for OSTree compose messages
  • pdc-updater should listen for fedora-atomic.git changes and kick off composes
* Create the PDC "component group"
  • How are we going to maintain this list?
  • So, there's a json "treefile" for atomic kept in a fedorahosted git repo. it defines at a high level what rpms go into atomic
  • pdc-updater should listen for when that git repo changes. it should parse that file, and then update the pdc "component group" for atomic.
  • that component group describes what should go into an atomic compose.
  • then, later, when one of those rpms gets rebuilt, releng-o-tron can query PDC for that list and realize that it should rebuild the ostree.
* Basic releng-o-tron implementation
  • Atomic plugin that listens for koji rawhide builds, fedora-atomic.git configuration changes, and bodhi mash fedmsgs
  • Query PDC for affected artifacts
  • Trigger composes in koji via pungi
  • Koji needs to know how to compose OSTrees
* Pungi needs to know how to trigger OSTree composes
;* Modify the composeinfo.json file with all of the packages in the OSTree
  • PDC will consume that file to know what went into each ostree repo.
  • Break up the Bodhi masher into releng-o-tron plugins for mashing & composing OSTrees
  • Port internal atomic-ci-jobs to Taskotron jobs

Phase 1 (drop-in)

  • Modify pdc-updater to maintain the Atomic "Component Group" from fedorahosted.
  • Drop in releng-o-tron to react to package & configuration changes and trigger composes
  • Implement/port Atomic CI tests to taskotron

Phase 2 (modify/refactor)

  • Modify pdc-updater to ingest atomic ostree composes.
  • Write koji composeOSTree plugin
  • Make pungi OSTree aware
  • Break the bodhi masher out into releng-o-tron plugins