User:Lmacken/AtomicWorkflowDraft

Functional Requirements

• Automated build/compose pipeline
  • Atomic Host build artifacts should only be refreshed/rebuilt when something in their package manifest changed/updated in a new compose
• Automated test pipeline
  • All Atomic Host build artifacts should be automatically tested with an acceptable level of test coverage that we trust the resulting "successful" marked artifacts are ready for release
• Automated Release pipeline
  • Once all of the artifacts pass the automated tests, they are then automatically signed and released.
• Websites
  • The Cloud download website should be automatically updated with the new content as it's released. https://stg.getfedora.org/en/cloud/download/atomic.html
• Updates
  • Updates for the components of Atomic Host's OSTrees should have automated tests in taskotron
  • Updates for the OSTree of Atomic Host should be refreshed as soon as one of the components of it's manifest hit's the Fedora or Fedora Updates stable repository.
  • updates-testing OSTrees should be visible in Bodhi. This means that the automated test results will be shown, along with manual tests that can be run by community testers, as well as karma/test case/bug gating.

Impacted Tools

• Product Definition Center
  • https://fedoraproject.org/wiki/Changes/ProductDefinitionCenter
  • Implement a pdc-updater service that listens to fedmsg and updates PDC accordingly Under development: https://github.com/fedora-infra/pdc-updater
  • Map packages to each compose artifact so our "releng-o-tron" tool can query it to determine what artifacts need to be recomposed when certain packages are updated
• Sigul
  • https://fedorahosted.org/sigul/
  • Ability to automate signing
  • Sigul needs development resources in order to fix existing bugs and add new features needed for automation.
  • Fix needs to be deployed: Server and bridge regularly crash when releng is signing rpms in batched mode https://bugzilla.redhat.com/show_bug.cgi?id=1272535
• Koji
  • Additional Koji artifacts
  • Mashed repositories https://fedoraproject.org/wiki/Changes/KojiSignedRepos Under development: https://github.com/Zyzyx/koji/tree/signed
  • OSTrees
• Taskotron, Tunir, Autocloud
  • Consolidation of these tools
  • Implement tests for each compose artifact
  • Listen to fedmsg and automatically trigger the appropriate tests
  • Write tests for the installer, AMIs, etc.
  • Integrate Atomic CI jobs http://git.app.eng.bos.redhat.com/git/atomic-ci-jobs.git
• Bodhi
  • Additional artifact support in Bodhi
  • Facilitate automated tests, community feedback, and test case/bug/karma based gating for OSTrees
• Creation of new "releng-o-tron" tool to handle kicking off composes as needed

"Releng-o-tron" (for lack of a better name)

A fedmsg consumer that listens for changes, queries PDC for products that are affected, and triggers composes & automatic tests accordingly.

Updates/updates-testing Workflow

• Workflow diagram: http://asciiflow.com/#0B5gxSIOofgpFUkw0ZHUxOWVTVUk
• Package maintainer commits package changes to git, builds it in koji, and submits an update to bodhi
• Taskotron automatically kicks off the RPM-based tests
• Updates are signed (currently by hand, which should eventually be automated)
• Bodhi "pushes" the update in a batch with others, which currently involves mashing a repository and composing the Atomic OSTrees.
  • Ideally, we want to create koji tasks for both mash and ostree composes, and have bodhi simply send a single fedmsg with the batch of updates listed, and wait for releng-o-tron to do the coordination.
• Releng-o-tron picks up the fedmsg with all of the updates for the push
  • Triggers mashing updates/updates-testing tasks in koji
  • For each update in the batch, determine which artifacts are affected by querying PDC
  • Upon completion of the repositories, trigger artifact composes against them
  • Send a fedmsg upon completion. Really, it will store the results in a releng-o-tron "resultsdb" and the resultsdb is already instrumented to send a fedmsg.
• Task-o-tron picks up the fedmsg and tests the artifacts, and kicks off tests accordingly
  • Tests OSTree, ISO, AMI, etc
• After Taskotron gives everything the "thumbs-up" (via fedmsg), releng-o-tron symlinks/rsyncs everything live, and then sends a fedmsg
• Bodhi picks up the fedmsg and "finishes" the push (sends emails, updates/closes bugs, comments on updates, etc)
  • A question: do we then only "finish" a push after compose artifacts are completed? That's new territory if so. We currently only finish a push when both the mash and rpm-ostree are completed anyway. So it's already an "all or nothing" model.

Rawhide Gating

• Packagers build packages in a 'rawhide-pending' tag, or something like that. We disallow building in the traditional rawhide tag directly to enforce this.
  • We could potentially streamline the workflow of getting new upstream releases into rawhide and tested with anitya->
• koji publishes a message when a new rawhide-pending build is built.
• qa taskotron notices this and runs some series of integrity checks on rawhide-pending. It publishes a message about this.
• If good, releng-o-tron notices qa-taskotron's result and it moves all the rawhide-pending builds into rawhide-proper (or whatever its called).
• If bad, it moves suspect builds from rawhide-pending into rawhide-badnewsbears.
• qa-taskotron can be set up to notice this as well, and start the integrity-check again.
• We can configure FMN to send emails to packagers when their packages go to rawhide-badnewsbears.

Requirements

• releng-o-tron should know how to compose everything, ideally with a simple plugin framework that can allows us to easily add new artifacts.
• releng-o-tron should be the final gate-keeper of all of the bits
• It should perform/trigger the syncing of bits to the master mirror, instead of relying on a cron job to do so.
• We should ensure no fedmsgs ever get dropped (via gilmsg)
• Along with knowing how to compose OSTrees, it should also be able to handle merging, tagging, and archival/garbage collection of branches.
• Automated cloning/creation of new releases. Handles interacting with PDC, pkgdb, bz, bodhi, etc.
• Streamline the End Of Life process. With a single command/click of a button it should update the pkgdb, update bugs, carve out OSTrees, and archive content.

Questions/Concerns

• We may need to rethink how we push bits to the users. With our current mirroring setup, there are some mirrors that only pull twice a day. If we're hoping to asynchronously push bits out all day, we might want to investigate a proper CDN setup.

Transitioning from the current "Two Week Atomic" flow

• Port push-two-week-atomic.py to a releng-o-tron plugin https://pagure.io/releng/blob/master/f/scripts/push-two-week-atomic.py
• Port mark-atomic-bad to releng-o-tron as well https://pagure.io/mark-atomic-bad/blob/master/f/bad-builds.json