- 1 Blockers / dependencies for redoing the Atomic workflow (try to prioritize these projects):
- 2 Things that are not blocked (add to the short term epic):
- 3 Functional Requirements
- 4 "Releng-o-tron" (for lack of a better name)
Blockers / dependencies for redoing the Atomic workflow (try to prioritize these projects):
- PDC implementation
- Implement a pdc-updater service that listens to fedmsg and updates PDC accordingly
- Map packages to each compose artifact
- Ability to automate signing
- Sigul needs development resources in order to fix existing bugs and add new features needed for automation.
- Server and bridge regularly crash when releng is signing rpms in batched mode https://bugzilla.redhat.com/show_bug.cgi?id=1272535
- Additional Koji artifacts
- Mashed repositories https://fedoraproject.org/wiki/Changes/KojiSignedRepos
- Taskotron & consolidating Tunir & Autocloud
- Implement tests for each compose artifact
- Listen to fedmsg and automatically trigger the appropriate tests
- Write tests for the installer, AMIs, etc.
- Integrate Atomic CI jobs http://git.app.eng.bos.redhat.com/git/atomic-ci-jobs.git
Things that are not blocked (add to the short term epic):
- better test case coverage
- better ostree management
- Single Atomic Host OSTree repository. https://fedorahosted.org/rel-eng/ticket/6125
- creating new ostrees starts with an empty tree and merges this into master if there are changes
- tagging, garbage collection, etc
- automated build, compose, test, release process for all artifacts: pxe2live, AMIs, Installer ISO, GCE (?)
- Automated build/compose pipeline
- Atomic Host build artifacts should only be refreshed/rebuilt when something in their package manifest changed/updated in a new compose
- Automated test pipeline
- All Atomic Host build artifacts should be automatically tested with an acceptable level of test coverage that we trust the resulting "successful" marked artifacts are ready for release
- Automated Release pipeline
- Once all of the artifacts pass the automated tests, they are then automatically signed and released.
- The Cloud download website should be automatically updated with the new content as it's released. https://stg.getfedora.org/en/cloud/download/atomic.html
- Updates for the components of Atomic Host's OSTrees should have automated tests in taskotron
- Updates for the OSTree of Atomic Host should be refreshed as soon as one of the components of it's manifest hit's the Fedora or Fedora Updates stable repository.
"Releng-o-tron" (for lack of a better name)
A fedmsg consumer that listens for changes, queries PDC for products that are affected, and triggers composes & automatic tests accordingly.
- Package maintainer commits package changes to git, builds it in koji, and submits an update to bodhi
- Taskotron automatically kicks off the RPM-based tests
- Updates are signed (currently by hand, which should eventually be automated)
- Bodhi "pushes" the update in a batch with others, which currently involves mashing a repository and composing the Atomic OSTrees.
- Ideally, we want to create koji tasks for both mash and ostree composes, and have bodhi simply send a single fedmsg with the batch of updates listed, and wait for releng-o-tron to do the coordination.
- Releng-o-tron picks up the fedmsg with all of the updates for the push
- Triggers mashing updates/updates-testing tasks in koji
- For each update in the batch, determine which artifacts are affected by querying PDC
- Upon completion of the repositories, trigger artifact composes against them
- Send a fedmsg upon completion. Really, it will store the results in a releng-o-tron "resultsdb" and the resultsdb is already instrumented to send a fedmsg.
- Task-o-tron picks up the fedmsg and tests the artifacts, and kicks off tests accordingly
- Tests OSTree, ISO, AMI, etc
- After Taskotron gives everything the "thumbs-up" (via fedmsg), releng-o-tron symlinks/rsyncs everything live, and then sends a fedmsg
- Bodhi picks up the fedmsg and "finishes" the push (sends emails, updates/closes bugs, comments on updates, etc)
- A question: do we then only "finish" a push after compose artifacts are completed? That's new territory if so. We currently only finish a push when both the mash and rpm-ostree are completed anyway. So it's already an "all or nothing" model.
- Packagers build packages in a 'rawhide-pending' tag, or something like that. We disallow building in the traditional rawhide tag directly to enforce this.
- We could potentially streamline the workflow of getting new upstream releases into rawhide and tested with anitya->new-hotness
- koji publishes a message when a new rawhide-pending build is built.
- qa taskotron notices this and runs some series of integrity checks on rawhide-pending. It publishes a message about this.
- If good, releng-o-tron notices qa-taskotron's result and it moves all the rawhide-pending builds into rawhide-proper (or whatever its called).
- If bad, it moves suspect builds from rawhide-pending into rawhide-badnewsbears.
- qa-taskotron can be set up to notice this as well, and start the integrity-check again.
- We can configure FMN to send emails to packagers when their packages go to rawhide-badnewsbears.
- releng-o-tron should know how to compose everything, ideally with a simple plugin framework that can allows us to easily add new artifacts.
- releng-o-tron should be the final gate-keeper of all of the bits
- It should perform/trigger the syncing of bits to the master mirror, instead of relying on a cron job to do so.
- We should ensure no fedmsgs ever get dropped (via gilmsg)
- Along with knowing how to compose OSTrees, it should also be able to handle merging, tagging, and archival/garbage collection of branches.
- Streamline the End Of Life process. With a single command/click of a button it should update the pkgdb, update bugs, carve out OSTrees, and archive content.
- Do we want to use the Bodhi web interface to list all of the "testing" compose artifacts and facilitate community testing/feedback/karma?
- We may need to rethink how we push bits to the users. With our current mirroring setup, there are some mirrors that only pull twice a day. If we're hoping to asynchronously push bits out all day, we might want to investigate a proper CDN setup.