From Fedora Project Wiki
This page is a draft only
It is still under construction and content may change. Do not rely on the information on this page.

Gating Rawhide - Multi package updates

Summary

We want to gate packages on test results before they can land in rawhide. This will reduce the amount of broken dependency, uninstallable packages and broken composes leading to a more stable rawhide as well as less work on the infrastructure and rel-eng teams to keep composes working.

This project will be split in two phases, at first only single package updates will be supported, in a second stage, we will add support for multi-packages updates.

This proposal is about the phase 2 of this project.

Owner

People who are/will be involved in this:

Current status

  • Targeted release: N/A
  • Last updated: 2019-02-28
  • Tracker bug: <will be assigned by the Wrangler>

Detailed Description

Querying the Bodhi database, it was revealed that 95% of all our updates involve a single package.

To be exhaustive, here are the exact number found:

Of all time:

  Single build updates: 123,657 (94.1%)
  Multi  build updates:   7,766 ( 5.9%)
 
  Total:                131,423

Per release:

  Fedora 29:
 
  Single:  4,675 (93.7%)
  Multi:     316 ( 6.3%)
  Fedora 28:
 
  Single:  9,153 (94.5%)
  Multi:     536 ( 5.5%)
  EPEL 7:
 
  Single: 12,664 (94.0%)
  Multi:     814 ( 6.0%)


When considering gating rawhide package updates on test results, we need to consider two workflows: single package updates, multi-package updates. This proposal only considers the multi package updates workflow. Another proposal was submitted to support the single package updates workflow.

At first we want to build the system allowing to gating rawhide, packagers will opt-in into gating using greenwave's opt-in gating mechanism as they want.

We do not aim at making any tests mandatory as part of these proposals. Once the two proposals have been deployed it will be up to the community and likely FESCo to decide if they would like to enforce some rules on all packages and which ones.

Workflows

Benefit to Fedora

  • As more packagers opt-into gating and add tests to their packages, rawhide becomes more stable
  • More stable rawhide will lead to easier composes and a smoother release process
  • Ability to use chain-build in rawhide and stable releases

Scope

The scope of this work is adding a mechanism to gate multi packages updates before they enter the rawhide buildroot (and thus become accessible to others) so broken packages are kept out and cannot affect other packages or the compose until they are fixed by their maintainers.

The CI system, the tests and the decision on which tests are used to gate upon are out of scope for the present document.


  • Proposal owners: pingou will be coordinating the work among the different stack holder
  • Other developers:
    • Bodhi developers: Implement the needed changes
    • Koji developers: Implement the needed changes
    • fedpkg developers: Implement the needed changes
    • infrastructure: deploy the new version of bodhi and the new koji plugin
  • Policies and guidelines: Once the system is in place, FESCo may want to set a policy on distribution-wide gating (ie: tests that would be enforced for all packages in the distributions). This is however out of scope of this proposal.
  • Trademark approval: N/A (not needed for this Change)

Application impacted

Bodhi

  • Multi package update
    • We will need to allow bodhi to create a new update from a specified side-tag instead of a list of builds.
    • We will need to write a message-bus listener that, upon greenwave notifications about an update created from a side-tag passing CI, will move all of its builds into the rawhide-candidate tag and clean the side-tag created for this update.

Koji

  • We will need a koji plugin allowing packagers to create new side-tags
  • We will need to optimize how side-tags are created so they do not take hours to be available to the user requesting them

fedpkg/rpkg

  • We will need to add support for the command: fedpkg side-tag create which will create a new side-tag in koji for this user using the new plugin
  • We will need to add support for the command: fedpkg side-tag merge which will call bodhi to ask it to create an update corresponding to this side-tag
    • Alternatively: Expand the existing fedpkg update command to add support for creating a new update from a side-tag

Greenwave / WaiverDB

Nothing should change for these tools but we will have to confirm this and test in staging that they behave as expected.

CI system

Nothing should change for the CI system but we will have to confirm this and test in staging that they behave as expected.

Other

  • We will need to write a script that regularly clean old side-tags left around

Upgrade/compatibility impact

N/A

How To Test

Multi-packages update

  1. Add tests to a package you maintain in Fedora using the CI/Quick_Start_Guide documentation (see the full CI documentation for more information).
  2. Request a new side-tag in koji for your rebuilds
  3. Update your packages in this side-tag
  4. Request for this side-tag to be merged either using fedpkg or bodhi's UI (create a new update from a side-tag)
  5. Check that the tests have properly ran (see the Tests tab in the bodhi update)
  6. Ensure that the bodhi update went through to rawhide once the tests passed
  7. Check that the side-tag was correctly cleaned afterward

User Experience

Single package update

   fedpkg clone rpms/foobar
   cd foobar/
   vim foobar.spec
   git commit -am "Git commit message"
   fedpkg build
   

If the CI tests pass, the package will land in rawhide, if they fail the user will be able to go to bodhi to see what failed and why as well as waiving the failed tests if needed.

Multi package update with gating

   fedpkg clone rpms/foobar
   cd foobar/
   fedpkg side-tag create foo1.2
   vim foobar.spec
   git commit -am "Update of foobar to update foo to 1.2"
   fedpkg push
   fedpkg build --target=user_foo1.2
   cd ..
   fedpkg clone rpms/foobaz
   cd foobaz
   vim foobaz.spec
   git commit -am "Update of foobaz to update foo to 1.2"
   fedpkg push
   fedpkg build --target=user_foo1.2
   cd ..
   fedpkg clone rpms/foo
   cd foo
   vim foo.spec
   git commit -am "Update foo to 1.2"
   fedpkg push
   fedpkg build --target=user_foo1.2
   fedpkg side-tag merge foo1.2

If the CI tests pass, the package will land in rawhide, if they fail the user will be able to go to bodhi to see what failed and why as well as waiving the failed tests if needed.

Dependencies

  • Bodhi
    • add support for creating an update from a side-tag
    • add support for moving builds from a side-tag to the "main" tag and cleaning up this side-tag when greenwave announces that the tests passed
  • koji
    • new koji plugins to allow packagers to request a new side-tag
    • optimize how side-tags are created in koji
  • fedpkg
    • new command to request a side-tag in koji
    • new command to request a side-tag be merged

Contingency Plan

We keep on gating only single package updates.

This implies that it will not be possible at any point in the future to enforce some distribution wide tests as we would need to keep a way to opt-out of the gating workflow.

Documentation

To be written/updated

List of documentation pages to update:

Release Notes