From Fedora Project Wiki
m (layout fix)
m (Add tracker)
 
(24 intermediate revisions by 2 users not shown)
Line 1: Line 1:
= Gating Rawhide - Single package updates =
= Gating Rawhide Packages =


== Summary ==
== Summary ==
We want to gate packages on test results before they can land in rawhide. This will reduce the amount of broken dependency, uninstallable packages and broken composes leading to a more stable rawhide as well as less work on the infrastructure and rel-eng teams to keep composes working.
We want to gate packages on test results before they can land in rawhide. This will reduce the amount of broken dependency, uninstallable packages and broken composes leading to a more stable rawhide as well as less work on the infrastructure and rel-eng teams to keep composes working.


This project will be split in two phases, at first only single package updates will be supported, in a second stage, we will add support for multi-packages updates.
This proposal is about the phase 1 of this project.


== Owner ==
== Owner ==
Line 18: Line 15:
People who are/will be involved in this:
People who are/will be involved in this:
* Coordinator: [[User:pingou|Pierre-Yves Chibon]]
* Coordinator: [[User:pingou|Pierre-Yves Chibon]]
* Bodhi: [[User:bowlofeggs|Randy Barlow]] and [[User:abompard|Aurélien Bompard]]
* Bodhi: [[User:bowlofeggs|Randy Barlow]], [[User:abompard|Aurélien Bompard]] and [[User:cverna|Clément Verna]]
* koji: [[User:mizdebsk|Mikolaj Izdebski]] and [[User:mikem|Mike McLean]]
* fedpkg/rpkg: [[User:cqi|Chenxiong ]] and [[User:lsedlar|Lubomír Sedlář]]
* Fedora CI: [[User:dperpeet|Dominik Perpeet]]
* Fedora CI: [[User:dperpeet|Dominik Perpeet]]
* Release notes owner: <!--- To be assigned by docs team [[User:FASAccountName| Release notes owner name]] <email address> -->
* Release notes owner: <!--- To be assigned by docs team [[User:FASAccountName| Release notes owner name]] <email address> -->
Line 27: Line 26:
== Current status ==
== Current status ==
* Targeted release: N/A
* Targeted release: N/A
* Targeted time period: <To be updated week 10 2019>
* Last updated: <!-- this is an automatic macro — you don't need to change this line -->  {{REVISIONYEAR}}-{{REVISIONMONTH}}-{{REVISIONDAY2}}  
* Last updated: <!-- this is an automatic macro — you don't need to change this line -->  {{REVISIONYEAR}}-{{REVISIONMONTH}}-{{REVISIONDAY2}}  
<!-- After the change proposal is accepted by FESCo, tracking bug is created in Bugzilla and linked to this page  
<!-- After the change proposal is accepted by FESCo, tracking bug is created in Bugzilla and linked to this page  
Line 35: Line 35:
ON_QA -> change is code completed and could be tested in the Beta release (optionally by QA)
ON_QA -> change is code completed and could be tested in the Beta release (optionally by QA)
CLOSED as NEXTRELEASE -> change is completed and verified and will be delivered in next release under development
CLOSED as NEXTRELEASE -> change is completed and verified and will be delivered in next release under development
* Tracker bug: <will be assigned by the Wrangler>
-->
-->
* Tracker bug: [https://bugzilla.redhat.com/show_bug.cgi?id=1698501 #1698501]


== Detailed Description ==
== Detailed Description ==


Querying the Bodhi database, it was revealed that 95% of all our updates involve a single package.
Rawhide is an unique place in the Fedora eco-system. It is the only place where large changes and rebuilds can be done. In stable Fedora releases, the rebuilds of large sets of packages is discouraged by the packaging guidelines, so the ratio of number of updates in bodhi which a single build vs multiple builds (95% to 5%) is not reflective of the reality of the rawhide ecosystem.
 
To be exhaustive, here are the exact number found:
 
Of all time:


  Single build updates: 123,657 (94.1%)
The basic principal of this proposal is to provide an environment in which packages can be built and tested without affecting other packages.
  Multi  build updates:   7,766 ( 5.9%)
 
  Total:                131,423


Per release:
When considering gating rawhide package updates on test results, we need to consider two workflows: single build updates, multi-builds updates. For single build, the easiest solution is to provide a dedicated koji tag in which these packages are built and where they will wait for their tests to pass before they can enter the buildroot. For multi-builds, the solution will be to rely on side-tags in koji. These side-tags are basically tags created by the packager and available to them to do their work, in this case rebuild all the packages desired. The side-tag can then sent to the test system as being one unit of change.


  Fedora 29:
To use a, now well-known, analogy, a single-build update is like sending a commit to a mailing list, it waits there to be reviewed and tested before being merged into the main repository. Multi-build updates are like pull-requests, they can contain one or more builds and which are reviewed and tested all together as one change before being merged.
 
  Single:  4,675 (93.7%)
  Multi:     316 ( 6.3%)


  Fedora 28:
Within this proposal, we aim at building the infrastructure allowing to gate packages in rawhide. The goal is for packages to go through bodhi, be tested and if the tests pass, land in the rawhide buildroot as they do today. In the simplest case, the packager workflow will not be affected by this proposal, less simple situation will require adjustements to the packager workflow that we would like to try to be as minimal as possible.
 
  Single:  9,153 (94.5%)
  Multi:     536 ( 5.5%)


  EPEL 7:
We do not aim at making any tests mandatory as part of these proposals. Once packages gating is in place it will be up to the community and likely FESCo to decide if they would like to enforce some rules on all packages and which ones.
 
  Single: 12,664 (94.0%)
  Multi:     814 ( 6.0%)


Note that this proposal completes previous initiative such as [[Changes/NoMoreAlpha]].


When considering gating rawhide package updates on test results, we need to consider two workflows: single package updates, multi-package updates. This proposal only considers the single package updates workflow. [ Another proposal] will be submitted in the future to support the multi package updates workflow.
At first we want to build the system allowing to gating rawhide, packagers will '''opt-in into gating''' [https://docs.pagure.org/greenwave/package-specific-policies.html using greenwave's opt-in gating mechanism] as they want. We will also offer a way to '''opt-out''' so packages depending on other packages can still be built without issues.
We do not aim at making any tests mandatory as part of these proposals. Once the two proposals have been deployed it will be up to the community and likely FESCo to decide if they would like to enforce some rules on all packages and which ones.


=== Workflows ===
=== Workflows ===
Line 80: Line 60:
<gallery>
<gallery>
Single_package_GatingRawhide_bodhi.png|Single package update with gating
Single_package_GatingRawhide_bodhi.png|Single package update with gating
GatingRawhide_MultiplePkgs_bodhi.png|Multi-packages update with gating
</gallery>
</gallery>


Line 89: Line 70:


== Scope ==
== Scope ==
The scope of this work is adding a mechanism to gate single package updates before they enter the rawhide buildroot (and thus become accessible to others) so broken packages are kept out and cannot affect other packages or the compose until they are fixed by their maintainers.
The scope of this work is adding a mechanism to gate builds/updates before they enter the rawhide buildroot (and thus become accessible to others) so broken packages are kept out and cannot affect other packages or the compose until they are fixed by their maintainers.


'''The CI system, the tests and the decision on which tests are used to gate upon are out of scope for the present document.'''  
'''The CI system, the tests and the decision on which tests are used to gate upon are out of scope for the present document.'''  




* Proposal owners: pingou will be coordinating the work among the different stack holder
* Proposal owners: pingou will be coordinating the work among the different stack holders


* Other developers:  
* Other developers:  
** Bodhi developers: Implement the needed changes
** Bodhi developers: Implement the needed changes
** infrastructure: deploy the new version of bodhi and the new koji plugin
** koji developers: Implement the needed changes
** fedpkg/rpkg developers: Implement the needed changes
** infrastructure: deploy the new version of bodhi and koji


* Release engineering: [https://pagure.io/releng/issues #Releng issue number] (a check of an impact with Release Engineering is needed)
* Release engineering: [https://pagure.io/releng/issue/8183 #8183]  


* Policies and guidelines: Once the entire system is in place, FESCo may want to set a policy on distribution-wide gating (ie: tests that would be enforced for all packages in the distributions). This is however out of scope of this proposal.
* Policies and guidelines: Once the entire system is in place, FESCo may want to set a policy on distribution-wide gating (ie: tests that would be enforced for all packages in the distributions). This is however out of scope of this proposal.


* Trademark approval: N/A (not needed for this Change)
* Trademark approval: N/A (not needed for this Change)
=== Application impacted ===
=== Application impacted ===
==== Bodhi ====
==== Bodhi ====


* Single package update
* A message-bus listener will automatically create a bodhi update for each package built in the rawhide-gated tag.
** We will need to write a message-bus listener that will automatically create a bodhi update for each package built in the rawhide-gated tag.
* A message-bus listener will automatically push bodhi updates created for rawhide that have passed CI (the decision being announced by Greenwave).
** We will need to write a message-bus listener to automatically push bodhi updates created for rawhide that have passed CI (the decision being announced by Greenwave).
* A message-bus listener will automatically comment on bodhi update about update in greenwave's decision
** This will notify users about the test results and the corresponding gating status.
* bodhi will support creating a new update from a side-tag instead of a list of builds.
 
==== Koji ====
* A koji plugin will allow packagers to create new side-tags
* How side-tags are created will be optimized so they do not take hours to be available to the user requesting them


* Bodhi will comment on the updates when greenwave's decision about an update is known
==== fedpkg/rpkg ====
** This will notify users about the test results and the corresponding gating status.
* The command: `fedpkg side-tag create` will be added to create a new side-tag in koji for this user using the new plugin
* The command: `fedpkg side-tag merge` will be added to create a bodhi update corresponding from this side-tag
** Alternatively: Expand the existing `fedpkg update` command to add support for creating a new update from a side-tag


==== Greenwave / WaiverDB ====
==== Greenwave / WaiverDB ====
Line 126: Line 118:
== How To Test ==
== How To Test ==


=== Single package update ===
=== Single build ===


# Add tests to a package you maintain in Fedora using the [[ CI/Quick_Start_Guide ]] documentation (see the full [[CI|CI documentation ]] for more information).
==== With opting in into gating ====
# Add tests to a package you maintain in Fedora using the [https://docs.fedoraproject.org/en-US/ci/quick-start-guide/ ci/quick-start-guide] documentation (see the full [https://docs.fedoraproject.org/en-US/ci/ CI documentation] for more information).
# Make your package gated on some tests using [https://docs.pagure.org/greenwave/package-specific-policies.html package-specific policies] (in a `gating.yaml`)
# Update your package in rawhide
# Update your package in rawhide
# Ensure that the bodhi update is automatically created
# Ensure that the bodhi update is automatically created
# Check that the tests have properly ran (see the Tests tab in the bodhi update)
# Check that the tests have properly ran (see the Tests tab in the bodhi update)
# Ensure that the bodhi update went through to rawhide once the tests passed
# Ensure that the bodhi update went through to rawhide once the tests passed
If one or more tests failed and you wish to waive them
# Waive the failing tests using `bodhi updates waive <id>` see `bodhi updates waive --help` for more information
==== Without opting in into gating ====
# Ensure your package does not have a `gating.yaml` in its dist-git repository
# Build your package in rawhide as you do today
# Ensure that the bodhi update is automatically created
# Ensure that the bodhi update goes through to rawhide automatically
=== Multi-builds ===
# Add tests to a package you maintain in Fedora using the [https://docs.fedoraproject.org/en-US/ci/quick-start-guide/ ci/quick-start-guide] documentation (see the full [https://docs.fedoraproject.org/en-US/ci/ CI documentation] for more information).
# Make your package gated on some tests using [https://docs.pagure.org/greenwave/package-specific-policies.html package-specific policies] (in a `gating.yaml`)
# Request a new side-tag in koji for your rebuilds
# Update your packages in this side-tag
# Request for this side-tag to be merged either using fedpkg or bodhi's UI (create a new update from a side-tag)
# Check that the tests have properly ran (see the Tests tab in the bodhi update)
# Ensure that the bodhi update went through to rawhide once the tests passed
# Check that the side-tag was correctly cleaned afterward
==== Without opting in into gating ====
# Ensure all your packages do not have a `gating.yaml` in its dist-git repository
# Build your packages in rawhide as you do today
# Ensure that the bodhi updates are automatically created
# Ensure that the bodhi updates go through to rawhide automatically


== User Experience ==
== User Experience ==


=== Single package update ===
=== Musts ===
 
Here below are the list of requirements for this proposal:
* Packagers must be notified when the update corresponding to their build(s) passes or fails CI
* Packagers must be able to see what is blocking an update (ie missing requirements/test results)
* Packagers must be able to re-trigger a test run (via either an UI or a CLI)
* Packagers must be able to waive false-negative (via either an UI or a CLI)
 
=== Nice to have ===
 
Here below is a list of ideas that would make the user experience more friendly but will not be part of the initial release:
* Packagers would like to follow the progress of the tests
** This could be implemented by notifying the packagers when a test run starts and provide an URL where they could see and follow them
* Packagers would like to be able to re-trigger a test run from an UI and a CLI
* Packagers would like to be able to waive false-negative from an UI and a CLI
* Packagers would like the ability to transform a bodhi update automatically created for a single build into a side-tag-specifi bodhi update
** This would allow the workflow where a packager makes a build, which fails CI because another package needs to be rebuilt as well. The packager can then convert the bodhi update into a side-tag based update, this would create the corresponding side-tag and move the existing build into it, allowing the packager to do the missing rebuilds before telling bodhi to process the update. This is entirely doable by hand, which is why this feature while convenient is considered a nice to have
* Packagers would like fedpkg chain-build to create the corresponding side-tag automatically
 
 
=== Single build update ===


     fedpkg clone rpms/foobar
     fedpkg clone rpms/foobar
Line 148: Line 189:
failed tests if needed.
failed tests if needed.


=== Multi package update with single package update gating ===
 
=== Multi builds with gating ===


     fedpkg clone rpms/foobar
     fedpkg clone rpms/foobar
     cd foobar/
     cd foobar/
    fedpkg side-tag create foo1.2
     vim foobar.spec
     vim foobar.spec
     git commit -am "Update of foobar to update foo to 1.2"
     git commit -am "Update of foobar to update foo to 1.2"
     fedpkg push
     fedpkg push
     fedpkg build --target=rawhide-candidate
     fedpkg build --target=user_foo1.2
     cd ..
     cd ..
     fedpkg clone rpms/foobaz
     fedpkg clone rpms/foobaz
Line 162: Line 205:
     git commit -am "Update of foobaz to update foo to 1.2"
     git commit -am "Update of foobaz to update foo to 1.2"
     fedpkg push
     fedpkg push
     fedpkg build --target=rawhide-candidate
     fedpkg build --target=user_foo1.2
     cd ..
     cd ..
     fedpkg clone rpms/foo
     fedpkg clone rpms/foo
Line 169: Line 212:
     git commit -am "Update foo to 1.2"
     git commit -am "Update foo to 1.2"
     fedpkg push
     fedpkg push
     fedpkg build --target=rawhide-candidate
     fedpkg build --target=user_foo1.2
 
    fedpkg side-tag merge foo1.2
Users are practically by-passing the gating process by specifying a specific build target.




Line 177: Line 219:


* Bodhi
* Bodhi
** bus listener to automatically create updates when a build happens in rawhide
* koji
** bus listener to automatically push updates who passed gating according to greenwave
* fedpkg
** comment on updates about gating status change/announce to notify packagers about it


== Contingency Plan ==
== Contingency Plan ==
Line 201: Line 242:
-->
-->


[[Category:ChangePageIncomplete]]
 
<!-- When your change proposal page is completed and ready for review and announcement -->
<!-- When your change proposal page is completed and ready for review and announcement -->
<!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler -->
<!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler -->
<!-- The Wrangler announces the Change to the devel-announce list and changes the category to Category:ChangeAnnounced (no action required) -->  
<!-- The Wrangler announces the Change to the devel-announce list and changes the category to Category:ChangeAnnounced (no action required) -->  
<!-- After review, the Wrangler will move your page to Category:ChangeReadyForFesco... if it still needs more work it will move back to Category:ChangePageIncomplete-->
<!-- After review, the Wrangler will move your page to Category:ChangeReadyForFesco... if it still needs more work it will move back to Category:ChangePageIncomplete-->
 
[[Category:ChangeAcceptedF31]]
[[Category:SystemWideChange]]
[[Category:SystemWideChange]]

Latest revision as of 13:57, 10 April 2019

Gating Rawhide Packages

Summary

We want to gate packages on test results before they can land in rawhide. This will reduce the amount of broken dependency, uninstallable packages and broken composes leading to a more stable rawhide as well as less work on the infrastructure and rel-eng teams to keep composes working.


Owner

People who are/will be involved in this:

Current status

  • Targeted release: N/A
  • Targeted time period: <To be updated week 10 2019>
  • Last updated: 2019-04-10
  • Tracker bug: #1698501

Detailed Description

Rawhide is an unique place in the Fedora eco-system. It is the only place where large changes and rebuilds can be done. In stable Fedora releases, the rebuilds of large sets of packages is discouraged by the packaging guidelines, so the ratio of number of updates in bodhi which a single build vs multiple builds (95% to 5%) is not reflective of the reality of the rawhide ecosystem.

The basic principal of this proposal is to provide an environment in which packages can be built and tested without affecting other packages.

When considering gating rawhide package updates on test results, we need to consider two workflows: single build updates, multi-builds updates. For single build, the easiest solution is to provide a dedicated koji tag in which these packages are built and where they will wait for their tests to pass before they can enter the buildroot. For multi-builds, the solution will be to rely on side-tags in koji. These side-tags are basically tags created by the packager and available to them to do their work, in this case rebuild all the packages desired. The side-tag can then sent to the test system as being one unit of change.

To use a, now well-known, analogy, a single-build update is like sending a commit to a mailing list, it waits there to be reviewed and tested before being merged into the main repository. Multi-build updates are like pull-requests, they can contain one or more builds and which are reviewed and tested all together as one change before being merged.

Within this proposal, we aim at building the infrastructure allowing to gate packages in rawhide. The goal is for packages to go through bodhi, be tested and if the tests pass, land in the rawhide buildroot as they do today. In the simplest case, the packager workflow will not be affected by this proposal, less simple situation will require adjustements to the packager workflow that we would like to try to be as minimal as possible.

We do not aim at making any tests mandatory as part of these proposals. Once packages gating is in place it will be up to the community and likely FESCo to decide if they would like to enforce some rules on all packages and which ones.

Note that this proposal completes previous initiative such as Changes/NoMoreAlpha.


Workflows

Benefit to Fedora

  • As more packagers opt-into gating and add tests to their packages, rawhide becomes more stable
  • More stable rawhide will lead to easier composes and a smoother release process
  • Ability to use chain-build in rawhide and stable releases

Scope

The scope of this work is adding a mechanism to gate builds/updates before they enter the rawhide buildroot (and thus become accessible to others) so broken packages are kept out and cannot affect other packages or the compose until they are fixed by their maintainers.

The CI system, the tests and the decision on which tests are used to gate upon are out of scope for the present document.


  • Proposal owners: pingou will be coordinating the work among the different stack holders
  • Other developers:
    • Bodhi developers: Implement the needed changes
    • koji developers: Implement the needed changes
    • fedpkg/rpkg developers: Implement the needed changes
    • infrastructure: deploy the new version of bodhi and koji
  • Release engineering: #8183
  • Policies and guidelines: Once the entire system is in place, FESCo may want to set a policy on distribution-wide gating (ie: tests that would be enforced for all packages in the distributions). This is however out of scope of this proposal.
  • Trademark approval: N/A (not needed for this Change)

Application impacted

Bodhi

  • A message-bus listener will automatically create a bodhi update for each package built in the rawhide-gated tag.
  • A message-bus listener will automatically push bodhi updates created for rawhide that have passed CI (the decision being announced by Greenwave).
  • A message-bus listener will automatically comment on bodhi update about update in greenwave's decision
    • This will notify users about the test results and the corresponding gating status.
  • bodhi will support creating a new update from a side-tag instead of a list of builds.

Koji

  • A koji plugin will allow packagers to create new side-tags
  • How side-tags are created will be optimized so they do not take hours to be available to the user requesting them

fedpkg/rpkg

  • The command: fedpkg side-tag create will be added to create a new side-tag in koji for this user using the new plugin
  • The command: fedpkg side-tag merge will be added to create a bodhi update corresponding from this side-tag
    • Alternatively: Expand the existing fedpkg update command to add support for creating a new update from a side-tag

Greenwave / WaiverDB

Nothing should change for these tools but we will have to confirm this and test in staging that they behave as expected.

CI system

Nothing should change for the CI system but we will have to confirm this and test in staging that they behave as expected.

Upgrade/compatibility impact

N/A

How To Test

Single build

With opting in into gating

  1. Add tests to a package you maintain in Fedora using the ci/quick-start-guide documentation (see the full CI documentation for more information).
  2. Make your package gated on some tests using package-specific policies (in a gating.yaml)
  3. Update your package in rawhide
  4. Ensure that the bodhi update is automatically created
  5. Check that the tests have properly ran (see the Tests tab in the bodhi update)
  6. Ensure that the bodhi update went through to rawhide once the tests passed

If one or more tests failed and you wish to waive them

  1. Waive the failing tests using bodhi updates waive <id> see bodhi updates waive --help for more information

Without opting in into gating

  1. Ensure your package does not have a gating.yaml in its dist-git repository
  2. Build your package in rawhide as you do today
  3. Ensure that the bodhi update is automatically created
  4. Ensure that the bodhi update goes through to rawhide automatically


Multi-builds

  1. Add tests to a package you maintain in Fedora using the ci/quick-start-guide documentation (see the full CI documentation for more information).
  2. Make your package gated on some tests using package-specific policies (in a gating.yaml)
  3. Request a new side-tag in koji for your rebuilds
  4. Update your packages in this side-tag
  5. Request for this side-tag to be merged either using fedpkg or bodhi's UI (create a new update from a side-tag)
  6. Check that the tests have properly ran (see the Tests tab in the bodhi update)
  7. Ensure that the bodhi update went through to rawhide once the tests passed
  8. Check that the side-tag was correctly cleaned afterward

Without opting in into gating

  1. Ensure all your packages do not have a gating.yaml in its dist-git repository
  2. Build your packages in rawhide as you do today
  3. Ensure that the bodhi updates are automatically created
  4. Ensure that the bodhi updates go through to rawhide automatically

User Experience

Musts

Here below are the list of requirements for this proposal:

  • Packagers must be notified when the update corresponding to their build(s) passes or fails CI
  • Packagers must be able to see what is blocking an update (ie missing requirements/test results)
  • Packagers must be able to re-trigger a test run (via either an UI or a CLI)
  • Packagers must be able to waive false-negative (via either an UI or a CLI)

Nice to have

Here below is a list of ideas that would make the user experience more friendly but will not be part of the initial release:

  • Packagers would like to follow the progress of the tests
    • This could be implemented by notifying the packagers when a test run starts and provide an URL where they could see and follow them
  • Packagers would like to be able to re-trigger a test run from an UI and a CLI
  • Packagers would like to be able to waive false-negative from an UI and a CLI
  • Packagers would like the ability to transform a bodhi update automatically created for a single build into a side-tag-specifi bodhi update
    • This would allow the workflow where a packager makes a build, which fails CI because another package needs to be rebuilt as well. The packager can then convert the bodhi update into a side-tag based update, this would create the corresponding side-tag and move the existing build into it, allowing the packager to do the missing rebuilds before telling bodhi to process the update. This is entirely doable by hand, which is why this feature while convenient is considered a nice to have
  • Packagers would like fedpkg chain-build to create the corresponding side-tag automatically


Single build update

   fedpkg clone rpms/foobar
   cd foobar/
   vim foobar.spec
   git commit -am "Git commit message"
   fedpkg build
   

If the CI tests pass, the package will land in rawhide, if they fail the user will be able to go to bodhi to see what failed and why as well as waiving the failed tests if needed.


Multi builds with gating

   fedpkg clone rpms/foobar
   cd foobar/
   fedpkg side-tag create foo1.2
   vim foobar.spec
   git commit -am "Update of foobar to update foo to 1.2"
   fedpkg push
   fedpkg build --target=user_foo1.2
   cd ..
   fedpkg clone rpms/foobaz
   cd foobaz
   vim foobaz.spec
   git commit -am "Update of foobaz to update foo to 1.2"
   fedpkg push
   fedpkg build --target=user_foo1.2
   cd ..
   fedpkg clone rpms/foo
   cd foo
   vim foo.spec
   git commit -am "Update foo to 1.2"
   fedpkg push
   fedpkg build --target=user_foo1.2
   fedpkg side-tag merge foo1.2


Dependencies

  • Bodhi
  • koji
  • fedpkg

Contingency Plan

We can keep doing rawhide as we are doing it today.

Documentation

To be written/updated

List of documentation pages to update:

Release Notes