From Fedora Project Wiki

(tweak 'what builds to include' a bit to differentiate between pre- and post- change deadline TCs)
(releng moved to pagure, replace trac links and references with pagure ones)
 
(13 intermediate revisions by 2 users not shown)
Line 1: Line 1:
{{header|qa}}
{{header|qa}}


This SOP explains the procedure for requesting composes from the [[ReleaseEngineering]] team. This task is usually performed by QA.
This SOP explains the procedure for requesting 'candidate' composes from the [[ReleaseEngineering]] team. This task is usually performed by QA.


== What composes should be requested when ==
== What composes should be requested when ==


There are three milestones in the Fedora release cycle: Alpha, Beta, and Final. For each milestone, a minimum of one release candidate (RC) must be produced. At least one test compose (TC) should also always be produced under normal circumstances.
There are three milestones in the Fedora release cycle: Alpha, Beta, and Final. For each milestone, a minimum of one 'candidate' (known in release engineering terms as a 'production compose') must be produced. Usually, there will be more than one.


The Fedora [[schedule]] specifies dates for the first test compose and for the first release candidate, for each milestone: the compose dates are listed on the release engineering schedule and usually also on the QA schedule. The requests should be filed on or (ideally) shortly prior to these dates. However, a release candidate can only be composed when all blocker bugs are addressed, and this requirement supersedes the schedule: see below for more on this.
In the normal course of events, 'nightly' composes are regularly produced by release engineering and validation testing is performed on these. There are two reasons to request a 'candidate' compose. The first is to allow testing of an update before it is pushed stable if it cannot be tested fully by updating an installed system (e.g. installer updates, or GNOME/Plasma updates to test the live images). The second is to produce a compose that can be released: 'nightly' composes have some properties that mean we cannot ship them as Alpha, Beta or Final releases, so a 'candidate' / 'production' compose must be built for release. You could refer to a 'candidate' of this type as a 'release candidate'.


The schedule for the next Fedora release, with the designated compose dates, can be found [http://rbergero.fedorapeople.org/schedules/f-{{FedoraVersionNumber|next}}/f-{{FedoraVersionNumber|next}}-releng-tasks.html here].
Only a 'candidate' composed '''after''' the [[Milestone freezes|milestone freeze]] ''and'' with '''all blocker bugs addressed''' can be released, and this requirement supersedes the schedule: see below for more on this.


=== Test composes ===
The schedule for the next Fedora release, with the milestone freeze dates, can be found [https://fedorapeople.org/groups/schedule/f-{{FedoraVersionNumber|next}}/f-{{FedoraVersionNumber|next}}-releng-tasks.html here].


A '''test compose''' is defined as a set of Fedora images built, from the current [[Releases/Branched|Branched]] stable tree ('''not''' including updates-testing), shortly prior to the Change Deadline (freeze) for one of the three Fedora milestones (Alpha, Beta and Final), for the purposes of performing [[:Category:Release_validation|release validation testing]]. It differs from a '''release candidate''' in that it is built before, not after, the Change Deadline and hence there is no possibility of its being declared gold and released as the Alpha, Beta or Final release.
=== Candidates for updates testing ===


Test composes beyond TC1 are done at the discretion of the QA team: there is no strict requirement for a certain number of test composes, or a strict trigger for building a new test compose. It usually makes sense to request a new test compose when significant changes to key components such as the installer, the kernel, X.org, the default bootloader and so on would be included: simply put, request a new test compose when it becomes clear that it may provide significantly different results to the current test compose.
Prior to the Fedora 24 cycle, there were 'test compose' and 'release candidate' composes. This is no longer the case. In the usual course of events, validation testing before the freeze date for each milestone is simply performed on nightly composes. 'Candidate' composes can be requested prior to the freeze date if they are required to test one or more updates before they can be pushed stable, but ideally this should be avoided to save unnecessary work on the release engineering team's part.
 
There is no requirement for blocker bugs to be addressed in order for a test compose to be produced. It is a good idea, however, to make a best effort before filing a TC compose request to ensure that the compose will succeed and result in images that do not fail immediately - if you know there is a package conflict in the repositories that will result in live image composes failing, or a showstopper bug which makes the installer crash at the first page, there is very little point in requesting a TC compose. It is better for TC1 to be a day or two late than for it to be on time but untestable.


=== Release candidates ===
=== Release candidates ===


A '''release candidate''' is defined as a set of Fedora images built, from the current [[Releases/Branched|Branched]] tree ('''not''' including updates-testing), after the Change Deadline (freeze) for one of the three Fedora milestones (Alpha, Beta and Final) and using a package set which is not known to contain any [[QA:SOP_blocker_bug_process|blocker bugs]], for the purposes of performing [[:Category:Release_validation|release validation testing]]. It differs from a '''test compose''' in that it is built after the Change Deadline and may be declared gold and released as the Alpha, Beta or Final release if it passes all validation tests.
A '''release candidate''' is defined as a set of Fedora images built, from the current [[Releases/Branched|Branched]] tree ('''not''' including updates-testing), after the [[Milestone freezes|milestone freeze]] for one of the three Fedora milestones (Alpha, Beta and Final) and using a package set which is not known to contain any [[QA:SOP_blocker_bug_process|blocker bugs]], for the purposes of performing [[:Category:Release_validation|release validation testing]]. It may be declared gold and released as the Alpha, Beta or Final release if it passes all validation tests.


A release candidate - including release candidates beyond RC1 - can be requested only after the Change Deadline (and only when the freeze has actually been put into place) and when all [[QA:SOP_blocker_bug_process#Reviewing_blocker_bugs|accepted blocker bugs]] are addressed. In most cases, 'addressed' means that the bug would not be present in the release candidate compose. Exceptions to this are cases where the blocker is not actually in a component present on the compose: the classic example is a bug in preupgrade, because in this case, a bug in Fedora N can block the release of Fedora N+1. In this case, 'addressed' can be taken to mean that the bug is properly assigned and being worked on. The term 'addressed' is used in preference to 'resolved' because fixes for blocker bugs can be pulled into composes before they have cleared the [[Updates_Policy|update testing process]] and been marked as stable (and hence before the bug is actually closed).
A 'candidate' compose can be considered a 'release candidate' only after the milestone freeze (and only when the freeze has actually been put into place) and when all [[QA:SOP_blocker_bug_process#Reviewing_blocker_bugs|accepted blocker bugs]] are addressed. In most cases, 'addressed' means that the bug would not be present in the release candidate compose. Exceptions to this are cases where the blocker is not actually in a component present on the compose: the [[QA:SOP_blocker_bug_process|blocker bug process]] designates two special blocker types, ''Accepted0Day'' and ''AcceptedPreviousRelease'', to handle such cases. In this case, 'addressed' can be taken to mean that the bug is properly assigned and being worked on. The term 'addressed' is used in preference to 'resolved' because fixes for blocker bugs can be pulled into composes before they have cleared the [[Updates_Policy|update testing process]] and been marked as stable (and hence before the bug is actually closed).


The procedure for release candidates is to request the RC1 compose as close as possible to the target date, but only after all accepted blockers are addressed. RC1 can be requested early if all accepted blockers are addressed, but it must be requested after the freeze date ('Change Deadline' on the schedule pages).  
The procedure for release candidates is to request the first compose as close as soon as possible after the milestone freeze date, but only after all accepted blockers are addressed.


If testing of RC1 results in further blocker bugs being proposed and accepted, an RC2 should be requested as soon as all the blockers accepted after RC1 testing have been addressed. This cycle should continue until a release candidate build passes validation testing with no new blockers being discovered.  
If testing of the first 'release candidate' results in further blocker bugs being proposed and accepted, another should be requested as soon as all the blockers accepted after testing have been addressed. This cycle should continue until a release candidate build passes validation testing with no new blockers being discovered.  


A new release candidate should never be requested to address freeze exception bugs only: a request should only be made if blocker issues remain. See below and the [[QA:SOP_freeze_exception_bug_process|freeze exception SOP]] for details on freeze exception bugs.
A new release candidate should never be requested to address freeze exception bugs only: a request should only be made if blocker issues remain. See below and the [[QA:SOP_freeze_exception_bug_process|freeze exception SOP]] for details on freeze exception bugs.


The [[Current_Release_Blockers]] page can be helpful in checking the status of proposed and accepted blockers at a glance, but please be aware it is updated only hourly. When the blocker situation is changing rapidly, it is best to use direct Bugzilla searches to ensure your list is entirely accurate.
The [http://qa.fedoraproject.org/blockerbugs/current blocker bugs web application] can be helpful in checking the status of proposed and accepted blockers at a glance, but please be aware it is updated only half-hourly. When the blocker situation is changing rapidly, it may be best to use direct Bugzilla searches to ensure your list is entirely accurate.


== How to request a compose ==
== How to request a compose ==


Compose requests are filed as tickets in [https://fedorahosted.org/rel-eng the release engineering team's trac instance]. Only one ticket should be filed for each milestone (Alpha, Beta, Final). So at each milestone, file a new ticket to request the first test compose, and then add comments to that ticket for subsequent test composes and the release candidates. The release engineering team will usually close the ticket each time they complete a compose - just re-open it to request the next one.
Compose requests are filed as tickets in [https://pagure.io/releng the release engineering team's Pagure project]. Only one ticket should be filed for each milestone (Alpha, Beta, Final). So at each milestone, file a new ticket to request the first candidate, and then add comments to that ticket for subsequent candidates. The release engineering team will usually close the ticket each time they complete a compose - just re-open it to request the next one.


* [https://fedorahosted.org/rel-eng/ticket/5602 Example compose request ticket (Fedora 19 Beta)]
* [https://pagure.io/releng/issue/135 Example compose request ticket (Fedora 25 Final)]


== What to include in a compose request ==
== What to include in a compose request ==


There is no formal template for compose requests, but looking at previous ones (see the examples above) is a good way to learn the basic format. A compose request should consist of a simple introduction stating what is requested - for example, a test compose for Fedora 18 Beta - followed by a list of builds which should be explicitly pulled into the compose, and any further special instructions to the release engineering team. It is good practice to include a link to the bugs addressed since the previous compose (if any), in order to form a changelog and to make it easier to track exactly what issues each compose addresses.
There is no formal template for compose requests, but looking at previous ones (see the examples above) is a good way to learn the basic format. A compose request should consist of a simple introduction stating what is requested - for example, a candidate compose for Fedora 24 Beta - followed by a list of builds which should be explicitly pulled into the compose, and any further special instructions to the release engineering team. It is good practice to include a link to the bugs addressed since the previous compose (if any), in order to form a changelog and to make it easier to track exactly what issues each compose addresses.


=== Understanding composes ===
=== Understanding composes ===
Line 52: Line 50:
using:
using:


* The packages present in the current 'stable' tree for the release (that is, in the repositories at development/(releasever) on the mirrors) - '''not''' including packages in updates-testing
* The packages present in the current ''[[Repositories#stable|stable]]'' tree for the release (that is, in the repositories at development/(releasever) on the mirrors) - '''not''' including packages in ''[[Repositories#updates-testing|updates-testing]]''
* The packages present in the [https://kojipkgs.fedoraproject.org/mash/bleed/ bleed repository], which is populated with packages explicitly requested to be included in composes (see [[#explicit-build-requests|below]])
* Any packages previously explicitly requested for inclusion in earlier composes (see [[#explicit-build-requests|below]])
* The versions of the compose tools currently in that same 'stable' tree - that is, {{package|livecd-tools}} from that repository, {{package|pungi}} from that repository, and so on
* The versions of the compose tools currently in that same 'stable' tree - that is, {{package|livecd-tools}} from that repository, {{package|pungi}} from that repository, and so on
* The current HEAD of the relevant branch in [https://git.fedorahosted.org/cgit/spin-kickstarts.git/ spin-kickstarts git] - not the master branch, but the f(releasever) branch, e.g. [https://git.fedorahosted.org/cgit/spin-kickstarts.git/log/?h=f{{FedoraVersionNumber|next}} here for Fedora {{FedoraVersionNumber|next}}]
* The current HEAD of the relevant branch in [https://pagure.io/fedora-kickstarts fedora-kickstarts git] - not the master branch, but the f(releasever) branch, e.g. [https://pagure.io/fedora-kickstarts/commits/f{{FedoraVersionNumber|next}} here for Fedora {{FedoraVersionNumber|next}}]
* The [[How_to_use_and_edit_comps.xml_for_package_groups| comps]] information included in the current 'stable' tree, which is generated daily from the comps-f(releasever).xml file in [https://git.fedorahosted.org/cgit/comps.git comps git], e.g. [https://git.fedorahosted.org/cgit/comps.git/tree/comps-f{{FedoraVersionNumber|next}}.xml.in this file for Fedora {{FedoraVersionNumber|next}}]
* The [[How_to_use_and_edit_comps.xml_for_package_groups|comps]] information included in the current 'stable' tree, which is generated daily from the comps-f(releasever).xml file in [https://pagure.io/fedora-comps fedora-comps git], e.g. [https://pagure.io/fedora-comps/blob/master/f/comps-f{{FedoraVersionNumber|next}}.xml.in this file for Fedora {{FedoraVersionNumber|next}}]


and place the results in the [https://dl.fedoraproject.org/pub/alt/stage/ staging tree].
and place the results in the [https://dl.fedoraproject.org/pub/alt/stage/ staging tree].
Line 63: Line 61:


{{anchor|explicit-build-requests}}
{{anchor|explicit-build-requests}}
=== Explicitly listing required builds ===
=== Explicitly listing required builds ===


Any build which needs to be included in the compose but which has not yet been added to the 'stable' repository for the release must be listed explicitly in the ticket, or it will not appear in the compose. There are usually several such builds for each compose - especially release candidates - as there is often not enough time for fixes to be given the necessary karma, submitted to stable, pushed to stable, and included in an actual tree compose prior to the image compose being requested. You should certainly explicitly list any required build which has not yet received the message 'This update has been pushed to stable' in Bodhi; it is good practice to explicitly list any build which received that message only within the 24 hours prior to the compose request being filed, in case the build has not yet been included in a tree compose.
Any build which needs to be included in the compose but which has not yet been added to the 'stable' repository for the release must be listed explicitly in the ticket, or it will not appear in the compose. This is the most common form of request. There are usually several such builds for each compose - especially release candidates - as there is often not enough time for fixes to be given the necessary karma, submitted to stable, pushed to stable, and included in an actual tree compose prior to the image compose being requested. You should certainly explicitly list any required build which has not yet received the message 'This update has been pushed to stable' in Bodhi; it is good practice to explicitly list any build which received that message only within the 24 hours prior to the compose request being filed, in case the build has not yet been included in a tree compose.


When explicitly listing required builds, link to the Bodhi page for the build where possible (not the Koji page). However, prior to branching, Bodhi is not in effect, so early in the Alpha phase you will have to link to Koji rather than Bodhi.
When explicitly listing required builds, link to the Bodhi page for the build where possible (not the Koji page). However, prior to branching, Bodhi is not in effect, so early in the Alpha phase you will have to link to Koji rather than Bodhi.
Line 72: Line 69:
=== What builds to ensure are included ===
=== What builds to ensure are included ===


When building test composes prior to the [[Change_deadlines|change deadline]] for the milestone, this is essentially a judgement call. Try and ensure all possible blocker fixes are included in test composes (so list out any that are still in updates-testing or that have only recently been pushed stable). It is usually a good idea to explicitly include any pending updates that you know will have a significant impact on testing - so if a big anaconda update is pending, you should probably request that it be pulled in.
When building a candidate to test update fixes, of course, list all updates that need a candidate compose in order to be fully tested. Try and ensure all possible blocker fixes are included in test composes (so list out any that are still in updates-testing or that have only recently been pushed stable).


When building test composes after the change deadline, you should only list packages that fix accepted release blocking or freeze exception bugs.
When requesting a candidate compose after the freeze, you must only list packages that fix accepted release blocking or freeze exception bugs.


When building release candidates (which by definition always happens after the change deadline), this is prescribed by policy. You can, and indeed must, require that all builds that fix accepted release blocking bugs are included: as explained above, all accepted release blocking bugs must be fixed in all release candidate builds. So ensure that, if any release blocking bug is still open when you request a release candidate, a build that fixes it is explicitly listed in the compose request. You should always double-check the entire list of open (and recently closed) accepted blockers and compare it to the ticket before hitting the 'submit' button.
When requesting a candidate intended as a potential 'release candidate' (which by definition always happens after the freeze), this is prescribed by policy. You can, and indeed must, require that all builds that fix accepted release blocking bugs are included: as explained above, all accepted release blocking bugs must be fixed in all release candidate builds. So ensure that, if any release blocking bug is still open when you request a release candidate, a build that fixes it is explicitly listed in the compose request. You should always double-check the entire list of open (and recently closed) accepted blockers and compare it to the ticket before hitting the 'submit' button.


You may also request that pending fixes for freeze exception bugs are included in release candidate composes. Freeze exception bugs are those that do not block the release, but which have been accepted as being of sufficient importance that fixes may be allowed through the freeze: see the [[QA:SOP_freeze_exception_bug_process]] for further details. During release freezes, release engineering may push freeze exception fixes to the stable repository, but if a fix for a freeze exception issue is still pending stable status, and you judge the issue to be of sufficient importance and the fix to be sufficiently safe, you may list it for inclusion in the release candidate compose. Release engineering may question this decision if they consider it risky. Err on the side of caution when requesting inclusion of pending freeze exception fixes in release candidate composes.
You may also request that pending fixes for freeze exception bugs are included in release candidate composes. Freeze exception bugs are those that do not block the release, but which have been accepted as being of sufficient importance that fixes may be allowed through the freeze: see the [[QA:SOP_freeze_exception_bug_process]] for further details. During release freezes, release engineering may push freeze exception fixes to the stable repository, but if a fix for a freeze exception issue is still pending stable status, and you judge the issue to be of sufficient importance and the fix to be sufficiently safe, you may list it for inclusion in the release candidate compose. Release engineering may question this decision if they consider it risky. Err on the side of caution when requesting inclusion of pending freeze exception fixes in release candidate composes.


Try and bear in mind that the ''ideal'' process would be for all builds that will be included in a compose to be fully tested, pushed and mashed into the stable repository before a compose happens, and we only pull packages in manually because the process is often too rushed to allow that to happen without delaying the release: this is a hack, so avoid it when possible, but do it when necessary.
Try and bear in mind that the ''ideal'' process would be for all builds that will be included in a compose to be fully tested, pushed and mashed into the stable repository before a compose happens, and we only pull packages in manually because the process is often too rushed to allow that to happen without delaying the release (and because sometimes updates cannot be tested without a compose): this is a hack, so avoid it when possible, but do it when necessary.


It is never acceptable to explicitly list a build for inclusion in a release candidate which is not a fix for an accepted release blocker or freeze exception bug. If you think a build ought to be included but the bug with which it is associated is not an accepted release blocker or freeze exception issue, do not just list it anyway: propose the bug as a release blocker or freeze exception issue, and contact other QA team members, release engineering team members, development team members, and/or the FPL to vote on this status, as would happen during a blocker review meeting.
It is never acceptable to explicitly list a build for inclusion in a release candidate which is not a fix for an accepted release blocker or freeze exception bug. If you think a build ought to be included but the bug with which it is associated is not an accepted release blocker or freeze exception issue, do not just list it anyway: propose the bug as a release blocker or freeze exception issue, and contact other QA team members, release engineering team members, development team members, and/or the FPL to vote on this status, as would happen during a blocker review meeting.
Line 87: Line 84:


There are sometimes issues that arise in the compose process itself and hence a little outside the scope simply of listing builds for inclusion, and it is a good idea to remind the release engineering team of these in the ticket. For instance, they may have to remember to have a certain package version installed on the compose host or in the compose environment, rather than being included in the compose itself (spin-kickstarts, lorax) or they may have to explicitly block a package from being included to avoid conflicts being present on the media. Whoever is handling a release from the QA side should naturally be in close contact with release engineering and aware of such issues; they should be mentioned in the ticket just to ensure no-one inadvertently forgets about them.
There are sometimes issues that arise in the compose process itself and hence a little outside the scope simply of listing builds for inclusion, and it is a good idea to remind the release engineering team of these in the ticket. For instance, they may have to remember to have a certain package version installed on the compose host or in the compose environment, rather than being included in the compose itself (spin-kickstarts, lorax) or they may have to explicitly block a package from being included to avoid conflicts being present on the media. Whoever is handling a release from the QA side should naturally be in close contact with release engineering and aware of such issues; they should be mentioned in the ticket just to ensure no-one inadvertently forgets about them.
=== The {{command|candpush}} tool ===
There is a tool called {{command|candpush}} which can help you generate compose requests. You can check it out and run it from the [https://pagure.io/fedora-qa/qa-misc qa-misc repository]:
git clone https://pagure.io/fedora-qa/qa-misc.git
cd qa-misc
dnf install python3-fedora python3-bugzilla python3-fedfind
./candpush
It will print proposed comment text for a candidate compose (and also for a stable push request - you can ignore that part). It will also print warnings for updates with <0 karma and updates that fix bugs which are not accepted blockers or freeze exceptions, for you to inspect manually.
Please only use this text as a starting point, and review it according to the above instructions. You may need to add or remove requests, or tweak the text. The tool is only a helper. It will list all updates which are marked in Bodhi as being for the correct release and associated with at least one accepted blocker or freeze exception bug.
== What comes next ==
Once a candidate is built, it must be tested! This is done according to the [[QA:Release_validation_test_plan|release validation test plan]]. Normally, the necessary result pages are created and the validation test event announced automatically by [https://git.fedorahosted.org/cgit/fedora-qa.git/tree/relvalconsumer relvalconsumer] (the production relvalconsumer instance is maintained by [[User:Adamwill|Adam Williamson]]). The [[QA/SOP_Release_Validation_Test_Event|release validation test event SOP]] covers the process of doing this manually if that is for any reason necessary.
=== Ensuring requested builds go stable ===
It is important that any builds pulled into a candidate via this process are then properly pushed to the stable repository very soon afterwards. This is particularly crucial for the Final milestone: we strongly desire that the final frozen release repository and the release media are derived from the same package set.
QA and release engineering share responsibility for ensuring the list of explicitly-included builds is tracked, and all such builds are pushed to stable (or, if they are broken, superseded or withdrawn from subsequent composes). The QA process for this is not yet documented, but usually every few days someone will check the list of builds included in the current candidate but not yet pushed stable, file a ticket in [https://pagure.io/releng release engineering Pagure] requesting stable push of all those with sufficient karma, and send out a mail requesting high-priority testing of any which do not have sufficient karma. An example 'stable push request' ticket can be found [https://pagure.io/releng/issue/134 here].


[[Category:QA SOPs]]
[[Category:QA SOPs]]

Latest revision as of 01:05, 17 November 2016

QA.png


This SOP explains the procedure for requesting 'candidate' composes from the ReleaseEngineering team. This task is usually performed by QA.

What composes should be requested when

There are three milestones in the Fedora release cycle: Alpha, Beta, and Final. For each milestone, a minimum of one 'candidate' (known in release engineering terms as a 'production compose') must be produced. Usually, there will be more than one.

In the normal course of events, 'nightly' composes are regularly produced by release engineering and validation testing is performed on these. There are two reasons to request a 'candidate' compose. The first is to allow testing of an update before it is pushed stable if it cannot be tested fully by updating an installed system (e.g. installer updates, or GNOME/Plasma updates to test the live images). The second is to produce a compose that can be released: 'nightly' composes have some properties that mean we cannot ship them as Alpha, Beta or Final releases, so a 'candidate' / 'production' compose must be built for release. You could refer to a 'candidate' of this type as a 'release candidate'.

Only a 'candidate' composed after the milestone freeze and with all blocker bugs addressed can be released, and this requirement supersedes the schedule: see below for more on this.

The schedule for the next Fedora release, with the milestone freeze dates, can be found here.

Candidates for updates testing

Prior to the Fedora 24 cycle, there were 'test compose' and 'release candidate' composes. This is no longer the case. In the usual course of events, validation testing before the freeze date for each milestone is simply performed on nightly composes. 'Candidate' composes can be requested prior to the freeze date if they are required to test one or more updates before they can be pushed stable, but ideally this should be avoided to save unnecessary work on the release engineering team's part.

Release candidates

A release candidate is defined as a set of Fedora images built, from the current Branched tree (not including updates-testing), after the milestone freeze for one of the three Fedora milestones (Alpha, Beta and Final) and using a package set which is not known to contain any blocker bugs, for the purposes of performing release validation testing. It may be declared gold and released as the Alpha, Beta or Final release if it passes all validation tests.

A 'candidate' compose can be considered a 'release candidate' only after the milestone freeze (and only when the freeze has actually been put into place) and when all accepted blocker bugs are addressed. In most cases, 'addressed' means that the bug would not be present in the release candidate compose. Exceptions to this are cases where the blocker is not actually in a component present on the compose: the blocker bug process designates two special blocker types, Accepted0Day and AcceptedPreviousRelease, to handle such cases. In this case, 'addressed' can be taken to mean that the bug is properly assigned and being worked on. The term 'addressed' is used in preference to 'resolved' because fixes for blocker bugs can be pulled into composes before they have cleared the update testing process and been marked as stable (and hence before the bug is actually closed).

The procedure for release candidates is to request the first compose as close as soon as possible after the milestone freeze date, but only after all accepted blockers are addressed.

If testing of the first 'release candidate' results in further blocker bugs being proposed and accepted, another should be requested as soon as all the blockers accepted after testing have been addressed. This cycle should continue until a release candidate build passes validation testing with no new blockers being discovered.

A new release candidate should never be requested to address freeze exception bugs only: a request should only be made if blocker issues remain. See below and the freeze exception SOP for details on freeze exception bugs.

The blocker bugs web application can be helpful in checking the status of proposed and accepted blockers at a glance, but please be aware it is updated only half-hourly. When the blocker situation is changing rapidly, it may be best to use direct Bugzilla searches to ensure your list is entirely accurate.

How to request a compose

Compose requests are filed as tickets in the release engineering team's Pagure project. Only one ticket should be filed for each milestone (Alpha, Beta, Final). So at each milestone, file a new ticket to request the first candidate, and then add comments to that ticket for subsequent candidates. The release engineering team will usually close the ticket each time they complete a compose - just re-open it to request the next one.

What to include in a compose request

There is no formal template for compose requests, but looking at previous ones (see the examples above) is a good way to learn the basic format. A compose request should consist of a simple introduction stating what is requested - for example, a candidate compose for Fedora 24 Beta - followed by a list of builds which should be explicitly pulled into the compose, and any further special instructions to the release engineering team. It is good practice to include a link to the bugs addressed since the previous compose (if any), in order to form a changelog and to make it easier to track exactly what issues each compose addresses.

Understanding composes

It will obviously help you to write a good compose request if you understand more or less what happens in a compose. If you were to simply request a compose with no special instructions, ReleaseEngineering would produce:

  • A full set of release media
  • Install trees for all installable products

using:

  • The packages present in the current stable tree for the release (that is, in the repositories at development/(releasever) on the mirrors) - not including packages in updates-testing
  • Any packages previously explicitly requested for inclusion in earlier composes (see below)
  • The versions of the compose tools currently in that same 'stable' tree - that is, Package-x-generic-16.pnglivecd-tools from that repository, Package-x-generic-16.pngpungi from that repository, and so on
  • The current HEAD of the relevant branch in fedora-kickstarts git - not the master branch, but the f(releasever) branch, e.g. here for Fedora 40
  • The comps information included in the current 'stable' tree, which is generated daily from the comps-f(releasever).xml file in fedora-comps git, e.g. this file for Fedora 40

and place the results in the staging tree.

If the compose should differ in any respect, you should explain in the compose request. This section explains some common cases where you might add particular requests.

Explicitly listing required builds

Any build which needs to be included in the compose but which has not yet been added to the 'stable' repository for the release must be listed explicitly in the ticket, or it will not appear in the compose. This is the most common form of request. There are usually several such builds for each compose - especially release candidates - as there is often not enough time for fixes to be given the necessary karma, submitted to stable, pushed to stable, and included in an actual tree compose prior to the image compose being requested. You should certainly explicitly list any required build which has not yet received the message 'This update has been pushed to stable' in Bodhi; it is good practice to explicitly list any build which received that message only within the 24 hours prior to the compose request being filed, in case the build has not yet been included in a tree compose.

When explicitly listing required builds, link to the Bodhi page for the build where possible (not the Koji page). However, prior to branching, Bodhi is not in effect, so early in the Alpha phase you will have to link to Koji rather than Bodhi.

What builds to ensure are included

When building a candidate to test update fixes, of course, list all updates that need a candidate compose in order to be fully tested. Try and ensure all possible blocker fixes are included in test composes (so list out any that are still in updates-testing or that have only recently been pushed stable).

When requesting a candidate compose after the freeze, you must only list packages that fix accepted release blocking or freeze exception bugs.

When requesting a candidate intended as a potential 'release candidate' (which by definition always happens after the freeze), this is prescribed by policy. You can, and indeed must, require that all builds that fix accepted release blocking bugs are included: as explained above, all accepted release blocking bugs must be fixed in all release candidate builds. So ensure that, if any release blocking bug is still open when you request a release candidate, a build that fixes it is explicitly listed in the compose request. You should always double-check the entire list of open (and recently closed) accepted blockers and compare it to the ticket before hitting the 'submit' button.

You may also request that pending fixes for freeze exception bugs are included in release candidate composes. Freeze exception bugs are those that do not block the release, but which have been accepted as being of sufficient importance that fixes may be allowed through the freeze: see the QA:SOP_freeze_exception_bug_process for further details. During release freezes, release engineering may push freeze exception fixes to the stable repository, but if a fix for a freeze exception issue is still pending stable status, and you judge the issue to be of sufficient importance and the fix to be sufficiently safe, you may list it for inclusion in the release candidate compose. Release engineering may question this decision if they consider it risky. Err on the side of caution when requesting inclusion of pending freeze exception fixes in release candidate composes.

Try and bear in mind that the ideal process would be for all builds that will be included in a compose to be fully tested, pushed and mashed into the stable repository before a compose happens, and we only pull packages in manually because the process is often too rushed to allow that to happen without delaying the release (and because sometimes updates cannot be tested without a compose): this is a hack, so avoid it when possible, but do it when necessary.

It is never acceptable to explicitly list a build for inclusion in a release candidate which is not a fix for an accepted release blocker or freeze exception bug. If you think a build ought to be included but the bug with which it is associated is not an accepted release blocker or freeze exception issue, do not just list it anyway: propose the bug as a release blocker or freeze exception issue, and contact other QA team members, release engineering team members, development team members, and/or the FPL to vote on this status, as would happen during a blocker review meeting.

Special instructions

There are sometimes issues that arise in the compose process itself and hence a little outside the scope simply of listing builds for inclusion, and it is a good idea to remind the release engineering team of these in the ticket. For instance, they may have to remember to have a certain package version installed on the compose host or in the compose environment, rather than being included in the compose itself (spin-kickstarts, lorax) or they may have to explicitly block a package from being included to avoid conflicts being present on the media. Whoever is handling a release from the QA side should naturally be in close contact with release engineering and aware of such issues; they should be mentioned in the ticket just to ensure no-one inadvertently forgets about them.

The candpush tool

There is a tool called candpush which can help you generate compose requests. You can check it out and run it from the qa-misc repository:

git clone https://pagure.io/fedora-qa/qa-misc.git
cd qa-misc
dnf install python3-fedora python3-bugzilla python3-fedfind
./candpush

It will print proposed comment text for a candidate compose (and also for a stable push request - you can ignore that part). It will also print warnings for updates with <0 karma and updates that fix bugs which are not accepted blockers or freeze exceptions, for you to inspect manually.

Please only use this text as a starting point, and review it according to the above instructions. You may need to add or remove requests, or tweak the text. The tool is only a helper. It will list all updates which are marked in Bodhi as being for the correct release and associated with at least one accepted blocker or freeze exception bug.

What comes next

Once a candidate is built, it must be tested! This is done according to the release validation test plan. Normally, the necessary result pages are created and the validation test event announced automatically by relvalconsumer (the production relvalconsumer instance is maintained by Adam Williamson). The release validation test event SOP covers the process of doing this manually if that is for any reason necessary.

Ensuring requested builds go stable

It is important that any builds pulled into a candidate via this process are then properly pushed to the stable repository very soon afterwards. This is particularly crucial for the Final milestone: we strongly desire that the final frozen release repository and the release media are derived from the same package set.

QA and release engineering share responsibility for ensuring the list of explicitly-included builds is tracked, and all such builds are pushed to stable (or, if they are broken, superseded or withdrawn from subsequent composes). The QA process for this is not yet documented, but usually every few days someone will check the list of builds included in the current candidate but not yet pushed stable, file a ticket in release engineering Pagure requesting stable push of all those with sufficient karma, and send out a mail requesting high-priority testing of any which do not have sufficient karma. An example 'stable push request' ticket can be found here.