User:Adamwill/Draft QA SOP compose request

This SOP explains the procedure for requesting composes from the ReleaseEngineering team. This task is usually performed by QA.

What composes should be requested when
There are three release points in the Fedora release cycle: Alpha, Beta, and Final. For each release point, a minimum of one release candidate (RC) must be produced. At least one test compose (TC) should also always be produced under normal circumstances.

The Fedora schedule specifies dates for the first test compose and for the first release candidate, for each release point: the compose dates are listed on the release engineering schedule and usually also on the QA schedule. The requests should be filed on or (ideally) shortly prior to these dates. However, a release candidate can only be composed when all blocker bugs are addressed, and this requirement supersedes the schedule: see below for more on this.

The schedule for the next Fedora release, with the designated compose dates, can be found here.

Test composes
Test composes beyond TC1 are done at the discretion of the QA team: there is no strict requirement for a certain number of test composes, or a strict trigger for building a new test compose. It usually makes sense to request a new test compose when significant changes to key components such as the installer, the kernel, X.org, the default bootloader and so on would be included: simply put, request a new test compose when it becomes clear that it may provide significantly different results to the current test compose.

There is no requirement for blocker bugs to be addressed in order for a test compose to be produced. It is a good idea, however, to make a best effort before filing a TC compose request to ensure that the compose will succeed and result in images that do not fail immediately - if you know there is a package conflict in the repositories that will result in live image composes failing, or a showstopper bug which makes the installer crash at the first page, there is very little point in requesting a TC compose. It is better for TC1 to be a day or two late than for it to be on time but untestable.

Release candidates
Release candidates are different. A release candidate - including release candidates beyond RC1 - can be requested only when all accepted blocker bugs are addressed. In most cases, 'addressed' means that the bug would not be present in the release candidate compose. Exceptions to this are cases where the blocker is not actually in a component present on the compose: the classic example is a bug in preupgrade, because in this case, a bug in Fedora N can block the release of Fedora N+1. In this case, 'addressed' can be taken to mean that the bug is properly assigned and being worked on. The term 'addressed' is used in preference to 'resolved' because fixes for blocker bugs can be pulled into composes before they have cleared the update testing process and been marked as stable (and hence before the bug is actually closed).

The procedure for release candidates is to request the RC1 compose as close as possible to the target date, but only after all accepted blockers are addressed. RC1 can be requested early if all accepted blockers are addressed, but it must be requested after the freeze date ('Change Deadline' on the schedule pages).

If testing of RC1 results in further blocker bugs being proposed and accepted, an RC2 should be requested as soon as all the blockers accepted after RC1 testing have been addressed. This cycle should continue until a release candidate build passes validation testing with no new blockers being discovered.

A new release candidate should never be requested to address NTH issues only: a request should only be made if blocker issues remain. See below and the NTH SOP for details on NTH bugs.

The Current_Release_Blockers page can be helpful in checking the status of proposed and accepted blockers at a glance, but please be aware it is updated only hourly. When the blocker situation is changing rapidly, it is best to use direct Bugzilla searches to ensure your list is entirely accurate.

It is theoretically possible that the situation could arise where a release candidate is requested and built, and results in multiple blockers being reported and accepted, some of which are easy to fix and some of which are not; in this case a subsequent compose to test the easy fixes may be desirable, but could not be denoted a release candidate due to the presence of the other un-addressed blockers. In this case a new test compose could be requested. This is obviously somewhat confusing, however, so it is best to try and get all the blockers fixed and request a new release candidate instead.

How to request a compose
Compose requests are filed as tickets in the release engineering team's trac instance. For each release point, one ticket should be created for test compose requests, and one for release candidate requests. So at each release point, file a new ticket to request the first test compose, and then add comments to that ticket for subsequent test composes; file another new ticket to request the first release candidate, and then add comments to that ticket for subsequent release candidates. The release engineering team will usually close the ticket each time they complete a compose - just re-open it to request the next one.


 * Example test compose request ticket (Fedora 17 Alpha)
 * Example release candidate request ticket (Fedora 17 Alpha)

What to include in a compose request
There is no formal template for compose requests, but looking at previous ones (see the examples above) is a good way to learn the basic format. A compose request should consist of a simple introduction stating what is requested - for example, a test compose for Fedora 18 Beta - followed by a list of builds which should be explicitly pulled into the compose, and any further special instructions to the release engineering team. It is good practice to include a link to the bugs addressed since the previous compose (if any), in order to form a changelog and to make it easier to track exactly what issues each compose addresses.

Explicitly listing required builds
Any build which needs to be included in the compose but which has not yet been added to the 'stable' repository for the release must be listed explicitly in the ticket, or it will not appear in the compose. There are usually several such builds for each compose - especially release candidates - as there is often not enough time for fixes to be given the necessary karma, submitted to stable, pushed to stable, and included in an actual tree compose prior to the image compose being requested. You should certainly explicitly list any required build which has not yet received the message 'This update has been pushed to stable' in Bodhi; it is good practice to explicitly list any build which received that message only within the 24 hours prior to the compose request being filed, in case the build has not yet been included in a tree compose.

When explicitly listing required builds, link to the Bodhi page for the build where possible (not the Koji page). However, prior to branching, Bodhi is not in effect, so early in the Alpha phase you will have to link to Koji rather than Bodhi.

What builds to ensure are included
At test compose stage, this is essentially a judgement call. Try and ensure all possible blocker fixes are included in test composes (so list out any that are still in updates-testing or that have only recently been pushed stable). It is usually a good idea to explicitly include any pending updates that you know will have a significant impact on testing - so if a big anaconda update is pending, you should probably request that it be pulled in.

At release candidate stage, this is prescribed by policy. You can, and indeed must, require that all builds that fix accepted release blocking bugs are included: as explained above, all accepted release blocking bugs must be fixed in all release candidate builds. So ensure that, if any release blocking bug is still open when you request a release candidate, a build that fixes it is explicitly listed in the compose request. You should always double-check the entire list of open (and recently closed) accepted blockers and compare it to the ticket before hitting the 'submit' button.

You may also request that pending fixes for NTH (nice-to-have) bugs are included in release candidate composes. NTH bugs are those that do not block the release, but which have been accepted as being of sufficient importance that fixes will be allowed through the freeze: see the QA:SOP_nth_bug_process for further details. During release freezes, release engineering will push NTH fixes to the stable repository, but if a fix for an NTH issue is still pending stable status, and you judge the issue to be of sufficient importance and the fix to be sufficiently safe, you may list it for inclusion in the release candidate compose. Release engineering may question this decision if they consider it risky. Err on the side of caution when requesting inclusion of pending NTH fixes in release candidate composes.

Try and bear in mind that the ideal process would be for all builds that will be included in a compose to be fully tested, pushed and mashed into the stable repository before a compose happens, and we only pull packages in manually because the process is often too rushed to allow that to happen without delaying the release: this is a hack, so avoid it when possible, but do it when necessary.

It is never acceptable to explicitly list a build for inclusion in a release candidate which is not a fix for an accepted release blocker or NTH bug. If you think a build ought to be included but the bug with which it is associated is not an accepted release blocker or NTH issue, do not just list it anyway: propose the bug as a release blocker or NTH issue, and contact other QA team members, release engineering team members, development team members, and/or the FPL to vote on this status, as would happen during a blocker review meeting.

Special instructions
There are sometimes issues that arise in the compose process itself and hence a little outside the scope simply of listing builds for inclusion, and it is a good idea to remind the release engineering team of these in the ticket. For instance, they may have to remember to have a certain package version installed on the compose host or in the compose environment, rather than being included in the compose itself (spin-kickstarts, lorax) or they may have to explicitly block a package from being included to avoid conflicts being present on the media. Whoever is handling a release from the QA side should naturally be in close contact with release engineering and aware of such issues; they should be mentioned in the ticket just to ensure no-one inadvertently forgets about them.