QA:SOP freeze exception bug process

From FedoraProject

(Difference between revisions)
Jump to: navigation, search
m (Use FedoraVersion template where possible)
(Automatic freeze exceptions)
(11 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 
== Background ==
 
== Background ==
  
Fedora uses a system of ''tracker bugs'' to keep track of ''nice-to-have (NTH) bugs'' - bugs which do not block a given pre-release or release, but which are considered high priority for tracking and fixing, and for which fixes will be accepted even during the freeze period for that release. This page defines the process by which bugs are proposed, reviewed and accepted as nice-to-have bugs, and how nice-to-have bugs are then tracked.
+
Fedora uses a system of ''tracker bugs'' to keep track of ''freeze exception bugs'' - bugs which do not block a given pre-release or release, but which are considered high priority for tracking and fixing, and for which fixes may be accepted even during the freeze period for that release. This page defines the process by which bugs are proposed, reviewed and accepted as freeze exception bugs, and how freeze exception bugs are then tracked.
  
See also the [[QA:SOP:Blocker_bug_process|blocker bug process]], which defines the similar process for ''blocker bugs'' - bugs that are blocking the release of its pre- and final releases and which must be fixed before these releases can proceed.
+
Note that freeze exception bugs used to be called ''nice-to-have'' or ''NTH'' bugs, and you may still find this name mentioned or used in old documentation and particularly in old bug reports.
  
== Nice-to-have bug principles ==
+
See also the [[QA:SOP_blocker_bug_process|blocker bug process]], which defines the similar process for ''blocker bugs'' - bugs that are blocking the release of its pre- and final releases and which must be fixed before these releases can proceed.
  
Strict [[Fedora_Release_Criteria]] are used to define whether a bug should block a Fedora release, but determining whether a bug is a nice-to-have is a more flexible process and should be done on a case-by-case basis. However, we have evolved some principles to be taken into account when evaluating a bug for nice-to-have status.
+
== Freeze exception bug principles ==
  
In general, nice-to-have bugs are usually bugs for which an update is not an optimal solution, and for which the fix is reasonably small and testable (this consideration becomes progressively more important as a release nears, so bugs may be downgraded from nice-to-have status late in the release process if it transpires that the fix is complex and hard to test).
+
Strict [[Fedora_Release_Criteria]] are used to define whether a bug should block a Fedora release, but determining whether a bug is worthy of a freeze exception is a more flexible process and should be done on a case-by-case basis. However, we have evolved some principles to be taken into account when evaluating a bug for freeze exception status.
  
Types of bugs which are typically likely to be accepted as nice-to-have bugs include:
+
In general, freeze exception bugs are usually bugs for which an update is not an optimal solution, and for which the fix is reasonably small and testable (this consideration becomes progressively more important as a release nears, so bugs may be downgraded from freeze exception status late in the release process if it transpires that the fix is complex and hard to test).
 +
 
 +
Types of bugs which are typically likely to be accepted as freeze exception bugs include:
  
 
# bugs which constitute infringements of the desktop-related [[Fedora_Release_Criteria]] as applied to non-default desktops
 
# bugs which constitute infringements of the desktop-related [[Fedora_Release_Criteria]] as applied to non-default desktops
Line 17: Line 19:
 
# significant installer bugs which do not meet the criteria to be blocker bugs
 
# significant installer bugs which do not meet the criteria to be blocker bugs
  
== Proposing nice-to-have bugs ==
+
== Proposing freeze exception bugs ==
 +
 
 +
To propose a bug as a freeze exception for a release, mark it as blocking the ''tracker bug'' for freeze exception bugs in that release. To do this, enter the alias or bug ID of the ''tracker bug'' into the '''Blocks:''' field in Bugzilla. The aliases for the freeze exception ''tracker bugs'' follow a consistent naming scheme. For the next release, the Alpha tracker will always be called '''AlphaFreezeException''', the Beta tracker will always be called '''BetaFreezeException''', and the final release tracker will always be called '''FinalFreezeException'''. Rarely, you may need to propose a bug as a freeze exception for the next release but one - in this case, prepend '''FXX''' (where XX is the release number) to the name of the alias and change '''Exception''' to '''Except''', e.g. '''F{{FedoraVersionNumber|next2}}AlphaFreezeExcept'''. So, to mark a bug as a freeze exception for the release of {{FedoraVersion|long|next}} Beta, you would set it to block the bug '''BetaFreezeException'''.
 +
 
 +
{{admon/important|Must use [http://bugzilla.redhat.com Red Hat Bugzilla]|Only bugs reported against a Fedora component in [http://bugzilla.redhat.com Red Hat Bugzilla] can be marked as freeze exceptions for a Fedora release. If the bug you wish to mark as a freeze exception is being tracked in an upstream bug tracking system, you must file a corresponding bug in the Fedora bug tracking system before proposing it.}}
 +
 
 +
== Reviewing freeze exception bugs ==
 +
 
 +
Proposed freeze exception bugs are reviewed and either accepted or rejected in collaboration between the three stakeholder groups: [[QA]], [[Development]] and [[ReleaseEngineering]]. This is mostly done during the weekly  [[QA:SOP_Blocker_Bug_Meeting|blocker review meeting]]. Blocker review meetings usually occur every Friday during release periods, but special review meetings can be scheduled at other times when necessary. When appropriate, proposed freeze exception bugs may also be reviewed between meetings or during other meetings, such as the [[Go_No_Go_Meeting|engineering readiness meeting]] (also known as a ''go/no-go meeting'') which is convened to decide whether a release candidate should be approved as a final release. In these cases, consensus between the three stakeholder groups should still be reached in order to accept or reject a bug as a freeze exception.
  
To propose a bug as a nice-to-have for a release, mark it as blocking the ''tracker bug'' for nice-to-have bugs in that release. To do this, enter the alias or bug ID of the ''tracker bug'' into the '''Blocks:''' field in Bugzilla. The aliases for the nice-to-have ''tracker bugs'' follow a consistent naming scheme. The Alpha tracker will always be called '''FXXAlpha-accepted''', the Beta tracker will always be called '''FXXBeta-accepted''', and the final release tracker will always be called '''FXX-accepted''', where ''XX'' is the number of the release in question. So, to mark a bug as a nice-to-have for the release of {{FedoraVersion|long|next}} Beta, you would set it to block the bug '''{{FedoraVersion|short|next}}Beta-accepted'''.
+
Bugs that are accepted as freeze exceptions for the relevant release will be marked with the Whiteboard field <code>AcceptedFreezeException</code>. Bugs which are rejected will be updated to no longer block the relevant ''tracker bug'', and have the <code>RejectedFreezeException</code> Whiteboard field added so that if they are proposed again, it is clear they have already been considered and rejected. Therefore, a bug which has been proposed but not accepted or rejected can be identified by the lack of a relevant Whiteboard field. All changes to freeze exception status should also be documented with a comment.
  
{{admon/important|Must use [http://bugzilla.redhat.com Red Hat Bugzilla]|Only bugs reported against a Fedora component in [http://bugzilla.redhat.com Red Hat Bugzilla] can be marked as nice-to-have for a Fedora release. If the bug you wish to mark as a nice-to-have is being tracked in an upstream bug tracking system, you must file a corresponding bug in the Fedora bug tracking system before proposing it.}}
+
== Automatic freeze exceptions ==
  
== Reviewing nice-to-have bugs ==
+
Certain types of bugs are considered ''automatic freeze exception bugs''. These bugs can be marked as AcceptedFreezeException by any member of one of the stakeholder groups without formal review. A comment should accompany this change, explaining that it has been made under the ''automatic freeze exception'' policy and linking to this section of this page. If anyone believes that a bug has been incorrectly marked as AcceptedFreezeException in this way, they may propose that it be formally reviewed by appending a comment to the bug or by raising it
 +
during a freeze exception review meeting. '''Only''' the following types of bug are considered ''automatic freeze exception bugs''. Note that where an item on this list applies to non-release-blocking images, a corresponding issue in a release-blocking image would likely be an ''automatic blocker'', under the corresponding [[QA:SOP_blocker_bug_process#Automatic_blockers|policy]].
  
Proposed nice-to-have bugs are reviewed and either accepted or rejected in collaboration between the [[QA]], [[Development]] and [[ReleaseEngineering]] groups. This is mostly done during the weekly  [[QA:SOP_NTH_Bug_Meeting|blocker review meeting]]. Blocker review meetings usually occur every Friday during release periods, but special review meetings can be scheduled at other times when necessary. When appropriate, proposed nice-to-have bugs may also be reviewed between meetings or during other meetings, such as the [[Go_No_Go_Meeting|engineering readiness meeting]] (also known as a ''go/no-go meeting'') which is convened to decide whether a release candidate should be approved as a final release. In these cases, consensus between the three stakeholder groups should still be reached in order to accept or reject a bug as a nice-to-have.
+
* Bugs which entirely prevent the composition of one or more of the non-release-blocking images expected to be built for a currently-pending (pre-)release
 +
* Incorrect checksums present on any of the non-release-blocking TC/RC images (failures of [[QA:Testcase_Mediakit_ISO_Checksums]])
 +
* Complete failure of any non-release-blocking TC/RC image to boot at all - "DOA" image (conditional failure is not an automatic blocker)
 +
* Any non-release-blocking Beta or Final TC/RC image exceeding its target size (failures of [[QA:Testcase_Mediakit_ISO_Size]])
  
Bugs that are accepted as nice-to-have for the relevant release will be marked with the Whiteboard field ''AcceptedNTH''. Bugs which are rejected will be updated to no longer block the relevant ''tracker bug'', and have the ''RejectedNTH'' Whiteboard field added so that if they are proposed again, it is clear they have already been considered and rejected. Therefore, a bug which has been proposed but not accepted or rejected can be identified by the lack of a relevant Whiteboard field. All changes to nice-to-have status should also be documented with a comment.
+
No other type of bug can be considered an ''automatic freeze exception bug'' under any circumstance. In particular, "I think it is obviously a freeze exception bug" is not a valid reason to use this procedure. If you believe another type of bug should be added to the list, please propose the change on the [https://lists.fedoraproject.org/mailman/listinfo/test test@ mailing list].
  
== Tracking nice-to-have bugs ==
+
== Tracking freeze exception bugs ==
  
Again, tracking nice-to-have bugs and making a best effort to fix them is a collaborative effort between the QA, Development and Release Engineering groups. The [[QA:SOP_NTH_Bug_Meeting]] process includes reviewing the status of existing blockers and ensuring that the appropriate resources to fix them are in place, as well as evaluating proposed nice-to-have bugs. After blocker bugs, QA group members are encouraged to prioritize testing of nice-to-have bug fixes, development group members are encouraged to prioritize developing fixes for nice-to-have bugs, and release engineering group members are encouraged to prioritize the release of fixes for nice-to-have bugs (after appropriate testing). Each group should have its own processes for ensuring its responsibilities in relation to nice-to-have bugs are met.
+
Again, tracking freeze exception bugs and making a best effort to fix them is a collaborative effort between the QA, Development and Release Engineering groups. After blocker bugs, QA group members are encouraged to prioritize testing of freeze exception bug fixes, development group members are encouraged to prioritize developing fixes for freeze exception bugs, and release engineering group members are encouraged to prioritize the release of fixes for freeze exception bugs (after appropriate testing). Each group should have its own processes for ensuring its responsibilities in relation to freeze exception bugs are met.
  
In Bugzilla, nice-to-have bugs should follow the normal [[BugZappers/BugStatusWorkFlow|workflow]], with special attention paid by the development group to submitting proposed fixes to the updates-testing repository so they reach '''MODIFIED''' and '''ON_QA''' status, and special attention paid by the QA group to testing proposed fixes and setting ones that are tested successfully to the VERIFIED status. No nice-to-have bug should be set to '''CLOSED ERRATA''' until a fix is actually released to the stable repository for the release in question: if a working fix is added to a test candidate or release candidate build, but not yet pushed to the stable repository, the bug should not be marked '''CLOSED ERRATA''', as this may result in the fix not being pushed to the stable repository and the fix accidentally omitted from the next candidate build as it is no longer possible to track the bug.
+
In Bugzilla, freeze exception bugs should follow the normal [[BugZappers/BugStatusWorkFlow|workflow]], with special attention paid by the development group to submitting proposed fixes to the updates-testing repository so they reach '''MODIFIED''' and '''ON_QA''' status, and special attention paid by the QA group to testing proposed fixes and setting ones that are tested successfully to the VERIFIED status. Builds that fix freeze exception bugs should be pulled into TC and RC builds by the release engineering team as long as they are submitted as proposed updates and meet the necessary karma requirements. No freeze exception bug should be set to '''CLOSED ERRATA''' until a fix is actually released to the stable repository for the release in question: if a working fix is added to a test candidate or release candidate build, but not yet pushed to the stable repository, the bug should not be marked '''CLOSED ERRATA''', as this may result in the fix not being pushed to the stable repository and the fix accidentally omitted from the next candidate build as it is no longer possible to track the bug.
  
[[Category:NTH_adjustment_drafts]]
 
 
[[Category:QA SOPs]]
 
[[Category:QA SOPs]]

Revision as of 00:16, 27 February 2013

Contents

Background

Fedora uses a system of tracker bugs to keep track of freeze exception bugs - bugs which do not block a given pre-release or release, but which are considered high priority for tracking and fixing, and for which fixes may be accepted even during the freeze period for that release. This page defines the process by which bugs are proposed, reviewed and accepted as freeze exception bugs, and how freeze exception bugs are then tracked.

Note that freeze exception bugs used to be called nice-to-have or NTH bugs, and you may still find this name mentioned or used in old documentation and particularly in old bug reports.

See also the blocker bug process, which defines the similar process for blocker bugs - bugs that are blocking the release of its pre- and final releases and which must be fixed before these releases can proceed.

Freeze exception bug principles

Strict Fedora_Release_Criteria are used to define whether a bug should block a Fedora release, but determining whether a bug is worthy of a freeze exception is a more flexible process and should be done on a case-by-case basis. However, we have evolved some principles to be taken into account when evaluating a bug for freeze exception status.

In general, freeze exception bugs are usually bugs for which an update is not an optimal solution, and for which the fix is reasonably small and testable (this consideration becomes progressively more important as a release nears, so bugs may be downgraded from freeze exception status late in the release process if it transpires that the fix is complex and hard to test).

Types of bugs which are typically likely to be accepted as freeze exception bugs include:

  1. bugs which constitute infringements of the desktop-related Fedora_Release_Criteria as applied to non-default desktops
  2. bugs which result in a system being unable to reach a graphical environment
  3. significant installer bugs which do not meet the criteria to be blocker bugs

Proposing freeze exception bugs

To propose a bug as a freeze exception for a release, mark it as blocking the tracker bug for freeze exception bugs in that release. To do this, enter the alias or bug ID of the tracker bug into the Blocks: field in Bugzilla. The aliases for the freeze exception tracker bugs follow a consistent naming scheme. For the next release, the Alpha tracker will always be called AlphaFreezeException, the Beta tracker will always be called BetaFreezeException, and the final release tracker will always be called FinalFreezeException. Rarely, you may need to propose a bug as a freeze exception for the next release but one - in this case, prepend FXX (where XX is the release number) to the name of the alias and change Exception to Except, e.g. F23AlphaFreezeExcept. So, to mark a bug as a freeze exception for the release of Fedora 22 Beta, you would set it to block the bug BetaFreezeException.

Important.png
Must use Red Hat Bugzilla
Only bugs reported against a Fedora component in Red Hat Bugzilla can be marked as freeze exceptions for a Fedora release. If the bug you wish to mark as a freeze exception is being tracked in an upstream bug tracking system, you must file a corresponding bug in the Fedora bug tracking system before proposing it.

Reviewing freeze exception bugs

Proposed freeze exception bugs are reviewed and either accepted or rejected in collaboration between the three stakeholder groups: QA, Development and ReleaseEngineering. This is mostly done during the weekly blocker review meeting. Blocker review meetings usually occur every Friday during release periods, but special review meetings can be scheduled at other times when necessary. When appropriate, proposed freeze exception bugs may also be reviewed between meetings or during other meetings, such as the engineering readiness meeting (also known as a go/no-go meeting) which is convened to decide whether a release candidate should be approved as a final release. In these cases, consensus between the three stakeholder groups should still be reached in order to accept or reject a bug as a freeze exception.

Bugs that are accepted as freeze exceptions for the relevant release will be marked with the Whiteboard field AcceptedFreezeException. Bugs which are rejected will be updated to no longer block the relevant tracker bug, and have the RejectedFreezeException Whiteboard field added so that if they are proposed again, it is clear they have already been considered and rejected. Therefore, a bug which has been proposed but not accepted or rejected can be identified by the lack of a relevant Whiteboard field. All changes to freeze exception status should also be documented with a comment.

Automatic freeze exceptions

Certain types of bugs are considered automatic freeze exception bugs. These bugs can be marked as AcceptedFreezeException by any member of one of the stakeholder groups without formal review. A comment should accompany this change, explaining that it has been made under the automatic freeze exception policy and linking to this section of this page. If anyone believes that a bug has been incorrectly marked as AcceptedFreezeException in this way, they may propose that it be formally reviewed by appending a comment to the bug or by raising it during a freeze exception review meeting. Only the following types of bug are considered automatic freeze exception bugs. Note that where an item on this list applies to non-release-blocking images, a corresponding issue in a release-blocking image would likely be an automatic blocker, under the corresponding policy.

  • Bugs which entirely prevent the composition of one or more of the non-release-blocking images expected to be built for a currently-pending (pre-)release
  • Incorrect checksums present on any of the non-release-blocking TC/RC images (failures of QA:Testcase_Mediakit_ISO_Checksums)
  • Complete failure of any non-release-blocking TC/RC image to boot at all - "DOA" image (conditional failure is not an automatic blocker)
  • Any non-release-blocking Beta or Final TC/RC image exceeding its target size (failures of QA:Testcase_Mediakit_ISO_Size)

No other type of bug can be considered an automatic freeze exception bug under any circumstance. In particular, "I think it is obviously a freeze exception bug" is not a valid reason to use this procedure. If you believe another type of bug should be added to the list, please propose the change on the test@ mailing list.

Tracking freeze exception bugs

Again, tracking freeze exception bugs and making a best effort to fix them is a collaborative effort between the QA, Development and Release Engineering groups. After blocker bugs, QA group members are encouraged to prioritize testing of freeze exception bug fixes, development group members are encouraged to prioritize developing fixes for freeze exception bugs, and release engineering group members are encouraged to prioritize the release of fixes for freeze exception bugs (after appropriate testing). Each group should have its own processes for ensuring its responsibilities in relation to freeze exception bugs are met.

In Bugzilla, freeze exception bugs should follow the normal workflow, with special attention paid by the development group to submitting proposed fixes to the updates-testing repository so they reach MODIFIED and ON_QA status, and special attention paid by the QA group to testing proposed fixes and setting ones that are tested successfully to the VERIFIED status. Builds that fix freeze exception bugs should be pulled into TC and RC builds by the release engineering team as long as they are submitted as proposed updates and meet the necessary karma requirements. No freeze exception bug should be set to CLOSED ERRATA until a fix is actually released to the stable repository for the release in question: if a working fix is added to a test candidate or release candidate build, but not yet pushed to the stable repository, the bug should not be marked CLOSED ERRATA, as this may result in the fix not being pushed to the stable repository and the fix accidentally omitted from the next candidate build as it is no longer possible to track the bug.