QA:Package Update Acceptance Test Plan

From FedoraProject

(Difference between revisions)
Jump to: navigation, search
(re-worded risks)
(comment out proven testers (proven tester group dormant))
 
(44 intermediate revisions by 3 users not shown)
Line 1: Line 1:
{{draft}}
 
 
= Revision history =
 
 
2010-02-11 kparal - First draft
 
 
 
= Introduction =
 
= Introduction =
  
The purpose of this document is to present a set of rules that determine whether a package update is considered '''PASSED''' (i.e. it is ready to be committed to repository), '''FAILED''' (i.e. the build should be discarded and a fix should be requested) or '''NEEDS_INSPECTION''' (i.e. similar to FAILED at the moment, but it becomes PASSED if all errors are waived).
+
The purpose of this document is to present a set of rules that determine whether a package update is considered '''ACCEPTED''' (i.e. it is ready to be committed to repository), '''REJECTED''' (i.e. the build should be discarded and a fix should be requested) or '''NEEDS_INSPECTION''' (i.e. similar to REJECTED at the moment, but it becomes ACCEPTED if all errors are waived).
  
 
= Test Strategy =
 
= Test Strategy =
Line 18: Line 12:
 
= Test Priority =
 
= Test Priority =
  
Mandatory tests are executed as the first ones, in arbitrary order. After all mandatory tests have been executed and passed, the introspection and advisory tests are executed in arbitrary order.
+
All tests are executed in arbitrary order. If any of available tests fails, other tests still continue to be executed.
 
+
''Do we really need to have the strict requirement that introspection and advisory tests are executed only after mandatory tests have completed?''
+
  
 
= Scope =
 
= Scope =
Line 28: Line 20:
 
= Test Pass/Fail Criteria =
 
= Test Pass/Fail Criteria =
  
All mandatory tests must pass. If some of mandatory tests fail, the testing stops and the package update is considered '''FAILED'''.
+
All mandatory tests must pass. If some of mandatory tests fail then the package update is considered '''REJECTED'''.
  
All introspection tests must pass. If an introspection test fails, other tests continue to be executed. If some of introspection tests fail, the package is considered '''NEEDS_INSPECTION'''. An eligible person may inspect errors of an failed introspection test and waive them.
+
All introspection tests must pass. If some of introspection tests fails, the package is considered '''NEEDS_INSPECTION''' (provided that all mandatory tests passed, otherwise it stays REJECTED). An [[#Responsibilities_and_permissions|eligible person]] may inspect errors of a failed introspection test and waive them.
  
The package update is considered '''PASSED''' if there are no failed mandatory tests and all failed introspection tests are waived.
+
The package update is considered '''ACCEPTED''' when:
 +
* all mandatory and introspection tests have finished cleanly (not crashed or otherwise aborted)
 +
* there are no failed mandatory tests
 +
* all failed introspection tests are waived
  
The results of advisory tests have no impact on package update validity.
+
The results of advisory tests have no impact on package update acceptance.
 
+
''Do all advisory tests must be completed before we mark package as PASSED? In order to avoid a situation where package is marked as PASSED but we still can't inspect all the advisory tests' output.''
+
  
 
= Test Deliverables =
 
= Test Deliverables =
  
* A decision if a particular package update has PASSED, FAILED or NEEDS_INSPECTION
+
* A decision if a particular package update has been ACCEPTED, REJECTED or NEEDS_INSPECTION.
* A full log of individual test cases' output containing infos, errors or warnings
+
* A full log of individual test cases' output containing infos, errors or warnings.
* A special output marked as informative from any test case that PASSED but wants to convey additional information
+
* A special output marked as informative from any test case that passed but wants to convey additional information.
* Any test scripts used for automation or verification
+
* Any test scripts used for automation or verification.
  
 
= Test Cases  =
 
= Test Cases  =
  
Note: More tests are going to be added in the future (init-scripts test, manpage test, desktop-file test, etc).
+
''Note: More tests are going to be added in the future (manpage test, desktop-file test, etc).''
  
 
== Mandatory tests ==
 
== Mandatory tests ==
  
* Test package sanity - must be installable, uninstallable, upgradeable, etc
+
* [[QA:Package_Sanity_Test_Plan|Package sanity]] - must be installable, uninstallable, upgradeable, etc
* Repo sanity - the update must not break dependencies of other packages
+
* [[QA:Depcheck Test Case|Depcheck]] - the update must not break dependencies of other packages
* Conflicts - there must be no file/package conflicts
+
* [[QA:Conflicts Test Case|Conflicts]] - there must be no file/package conflicts
* Upgrade path - the update must not break upgrade path
+
* [[QA:Upgrade Path Test Case|Upgrade path]] - the update must not break upgrade path
  
 
== Introspection tests ==
 
== Introspection tests ==
  
* Rpmlint - no errors present, warnings count is under certain threshold
+
* [[QA:Rpmlint Test Case|Rpmlint]] - no errors present
 +
* [[QA:Initscripts Test Case|Initscripts]] - no errors present
  
 
== Advisory tests ==
 
== Advisory tests ==
  
* Rpmlint - no warnings present
+
* [[QA:Rpmlint Test Case|Rpmlint]] - no warnings present
* Rpmguard - no warnings present
+
* [[QA:Rpmguard Test Case|Rpmguard]] - no warnings present
 
+
''Can we move some [https://fedorahosted.org/autoqa/wiki/RpmguardChecks rpmguard checks] into introspection tests?''
+
  
 
= Test Environment =
 
= Test Environment =
Line 73: Line 65:
 
* Test cases will be executed on all hardware platforms relevant for that particular package build. The only exception being test cases which can be executed in a platform-neutral way.
 
* Test cases will be executed on all hardware platforms relevant for that particular package build. The only exception being test cases which can be executed in a platform-neutral way.
  
* For those test cases, for which it makes sense, their execution should be done in a software environment as close as is feasible to that which will exist after the package is pushed to '-updates' repository. Generally this will be with a system that is up to date from 'release' and '-updates' repositories, but does not contain other packages from '-updates-testing' repository (except for the dependencies or other packages built from the affected source package - they should be updated in generally installed).
+
* The test cases should strive for testing updates in such a way that is as close as possible to regular update process for the end-user. That means that generally the system should be fully up to date from 'fedora' and 'updates' repositories before running the test case. Also if there is a whole set of pending updates that are going to be accepted and pushed to stable repositories together, the testing should consider that set as a whole, not just individual packages in it.
  
= Responsibilities =
+
= Responsibilities and permissions =
  
Fedora QA team members are responsible for executing this test plan.
+
* [[QA|Quality Assurance]] team members are responsible for executing this test plan.
  
Introspection tests failures may be waived by:
+
* Introspection tests failures may be waived by:
* Fedora QA team members
+
** package maintainer
* Release Engineering team members
+
** package update builder
* package update builder
+
** update ticket author
* package maintainer
+
<!--[proventesters dormant] ** [[Proven_tester|ProvenTesters]] team members-->
  
When all introspection tests failures are waived, the state change from NEEDS_INSPECTION to PASSED is to be handled automatically by Fedora QA team tools.
+
* It is the responsibility of the waiver to ensure that he/she waives only false errors, not real problems.
  
The aforementioned four groups or persons may also re-execute the test plan on a particular package.
+
* When all introspection tests failures are waived, the state change from NEEDS_INSPECTION to ACCEPTED is to be handled automatically by Quality Assurance team tools.
 +
 
 +
* Groups of people that are allowed to waive errors may also re-execute the test plan on a particular package.
  
 
= Schedule =
 
= Schedule =
  
This test plan is executed on every package update build and on all new packages builds.
+
This test plan is executed on every package update or new package build that is submitted to 'updates-testing' or 'updates' repository through [https://admin.fedoraproject.org/updates/ Fedora Updates].
  
 
= Risks =
 
= Risks =
  
* If [[AutoQA]] needed for automated testing is not available (outage, crash) and test plan execution is not possible, the package updates may be stalled, automatically refused or automatically accepted, depending on Fedora QA team decision.
+
* If [[AutoQA]] needed for automated testing is not available (outage, crash) and test plan execution is not possible, the package updates may be queued, automatically refused or automatically accepted, depending on [[QA|Quality Assurance]] team decision.
 
+
= Reviewers =
+
 
+
{{{reviewedby|}}}
+
  
= References =
 
  
<references/>
+
[[Category:Test Plans]]

Latest revision as of 03:55, 6 February 2013

Contents

[edit] Introduction

The purpose of this document is to present a set of rules that determine whether a package update is considered ACCEPTED (i.e. it is ready to be committed to repository), REJECTED (i.e. the build should be discarded and a fix should be requested) or NEEDS_INSPECTION (i.e. similar to REJECTED at the moment, but it becomes ACCEPTED if all errors are waived).

[edit] Test Strategy

The test plan is divided into three main sections:

  1. Mandatory tests
  2. Introspection tests
  3. Advisory tests

[edit] Test Priority

All tests are executed in arbitrary order. If any of available tests fails, other tests still continue to be executed.

[edit] Scope

This test plan contains only tests that can be automated, i.e. executed without human intervention. No manual tests will be performed. (But this does not reject human intervention when inspecting the test results, like waiving false errors.)

[edit] Test Pass/Fail Criteria

All mandatory tests must pass. If some of mandatory tests fail then the package update is considered REJECTED.

All introspection tests must pass. If some of introspection tests fails, the package is considered NEEDS_INSPECTION (provided that all mandatory tests passed, otherwise it stays REJECTED). An eligible person may inspect errors of a failed introspection test and waive them.

The package update is considered ACCEPTED when:

  • all mandatory and introspection tests have finished cleanly (not crashed or otherwise aborted)
  • there are no failed mandatory tests
  • all failed introspection tests are waived

The results of advisory tests have no impact on package update acceptance.

[edit] Test Deliverables

  • A decision if a particular package update has been ACCEPTED, REJECTED or NEEDS_INSPECTION.
  • A full log of individual test cases' output containing infos, errors or warnings.
  • A special output marked as informative from any test case that passed but wants to convey additional information.
  • Any test scripts used for automation or verification.

[edit] Test Cases

Note: More tests are going to be added in the future (manpage test, desktop-file test, etc).

[edit] Mandatory tests

  • Package sanity - must be installable, uninstallable, upgradeable, etc
  • Depcheck - the update must not break dependencies of other packages
  • Conflicts - there must be no file/package conflicts
  • Upgrade path - the update must not break upgrade path

[edit] Introspection tests

[edit] Advisory tests

[edit] Test Environment

  • AutoQA (the automated test execution framework) must be packaged and deployed in Fedora infrastructure. It will be used for test execution.
  • Test cases will be executed on all hardware platforms relevant for that particular package build. The only exception being test cases which can be executed in a platform-neutral way.
  • The test cases should strive for testing updates in such a way that is as close as possible to regular update process for the end-user. That means that generally the system should be fully up to date from 'fedora' and 'updates' repositories before running the test case. Also if there is a whole set of pending updates that are going to be accepted and pushed to stable repositories together, the testing should consider that set as a whole, not just individual packages in it.

[edit] Responsibilities and permissions

  • Introspection tests failures may be waived by:
    • package maintainer
    • package update builder
    • update ticket author
  • It is the responsibility of the waiver to ensure that he/she waives only false errors, not real problems.
  • When all introspection tests failures are waived, the state change from NEEDS_INSPECTION to ACCEPTED is to be handled automatically by Quality Assurance team tools.
  • Groups of people that are allowed to waive errors may also re-execute the test plan on a particular package.

[edit] Schedule

This test plan is executed on every package update or new package build that is submitted to 'updates-testing' or 'updates' repository through Fedora Updates.

[edit] Risks

  • If AutoQA needed for automated testing is not available (outage, crash) and test plan execution is not possible, the package updates may be queued, automatically refused or automatically accepted, depending on Quality Assurance team decision.