2010-02-11 kparal - First draft
The purpose of this document is to present a set of rules that determine whether a package update is considered PASSED (i.e. it is ready to be committed to repository), FAILED (i.e. the build should be discarded and a fix should be requested) or NEEDS_INSPECTION (i.e. similar to FAILED at the moment, but it becomes PASSED if all errors are waived).
The test plan is divided into three main sections:
- Mandatory tests
- Introspection tests
- Advisory tests
Mandatory tests are executed as the first ones, in arbitrary order. After all mandatory tests have been executed and passed, the introspection and advisory tests are executed in arbitrary order.
Do we really need to have the strict requirement that introspection and advisory tests are executed only after mandatory tests have completed?
This test plan contains only tests that can be automated, i.e. executed without human intervention. No manual tests will be performed. (But this does not reject human intervention when inspecting the test results, like waiving false errors.)
Test Pass/Fail Criteria
All mandatory tests must pass. If some of mandatory tests fail, the testing stops and the package update is considered FAILED.
All introspection tests must pass. If an introspection test fails, other tests continue to be executed. If some of introspection tests fail, the package is considered NEEDS_INSPECTION. An eligible person may inspect errors of an failed introspection test and waive them.
The package update is considered PASSED if there are no failed mandatory tests and all failed introspection tests are waived.
The results of advisory tests have no impact on package update validity.
Do all advisory tests must be completed before we mark package as PASSED? In order to avoid a situation where package is marked as PASSED but we still can't inspect all the advisory tests' output.
- A decision if a particular package update has PASSED, FAILED or NEEDS_INSPECTION
- A full log of individual test cases' output containing infos, errors or warnings
- A special output marked as informative from any test case that passed but wants to convey additional information
- Any test scripts used for automation or verification
Note: More tests are going to be added in the future (init-scripts test, manpage test, desktop-file test, etc).
- Test package sanity - must be installable, uninstallable, upgradeable, etc
- Repo sanity - the update must not break dependencies of other packages
- Conflicts - there must be no file/package conflicts
- Upgrade path - the update must not break upgrade path
- Rpmlint - no errors present, warnings count is under certain threshold
- Rpmlint - no warnings present
- Rpmguard - no warnings present
Can we move some rpmguard checks into introspection tests?
- AutoQA (the automated test execution framework) must be packaged and deployed in Fedora infrastructure. It will be used for test execution.
- Test cases will be executed on all hardware platforms relevant for that particular package build. The only exception being test cases which can be executed in a platform-neutral way.
- For those test cases, for which it makes sense, their execution should be done in a software environment as close as is feasible to that which will exist after the package is pushed to 'updates' repository. Generally this will be with a system that is up to date from 'fedora' and 'updates' repositories, but does not contain other packages from 'updates-testing' repository (except for the dependencies or other packages built from the affected source package - they should be updated in generally installed).
Quality Assurance team members are responsible for executing this test plan.
Introspection tests failures may be waived by:
- Quality Assurance team members
- Release Engineering team members
- package update builder
- package maintainer
When all introspection tests failures are waived, the state change from NEEDS_INSPECTION to PASSED is to be handled automatically by Quality Assurance team tools.
The aforementioned four groups of people may also re-execute the test plan on a particular package.
This test plan is executed on every package update build and on all new packages builds.
- If AutoQA needed for automated testing is not available (outage, crash) and test plan execution is not possible, the package updates may be stalled, automatically refused or automatically accepted, depending on Fedora QA team decision.