QA:Security Lab validation testing
Before an official Fedora release comes out, several alpha, beta and release candidate releases - known collectively as pre-releases - are made available. As part of the validation process for each pre-release, the QA group tests to see that various basic functions meet pre-defined quality standards. You can find the requirements for each release in the Fedora_Release_Criteria.
Scope and Approach
Testing will include:
- Manually executed test cases using Fedora Security Spin live image media
This is one of several release validation test plans. Each covers a particular type of testing - installation, base, desktop etc. This test plan on its own does not cover all release criteria, but the set of release validation test plans taken together ought to do so.
Timing of base test events
Base test events occur before milestone releases. Current events can be found at:
A set of test cases forms the basis of base validation testing. These test cases are designed to ensure that the release under testing complies with the Fedora_Release_Criteria: the test cases derive from the criteria, not the other way around. As and when the criteria are adjusted or updated, new test cases should be created and no longer valid ones removed as appropriate.
For each candidate, pre-release, and release build to be tested, the entire set of base validation test cases should be run. Tests can be performed by any QA group member, and multiple test runs are acceptable and encouraged. Each test case is associated with a particular release stage (Alpha, Beta or Final), depending on which set of release criteria the criterion it is intended to validate forms a part of. Even though all tests are run at each stage, only those associated with the current stage or an earlier stage will be considered as blocking the release. That is, if an Alpha test fails at Final stage, the failure blocks the final release; but if a Final test fails at Alpha stage, the Alpha release is not blocked.
Any failure of any test must be reported using the appropriate tracking system - usually Bugzilla. If the failure constitutes an infringement of the Fedora_Release_Criteria, as is usually the case for the default desktop tests, the bug should be marked as blocking the appropriate release (Alpha, Beta or Final, according to the level of the test and criterion involved), according to the blocker bug process.
Testing can be performed using the pre-release or release images, or candidate images for pre-releases or releases as provided to the QA group by the Release Engineering group. Testing must be performed on the primary architectures.
Results are recorded in a separate page for each test run. The results for the upcoming Fedora 21 release can be found in this category. The results for the current Fedora 20 release can be found in this category. These pages are created based on a template page.
The procedure for managing base validation testing is documented as part of the general release validation testing procedure.