Introduction
This document describes the tests that will be created and used to verify the functions/components of Fedora 21.
The goals of this plan are to:
- Evaluate from scratch the desired test coverage for a Fedora.next-based release
- Serve as a basis for deciding how much of that coverage is practically possible, and how it should be divided between teams
- Function as a reference document as we draw up new test cases and subsidiary plans to cover the Fedora.next products, and move forward with that testing for the first time
- Help us evaluate the test effort after the release of Fedora 21
Test Strategy
As has been the case for some time, there will be four broad strands of Fedora 21 testing:
- Automated testing (via Taskotron) - primarily focusing on CI-style validation tests of package builds and repository composes
- Ongoing ad hoc / casual / undirected user testing, both pre- and post-release, using Bodhi for update testing (see also QA:Update_feedback_guidelines, Updates_Policy), Bugzilla for bug reporting, and the test mailing list and #fedora-qa[?] IRC channel for discussion
- Planned manual validation testing of test composes and release candidates at each milestone, backed by the Release_criteria and QA:SOP_blocker_bug_process / QA:SOP_freeze_exception_bug_process
- Test Day events to co-ordinate group testing of particularly significant features or areas
Scope
Release deliverables
As regards Fedora 21 release deliverables, our goal is to verify to the best of our abilities whether the deliverables for each milestone meet the common and product-specific Release_criteria. Priority in this area should go to the deliverables considered most vital by the project as a whole. This is likely to include the Fedora.next product deliverables, the KDE live image, and generic (non-Product-specific) network install and DVD images if either/both are produced. See:
- Planned Server deliverables
- Planned Workstation deliverables
- Planned Cloud deliverables
- Fedora_21_Alpha_Release_Criteria
- Fedora_21_Beta_Release_Criteria
- Fedora_21_Final_Release_Criteria
- REQUIRED: product-specific release criteria
Packages and repositories
The goal of both automated and manual testing in this area is to prevent errors and bugs being introduced to the Fedora 21 repositories, both pre- and post-release. Our priorities are to catch updates which violate the Updates_Policy, break critical path functionality, prevent system updates from working correctly for end users (e.g. dependency problems or upgrade path violations), and prevent the composition of images (especially test compose / release candidate builds).
Responsibilities
QA team
The QA team's responsibilities are:
- Overall responsibility for Fedora 21 testing, as with all Fedora testing.
- Ensuring this test plan itself is implemented: i.e. to ensure that all the test mechanisms described herein are created, and that the testing envisaged under this plan is carried out (or, if there are insuperable problems in accomplishing either, alerting FESCo, the Board and/or the Project_Leader to the problems).
- Maintaining the non-Product-specific release criteria and release validation test cases. QA team will request input from other teams, particularly from the Base working group, developers/package maintainers, and from ReleaseEngineering, as appropriate.
- Maintaining the release validation process documentation.
- Performing non-Product-specific release validation testing.
- Performing non-Product-specific ad hoc / unplanned testing.
- Maintaining the automated testing infrastructure.
Product Working Groups
The Fedora.next#Working_groups are responsible for drawing up release criteria and release validation test cases specific to their products: that is, the Server WG is responsible for drawing up a set of Server-specific release criteria and test cases, the Workstation WG is responsible for drawing up a set of Workstation-specific release criteria and test cases, and so forth. The QA team will provide assistance with this work.
The QA team and Working Groups share responsibility for performing Product-specific validation testing. In particular, Working Groups will be expected to contribute strongly to testing which requires substantial Product-related expertise and/or equipment/configuration.
The QA team and Working Groups also share responsibility for performing ongoing ad hoc / unplanned testing of components that relate to Products. As a rule of thumb, the more deeply a component is tied to a Product, the more the Working Group rather than the QA team should be considered responsible for testing it.