Introduction
This document describes the tests that will be created and used to verify the functions/components of Fedora 21.
The goals of this plan are to:
- Evaluate from scratch the desired test coverage for a Fedora.next-based release
- Serve as a basis for deciding how much of that coverage is practically possible, and how it should be divided between teams
- Function as a reference document as we draw up new test cases and subsidiary plans to cover the Fedora.next products, and move forward with that testing for the first time
- Help us evaluate the test effort after the release of Fedora 21
Test Strategy
As has been the case for some time, there will be four broad strands of Fedora 21 testing:
- Automated testing (via Taskotron) - primarily focusing on CI-style validation tests of package builds and repository composes
- Ongoing ad hoc / casual / undirected user testing, both pre- and post-release, using Bodhi for update testing (see also QA:Update_feedback_guidelines, Updates_Policy), Bugzilla for bug reporting, and the test mailing list and #fedora-qa[?] IRC channel for discussion
- Planned manual validation testing of test composes and release candidates at each milestone, backed by the Release_criteria and QA:SOP_blocker_bug_process / QA:SOP_freeze_exception_bug_process
- Test Day events to co-ordinate group testing of particularly significant features or areas
Scope
Release deliverables
As regards Fedora 21 release deliverables, our goal is to verify to the best of our abilities whether the deliverables for each milestone meet the common and product-specific Release_criteria. Priority in this area should go to the deliverables considered most vital by the project as a whole. This is likely to include the Fedora.next product deliverables, the KDE live image, and generic (non-Product-specific) network install and DVD images if either/both are produced. See:
- Planned Server deliverables
- Planned Workstation deliverables
- Planned Cloud deliverables
- Fedora_21_Alpha_Release_Criteria
- Fedora_21_Beta_Release_Criteria
- Fedora_21_Final_Release_Criteria
- REQUIRED: product-specific release criteria
Packages and repositories
The goal of both automated and manual testing in this area is to prevent errors and bugs being introduced to the Fedora 21 repositories, both pre- and post-release. Our priorities are to catch updates which violate the Updates_Policy, break critical path functionality, prevent system updates from working correctly for end users (e.g. dependency problems or upgrade path violations), and prevent the composition of images (especially test compose / release candidate builds).