QA:Fedora 13 Install Test Plan

From FedoraProject

(Difference between revisions)
Jump to: navigation, search
(Testing Tasks)
(add Category:Install_Test_Plans)
(28 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 +
{{header|qa}}
 +
 
= Introduction =
 
= Introduction =
 
This document describes the tests that will be created and used to verify the functions/components of {{FedoraVersion|long|13}}.
 
This document describes the tests that will be created and used to verify the functions/components of {{FedoraVersion|long|13}}.
Line 13: Line 15:
  
 
As with Fedora 12, Fedora 13 will bring us some new features.  The following list outlines the larger changes that affect installation.  Test plans for these features will be designed/developed on each feature page.
 
As with Fedora 12, Fedora 13 will bring us some new features.  The following list outlines the larger changes that affect installation.  Test plans for these features will be designed/developed on each feature page.
 +
* [[Anaconda/Features/StorageFiltering|Continued Storage Filtering]] changes
 +
* [[Anaconda/Features/UICleanup|Storage UI]] changes
 +
* Replacing internal URL handling with {{package|libcurl}}
  
* N/A
 
 
Additional features outside the scope of testing can be found at:
 
Additional features outside the scope of testing can be found at:
 
* [[Anaconda/Features|List of anaconda features]]
 
* [[Anaconda/Features|List of anaconda features]]
Line 21: Line 25:
 
= Schedule/Milestones =
 
= Schedule/Milestones =
 
* The {{FedoraVersion|long|13}} release schedule is available at [[Releases/13/Schedule]]
 
* The {{FedoraVersion|long|13}} release schedule is available at [[Releases/13/Schedule]]
* Each major milestone(Beta,Preview,etc..) will demand a full regression run
+
* Each major milestone (Alpha, Beta, Final, etc..) will demand a full regression run
  
 
= Test Priority =
 
= Test Priority =
  
This test plan will use a 3 '''tier''' classification for test execution priority.
+
This test plan prioritizes tests according to the major release milestones for Fedora 13, including the Alpha, Beta and Final release milestones.  All test cases are intended for execution at every milestone.  However, priority should be given to tests specific to the milestone under test.
  
== [[QA/TestPlans/Fedora12InstallTier1|Tier#1 test cases]] ==
+
== [[QA/TestPlans/Fedora12InstallTier1|Alpha test cases]] ==
  
Tier1 is intended to verify that installation is possible on common hardware using common use cases.   
+
Alpha (formerly tier#1) priority tests are intended to verify that installation is possible on common hardware using common use cases.  These tests also attempt to validate [[Fedora_13_Alpha_Release_Criteria#Alpha_Release_Requirements|Alpha Release Requirements]].
  
 
Verification includes:
 
Verification includes:
 
* Common boot media
 
* Common boot media
 
* Common Installation source
 
* Common Installation source
* Installation using defaults installation options
+
* Installation using default installation options
* Default Partitioning
+
* Default Partition options
 +
* Default package installation
  
== [[QA/TestPlans/Fedora12InstallTier2|Tier#2 test cases]] ==
+
== [[QA/TestPlans/Fedora12InstallTier2|Beta test cases]] ==
  
Tier2 takes a step further to include more use cases.  Tier2 verification consists of:
+
Beta (formerly tier#2) priority tests take a step further to include additional use cases and installation methodsThese tests also attempt to validate [[Fedora_13_Beta_Release_Criteria#Beta_Release_Requirements|Beta Release Requirements]].
 +
 
 +
Beta verification consists of:
 
* All boot media
 
* All boot media
 
* All installation sources
 
* All installation sources
 
* All kickstart delivery methods
 
* All kickstart delivery methods
 
* Some architecture specific verification
 
* Some architecture specific verification
 +
* Some upgrade validation
  
== [[QA/TestPlans/Fedora12InstallTier3|Tier#3 test cases]] ==
+
== [[QA/TestPlans/Fedora12InstallTier3|Final test cases]] ==
  
Tier3 captures the remaining identified use cases:
+
Final (formerly tier#3) priority tests capture all remaining use cases and installation pathways.  These tests also attempt to validate [[Fedora_13_Final_Release_Criteria#Final_Release_Requirements|Final Release Requirements]].
 +
 
 +
Final verification consists of:
 
* More exhaustive partitioning schemes
 
* More exhaustive partitioning schemes
 
* More complex networking scenarios
 
* More complex networking scenarios
Line 65: Line 75:
 
* Any changes in composition of the installation tree are explainable by way of bugzilla
 
* Any changes in composition of the installation tree are explainable by way of bugzilla
  
== Alpha criteria ==
+
== Milestone specific criteria ==
* Entrance criteria have been met
+
* [[Fedora_13_Alpha_Release_Criteria]]
* All test tiers have have been executed
+
* [[Fedora_13_Beta_Release_Criteria]]
* All tier#1 tests pass
+
* [[Fedora_13_Final_Release_Criteria]]
 
+
== Beta criteria ==
+
* Alpha criteria have been met
+
* All tier#2 tests pass
+
 
+
== GA criteria ==
+
* Beta criteria have been met
+
* All test tiers must pass
+
* Any open defects have been documented as release notes
+
  
 
= Scope and Approach =
 
= Scope and Approach =
Line 83: Line 84:
 
Testing will include:
 
Testing will include:
  
* Manually execute test cases using a rawhide boot or live image media
+
* Manually execute test cases using DVD,CDROM,rawhide boot or live image media
* Where possible, the use of virtualization is encouraged to allow for greater test coverage and improve failure diagnosis
+
* Automatically execute test cases with auto-test system. More information about install auto-test ,please see [[Is_anaconda_broken_proposal]]
* Automated execution of tests is encouraged using kickstarts designed to execute specific test cases. At this time, kickstarts are not provided for every test case.
+
  
 
Items outside the scope of this test plan include:
 
Items outside the scope of this test plan include:
 
* Functional verification of software installed on the system
 
* Functional verification of software installed on the system
 
* Installation from media not generated by fedora release engineering
 
* Installation from media not generated by fedora release engineering
 +
* [[Architectures#Secondary_Architectures|Secondary hardware platforms]]
  
 
= Test Deliverables =
 
= Test Deliverables =
  
* [[QA:Fedora_12_Install_Test_Plan|This test plan]]
+
* This test plan
* [[:Category:Fedora_12_Test_Results|A test summary document for each major milestone]]
+
* test summary documents for each major milestone of F13: [[:Category:Fedora_13_Test_Results]]
 
* A list of defects filed
 
* A list of defects filed
 
* Any test scripts used for automation or verification
 
* Any test scripts used for automation or verification
Line 101: Line 102:
  
 
Testing will execute test cases to verify installation of Fedora 13 on different hardware platforms and gather installation test feedback.
 
Testing will execute test cases to verify installation of Fedora 13 on different hardware platforms and gather installation test feedback.
* [[QA:Fedora_12_Install_Results_Template#Test_Areas|Installation Test Cases]]
+
* [[QA:Fedora_13_Install_Results_Template#Test_Matrix|Manual installation Test Cases]]
* [[QA:Posting_Install_Test_Results|Instructions for adding test results]]
+
* [[Auto-installation Test Cases]]
 +
* [[QA:Create_Install_Test_Result_Page|Instructions for adding test result page]]
  
 
= Test Environment/Configs =
 
= Test Environment/Configs =
  
For {{FedoraVersion|long|12}},test cases will be executed on these hardware platforms:
+
For {{FedoraVersion|long|13}}, test cases will be executed on the [[Architectures#Primary_Architectures|''primary'' supported hardware platforms]].  This includes:
 
* i386
 
* i386
* ppc
 
 
* x86_64
 
* x86_64
  
Line 117: Line 118:
 
= Risks and Contingencies =
 
= Risks and Contingencies =
  
* If no physical media is available for testing, it is acceptable to use a <code>boot.iso</code> or PXE boot images from a current Rawhide release. For more information, see [[Releases/Rawhide#Direct_Rawhide_install]].
+
* If ''new'' physical media are provided for an already ''inprogress'' test run, a new test run '''must''' be initiated.  Test results from the previous run may be carried forward to the new test run if they are not affected by the changes introduced by the new physical media.
 +
* If physical media are '''not''' available for testing, it is acceptable to use a {{filename|boot.iso}} or PXE boot images from Rawhide. For more information, see [[Releases/Rawhide#Using_daily_Rawhide_Anaconda_build]].
  
 
= Reporting Bugs and Debugging Problems =
 
= Reporting Bugs and Debugging Problems =
Line 126: Line 128:
  
 
= Reviewers =
 
= Reviewers =
*
+
* [[User:Jlaska|jlaska]] 19:52, 7 February 2010 (UTC)
  
 
= References =
 
= References =
* [http://docs.fedoraproject.org/install-guide/f11/ {{FedoraVersion|long|11}} Installation Guide]
+
* [http://docs.fedoraproject.org/install-guide/f{{FedoraVersion||12}}/en-US/html/ {{FedoraVersion|long|12}} Installation Guide]
* Previous test plans available at [[:Category:Test_Plans]]  
+
 
* [https://fedoraproject.org/wiki/Category:Installer_Test_Cases Fedora Install test cases]
 
* [https://fedoraproject.org/wiki/Category:Installer_Test_Cases Fedora Install test cases]
 +
* Previous test plans available at [[:Category:Test_Plans]]
 
* Anaconda Documentation
 
* Anaconda Documentation
 
** [[Anaconda/Options|Supported command-line options]]
 
** [[Anaconda/Options|Supported command-line options]]
Line 139: Line 141:
  
 
[[Category:Test Plans]]
 
[[Category:Test Plans]]
 +
[[Category:Install_Test_Plans]]

Revision as of 06:04, 13 July 2011

QA.png


Contents

Introduction

This document describes the tests that will be created and used to verify the functions/components of Fedora 13.

The goals of this plan are to:

  • Organize the test effort
  • Communicate the planned tests to all relevant stake-holders for their input and approval
  • Serve as a base for the test planning for future Fedora 13 releases

Test Strategy

Instead of outlining all possible installation inputs and outputs, this test plan will focus on defining inputs and outputs at different stages in anaconda. This will also allow different tests to be performed independently during a single installation. For example, one may execute a kickstart delivery via HTTP, raid0 partitioning using 3 physical disks, and a minimal package installation on a para-virtualized xen guest all in single installation. Scenarios where the stages are dependent will be indicated as such in the test case.

New features of Fedora 13

As with Fedora 12, Fedora 13 will bring us some new features. The following list outlines the larger changes that affect installation. Test plans for these features will be designed/developed on each feature page.

Additional features outside the scope of testing can be found at:

Schedule/Milestones

  • The Fedora 13 release schedule is available at Releases/13/Schedule
  • Each major milestone (Alpha, Beta, Final, etc..) will demand a full regression run

Test Priority

This test plan prioritizes tests according to the major release milestones for Fedora 13, including the Alpha, Beta and Final release milestones. All test cases are intended for execution at every milestone. However, priority should be given to tests specific to the milestone under test.

Alpha test cases

Alpha (formerly tier#1) priority tests are intended to verify that installation is possible on common hardware using common use cases. These tests also attempt to validate Alpha Release Requirements.

Verification includes:

  • Common boot media
  • Common Installation source
  • Installation using default installation options
  • Default Partition options
  • Default package installation

Beta test cases

Beta (formerly tier#2) priority tests take a step further to include additional use cases and installation methods. These tests also attempt to validate Beta Release Requirements.

Beta verification consists of:

  • All boot media
  • All installation sources
  • All kickstart delivery methods
  • Some architecture specific verification
  • Some upgrade validation

Final test cases

Final (formerly tier#3) priority tests capture all remaining use cases and installation pathways. These tests also attempt to validate Final Release Requirements.

Final verification consists of:

  • More exhaustive partitioning schemes
  • More complex networking scenarios
  • More architecture specific verification
  • Network device
  • Storage device
  • Upgrade testing

Test Pass/Fail Criteria

The milestone release of Fedora 13 should conform these criteria:

Entrance criteria

  • Trees must be generated using release engineering tools (not hand crafted)
  • There must be no unresolved dependencies for packages included in the installation tree
  • There must be no dependency conflicts for packages included in the installation tree
  • Any changes in composition of the installation tree are explainable by way of bugzilla

Milestone specific criteria

Scope and Approach

Testing will include:

  • Manually execute test cases using DVD,CDROM,rawhide boot or live image media
  • Automatically execute test cases with auto-test system. More information about install auto-test ,please see Is_anaconda_broken_proposal

Items outside the scope of this test plan include:

  • Functional verification of software installed on the system
  • Installation from media not generated by fedora release engineering
  • Secondary hardware platforms

Test Deliverables

  • This test plan
  • test summary documents for each major milestone of F13: Category:Fedora_13_Test_Results
  • A list of defects filed
  • Any test scripts used for automation or verification

Testing Tasks

Testing will execute test cases to verify installation of Fedora 13 on different hardware platforms and gather installation test feedback.

Test Environment/Configs

For Fedora 13, test cases will be executed on the primary supported hardware platforms. This includes:

  • i386
  • x86_64

Responsibilities

Fedora QA team members are responsible for executing this test plan. Contributions from Rawhide testers and other interested parties are encouraged.

Risks and Contingencies

  • If new physical media are provided for an already inprogress test run, a new test run must be initiated. Test results from the previous run may be carried forward to the new test run if they are not affected by the changes introduced by the new physical media.
  • If physical media are not available for testing, it is acceptable to use a boot.iso or PXE boot images from Rawhide. For more information, see Releases/Rawhide#Using_daily_Rawhide_Anaconda_build.

Reporting Bugs and Debugging Problems

If defects/problems are encountered, please go ahead and file the bugs following the guide below:

Reviewers

  • jlaska 19:52, 7 February 2010 (UTC)

References