From Fedora Project Wiki

(Test Strategy)
(obsolete (old release))
 
(63 intermediate revisions by 4 users not shown)
Line 1: Line 1:
{{admon/note|DRAFT|This page is currently a draft.  This test plan has not been finalized yet.  Stay tuned ...}}
 
 
 
= Introduction =
 
= Introduction =
 
This document describes the tests that will be created and used to verify the functions/components of {{FedoraVersion|long|12}}.
 
This document describes the tests that will be created and used to verify the functions/components of {{FedoraVersion|long|12}}.
Line 9: Line 7:
 
* Serve as a base for the test planning for future Fedora 12 releases
 
* Serve as a base for the test planning for future Fedora 12 releases
 
= Test Strategy =
 
= Test Strategy =
Testing will include:
+
Instead of outlining all possible installation inputs and outputs, this test plan will focus on defining inputs and outputs at different ''stages'' in anaconda.  This will also allow different tests to be performed independently during a single installation.  For example, one may execute a kickstart delivery via HTTP, raid0 partitioning using 3 physical disks, and a minimal package installation on a para-virtualized xen guest all in single installation. Scenarios where the ''stages'' are dependent will be indicated as such in the test case.
* Manually execute test cases using a live image of rawhide or DVD/CD install media
 
* Manually/kickstart boot the installation program to execute the installation
 
* Automatically execute installation testing via cobbler/koan etc..,cobbler guide and scripts,please refer to [http://jlaska.livejournal.com/tag/cobbler cobbler]
 
 
 
Items outside the scope of this test plan include:
 
* Functional verification of software installed on the system
 
* Installation from media not generated by fedora release engineering
 
  
 
= New features of {{FedoraVersion|long|12}} =
 
= New features of {{FedoraVersion|long|12}} =
  
{{admon/important|FIXME|Need confirmation on the following list of features for {{FedoraVersion|long|12}}}}
+
As with Fedora 11, Fedora 12 will bring us some new features.  The following list outlines the larger changes that affect installation.  Test plans for these features will be designed/developed on each feature page.
  
As Fedora 11,Fedora 12 will bring us some new features.The following list of features was obtained from Anaconda/Features.  Test plans for these features will be designed/developed on each feature page.
+
* [[Anaconda/Features/MDRaid|Use MDRaid instead of dmraid]]
 +
* [[Anaconda/Features/StorageFiltering|Select devices better, plus storage cleanup]]
 +
* [[Anaconda/Features/UICleanup|Storage UI Cleanup]]
  
* Anaconda/Features at [[Anaconda/Features]]
+
Additional features outside the scope of testing can be found at:
* Fedora 12 Accepted Features at [[Releases/12/FeatureList]]
+
* [[Anaconda/Features|List of anaconda features]]
 +
* [[Releases/12/FeatureList|Fedora 12 Accepted Features]]
  
 
= Schedule/Milestones =
 
= Schedule/Milestones =
 
* The {{FedoraVersion|long|12}} release schedule is available at [[Releases/12/Schedule]]
 
* The {{FedoraVersion|long|12}} release schedule is available at [[Releases/12/Schedule]]
* Each major milestone(Alpha,Beta,Preview,etc..) will demand a full regression run
+
* Each major milestone(Beta,Preview,etc..) will demand a full regression run
 +
 
 +
= Test Priority =
 +
 
 +
This test plan will use a 3 '''tier''' classification for test execution priority.
 +
 
 +
== [[QA/TestPlans/Fedora12InstallTier1|Tier#1 test cases]] ==
 +
 
 +
Tier1 is intended to verify that installation is possible on common hardware using common use cases. 
 +
 
 +
Verification includes:
 +
* Common boot media
 +
* Common Installation source
 +
* Installation using defaults installation options
 +
* Default Partitioning
 +
 
 +
== [[QA/TestPlans/Fedora12InstallTier2|Tier#2 test cases]] ==
 +
 
 +
Tier2 takes a step further to include more use cases.  Tier2 verification consists of:
 +
* All boot media
 +
* All installation sources
 +
* All kickstart delivery methods
 +
* Some architecture specific verification
 +
 
 +
== [[QA/TestPlans/Fedora12InstallTier3|Tier#3 test cases]] ==
 +
 
 +
Tier3 captures the remaining identified use cases:
 +
* More exhaustive partitioning schemes
 +
* More complex networking scenarios
 +
* More architecture specific verification
 +
* Network device
 +
* Storage device
 +
* Upgrade testing
 +
 
 +
= Test Pass/Fail Criteria =
 +
 
 +
The milestone release of Fedora 12 should conform these criteria:
 +
 
 +
== Entrance criteria ==
 +
* Trees must be generated using release engineering tools (not hand crafted)
 +
* There must be no unresolved dependencies for packages included in the installation tree
 +
* There must be no dependency conflicts for packages included in the installation tree
 +
* Any changes in composition of the installation tree are explainable by way of bugzilla
 +
 
 +
== Alpha criteria ==
 +
* Entrance criteria have been met
 +
* All test tiers have have been executed
 +
* All tier#1 tests pass
  
= Community Testing =
+
== Beta criteria ==
'''Test Day:'''
+
* Alpha criteria have been met
*We run Test Day to co-ordinate focused testing on a specific feature or component in community.
+
* All tier#2 tests pass
**The {{FedoraVersion|long|12}} Test Day is available at [[QA/Test_Days/F12]]
 
**As usual,Test Day will be announced in mailing list [https://www.redhat.com/mailman/listinfo/fedora-test-list fedora-test-list]
 
  
'''Milestone Install Testing:'''
+
== GA criteria ==
* We use Milestone Install Testing to verify installation against Fedora 12 milestone Release Candidate builds on different hardware platforms.
+
* Beta criteria have been met
**Fedora 12 milestones can be found at [[Releases/12/Schedule]].
+
* All test tiers must pass
**As usual,Milestone Install Testing will be announced in mailing list [https://www.redhat.com/mailman/listinfo/fedora-test-list fedora-test-list]
+
* Any open defects have been documented as release notes
**Test cases for Fedora 12 Milestone Install Testing ,please refer to [[QA:Fedora_12_Install_Results_Template#Test_Areas]]
 
**Please select and test the test cases that best match your test environment for Fedora 12 milestone Release Candidate,and post test results according to template at [[QA:Fedora_12_Install_Results_Template]]
 
**Reference of posting test results at [https://fedoraproject.org/wiki/QA:Fedora_12_Install_Results_Template#Posting_Test_Results post test results]
 
* This test plan will use  [https://fedoraproject.org/wiki/QA:Fedora_12_Install_Results_Template#Priority priority of 3 level] for Milestone Install Test cases
 
  
 
= Scope and Approach =
 
= Scope and Approach =
 +
 
Testing will include:
 
Testing will include:
* Manually execute test cases of Test Day using a live image of rawhide or updated system to rawhide
+
 
* Manually/kickstart boot the installation program to execute the installation
+
* Manually execute test cases using a rawhide boot or live image media
* Automatically execute installation testing via cobbler/koan etc..,cobbler guide and scripts,please refer to [http://jlaska.livejournal.com/tag/cobbler cobbler]
+
* Where possible, the use of virtualization is encouraged to allow for greater test coverage and improve failure diagnosis
 +
* Automated execution of tests is encouraged using kickstarts designed to execute specific test cases. At this time, kickstarts are not provided for every test case.
  
 
Items outside the scope of this test plan include:
 
Items outside the scope of this test plan include:
Line 56: Line 93:
 
* Installation from media not generated by fedora release engineering
 
* Installation from media not generated by fedora release engineering
  
= Reporting Bugs and Debugging Problems=
+
= Test Deliverables =
If you catch some bugs during testing,we encourage you to go ahead and file the bugs.If you a new person here,please follow the guide below
+
 
* Bug Reporting Process - [[Bugs_and_feature_requests]]
+
* [[QA:Fedora_12_Install_Test_Plan|This test plan]]
* Anaconda Bug Reporting Guide - [[Anaconda/BugReporting]]
+
* [[:Category:Fedora_12_Test_Results|A test summary document for each major milestone]]
 +
* A list of defects filed
 +
* Any test scripts used for automation or verification
 +
 
 +
= Testing Tasks =
 +
 
 +
Testing will execute test cases to verify installation of Fedora 12 on different hardware platforms and gather installation test feedback.
 +
* [[QA:Fedora_12_Install_Results_Template#Test_Areas|Installation Test Cases]]
 +
* [[QA:Create_Install_Test_Result_Page|Instructions for adding test result page]]
  
 
= Test Environment/Configs =
 
= Test Environment/Configs =
  
For {{FedoraVersion|long|12}}, we will execute test cases on these hardware platforms:
+
For {{FedoraVersion|long|12}},test cases will be executed on these hardware platforms:
 
* i386
 
* i386
 
* ppc
 
* ppc
 
* x86_64
 
* x86_64
  
= Communicate =
+
= Responsibilities =
If you want to participate test day of Fedora 12,you can get help or give feedback by any of these ways:
+
 
* IRC: #fedora-qa on irc.freenode.net
+
Fedora QA team members are responsible for executing this test plan. Contributions from Rawhide testers and other interested parties are encouraged.
* Mailing list: [https://www.redhat.com/mailman/listinfo/fedora-test-list fedora-test-list]
+
 
* Developers and QA will be available on test day at [[QA/Fedora_12_test_days]]
+
= Risks and Contingencies =
* Reference of ways to communicate at [[Communicate]]
+
 
 +
* If no physical media is available for testing, it is acceptable to use a <code>boot.iso</code> or PXE boot images from a current Rawhide release. For more information, see [[Releases/Rawhide#Direct_Rawhide_install]].
 +
 
 +
= Reporting Bugs and Debugging Problems =
 +
 
 +
If defects/problems are encountered, please go ahead and file the bugs following the guide below:
 +
* [[Bugs_and_feature_requests|Bug Reporting Process]]
 +
* [[How to debug installation problems]]
 +
 
 +
= Reviewers =
 +
* [[User:Jlaska|jlaska]] 19:08, 3 August 2009 (UTC)
  
 
= References =
 
= References =
 
* [http://docs.fedoraproject.org/install-guide/f11/ {{FedoraVersion|long|11}} Installation Guide]
 
* [http://docs.fedoraproject.org/install-guide/f11/ {{FedoraVersion|long|11}} Installation Guide]
 
* Previous test plans available at [[:Category:Test_Plans]]  
 
* Previous test plans available at [[:Category:Test_Plans]]  
* [https://fedoraproject.org/wiki/Category:Test_Cases Fedora Project QA test cases]
+
* [https://fedoraproject.org/wiki/Category:Installer_Test_Cases Fedora Install test cases]
 
* Anaconda Documentation
 
* Anaconda Documentation
 
** [[Anaconda/Options|Supported command-line options]]
 
** [[Anaconda/Options|Supported command-line options]]
Line 84: Line 139:
 
** [[Anaconda/Updates|How to Create and Use an updates.img]]
 
** [[Anaconda/Updates|How to Create and Use an updates.img]]
 
** [[Anaconda/SourceOverview|Source Code Overview]]
 
** [[Anaconda/SourceOverview|Source Code Overview]]
** [[Anaconda/Stage1DevelopmentGuide|Anaconda Stage1 Guide]]
 
** [[Anaconda/Stage2DevelopmentGuide|Anaconda Stage2 Guide]]
 
  
[[Category:Test Plans]]
+
[[Category:Obsolete_Test Plans]]

Latest revision as of 08:21, 6 August 2014

Introduction

This document describes the tests that will be created and used to verify the functions/components of Fedora 12.

The goals of this plan are to:

  • Organize the test effort
  • Communicate the planned tests to all relevant stake-holders for their input and approval
  • Serve as a base for the test planning for future Fedora 12 releases

Test Strategy

Instead of outlining all possible installation inputs and outputs, this test plan will focus on defining inputs and outputs at different stages in anaconda. This will also allow different tests to be performed independently during a single installation. For example, one may execute a kickstart delivery via HTTP, raid0 partitioning using 3 physical disks, and a minimal package installation on a para-virtualized xen guest all in single installation. Scenarios where the stages are dependent will be indicated as such in the test case.

New features of Fedora 12

As with Fedora 11, Fedora 12 will bring us some new features. The following list outlines the larger changes that affect installation. Test plans for these features will be designed/developed on each feature page.

Additional features outside the scope of testing can be found at:

Schedule/Milestones

  • The Fedora 12 release schedule is available at Releases/12/Schedule
  • Each major milestone(Beta,Preview,etc..) will demand a full regression run

Test Priority

This test plan will use a 3 tier classification for test execution priority.

Tier#1 test cases

Tier1 is intended to verify that installation is possible on common hardware using common use cases.

Verification includes:

  • Common boot media
  • Common Installation source
  • Installation using defaults installation options
  • Default Partitioning

Tier#2 test cases

Tier2 takes a step further to include more use cases. Tier2 verification consists of:

  • All boot media
  • All installation sources
  • All kickstart delivery methods
  • Some architecture specific verification

Tier#3 test cases

Tier3 captures the remaining identified use cases:

  • More exhaustive partitioning schemes
  • More complex networking scenarios
  • More architecture specific verification
  • Network device
  • Storage device
  • Upgrade testing

Test Pass/Fail Criteria

The milestone release of Fedora 12 should conform these criteria:

Entrance criteria

  • Trees must be generated using release engineering tools (not hand crafted)
  • There must be no unresolved dependencies for packages included in the installation tree
  • There must be no dependency conflicts for packages included in the installation tree
  • Any changes in composition of the installation tree are explainable by way of bugzilla

Alpha criteria

  • Entrance criteria have been met
  • All test tiers have have been executed
  • All tier#1 tests pass

Beta criteria

  • Alpha criteria have been met
  • All tier#2 tests pass

GA criteria

  • Beta criteria have been met
  • All test tiers must pass
  • Any open defects have been documented as release notes

Scope and Approach

Testing will include:

  • Manually execute test cases using a rawhide boot or live image media
  • Where possible, the use of virtualization is encouraged to allow for greater test coverage and improve failure diagnosis
  • Automated execution of tests is encouraged using kickstarts designed to execute specific test cases. At this time, kickstarts are not provided for every test case.

Items outside the scope of this test plan include:

  • Functional verification of software installed on the system
  • Installation from media not generated by fedora release engineering

Test Deliverables

Testing Tasks

Testing will execute test cases to verify installation of Fedora 12 on different hardware platforms and gather installation test feedback.

Test Environment/Configs

For Fedora 12,test cases will be executed on these hardware platforms:

  • i386
  • ppc
  • x86_64

Responsibilities

Fedora QA team members are responsible for executing this test plan. Contributions from Rawhide testers and other interested parties are encouraged.

Risks and Contingencies

  • If no physical media is available for testing, it is acceptable to use a boot.iso or PXE boot images from a current Rawhide release. For more information, see Releases/Rawhide#Direct_Rawhide_install.

Reporting Bugs and Debugging Problems

If defects/problems are encountered, please go ahead and file the bugs following the guide below:

Reviewers

  • jlaska 19:08, 3 August 2009 (UTC)

References