Latest revision as of 09:11, 6 August 2014

Obsolete (but updateable)
This test plan is obsolete in many details; it has not been significantly updated since 2009 and refers to many obsolete details of Fedora installation, tree creation etc. The basic concept is still valid, however, and should still be implemented in Taskotron.

Revision history

First draft: WillWoods 20:54, 17 June 2009 (UTC)

Introduction

This test plan documents the process used to check the basic requirements for a Rawhide tree to be acceptable for further testing. It aims to check whether Rawhide is installable, usable as a package repo for updating, and whether critical packages are present and functional.

In short, this is how we decide if Rawhide is broken or not.

Test Strategy

There are three main components here: Repo sanity, Installability, and Basic Functionality. These three categories can be tested mostly independent of one another.

Scope

This plan seeks to answer three basic questions:

Can current Rawhide users update their systems using this repo?
Can this Rawhide tree be installed?
Does the basic system (a subset of the "critical path") work as expected for simple testing?

It is not intended to be an exhaustive test of any part of the system.

Test Pass/Fail Criteria

Rawhide will be considered Good for each purpose if all of the underlying conditions are met:

Repo Sanity

Contains valid yum metadata (repodata)
Contains key packages (kernel, glibc, coreutils)
No unresolved dependencies in critical packages
comps.xml exists and is valid

Installability

installer images (kernel, initrd,and boot.iso) exist
kernel boots on most machines of the primary architectures
initrd is able to find stage2 by at least one method (network, local CD)
stage2 is able to detect the presence of disks attached to most common controllers
installer can write packages to disk storage
installer can setup up bootloader correctly for common boot systems

Basic Functionality

Kernel/X is able to set up common display configurations
- At least 2 out of 3 of the most common video drivers (intel, nouveau, radeon) must be functioning
Kernel/X properly handle input from standard USB keyboard/mouse
System is able to establish a wired network connection
- As with video, this should work for most common network drivers. Individual chipset failures may be waived.
yum is able to install simple updates

Test Deliverables

This test plan should produce:

A summary report on whether Rawhide is broken or not
Bug reports for broken dependencies / missing files / etc.
A list of test cases used to verify the expected results. (see below)

Test Cases

TODO: Automate these test cases. See tickets for each test here.

Repo Sanity

Installability

Basic Functionality

Test Environment

The tests will be run on a host which has the Rawhide tree accessible by HTTP or FTP.
- The tests will download a lot of data, so ideally the host should have a very fast network link to the HTTP/FTP mirror.
The test host will be no older than RHEL5/CentOS5 or currently supported Fedora releases.
The test host will have at least 1GB free disk space:
- critical path packages (~300MB)
- their associated metadata (~200MB)
- all of the boot images (~300MB)
- plus some extra scratch space for rebuilding images etc.

Responsibilities

Fedora QA team members are responsible for executing this test plan. Contributions from Rawhide testers and other interested parties are encouraged.

Schedule

Ideally this test plan should be run for every new Rawhide tree - i.e. daily.

Risks

Testing kernel boot and X startup is currently tricky to automate.
Critical_Path_Packages_Proposal - while exact list of packages may change, it is possible that no agreement on the exact list can be made. In this event, a temporary subset of critical packages will be used

Reviewers

jlaska 11:13, 19 June 2009 (UTC)
viking-ice 16:14, 24 June 2009 (UTC)

References

JohnPoelstra/ImproveRawhideF10#Defining_GOOD - A previous discussion of Rawhide requirements
Critical Path Packages Proposal - Proposal to define the most critical packages - required for some repo sanity test cases
Israwhidebroken.com Proposal - This test plan is a central part of this proposal.

@@ Line 1: / Line 1: @@
+{{admon/note|Obsolete (but updateable)|This test plan is obsolete in many details; it has not been significantly updated since 2009 and refers to many obsolete details of Fedora installation, tree creation etc. The basic concept is still valid, however, and should still be implemented in [[Taskotron]].}}
 = Revision history =
-Detail key dates with respect to this document.  Including it's creation date and significant updates since then.
+First draft: [[User:Wwoods|WillWoods]] 20:54, 17 June 2009 (UTC)
 = Introduction =
-Brief description of this document.
+This test plan documents the process used to check the basic requirements for a Rawhide tree to be acceptable for further testing. It aims to check whether Rawhide is installable, usable as a package repo for updating, and whether critical packages are present and functional.
-* The goals of this plan are to:
-* ''Organize'' the test effort
-* ''Communicate'' the strategy, scope and priorities of the planned tests to all relevant stake-holders for their input and approval
-* Serve as a base for the test planning for future Fedora releases
+In short, this is how we decide if Rawhide is broken or not.
 = Test Strategy =
-Describe how the testing will be organized and executed.  Is there any special order or sequence to perform actions.
+There are three main components here: Repo sanity, Installability, and Basic Functionality. These three categories can be tested mostly independent of one another.
+<!-- Not really sure what this section would be useful for, but feel free to fill it in
 = Test Priority =
-Often used to outline tiers of tests arranged from most important (tier#1) to infrequently used tests (tier#3).
+TBD. No clear priorities exist yet.
+-->
+= Scope =
-= Scope =
+This plan seeks to answer three basic questions:
-What will and won't be tested.
+# Can current Rawhide users update their systems using this repo?
+# Can this Rawhide tree be installed?
+# Does the basic system (a subset of the [[Critical_Path_Packages_Proposal|"critical path"]]) work as expected for simple testing?
+It is ''not'' intended to be an exhaustive test of any part of the system.
 = Test Pass/Fail Criteria =
-When do you consider testing completed or stopped?  How do you know when to stop?
+Rawhide will be considered ''Good'' for each purpose if all of the underlying conditions are met:
+=== Repo Sanity ===
+* Contains valid yum metadata (repodata)
+* Contains key packages (kernel, glibc, coreutils)
+* No unresolved dependencies in critical packages
+* comps.xml exists and is valid
+=== Installability ===
+* installer images (kernel, initrd,and boot.iso) exist
+* kernel boots on ''most'' machines of the primary architectures
+* initrd is able to find stage2 by at least one method (network, local CD)
+* stage2 is able to detect the presence of disks attached to most common controllers
+* installer can write packages to disk storage
+* installer can setup up bootloader correctly for common boot systems
+=== Basic Functionality ===
+* Kernel/X is able to set up common display configurations
+** At least 2 out of 3 of the most common video drivers (intel, nouveau, radeon) must be functioning
+* Kernel/X properly handle input from standard USB keyboard/mouse
+* System is able to establish a wired network connection
+** As with video, this should work for ''most'' common network drivers. Individual chipset failures may be waived.
+* {{package|yum}} is able to install simple updates
 = Test Deliverables =
-Describe what the consumables will be from testing.  Examples include:
+This test plan should produce:
-* This test plan
+* A summary report on whether Rawhide is broken or not
-* A test summary document for each major milestone
+* Bug reports for broken dependencies / missing files / etc.
-* A list of defects filed
+* A list of test cases used to verify the expected results. (see below)
-* Any test scripts used for automation or verification
 = Test Cases  =
-Using the level of detail required by this test plan, outline test areas and test cases to be exercised.
+''TODO: Automate these test cases. See [https://fedorahosted.org/autoqa/query?status=new&status=assigned&status=reopened&component=tests&summary=~Automate&milestone=israwhidebroken.com&order=priority tickets for each test here].
+=== Repo Sanity ===
+# [[QA:Repodata validity test case|Repodata validity]]
+# [[QA:Comps Validity Test Case|comps.xml validity]]
+# [[QA:Core package dependency closure test case|Core package dependency closure]]
+# [[QA:Core package existence test case|Core package existence]]
+=== Installability ===
+# [[QA:Installer image presence test case|Installer image existence]]
+# [[QA:Kernel simple boot test case|Kernel boot]]
+# [[QA:Anaconda stage2 fetch test case|Anaconda loader fetching stage2]]
+# [[QA:Anaconda storage probe test case|Anaconda stage2 disk probe]]
+# [[QA:Anaconda package install test case|Anaconda package install]]
+# [[QA:Anaconda bootloader setup test case|Anaconda bootloader setup]]
+=== Basic Functionality ===
+# [[QA:X basic display test case|X startup/basic display configuration]]
+# [[QA:X basic input handling test case|X basic input handling]]
+# [[QA:Network basic test case|Basic network connectivity]]
+# [[QA:Yum simple update test case|yum update functionality]]
 = Test Environment =
-Describe the environment tests will be executed in
+* The tests will be run on a host which has the Rawhide tree accessible by HTTP or FTP.
+** The tests will download a lot of data, so ideally the host should have a very fast network link to the HTTP/FTP mirror.
+* The test host will be no older than RHEL5/CentOS5 or currently supported Fedora releases.
+* The test host will have at least 1GB free disk space:
+** critical path packages (~300MB)
+** their associated metadata (~200MB)
+** all of the boot images (~300MB)
+** plus some extra scratch space for rebuilding images etc.
 = Responsibilities =
-High-level detail of who is accountable for different phases of testing.
+Fedora QA team members are responsible for executing this test plan. Contributions from Rawhide testers and other interested parties are encouraged.
 = Schedule =
-Outline the scheduled milestones.
+Ideally this test plan should be run for ''every new Rawhide tree'' - i.e. daily.
 = Risks =
-Given the current scope and test priority, what are some potential risks.  Are there any other contingencies should problems arrise?  How will issues be dealt with when they surface.
+* Testing kernel boot and X startup is currently tricky to automate.
+* [[Critical_Path_Packages_Proposal]] - while exact list of packages may change, it is possible that no agreement on the exact list can be made.  In this event, a temporary subset of critical packages will be used
 = Reviewers =
+<!-- Add a line like this:
+* ~~~~
+This will automatically be expanded to your username and timestamp. Preview to check it. -->
+* [[User:Jlaska|jlaska]] 11:13, 19 June 2009 (UTC) <!-- jlaska -->
+* [[User:Johannbg|viking-ice]] 16:14, 24 June 2009 (UTC)
 = References =
+* [[JohnPoelstra/ImproveRawhideF10#Defining_GOOD]] - A previous discussion of Rawhide requirements
+* [[Critical Path Packages Proposal]] - Proposal to define the most critical packages - required for some repo sanity test cases
+* [[Israwhidebroken.com Proposal]] - This test plan is a central part of this proposal.
+<references/>
-<references/>
+[[Category:Obsolete Test Plans]]
+[[Category:Rawhide Acceptance Test Cases]]

Search

QA:Rawhide Acceptance Test Plan: Difference between revisions

Latest revision as of 09:11, 6 August 2014

Contents

Revision history

Introduction

Test Strategy

Scope

Test Pass/Fail Criteria

Repo Sanity

Installability

Basic Functionality

Test Deliverables

Test Cases

Repo Sanity

Installability

Basic Functionality

Test Environment

Responsibilities

Schedule

Risks

Reviewers

References