User:Jlaska/Draft3

From FedoraProject

< User:Jlaska(Difference between revisions)
Jump to: navigation, search
(Draft)
 
m (Add questions section)
 
(40 intermediate revisions by 2 users not shown)
Line 1: Line 1:
This is a copy of [[QA:Rawhide_Acceptance_Test_Plan]] but uses [[Template:QA/Test_Plan]] instead.
+
{{header|qa}}
 +
{{draft}}
  
{{QA/Test_Plan
+
This page will outline criteria for success for a Fedora AMI release
|history=
+
* First draft: [[User:Wwoods|WillWoods]] 20:54, 17 June 2009 (UTC)
+
  
|introduction=
+
= Release Criteria =
This test plan documents the process used to check the basic requirements for a Rawhide tree to be acceptable for further testing. It aims to check whether Rawhide is installable, usable as a package repo for updating, and whether critical packages are present and functional.
+
  
In short, this is how we decide if Rawhide is broken or not.
+
In addition to the existing [[Fedora Release Criteria]], the following criteria apply to EC2 Fedora images. A <span style="color:green; font-weight:bold;">good</span> AMI '''must'''...
|strategy=
+
# allow non-root user login
There are three main components here: Repo sanity, Installability, and Basic Functionality. These three categories can be tested mostly independent of one another.
+
# have SELinux enabled and ''enforcing'' by default
 +
# not contain any package dependency problems
 +
# not include any packages built outside of Fedora infrastructure
  
|priority=
+
= How to Test =
TBD. No clear priorities exist yet.
+
  
|scope=
+
To ''create'' an EC2 image, one needs...
This plan seeks to answer three basic questions:
+
# an amazon account
 +
# the {{package|python-boto}} libraries installed
 +
# a python script to tell Amazon to build an instance
 +
# the ability to SSH into a machine to run tests, etc.
  
# Can current Rawhide users update their systems using this repo?
+
= Open Questions =
# Can this Rawhide tree be installed?
+
# Who is responsible for creating and building EC2 images?
# Does the basic system (a subset of the [[Critical_Path_Packages_Proposal|"critical path"]]) work as expected for simple testing?
+
#:
  
It is ''not'' intended to be an exhaustive test of any part of the system.
+
= Additional Reading =
 
+
* [[Publishing_image_to_EC2]]
|criteria=
+
* [[Cloud_SIG]]
Rawhide will be considered ''Good'' for each purpose if all of the underlying conditions are met:
+
* [[Cloud_SIG/EC2_Images]]
== Repo Sanity ==
+
* Contains valid yum metadata (repodata)
+
* Contains key packages (kernel, glibc, coreutils)
+
* No unresolved dependencies in critical packages
+
* comps.xml exists and is valid
+
 
+
== Installability ==
+
* installer images (kernel, initrd, install.img, and boot.iso) exist
+
* kernel boots on ''most'' machines of the primary architectures
+
* initrd is able to find stage2 (install.img) by at least one method (network, local CD)
+
* stage2 is able to detect the presence of disks attached to most common controllers
+
* installer can write packages to disk storage
+
* installer can setup up bootloader correctly for common boot systems
+
 
+
== Basic Functionality ==
+
* Kernel/X is able to set up common display configurations
+
** At least 2 out of 3 of the most common video drivers (intel, nouveau, radeon) must be functioning
+
* Kernel/X properly handle input from standard USB keyboard/mouse
+
* System is able to establish a wired network connection
+
** As with video, this should work for ''most'' common network drivers. Individual chipset failures may be waived.
+
* {{package|yum}} is able to install simple updates
+
 
+
|deliverables=
+
This test plan should produce:
+
* A summary report on whether Rawhide is broken or not
+
* Bug reports for broken dependencies / missing files / etc.
+
* A list of test cases used to verify the expected results. (see below)
+
 
+
|testcases=
+
''TODO'': Automate these test cases. See [https://fedorahosted.org/autoqa/query?status=new&status=assigned&status=reopened&component=tests&summary=~Automate&milestone=israwhidebroken.com&order=priority tickets for each test here].
+
 
+
== Repo Sanity ==
+
# [[QA:Repodata validity test case|Repodata validity]]
+
# [[QA:Comps Validity Test Case|comps.xml validity]]
+
# [[QA:Core package dependency closure test case|Core package dependency closure]]
+
# [[QA:Core package existence test case|Core package existence]]
+
 
+
== Installability ==
+
# [[QA:Installer image presence test case|Installer image existence]]
+
# [[QA:Kernel simple boot test case|Kernel boot]]
+
# [[QA:Anaconda stage2 fetch test case|Anaconda loader fetching stage2]]
+
# [[QA:Anaconda storage probe test case|Anaconda stage2 disk probe]]
+
# [[QA:Anaconda package install test case|Anaconda package install]]
+
# [[QA:Anaconda bootloader setup test case|Anaconda bootloader setup]]
+
 
+
== Basic Functionality ==
+
# [[QA:X basic display test case|X startup/basic display configuration]]
+
# [[QA:X basic input handling test case|X basic input handling]]
+
# [[QA:Network basic test case|Basic network connectivity]]
+
# [[QA:Yum simple update test case|yum update functionality]]
+
 
+
|environment=
+
* The tests will be run on a host which has the Rawhide tree accessible by HTTP or FTP.
+
** The tests will download a lot of data, so ideally the host should have a very fast network link to the HTTP/FTP mirror.
+
* The test host will be no older than RHEL5/CentOS5 or currently supported Fedora releases.
+
* The test host will have at least 1GB free disk space:
+
** critical path packages (~300MB)
+
** their associated metadata (~200MB)
+
** all of the boot images (~300MB)
+
** plus some extra scratch space for rebuilding images etc.
+
 
+
|resposibility=
+
Fedora QA team members are responsible for executing this test plan. Contributions from Rawhide testers and other interested parties are encouraged.
+
 
+
|schedule=
+
Ideally this test plan should be run for ''every new Rawhide tree'' - i.e. daily.
+
 
+
|risks=
+
 
+
* Testing kernel boot and X startup is currently tricky to automate.
+
* [[Critical_Path_Packages_Proposal]] - while exact list of packages may change, it is possible that no agreement on the exact list can be made.  In this event, a temporary subset of critical packages will be used
+
 
+
|reviewedby=
+
<!-- Add a line like this:
+
* ~~~~
+
This will automatically be expanded to your username and timestamp. Preview to check it. -->
+
* [[User:Jlaska|jlaska]] 11:13, 19 June 2009 (UTC) <!-- jlaska -->
+
* [[User:Johannbg|viking-ice]] 16:14, 24 June 2009 (UTC)
+
 
+
|references=
+
* [[JohnPoelstra/ImproveRawhideF10#Defining_GOOD]] - A previous discussion of Rawhide requirements
+
* [[Critical Path Packages Proposal]] - Proposal to define the most critical packages - required for some repo sanity test cases
+
* [[Israwhidebroken.com Proposal]] - This test plan is a central part of this proposal.
+
<references/>
+
}}
+

Latest revision as of 17:19, 9 March 2011

QA.png


Warning (medium size).png
This page is a draft only
It is still under construction and content may change. Do not rely on the information on this page.

This page will outline criteria for success for a Fedora AMI release

Contents

[edit] Release Criteria

In addition to the existing Fedora Release Criteria, the following criteria apply to EC2 Fedora images. A good AMI must...

  1. allow non-root user login
  2. have SELinux enabled and enforcing by default
  3. not contain any package dependency problems
  4. not include any packages built outside of Fedora infrastructure

[edit] How to Test

To create an EC2 image, one needs...

  1. an amazon account
  2. the Package-x-generic-16.pngpython-boto libraries installed
  3. a python script to tell Amazon to build an instance
  4. the ability to SSH into a machine to run tests, etc.

[edit] Open Questions

  1. Who is responsible for creating and building EC2 images?

[edit] Additional Reading