From Fedora Project Wiki

(Created page with '<!-- This is the FESCo proposal template. It is not necessary that you use this template but it is encouraged that you do. It will help FESCo focus on the issue being addressed...')
 
(Overview)
Line 3: Line 3:
 
== Overview ==
 
== Overview ==
  
Provide a single, well-known location with information about whether or not Rawhide is ''broken'', and a link to the ''last known-good'' Rawhide tree.
+
provide small auto-test system with a web-based interface which can help to customize autoinstall testing.The test system should be small enough for personal use.
  
 
=== Problem Space ===
 
=== Problem Space ===
 +
The purpose of auto test is to simplify testing,save time, improve efficiency.So the auto-test should help us to solve the following problems:
  
Sometimes Rawhide is ''broken''. Brokenness typically comes in one of these types:
+
* the system should be small enough for personal use
 +
* have a very simple interface for user to customize testing or configure testing
 +
* very easy interface for user to setup some services
 +
* test result is easy to check
  
* Installer
 
** Installer images missing
 
** Installer doesn't work
 
* Repo
 
** Metadata is missing
 
** Package/file conflicts
 
** Dependency problems
 
* Functional
 
** Major bugs with packages in the "Critical Path" (see [[Critical Path Packages Proposal]])
 
  
When Rawhide is broken it's much harder for people to become Rawhide testers, harder for developers to get fixes into the hands of testers, and other such badness.
 
  
 
=== Proposed Solution ===
 
=== Proposed Solution ===

Revision as of 03:10, 4 November 2009


Overview

provide small auto-test system with a web-based interface which can help to customize autoinstall testing.The test system should be small enough for personal use.

Problem Space

The purpose of auto test is to simplify testing,save time, improve efficiency.So the auto-test should help us to solve the following problems:

  • the system should be small enough for personal use
  • have a very simple interface for user to customize testing or configure testing
  • very easy interface for user to setup some services
  • test result is easy to check


Proposed Solution

First, we should create test cases for each of the types of breakage described previously. These should be automated or scripted when possible.

These tests should be run regularly (daily if possible) by QA team members and other interested parties. The results should be submitted and displayed prominently on a public website. Kyle McMartin registered israwhidebroken.com for this purpose.

Second: currently we keep copies of several weeks' worth of daily Rawhide trees in the Koji build system. When a tree is deemed good (i.e. when it passes all the tests) someone from QA should file a ticket with rel-eng to create a "known-good" symlink for that rawhide tree. This link should be public and well-known. Someone suggested the name "chewtoy" for this known-good Rawhide.

Scope

  • QA Team members (i.e. testers) will need to do regular (daily) testing
  • rel-eng will need to move the symlink when requested

Active Ingredients

  • Define requirements for Good vs. Broken
  • Write test cases to check each requirement
  • Automate test cases wherever possible
  • Set up a system (webapp?) to allow testers to submit results for tests that can't be automated
  • Public website that shows results of daily rawhide testing
  • Set up a known-good rawhide ("chewtoy") link based on those test results
  • Publicize a link to the "chewtoy" known-good tree
  • Hosting for israwhidebroken.com - or a redirect.

Roadmap

See https://fedorahosted.org/autoqa/milestone/israwhidebroken.com

Results

Discussion Points

Need to list requirements / solicit mockups for the webapp.

Comments?

To leave a comment, use the Talk page for this proposal.

Owner

User:wwoods User:jlaska