Israwhidebroken.com Proposal

From FedoraProject

(Difference between revisions)
Jump to: navigation, search
(template!)
 
(Results: add link to automated test cases)
 
(9 intermediate revisions by 3 users not shown)
Line 2: Line 2:
  
 
== Overview ==
 
== Overview ==
 +
 +
Provide a single, well-known location with information about whether or not Rawhide is ''broken'', and a link to the ''last known-good'' Rawhide tree.
  
 
=== Problem Space ===
 
=== Problem Space ===
<!-- Describe the problem this proposal seeks to solve -->
+
 
 +
Sometimes Rawhide is ''broken''. Brokenness typically comes in one of these types:
 +
 
 +
* Installer
 +
** Installer images missing
 +
** Installer doesn't work
 +
* Repo
 +
** Metadata is missing
 +
** Package/file conflicts
 +
** Dependency problems
 +
* Functional
 +
** Major bugs with packages in the "Critical Path" (see [[Critical Path Packages Proposal]])
 +
 
 +
When Rawhide is broken it's much harder for people to become Rawhide testers, harder for developers to get fixes into the hands of testers, and other such badness.
  
 
=== Proposed Solution ===
 
=== Proposed Solution ===
 
<!-- Describe proposed solution in detail -->
 
<!-- Describe proposed solution in detail -->
 +
 +
First, we should create test cases for each of the types of breakage described previously. These should be automated or scripted when possible.
 +
 +
These tests should be run regularly (daily if possible) by QA team members and other interested parties. The results should be submitted and displayed prominently on a public website. Kyle McMartin registered [http://israwhidebroken.com/ israwhidebroken.com] for this purpose.
 +
 +
Second: currently we keep copies of several weeks' worth of daily Rawhide trees in the Koji build system. When a tree is deemed good (i.e. when it passes all the tests) someone from QA should file a ticket with rel-eng to create a "known-good" symlink for that rawhide tree. This link should be public and well-known. Someone suggested the name "chewtoy" for this known-good Rawhide.
  
 
=== Scope ===
 
=== Scope ===
 
<!-- Describe the scope of any project work or properties that will be affected by the proposal -->
 
<!-- Describe the scope of any project work or properties that will be affected by the proposal -->
 +
 +
* QA Team members (i.e. testers) will ''need'' to do regular (daily) testing
 +
* rel-eng will need to move the symlink when requested
  
 
=== Active Ingredients ===
 
=== Active Ingredients ===
 
<!-- A list of any prerequisites or other materials needed to implement the solution -->
 
<!-- A list of any prerequisites or other materials needed to implement the solution -->
  
==== Component 1 ====
+
* Define requirements for Good vs. Broken
<!-- The first of a list of prerequisites, and what exactly needs to be done with it -->
+
* Write test cases to check each requirement
 +
* Automate test cases wherever possible
 +
* Set up a system (webapp?) to allow testers to submit results for tests that can't be automated
 +
* Public website that shows results of daily rawhide testing
 +
* Set up a known-good rawhide ("chewtoy") link based on those test results
 +
* Publicize a link to the "chewtoy" known-good tree
 +
* Hosting for [http://israwhidebroken.com/ israwhidebroken.com] - or a redirect.
  
==== Component 2 ====
+
=== Roadmap ===
<!-- The second of a list of prerequisites, and what exactly needs to be done with it. Add more subsections as needed for additional components. -->
+
 
 +
See https://fedorahosted.org/autoqa/milestone/israwhidebroken.com
 +
 
 +
== Results ==
 +
 
 +
* [[QA:Rawhide Acceptance Test Plan]] is used to define Good vs. Broken.
 +
* 12 test cases identified in that test plan are listed here: [[:Category:Rawhide Acceptance Test Cases]]
 +
* 8 of those test cases have been automated for use with [http://autotest.kernel.org/ autotest]
 +
** code: [https://fedorahosted.org/autoqa/browser/rawhide_acceptance https://fedorahosted.org/autoqa/browser/rawhide_acceptance]
  
 
== Discussion Points ==
 
== Discussion Points ==
 
<!-- Describe things which should be discussed regarding the proposal.  Specifics that need narrowing down, or contentions parts of the proposal. -->
 
<!-- Describe things which should be discussed regarding the proposal.  Specifics that need narrowing down, or contentions parts of the proposal. -->
 +
 +
Need to list requirements / solicit mockups for the webapp.
  
 
== Comments? ==
 
== Comments? ==
 
To leave a comment, use the Talk page for this proposal.
 
To leave a comment, use the Talk page for this proposal.
  
 +
== Owner ==
 +
[[User:wwoods]]
 +
[[User:jlaska]]
  
 
[[Category:Proposals]]
 
[[Category:Proposals]]
 +
[[Category:FAD]]

Latest revision as of 21:47, 31 July 2009


Contents

[edit] Overview

Provide a single, well-known location with information about whether or not Rawhide is broken, and a link to the last known-good Rawhide tree.

[edit] Problem Space

Sometimes Rawhide is broken. Brokenness typically comes in one of these types:

  • Installer
    • Installer images missing
    • Installer doesn't work
  • Repo
    • Metadata is missing
    • Package/file conflicts
    • Dependency problems
  • Functional

When Rawhide is broken it's much harder for people to become Rawhide testers, harder for developers to get fixes into the hands of testers, and other such badness.

[edit] Proposed Solution

First, we should create test cases for each of the types of breakage described previously. These should be automated or scripted when possible.

These tests should be run regularly (daily if possible) by QA team members and other interested parties. The results should be submitted and displayed prominently on a public website. Kyle McMartin registered israwhidebroken.com for this purpose.

Second: currently we keep copies of several weeks' worth of daily Rawhide trees in the Koji build system. When a tree is deemed good (i.e. when it passes all the tests) someone from QA should file a ticket with rel-eng to create a "known-good" symlink for that rawhide tree. This link should be public and well-known. Someone suggested the name "chewtoy" for this known-good Rawhide.

[edit] Scope

  • QA Team members (i.e. testers) will need to do regular (daily) testing
  • rel-eng will need to move the symlink when requested

[edit] Active Ingredients

  • Define requirements for Good vs. Broken
  • Write test cases to check each requirement
  • Automate test cases wherever possible
  • Set up a system (webapp?) to allow testers to submit results for tests that can't be automated
  • Public website that shows results of daily rawhide testing
  • Set up a known-good rawhide ("chewtoy") link based on those test results
  • Publicize a link to the "chewtoy" known-good tree
  • Hosting for israwhidebroken.com - or a redirect.

[edit] Roadmap

See https://fedorahosted.org/autoqa/milestone/israwhidebroken.com

[edit] Results

[edit] Discussion Points

Need to list requirements / solicit mockups for the webapp.

[edit] Comments?

To leave a comment, use the Talk page for this proposal.

[edit] Owner

User:wwoods User:jlaska