From Fedora Project Wiki

m (Wiki syntax cleanup)
m (Cleanup again)
Line 58: Line 58:
 
# runs - stores job information (time, what is being testing, NVRs etc...).  All information so that the same test could be initiated at a future date.
 
# runs - stores job information (time, what is being testing, NVRs etc...).  All information so that the same test could be initiated at a future date.
 
# results - all the test results produced by the run
 
# results - all the test results produced by the run
#*#* ''run'' (foreign key)
+
#* ''run'' (foreign key)
#*#* ''test'' (foreign key) - basically a link back to the specific test
+
#* ''test'' (foreign key) - basically a link back to the specific test
#*#* ''test'' result - PASS, INFO, FAIL, WAIVE, INSPECT
+
#* ''test'' result - PASS, INFO, FAIL, WAIVE, INSPECT
#*#* ''log'' - stdout associated with the test
+
#* ''log'' - stdout associated with the test
 
# waivers - contains information for all result waivers
 
# waivers - contains information for all result waivers
#*#* ''person'' - some reference to who is waiving
+
#* ''person'' - some reference to who is waiving
#*#* ''description'' - user supplied text describing waiver
+
#* ''description'' - user supplied text describing waiver
#*#* ''old_result'' - the test result prior to waiving
+
#* ''old_result'' - the test result prior to waiving
#*#* ''datestamp''
+
#* ''datestamp''
  
 
== autotest ==
 
== autotest ==

Revision as of 20:29, 25 February 2010

Warning.png
This page is a draft only
It is still under construction and content may change. Do not rely on the information on this page.

This document summarizes alternative approaches for storing and presenting automated test results used by different automation solutions.

Beaker

rpmdiff

Background

rpmdiff is a collection of tools ...

  • A test tool for static and comparative analysis of packages
  • A test scheduler for pushing tests to test systems
  • A web front-end for results reporting/waiving

Scheduling

Rpmdiff tests are initiated by ...

  • ad-hoc "make" invocations by maintainers
  • ad-hoc XML-RPC by maintainers (for a CLI tool)
  • web UI for manual scheduling
  • package updates - analogous to bodhi updates

There is more information recorded in the database around queuing and running tests on systems. I'm ignoring this since it's not something we're including in our discussion.

Execution

Not much to say here, rpmdiff compares two packages and reports feedback. There is a utility included with yum-utils called rpmdiff, these are similar, but not the same.

Results

Each test run produces a list of results. Each result consists of:

  • severity - INFO, GOOD, BAD, VERIFY
  • text - the message for the end-user

Reporting

All test results are sent to stdout. In addition, the results are saved to an xml data file which is sent back to the scheduler (via XML-RPC). The scheduler processes the input and ...

  • Updates database so that results are visible from the web front-end
  • Based on the results, emails failure information to maintainers

DB schema

Thankfully, the database doesn't appear very complex. The same database is used for storing scheduler and results information, which isn't something we'd need to worry about. So I'll focus just on the results ...

There are more tables, but the 4 key tables for test results appear to be:

  1. tests - just some metadata about the test, including a URL to a wiki page
  2. runs - stores job information (time, what is being testing, NVRs etc...). All information so that the same test could be initiated at a future date.
  3. results - all the test results produced by the run
    • run (foreign key)
    • test (foreign key) - basically a link back to the specific test
    • test result - PASS, INFO, FAIL, WAIVE, INSPECT
    • log - stdout associated with the test
  4. waivers - contains information for all result waivers
    • person - some reference to who is waiving
    • description - user supplied text describing waiver
    • old_result - the test result prior to waiving
    • datestamp

autotest