|
|
| Line 1: |
Line 1: |
| == What is ResultsDB == | | == What is ResultsDB == |
|
| |
|
| <!-- ResultsDB is more than a database, and more of an information storage system with clearly defined non-SQL APIs -->
| | ResultsDB is a system designed to hold the results of tests executed by the Taskotron framework. |
| ResultsDB is a system designed to hold the results of tests executed by the AutoQA framework. | |
|
| |
|
| === Motivation behind ResultsDB ===
| | ResultsDB is only a system for storing test results, different 'frontend' apps can be created using ResultsDB as a data source. These frontends could be anything from a simple tool aggregating recent results for builds, or anything more complicated like a tool for gathering test-related metrics (historical fail/pass ratio of a specific test in a Fedora release, failure rate of critpath packages etc.). |
| From the beginning of AutoQA development, all test results were stored on the autoqa-results mailing list <ref>https://fedorahosted.org/pipermail/autoqa-results/</ref>. While this is not a bad system for storing test results on a small scale, searching through these results (ie. find all test results for foobar-1.3-fc15) is overly difficult and cumbersome.
| |
|
| |
|
| The primary motivation behind ResultsDB is to provide a simple, easily accessible interface to test results through simple querying API. ResultsDB is also capable of storing test results but this is a secondary objective.
| | == Current state == |
|
| |
|
| Since ResultsDB is only a system for storing test results, different 'frontend' systems can be created using ResultsDB as a data source. These frontends could be a simple tool to aggregate recent results for packages (potentially a table for a package/update's results within bodhi or koji) or something more complicated like a tool for gathering test-related metrics (historical fail/pass ratio of a specific test in a Fedora release, failure rate of critpath packages etc.).
| | At the moment, there are two versions of ResultsDB. |
| | * Legacy version - first incarnation of ResultsDB, still required for tools like TestDays App, or AutoQA <https://fedorahosted.org/ResultsDB/>. |
| | * Current development version - intender for the Taskotron project. Reimplemented to provide JSON/RESTful api, and quite slimmed down, in order to remove the relics from the primary draught. |
|
| |
|
| == Current State of Development == | | == Current development version == |
|
| |
|
| At the time of this writing, most of the first version of ResultsDB has been completed:
| | The project is split into these parts: |
| * The underlying database schema has been designed.<ref name='database-schema'>https://fedoraproject.org/wiki/AutoQA_resultsdb_schema</ref> | | * ResultsDB core <https://bitbucket.org/rajcze/resultsdb>, which is the implementation of the datastore, and the API provider. Current dev instance is here <http://resultsdb.qa.fedoraproject.org/resultsdb/> |
| * The data input API to be used for submitting test results has been implemented.<ref>https://fedoraproject.org/wiki/AutoQA_resultsdb_API</ref>
| | * ResultsDB Client Library <https://bitbucket.org/rajcze/resultsdb_api> providing the easy-to-use python implementation of the ResultsDB API, described here <http://docs.resultsdb.apiary.io/> |
| * A proof-of-concept front end to display test results has been implemented using TurboGears2.<ref>http://git.fedorahosted.org/git/?p=ResultsDB.git;a=tree;f=resultsdb</ref> | | * ResultsDB Frontend <https://bitbucket.org/rajcze/resultsdb_frontend> - a simple frontend intended for browsing the results via web interface. Current dev instance is here <http://resultsdb.qa.fedoraproject.org/resultsdb_frontend/jobs> |
| * A second proof-of-concept frontend to create test plans (ie. The Package Update Acceptance Test Plan <ref name='PUATP'>https://fedoraproject.org/wiki/QA:Package_Update_Acceptance_Test_Plan</ref>) has been started<ref name='git-repo'>http://git.fedorahosted.org/git/?p=ResultsDB.git;a=tree;f=frontends/puatp</ref>. | |
|
| |
|
| <!-- may use this later
| |
| using mediawiki to store test plan requirements
| |
| (example ) has been started
| |
|
| |
| -->
| |
| == API ==
| |
|
| |
| {{admon/note|Down with direct SQL|During discussions, it was decided against using a direct SQL-based API. Instead, the preference was for a set of specific filter-methods with monitorable (?) arguments.}}
| |
|
| |
| At the time of this writing, ResultsDB has a well defined API for storing results but the API for retrieving results has yet to be designed. The current thought behind the retrieval API is that it could be better designed once a proper production dataset exists.
| |
|
| |
| When the Fedora Message Bus is deployed, that would provide another mechanism to monitor for and broadcast events.
| |
|
| |
| == Database Schema ==
| |
|
| |
| Visit AutoQA_resultsdb_schema <ref name='database-schema' /> to read up on the schema.
| |
|
| |
| == Generic Testplan Frontend ==
| |
|
| |
| The testplan frontends are designed to be simple applications with an MVC <ref>http://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller</ref> architecture.
| |
| * '''Model''' - The data from ResultsDB makes up the model
| |
| * '''Controller''' - Metadata stored in Mediawiki pages <ref name='PUATP_metadata'>https://fedoraproject.org/wiki/User:Jskladan/Sandbox:Package_Update_Acceptance_Test_Plan_Metadata</ref> makes up the Controller.
| |
| * '''View''' - Simple frontend implementations<ref name='git-repo'/> make up the view.
| |
|
| |
| It is anticipated that multiple testplan frontends will be created for ResultsDB and this architecture should be flexible enough to support the implementation of several independent frontends.
| |
|
| |
| ==== Example ====
| |
| A frontend was created for the Package Update Acceptance Testplan (PUATP) <ref name='PUATP'/> as an example. There are multiple 'levels' of acceptance, based on the results of different tests. Many AutoQA tests are run on most, if not all, packages and the same data can be presented in different contexts to allow for easier understanding of that data.
| |
|
| |
| At the time of this writing, FedoraQA is using MediaWiki for a TCMS. A method to store and retrieve test metadata on wiki pages was proposed<ref name='metadata_test_page'>https://fedoraproject.org/wiki/QA:Test_Plan_Metadata_Test_Page</ref> and is currently being used for the more complicated PUATP template<ref name='PUATP_metadata'/>.
| |
|
| |
| The frontend <ref name='git-repo'/> parses metadata from the wiki page, queries ResultsDB for test results from a specific NVR/testcase combination, and renders the information into an easily readable format.
| |
|
| |
| ==== Mediawiki Testcase Metadata ====
| |
|
| |
| The semantics of metadata for the PUATP testcase <ref name='PUATP_metadata'/> is quite simple.
| |
| * <code>testcases</code> (required): A dictionary that connects the testcase name(key) to the URL of the testcase metadata (value). Testcase names (i.e. rpmlint or initscripts) are the primary method of reference beyond this dictionary.
| |
| * <code>testcase_classes</code> (required): A list defining any number of 'classes'. Any class in this list must be farther defined in the metadata for the same testcase.
| |
| * <code>mandatory, introspection, advisory</code> (test classes, user defined): Each test class is a dictionary which describes which testcases are to be taken into account, which results will be accepted, and what result should the test class have, if either of the tests is out of the specified set.
| |
| ** <code>testcases</code>: List of testcase names that fall under the specified class
| |
| ** <code>pass</code>: List of testcase results that qualify as 'accepted'. If all possible test results are in this set, the overall result of the test class will always be "PASSED".
| |
| ** <code>on_fail</code>: Overall test class result if any of the specified testcases return a result not on the 'pass' list
| |
|
| |
| == Links ==
| |
| * [https://fedorahosted.org/ResultsDB/ Fedorahosted project homepage]
| |
|
| |
| == References ==
| |
| <references/>
| |
|
| |
|
| [[Category:ResultsDB]] | | [[Category:ResultsDB]] |
