|
|
| (15 intermediate revisions by 4 users not shown) |
| Line 1: |
Line 1: |
| {{admon/caution|Draft|This page is only draft, and will change in time.}}
| | == What is ResultsDB == |
|
| |
|
| = What is ResultsDB =
| | ResultsDB is a system designed to hold the results of tests executed by the Taskotron framework. |
|
| |
|
| ResultsDB is a database, in which we'll store results of tests executed by the AutoQA framework. | | ResultsDB is only a system for storing test results, different 'frontend' apps can be created using ResultsDB as a data source. These frontends could be anything from a simple tool aggregating recent results for builds, or anything more complicated like a tool for gathering test-related metrics (historical fail/pass ratio of a specific test in a Fedora release, failure rate of critpath packages etc.). |
|
| |
|
| == Motivation behind ResultsDB == | | == Current state == |
|
| |
|
| At the moment, all the results are stored on the autoqa-results mailing list [1]. Although this is not really a bad system for storing the results, searching for some particular result (even as simple query as "All results for foobar-1.3-fc15") is not really comfortable. ResultsDB's main purpose is not to store the data, but to provide easy access to the results via simple querying API. | | At the moment, there are two versions of ResultsDB. |
| | * Legacy version - first incarnation of ResultsDB, still required for tools like TestDays App, or AutoQA <https://fedorahosted.org/ResultsDB/>. |
| | * Current development version - intender for the Taskotron project. Reimplemented to provide JSON/RESTful api, and quite slimmed down, in order to remove the relics from the primary draught. |
|
| |
|
| This means, that using the ResultsDB as database backend, many 'frontends' can be created. Be it simple tools which will just aggregate most recent results for specified package (maybe a table inside koji/bodhi info pages), some statistical tool for gathering fail/pass ratio of *insert test name of your choice* for given Fedora release, etc.
| | == Current development version == |
|
| |
|
| = Current state =
| | The project is split into these parts: |
| | * ResultsDB core <https://bitbucket.org/fedoraqa/resultsdb>, which is the implementation of the datastore, and the API provider. |
| | * ResultsDB Client Library <https://bitbucket.org/fedoraqa/resultsdb_api> providing the easy-to-use python implementation of the ResultsDB API, described here <http://docs.resultsdb.apiary.io/> |
| | * ResultsDB Frontend <https://bitbucket.org/fedoraqa/resultsdb_frontend> - a simple frontend intended for browsing the results via web interface. |
|
| |
|
| From development point of view, there is quite well-defined database schema [2], 'input' API [3], and proof-of-concept TurboGears2 application [4], implementing these two.
| | [[Category:ResultsDB]] |
| | |
| I have also started to work on a proof-of-concept frontend [5], which will allow us to create test plans (like Package Update Acceptance Testplan [6]), using mediawiki pages to define the testplan's requirements [7][8] (this will be covered in more detail further in the text).
| |
| | |
| == API ==
| |
| | |
| We have quite well-defined 'input' API - i.e. the API for storing data into ResultsDB. The 'output' API needs to be designed. I have not gone to that particular area that much, because at the time being. My plan was to sneak ResultsDB reporting into the production to gain some reasonable dataset, and then adjust the output API based on the actual needs of our frontend(s).
| |
| | |
| Note: we all decided against 'sql-based' API, and prefer the set of specific filter-methods with monitorable (?) arguments.
| |
| | |
| Also, once Fedora Message Bus is up'n'running (if ever), we definitely want to send 'announces' via the bus too.
| |
| | |
| | |
| == Thoughts behind the database schema ==
| |
| | |
| It's safe to ignore the 'Job' idea at the moment - the idea behind Jobs was that we wanted to be able to internally group several test-runs (e.g. rpmgguard & conflicts for some particular package) into the job. This option is a) IMHO not really feasible with the current way of scheduling tests b) easily substituted with the 'generic testplan frontend' (see below).
| |
| | |
| Other than that - our primary goal was versatility - we wanted to be able to store results not only from the tests we did have at the moment, but to be able to meet the needs of 'almost anything'.
| |
| | |
| 'Testrun' table corresponds with a single execution of a test, all the data not covered by it's attributes (such as tested package/update, architecture, etc) are to be stored in the 'TestrunData' as key-value pairs. Metadata for each test should define a set of used keyval pairs (with possible information about 'always present' and 'sometimes present' keys) - e.g. [9].
| |
| | |
| Testruns can be devided into phases (e.g. Setup phase of the test, downloading packages from koji...) - hence the TestrunPhase table. If the test developer decides to split the test into multiple phases, each phase can have it's result. Also metadata can be used to specify how to treat failures/warnings/etc from each phase - warning in setup phase might be less important than warning in run-once, etc.
| |
| | |
| == Generic Testplan frontend ==
| |
| | |
| The idea behind this tool is quite simple - Model-View-Controller [10] architecture. ResultsDB will provide data (model), metadata on Mediawiki [8] would be Controller, and the simple-logic frontend [5] is the View part. Given that we probably will want to have multiple test plans, this architecture might suit us quite well.
| |
| | |
| The [6] is great example. There are multiple 'levels' of acceptance, based on the results of different tests. Our multi-purpose test are run on most (if not all) of the packages anyway, so why not just use the data, and present it in different contexts?
| |
| | |
| Our TCMS is MediaWiki at the moment, so wwoods proposed quite elegant way of storing (and retrieving, of course) metadata on some wiki page [7]. I just used that idea to store 'more complicated' kind of template - [8].
| |
| | |
| The semantics of metadata on [8] is quite simple. Let's take this particular
| |
| * <code>testcases</code> (required): A dictionary, which connects name of the testcase (key), and the url of testcase metadata (value). Only testcase names (i.e. rpmlint, initscripts, etc) are used from this point further.
| |
| * <code>testcase_classes</code> (required): A list, which defines any number of 'classes'. These are to be user-defined further in the metadata.
| |
| * <code>mandatory, introspection, advisory</code> (test classes, user defined): Each test class is a dictionary which describes which testcases are to be taken into account, which results will be accepted, and what result should the test class have, if either of the tests is out of the specified set.
| |
| ** <code>testcases</code>: List of testcase names to be used in this testclass
| |
| ** <code>pass</code>: List of 'accepted' results. If all of the test results are in this set, overall test class result is "PASSED".
| |
| ** <code>on_fail</code>: If any of the test results is not in the 'pass' list, the overall test class result is the value defined here.
| |
| | |
| The frontend [5] then simply parses the metadata, queries ResultsDB for test results for given NVR/testcase combination, and displays it.
| |
| | |
| = Links =
| |
| | |
| * [1] https://fedorahosted.org/pipermail/autoqa-results/
| |
| * [2] https://fedoraproject.org/wiki/AutoQA_resultsdb_schema
| |
| * [3] https://fedoraproject.org/wiki/AutoQA_resultsdb_API
| |
| * [4] https://www.assembla.com/code/resultsdb/git/nodes?rev=devel
| |
| * [5] https://www.assembla.com/code/resultsdb_puatp/git/nodes?rev=master
| |
| * [6] https://fedoraproject.org/wiki/QA:Package_Update_Acceptance_Test_Plan
| |
| * [7] https://fedoraproject.org/wiki/QA:Test_Plan_Metadata_Test_Page
| |
| * [8] https://fedoraproject.org/wiki/User:Jskladan/Sandbox:Package_Update_Acceptance_Test_Plan_Metadata
| |
| * [9] https://fedoraproject.org/wiki/User:Jskladan/Sandbox:Rpmguard_Testcase_Metadata
| |
| * [10] http://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller
| |
