Wikitcms is a term used to refer to Fedora's use of this Mediawiki instance as an ad hoc test management system ('TCMS' stands for 'Test Case Management System', but that precise initialism is in fact quite rarely used). It is also the name of a Python module which provides an interface to the 'system'.
- 1 Background
- 2 Features
- 3 Implementation
- 3.1 General caveat for things interfacing with wikitcms
- 3.2 Naming conventions
- 3.3 Contents
- 3.4 Auxiliary pages
- 3.5 Supplementary pages
- 4 Reference implementation
- 5 Contact: discussing and reporting bugs in Wikitcms
The use of the Fedora wiki to track test results dates back to at least the page QA/FC6Test2TreeTesting - at that time, the Fedora wiki was a MoinMoin instance, not this Mediawiki instance. The rough form of a table with different tests as the rows and results as columns is visible even there. Since then, this basic format has been gradually elaborated.
The following can be considered as features of the wikitcms 'system':
- Storage and organization (as pages, in categories) of test cases
- Advanced formatting and templating for test case content (via the mediawiki engine)
- Result reporting without authentication
- Capability to distinguish between test case and 'test instance'
- Results sorted by environment (with arbitrary 'environments')
- Tests for a given build grouped by 'test type' and section within test type
Fundamentally, wikitcms consists of a set of conventions and expectations about the names, contents and categorization of wiki pages related to test results. There are also several critical 'auxiliary' pages which are used in the generation of the primary pages; these are considered a part of the system and should be modified only with care.
General caveat for things interfacing with wikitcms
The component pages of the 'system' can often be human edited, and so variation must be expected in the content within the requirements specified below. Two considerations apply to most data you may want to take from or enter into the system:
- Mediawiki tends to tolerate and ignore leading and trailing whitespace: for instance, mediawiki displays and identically, and things interfacing with the system should treat them as equivalent
- In page names underscore (_) and space ( ) characters should be treated as identical and interchangeable: Foo_Bar and Foo Bar lead to the same page, and therefore any code handling page names should expect to cope with either format and consider them as identical
The wikitcms module follows these principles and handles most variations in the formatting of the data it can extract, so using it is recommended.
- Release validation result pages are in the Test Results namespace, and are named Fedora RELEASE MILESTONE COMPOSE TESTTYPE or Fedora RELEASE MILESTONE DATE TESTTYPE, where:
- RELEASE is a Fedora release version number
- MILESTONE is a valid Fedora milestone for the release, or a nightly compose type, currently Rawhide or Branched
- COMPOSE is a TC or RC compose identifier, e.g. TC1 or RC2 (nb: RC2.1 is a valid format here)
- DATE is a nightly compose date, in YYYYMMDD format
- TESTTYPE is one of the validation test types (see below)
- Test types are defined by the existence of a template page (in the Template namespace) named TESTTYPE test matrix
- Validation test category pages are named as follows:
- The top-level Category:Test_Results exists
- Each Fedora release has a Fedora RELEASE Test Results category that is a member of Category:Test_Results
- There is a category for each milestone for each Fedora release, which is a member of that release's category, with the name Fedora RELEASE MILESTONE Test Results
- If there are nightly validation pages for the release, there is a category with the name Fedora RELEASE Nightly Test Results, which is a member of the that release's category
- Test case pages are in the QA namespace and their name begins with Testcase_ (which, as noted above, is effectively equivalent to Testcase )
The result pages contain a section heading, Test Matrix, which acts as a separator. The page wikitext beyond that separator is considered the result text. Results take the form of one or more wiki tables, but separating tests into different tables is only cosmetic.
Result tables must follow these rules:
- They have a single header row, which appears as a single line in the wikitext. The format where each cell in the header row appears on a newline in the wikitext is not allowed for wikitcms purposes
- Each row other than the header row - a result row - represents a single test instance (usually just a test case, but see elsewhere in this section for the way in which test names and page sections can produce multiple test instances for a single test case)
- In contrast to the header row, result rows place each of their cells on a new line in the wikitext. The format where the entire row appears on a single line of wikitext is not allowed for wikitcms purposes
- The first column indicates the level (usually milestone) of the test
- One of the columns must be used for the purpose of identifying the test instance with which that row is associated
- For each row, the contents of that column must be formatted to begin with a link to the test case associated with the row, e.g.
- The link may be named, in which case the name portion is considered the test name, e.g. . This is an example of a test instance which is not simply a single test case. It is considered a test instance with the test case and the test name . The same test case could appear in another row with a different test name, which would constitute another test instance
- Each column after the identifier column is a result column: its column title represents a test environment, and its contents is one or more test results
- Test results must use the Template:Result template, e.g. (to indicate a failure from the user adamwill with the related bug #123456)
- Comments may appear following the test result with which they are associated; all text between one test result and the next (or the end of the line) is considered a comment associated with that test result
The result text for a given page may be separated by section, and if it is, the section in which a test instance appears is considered an identifying attribute of that test instance: that is, if there are two rows for the same test case (with no test name, or with the same test name) but they appear in separate sections, they are considered two separate test instances.
Test case pages
The contents of test case pages is not currently used by wikitcms the 'system'; it effectively simply expects that what they contain will be some sort of content from which a human being can generate a test result. The contents are 'parsed' only by the human testers.
In practice, it is strongly established that each page should contain a single test case using the Template:QA/Test_Case template, according to QA:SOP_test_case_creation. Some test cases used in wikitcms take advantage of mediawiki's advanced templating features, including, for example, content that changes depending on the nature of the current compose according to Template:CurrentFedoraCompose, or simply sharing common content between many test cases for consistency and convenience of editing. This is a significant difference from most test management systems, which have no such capabilities for their test cases.
Auxiliary pages are those which are used in the process of generating the other pages in the system. They are not used as stores of results data directly. These pages are critical to the functioning of the system and care must be taken when editing them. Sensible simple edits to the text in the templates are usually fine, but please be careful about touching any of the critical wiki syntax. Please contact QA before making changes to any of these pages if you are at all unsure of the consequences.
Not all template pages and so on that are used by pages that form a part of the system are not really crucial to its operation; only those that would cause the system described here not to function correctly are considered auxiliary pages.
Result page generation system
The result page generation system consists of a set of nested templates. It is set up such that one complete result page can be generated using a single template, e.g. to generate the Alpha TC1 Installation results page for Fedora 32. As long as it is invoked with correct parameters (i.e. a sensible Fedora release number, existing test type, valid milestone and compose or date), it should always generate result pages that comply with the requirements listed above. This style of invocation is the only intended interface to this mechanism; any other use of the component templates is not expected and may have unintended consequences. The wikitcms module, and the
relval tool based upon it, generates result pages using this system.
The pages which make up the system are:
- The result matrix templates for each test type: those are found in the Category:QA_test_matrix_templates category and must be named as specified in #Naming_conventions
Any situation in which the result page generation system produces pages that do not meet the expectations documented here should be considered a bug in the system and reported.
Template:CurrentFedoraCompose defines the current compose: the TC, RC or nightly compose which is considered to be under active testing by the QA team at present. It can be used as a template with various parameters to display various elements of the current compose's identity, and it is also intended to be used as the authoritative source of that data by things interfacing with wikitcms as a 'system'.
The wikitcms module and
relval use it in various ways, and its contents are usually expected to be updated by passing the parameter to
relval when creating a new validation event. Hand editing the page is not recommended; its format is quite specific and mistakes may cause it to be processed incorrectly.
Things interfacing with wikitcms as a 'system' should expect each line in the page's wikitext that begins with a pipe symbol (|) to represent a different attribute of the 'current' compose's identification. The pipe symbol may be followed by any number of spaces (including none), then a string containing no spaces that identifies an attribute, then any number of spaces (including none), then an equals sign (=), then any number of spaces (including none), then a string which may contain spaces that specifies the value for that attribute. The value ends at the end of the line, or - if any number of spaces directly precedes the end of the line - at the last character before the first of those spaces. For example, the line:
|full =22 Rawhide 20150107
(which ends with two space characters) means theattribute has the value .
The following attributes should always be present:, , , , . The value for one of or should always be empty; being empty indicates that the current compose is a nightly build (in which case the value of should be either Rawhide or Nightly), while being empty indicates that the current compose is a TC or RC (in which case the value of should be a valid release milestone). should always be a valid Fedora release number. combines all the non-empty attributes in a human-readable string defining the complete identification of the current compose.
These two pages are used throughout the wiki and are not maintained as part of the wikitcms system, but they are used extensively by pages that form a part of wikitcms and so they are considered auxiliary pages for its purposes. Changes to their behaviour could cause unexpected results for wikitcms.
Supplementary pages are those that are provided only as conveniences for human interaction with the system, not for storage or organization of results data. Things that treat wikitcms as a 'system' should not usually rely on their existence, though in practice this assumption is usually safe.
Results summary page
For each validation test event (a TC, RC, or nominated nightly compose), there is a page named as above, but with Summary in place of the TESTTYPE. It is produced by transclusion of the result pages, which are set up (by the generation system) such that transcluding them shows only their result text. Thus the summary page combines the result text from all the result pages.
Current redirect pages
For each test type and also for the results summary page, a page exists in the Test Results namespace, named Current TESTTYPE Test (or Current Summary for the summary page). These pages should always redirect to the result page for the current compose, as defined by the Template:CurrentFedoraCompose page defined above. In the case of a conflict, Template:CurrentFedoraCompose should be treated as the authoritative definition of what is current, not the destinations of these redirect pages, though conflicts should happen rarely if at all so long as the
relval utility is used - it updates both Template:CurrentFedoraCompose and the redirects, or neither.
The python-wikitcms module is considered the 'reference implementation' for things that interface with the Wikitcms 'system'. When reading existing content, it is intended to follow the expectations described in this page: that is, any wiki content consistent with the expectations specified here should be correctly processed by python-wikitcms. Similarly, when creating content in the Wikitcms 'system' - creating new result pages, or entering results into existing pages - it is intended to create them in accordance with these expectations.
Any divergence between the expectations described here and the behaviour of python-wikitcms should be considered a bug in one or the other, and reported as such.
Note that python-wikitcms is tolerant of a wider range of behaviours than described above in many cases. For instance, it understands several alternatives to Test Matrix as the result text separator in a result page, it can process the older test result templates as well as the current one, and it accepts several variants on the test case page name convention described above. It does this for the purpose of backwards compatibility with content created for older releases, and these behaviours should not be used in newly-created content. Please ensure all newly-created content in the system follows the expectations described in this page.
Contact: discussing and reporting bugs in Wikitcms
The Wikitcms 'system' is maintained by the QA team. If you have any questions or concerns about its behaviour, or want to report anything that can be considered a 'bug' in the system, please contact the QA group via one of the methods described at QA#Communicate. At present Adam Williamson is the primary 'maintainer' of the 'system', and is happy to discuss it via IRC (adamw) or email.