From Fedora Project Wiki

(some more content. moar! content!)
(moar content)
Line 40: Line 40:
 
Fundamentally, wikitcms consists of a set of conventions and expectations about the ''names'', ''contents'' and ''categorization'' of wiki pages related to test results. There are also several critical 'auxiliary' pages which are used in the generation of the primary pages; these are considered a part of the system and should be modified only with care.
 
Fundamentally, wikitcms consists of a set of conventions and expectations about the ''names'', ''contents'' and ''categorization'' of wiki pages related to test results. There are also several critical 'auxiliary' pages which are used in the generation of the primary pages; these are considered a part of the system and should be modified only with care.
  
=== Names ===
+
=== Naming conventions ===
  
 
* Release validation result pages are in the ''Test Results'' namespace, and are named ''Fedora '''RELEASE''' '''MILESTONE''' '''COMPOSE''' '''TESTTYPE''''' or ''Fedora '''RELEASE''' '''MILESTONE''' '''DATE''' '''TESTTYPE''', where:
 
* Release validation result pages are in the ''Test Results'' namespace, and are named ''Fedora '''RELEASE''' '''MILESTONE''' '''COMPOSE''' '''TESTTYPE''''' or ''Fedora '''RELEASE''' '''MILESTONE''' '''DATE''' '''TESTTYPE''', where:
Line 48: Line 48:
 
*: '''DATE''' is a nightly compose date, in ''YYYYMMDD'' format
 
*: '''DATE''' is a nightly compose date, in ''YYYYMMDD'' format
 
*: '''TESTTYPE''' is one of the validation test types (see below)
 
*: '''TESTTYPE''' is one of the validation test types (see below)
 +
* For each 'event' (a TC, RC, or nominated nightly compose), there is a page named as above, but with ''Summary'' in place of the '''TESTTYPE'''
 +
* Test types are defined by the existence of a template page (in the ''Template'' namespace) named '''''TESTTYPE''' test matrix''
 +
* For each test type, a page exists in the ''Test Results'' namespace, named ''Current '''TESTTYPE''' Test''
 
* Validation test category pages are named as follows:
 
* Validation test category pages are named as follows:
 
*: The top-level [[:Category:Test_Results]] exists
 
*: The top-level [[:Category:Test_Results]] exists
Line 53: Line 56:
 
*: There is a category for each milestone for each Fedora release, which is a member of that release's category, with the name ''Fedora '''RELEASE''' '''MILESTONE''' Test Results''
 
*: There is a category for each milestone for each Fedora release, which is a member of that release's category, with the name ''Fedora '''RELEASE''' '''MILESTONE''' Test Results''
 
*: If there are nightly validation pages for the release, there is a category with the name ''Fedora '''RELEASE''' Nightly Test Results'', which is a member of the that release's category
 
*: If there are nightly validation pages for the release, there is a category with the name ''Fedora '''RELEASE''' Nightly Test Results'', which is a member of the that release's category
 +
 +
=== Contents ===
 +
 +
==== Result pages ====
 +
 +
The result pages contain a section heading, ''Test Matrix'', which acts as a separator. The page wikitext beyond that separator is considered the ''result text''. Results take the form of one or more wiki tables, but separating tests into different ''tables'' is only cosmetic.
 +
 +
Result tables must follow these rules:
 +
 +
* They have a ''single'' header row, which appears as a '''single line''' in the wikitext. The format where each cell in the ''header'' row appears on a newline in the wikitext is not allowed for wikitcms purposes
 +
* Each row other than the header row - a ''result row'' - represents a single ''test instance'' (usually just a test case, but see elsewhere in this section for the way in which ''test names'' and ''page sections'' can produce multiple ''test instances'' for a single ''test case'')
 +
* In contrast to the ''header row'', ''result rows'' place '''each of their cells on a new line''' in the wikitext. The format where the entire row appears on a single line of wikitext is not allowed for wikitcms purposes
 +
* The first column indicates the ''level'' (usually milestone) of the test
 +
* One of the columns must be used for the purpose of identifying the ''test instance'' with which that row is associated
 +
*: For each row, the contents of that column must be formatted to begin with a link to the test case associated with the row, e.g. {{code|<nowiki>[[QA:Testcase_Mediakit_Checksums]]</nowiki>}}
 +
*: The link '''may''' be named, in which case the name portion is considered the ''test name'', e.g. {{code|<nowiki>[[QA:Testcase_Boot_default_install|Workstation live]]</nowiki>}}. This is an example of a ''test instance'' which is not simply a single test case: the same test case could appear in another row with a different ''test name'', which would constitute another ''test instance''
 +
* Each column ''after'' the identifier column is a ''result column'': its column title represents a ''test environment'', and its contents is one or more ''test results''
 +
* ''Test results'' must use the [[Template:Result]] template, e.g. {{code|<nowiki>{{result|fail|adamwill|123456}}</nowiki>}} (to indicate a failure from the user ''adamwill'' with the related bug #123456)
 +
*: Comments ''may'' appear following the test result with which they are associated; all text between one test result and the next (or the end of the line) is considered a comment associated with that test result
 +
 +
The result text for a given page ''may'' be separated by section, and if it is, the section in which a test instance appears is considered an identifying attribute of that test instance: that is, if there are two rows for the same test case (with no ''test name'', or with the same ''test name'') but they appear in separate sections, they are considered two separate ''test instances''.

Revision as of 19:32, 21 January 2015

Wikitcms is a term used to refer to Fedora's use of this Mediawiki instance as an ad hoc test management system ('TCMS' stands for 'Test Case Management System', but that precise initialism is in fact quite rarely used). It is also the name of a Python module which provides an interface to the 'system'.

Background

The use of the Fedora wiki to track test results dates back to at least the page QA/FC6Test2TreeTesting - at that time, the Fedora wiki was a MoinMoin instance, not this Mediawiki instance. The rough form of a table with different tests as the rows and results as columns is visible even there. Since then, this basic format has been gradually elaborated.

Features

The following can be considered as features of the wikitcms 'system':

  • Storage and organization (as pages, in categories) of test cases
  • Advanced formatting and templating for test case content (via the mediawiki engine)
  • Result reporting without authentication
  • Capability to distinguish between test case and 'test instance'
  • Results sorted by environment (with arbitrary 'environments')
  • Tests for a given build grouped by 'test type' and section within test type

Implementation

Fundamentally, wikitcms consists of a set of conventions and expectations about the names, contents and categorization of wiki pages related to test results. There are also several critical 'auxiliary' pages which are used in the generation of the primary pages; these are considered a part of the system and should be modified only with care.

Naming conventions

  • Release validation result pages are in the Test Results namespace, and are named Fedora RELEASE MILESTONE COMPOSE TESTTYPE or Fedora RELEASE MILESTONE DATE TESTTYPE, where:
    RELEASE is a Fedora release version number
    MILESTONE is a valid Fedora milestone for the release, or a nightly compose type, currently Rawhide or Branched
    COMPOSE is a TC or RC compose identifier, e.g. TC1 or RC2 (nb: RC2.1 is a valid format here)
    DATE is a nightly compose date, in YYYYMMDD format
    TESTTYPE is one of the validation test types (see below)
  • For each 'event' (a TC, RC, or nominated nightly compose), there is a page named as above, but with Summary in place of the TESTTYPE
  • Test types are defined by the existence of a template page (in the Template namespace) named TESTTYPE test matrix
  • For each test type, a page exists in the Test Results namespace, named Current TESTTYPE Test
  • Validation test category pages are named as follows:
    The top-level Category:Test_Results exists
    Each Fedora release has a Fedora RELEASE Test Results category that is a member of Category:Test_Results
    There is a category for each milestone for each Fedora release, which is a member of that release's category, with the name Fedora RELEASE MILESTONE Test Results
    If there are nightly validation pages for the release, there is a category with the name Fedora RELEASE Nightly Test Results, which is a member of the that release's category

Contents

Result pages

The result pages contain a section heading, Test Matrix, which acts as a separator. The page wikitext beyond that separator is considered the result text. Results take the form of one or more wiki tables, but separating tests into different tables is only cosmetic.

Result tables must follow these rules:

  • They have a single header row, which appears as a single line in the wikitext. The format where each cell in the header row appears on a newline in the wikitext is not allowed for wikitcms purposes
  • Each row other than the header row - a result row - represents a single test instance (usually just a test case, but see elsewhere in this section for the way in which test names and page sections can produce multiple test instances for a single test case)
  • In contrast to the header row, result rows place each of their cells on a new line in the wikitext. The format where the entire row appears on a single line of wikitext is not allowed for wikitcms purposes
  • The first column indicates the level (usually milestone) of the test
  • One of the columns must be used for the purpose of identifying the test instance with which that row is associated
    For each row, the contents of that column must be formatted to begin with a link to the test case associated with the row, e.g. [[QA:Testcase_Mediakit_Checksums]]
    The link may be named, in which case the name portion is considered the test name, e.g. [[QA:Testcase_Boot_default_install|Workstation live]]. This is an example of a test instance which is not simply a single test case: the same test case could appear in another row with a different test name, which would constitute another test instance
  • Each column after the identifier column is a result column: its column title represents a test environment, and its contents is one or more test results
  • Test results must use the Template:Result template, e.g. {{result|fail|adamwill|123456}} (to indicate a failure from the user adamwill with the related bug #123456)
    Comments may appear following the test result with which they are associated; all text between one test result and the next (or the end of the line) is considered a comment associated with that test result

The result text for a given page may be separated by section, and if it is, the section in which a test instance appears is considered an identifying attribute of that test instance: that is, if there are two rows for the same test case (with no test name, or with the same test name) but they appear in separate sections, they are considered two separate test instances.