From Fedora Project Wiki

< CI

(Link to Examples page)
(Moved to Pagure.)
 
(8 intermediate revisions by the same user not shown)
Line 1: Line 1:
= Requirements =
Fedora CI metadata specification moved to https://pagure.io/fedora-ci/metadata.
 
In order to use the [[Flexible_Metadata_Format|Flexible Metadata Format]] effectively for the CI testing we need to agree on the essential set of attributes to be used. For each attribute we need to standardize:
 
* '''Name''' ... unique, well chosen, possibly with a prefix
* '''Type''' ... expected content type: string, number, list, dictionary
* '''Purpose''' ... description of the attribute purpose
 
For names we should probably consider using namespace prefix (such as test-description, requirement-description) to prevent future collisions with other attributes. Each attribute definition should contain at least one apt example of the usage. Or better, a set of user stories to be covered.
 
= Attributes =
 
In this section there are attributes proposed so far. Material for discussion. Nothing final for now.
 
== Summary ==
 
In order to efficiently collaborate on test maintenance it's crucial to have a short summary of what the test does.
 
* Name ... summary
* Type ... string (one line, up to 50 characters)
* Purpose ... concise summary of what the test does
 
User stories:
 
* As a developer reviewing 10 failed tests I would like to get quickly an idea of what my change broke.
 
Notes:
 
* Shall we recommend 50 characters or less? Like there is for [https://stackoverflow.com/questions/2290016/git-commit-messages-50-72-formatting commits]? Yes.
 
Example:
 
summary: Test wget recursive download options
 
== Description ==
 
For complex tests it makes sense to provide more detailed description to better clarify what is covered by the test.
 
* Name ... description
* Type ... string (multi line, plain text)
* Purpose ... detailed description of what the test does
 
User stories:
 
* As a tester I come to a test code I wrote 10 years ago (so I have absolutely no idea about it) and would like to quickly understand what it does.
* As a developer I review existing test coverage for my component and would like to get an overall idea what is covered without having to read the whole test code.
 
Example:
 
description: |
  This test checks all available wget options related to
  downloading files recursively. First a tree directory
  structure is created for testing. Then a file download
  is performed for different recursion depth specified by
  the "--level=depth" option.
 
== Tags ==
 
Throughout the years, free-form tags proved to be useful for many, many scenarios. Primarily to provide an easy way how to select a subset of objects.
 
* Name: tags
* Type: list
* Purpose: free-form tags for easy filtering
 
Notes:
 
* Tags are case-sensitive.
* Using lowercase is recommended.
 
User stories:
 
* A a developer/tester I would like to run only a subset of available tests.
 
Example:
 
tags: [Tier1, fast]
 
== Test ==
 
This is the [http://fmf.readthedocs.io/en/latest/features.html#leaves key content] attribute defining how the test is to be executed.
 
* Name: test
* Type: string
* Purpose: shell command which executes the test
 
User stories:
 
* As a developer/tester I want to easily execute all available tests with just one command.
* As a test writer I want to run a single test script in multiple ways (e.g. providing different parameters)
 
Example:
 
test: ./runtest.sh
 
== Path ==
 
As the object hierarchy does not need to copy the filesystem structure (e.g. [http://fmf.readthedocs.io/en/latest/features.html#virtual virtual] test cases) we need a way how to define where the test is located.
 
* Name: path
* Type: string
* Purpose: filesystem directory to be entered before executing the test
 
User stories:
 
* As a test writer I define two virtual test cases, both using the same script for executing. See also the [[#Virtual_Tests|Virtual Tests]] example.
 
Example:
 
path: wget/recursion
 
== Time ==
 
In order to prevent stuck tests consuming resources we should be able to define a maximum time for test execution.
 
* Name: time
* Type: string
* Purpose: maximum time for test execution after which a running test is killed by the test harness
 
Notes:
 
* Let's use the same format as the ``sleep`` command. For example: 3m, 2h, 1d.
 
User stories:
 
* As a deverloper/tester I want to prevent resource wasting by stuck tests.
* As a test harness I need to know after how long time I should kill test if it is still running.
 
Example:
 
time: 5m
 
== Relevancy ==
 
Sometimes a test case is only relevant for specific environment. Test Case Relevancy allows to filter irrelevant test cases out.
 
* Name: relevancy
* Type: list
* Purpose: Test Case Relevancy rules used for filtering relevant test cases for given environment.
 
User stories:
 
* As a tester I want to skip execution of a particular test case in given test environment.
 
Notes:
 
Environment is defined by one or more environment dimensions such as <code>product</code>, <code>distro</code>, <code>collection</code>, <code>variant</code>, <code>arch</code>, <code>component</code>. Relevancy consists of a set of rules of the form <code>condition: decision</code>. For more details see the [[CI/Test_Case_Relevancy|Test Case Relevancy]] documentation.
 
Example:
 
relevancy:
    - "distro < f-28: False"
    - "distro = rhel-7 & arch = ppc64: False"
 
= Examples =
 
Below you can find some basic examples using the metadata definen above. Separate [[CI/Metadata/Examples|Examples]] page illustrates integration with [[CI/Standard_Test_Roles|Standard Test Roles]] on some real-life components.
 
== BeakerLib Tests ==
 
Three beakerlib tests, each in it's own directory:
 
main.fmf
 
  test: ./runtest.sh
  /one:
      path: tests/one
  /two:
      path: tests/two
  /three:
      path: tests/three
 
fmf
 
  tests/one
  path: tests/one
  test: ./runtest.sh
 
  tests/two
  path: tests/two
  test: ./runtest.sh
 
  tests/three
  path: tests/three
  test: ./runtest.sh
 
== Three Scripts ==
 
Three different script residing in a single directory:
 
main.fmf
 
path: tests
/one:
    test: ./one
/two:
    test: ./two
/three:
    test: ./three
 
fmf
 
tests/one
path: tests
test: ./one
 
tests/two
path: tests
test: ./two
 
tests/three
path: tests
test: ./three
 
== Virtual Tests ==
 
Thre virtual test cases based on a single test script:
 
main.fmf
 
path: tests/virtual
/one:
    test: ./script --one
/two:
    test: ./script --two
/three:
    test: ./script --three
 
fmf
 
tests/one
path: tests
test: ./script --one
 
tests/two
path: tests
test: ./script --two
 
tests/three
path: tests
test: ./script --three

Latest revision as of 14:15, 18 October 2018

Fedora CI metadata specification moved to https://pagure.io/fedora-ci/metadata.