From Fedora Project Wiki

(Add links to the initial proof of concept)
(Include essential summary and links only)
Line 5: Line 5:
 
which define some aspects of how the test coverage is executed.
 
which define some aspects of how the test coverage is executed.
  
This document proposes a flexible format for defining metadata in
+
This concept proposes a flexible format for defining metadata in
 
plain text files which can be stored close to the test code and
 
plain text files which can be stored close to the test code and
 
structured in a hiearchical way with support for inheritance.
 
structured in a hiearchical way with support for inheritance.
 
Although the proposal initially originated from user stories
 
centered around test execution, the format is general and thus
 
can be used in broader scenarios, e.g. test coverage mapping.
 
 
Using this approach it's also possible to combine both test
 
execution metadata and test coverage information. Thanks to
 
elasticity and hiearchy it provides ability to organize data
 
into well-sized text documents while preventing duplication.
 
  
 
'''Contact:''' [[User:Psss|Petr Šplíchal]]
 
'''Contact:''' [[User:Psss|Petr Šplíchal]]
  
== Stones ==
+
== Features ==
 
 
These are essential corner stones for the design:
 
 
 
* Text files under version control
 
* Keep common uses cases simple
 
* Use hiearchy to organize content
 
* Prevent duplication where possible
 
* Metadata close to the test code
 
* Solution should be open source
 
* Focus on essential use cases
 
 
 
== Stories ==
 
 
 
Important user stories to be covered:
 
 
 
* As a tester or developer I want to easy read and modify metadata and see history.
 
* As a tester I want to select a subset of test cases for execution by specifying a tag.
 
* As a tester I want to define a maximum time for a test case to run.
 
* As a tester I want to specify which environment is relevant for testing.
 
* As a user I want to easily define common metadata for multiple cases to simplify maintenance.
 
* As a user I want to provide specific metadata for selected tests to complement common metadata.
 
* As an individual tester and test contributor I want to execute specific single test case.
 
* As an automation tool I need a metadata storage with good api, extensible, quick for reading.
 
 
 
== Choices ==
 
 
 
* Use git for version control and history of changes.
 
* Yaml format easily readable for both machines and humans.
 
 
 
== Naming ==
 
 
 
A dedicated file name extension <code>fmf</code> as an
 
abbreviation of Flexible Metadata Format to easily find all
 
metadata files:
 
 
 
* main.fmf
 
* smoke.fmf
 
 
 
The format does not define attribute naming in any way. This is up
 
to individual projects. The only exception is the special name
 
<code>main</code> which is reserved for main directory index.
 
 
 
Attribute namespacing can be introduced as needed to prevent
 
collisions between similar attributes. For example:
 
 
 
* test-description, requirement-description
 
* test:description, requirement:description
 
* test_description, requirement_description
 
 
 
== Objects ==
 
  
Objects are identified by path from the git root directory.
+
Here are some of the key features of the Flexible Metadata Format:
Special file name <code>main.fmf</code> works similarly as
 
<code>index.html</code>:
 
  
{| class="wikitable"
+
* [https://fmf.readthedocs.io/en/latest/features.html#simple Simple] ... common use cases super simple to read & write
!Location
+
* [https://fmf.readthedocs.io/en/latest/features.html#hierarchy Hiearchy] ... metadata structured in a hierarchy
!Identifier
+
* [https://fmf.readthedocs.io/en/latest/features.html#inheritance Inheritance] ... data inheritance prevent duplication
|-
+
* [https://fmf.readthedocs.io/en/latest/features.html#elasticity Elasticity] ... single file or data scattered across hierarchy
|wget/main.fmf
+
* [https://fmf.readthedocs.io/en/latest/features.html#virtual Virtual] ... support for maintaining virtual test cases
|wget
 
|-
 
|wget/download/main.fmf
 
|wget/download
 
|-
 
|wget/download/smoke.fmf
 
|wget/download/smoke
 
|}
 
  
= Features =
+
= Links =
  
Let's demonstrate the features on a simple wget example with the
+
A Python module and a simple command line tool are available for
following directory structure:
+
experimenting. Detailed documentation is included as well:
 
 
wget
 
├── download
 
├── protocols
 
│  ├── ftp
 
│  ├── http
 
│  └── https
 
├── recursion
 
└── smoke
 
 
 
== Simple ==
 
 
 
The most common use cases super simple to read & write. Test
 
metadata for a single test look like this:
 
 
 
description: Check basic download options
 
tester: Petr Šplíchal <psplicha@redhat.com>
 
tags: [Tier2, TierSecurity]
 
test: runtest.sh
 
time: 3 min
 
 
 
== Hiearchy ==
 
 
 
Hiearchy is defined by directory structure (see example above) and
 
explicit nesting using attributes starting with <code>/</code>.
 
Defining metadata for several tests in a single file is
 
straightforward:
 
 
 
/download:
 
    description: Check basic download options
 
    tester: Petr Šplíchal <psplicha@redhat.com>
 
    tags: [Tier2, TierSecurity]
 
    test: runtest.sh
 
    time: 3 min
 
/recursion:
 
    description: Check recursive download options
 
    tester: Petr Šplíchal <psplicha@redhat.com>
 
    tags: [Tier2, TierSecurity]
 
    test: runtest.sh
 
    time: 20 min
 
 
 
Content above would be stored in <code>wget/main.fmf</code> file.
 
 
 
== Inheritance ==
 
 
 
Metadata is inherited from parent objects:
 
 
 
tester: Petr Šplíchal <psplicha@redhat.com>
 
tags: [Tier2, TierSecurity]
 
test: runtest.sh
 
 
/download:
 
    description: Check basic download options
 
    time: 3 min
 
/recursion:
 
    description: Check recursive download options
 
    time: 20 min
 
 
 
This nicely prevents unnecessary duplication.
 
 
 
== Elasticity ==
 
 
 
Use a single file or scatter metadata across the hiearchy,
 
whatever is more desired for the project.
 
 
 
File <code>wget/main.fmf</code>:
 
 
 
tester: Petr Šplíchal <psplicha@redhat.com>
 
tags: [Tier2, TierSecurity]
 
test: runtest.sh
 
 
 
File <code>wget/download/main.fmf</code>:
 
 
 
description: Check basic download options
 
time: 3 min
 
 
 
File: <code>wget/recursion/main.fmf</code>:
 
 
 
description: Check recursive download options
 
time: 20 min
 
 
 
This allows reasonable structure for both small and large
 
projects.
 
 
 
== Scatter ==
 
 
 
Thanks to elasticity, metadata can be scattered across several
 
files. For example <code>wget/download</code> metadata can be
 
defined in the following three files:
 
 
 
File <code>wget/main.fmf</code>:
 
 
 
/download:
 
    description: Check basic download options
 
    test: runtest.sh
 
 
 
File <code>wget/download.fmf</code>:
 
 
 
description: Check basic download options
 
test: runtest.sh
 
 
 
File <code>wget/download/main.fmf</code>:
 
 
 
description: Check basic download options
 
test: runtest.sh
 
 
 
Parsing is done from top to bottom (in the order of examples
 
above). Later/lower defined attributes replace values defined
 
earlier/higher in the structure.
 
 
 
== Leaves ==
 
 
 
When searching, '''key content''' is used to define which leaves
 
from the metadata tree will be selected. For example, every test
 
case to be executed must have the <code>test</code> attribute
 
defined, every requirement to be considered for test coverage
 
evaluation must have the <code>requirement</code> attribute
 
defined. Otherwise object data is used for inheritance only.
 
 
 
description: Check basic download options
 
test: runtest.sh
 
time: 3 min
 
 
 
The key content attributes are not supposed to be hard-coded in
 
the Flexible Metadata Format but freely configurable. Multiple key
 
content attributes (e.g. script & backend) could be used as well.
 
 
 
== Virtual ==
 
 
 
Using a single test code for testing multiple scenarios can be
 
easily implemented using leaves inheriting from the same parent:
 
 
 
description: Check basic download options
 
test: runtest.sh
 
 
/fast:
 
    description: Check basic download options (quick smoke test)
 
    environment: MODE=fast
 
    tags: [Tier1]
 
    time: 1 min
 
/full:
 
    description: Check basic download options (full test set)
 
    environment: MODE=full
 
    tags: [Tier2]
 
    time: 3 min
 
 
 
In this way we can efficiently create virtual test cases.
 
 
 
 
 
= Examples =
 
 
 
== Relevancy ==
 
 
 
Test Case Relevancy can be naturaly integrated:
 
 
 
description: Check basic download options
 
tags: [Tier2, TierSecurity]
 
relevancy:
 
- "distro < rhel-7: False"
 
- "arch = s390x: False"
 
 
 
Note that, because of YAML parsing, relevancy rules have to be
 
enclosed in quotes. Another option is to use text format:
 
 
 
description: Check basic download options
 
tags: [Tier2, TierSecurity]
 
relevancy: |
 
    distro < rhel-7: False
 
    arch = s390x: False
 
 
 
Which seems a bit more clear.
 
 
 
== Coverage ==
 
 
 
Test coverage information can be stored in a single file, for
 
example <code>wget/requirements.fmf</code>:
 
 
 
/protocols:
 
    priority: high
 
    /ftp:
 
        requirement: Download a file using the ftp protocol.
 
        coverage: wget/protocols/ftp
 
    /http:
 
        requirement: Download a file using the http protocol.
 
        coverage: wget/protocols/http
 
    /https:
 
        requirement: Download a file using the https protocol.
 
        coverage: wget/protocols/https
 
 
/download:
 
    priority: medium
 
    /output-document-pipe:
 
        requirement: Save content to pipe.
 
        coverage: wget/download
 
    /output-document-file:
 
        requirement: Save content to a file.
 
        coverage: wget/download
 
 
/upload:
 
    priority: medium
 
    /post-file:
 
        requirement: Upload a file to the server
 
        coverage: wget/protocols/http
 
    /post-data:
 
        requirement: Upload a string to the server
 
        coverage: wget/protocols/http
 
 
 
Or split by functionality area into separate files as desired, for
 
example <code>wget/download/requirements.fmf</code>:
 
 
 
priority: medium
 
/output-document-pipe:
 
    requirement: Save content to pipe.
 
    coverage: wget/download
 
/output-document-file:
 
    requirement: Save content to a file.
 
    coverage: wget/download
 
 
 
Or integrated with test case metadata, e.g.
 
<code>wget/download/main.fmf</code>:
 
 
 
description: Check basic download options
 
tags: [Tier2, TierSecurity]
 
test: runtest.sh
 
time: 3 min
 
 
/requirements
 
    requirement: Various download options working correctly
 
    priority: low
 
    /get-file:
 
        coverage: wget/download
 
    /output-document:
 
        coverage: wget/download
 
    /continue:
 
    /timestamping:
 
    /tries:
 
    /no-clobber:
 
        coverage: wget/download
 
    /progress:
 
    /quota:
 
    /server-response:
 
    /bind-address:
 
    /spider:
 
 
 
In the example above three requirements are already covered,
 
the rest still await for test coverage (attributes value is null).
 
 
 
== Strategist ==
 
 
 
Here's an example implementation of
 
[https://github.com/dahaic/test-strategist test-strategist] data for
 
openscap using the Flexible Metadata Format:
 
 
 
/probes:
 
    description: Probes
 
    /offline:
 
        description: Offline scanning
 
    /online:
 
        description: Online scanning
 
/scanning:
 
    description: Reading and understanding source datastreams
 
    /oval:
 
        influencers:
 
        - openscap/probes/offline
 
        - openscap/probes/online
 
    /ds:
 
        influencers:
 
        - openscap/scanning/oval
 
        - openscap/scanning/cpe
 
    /cpe:
 
        influencers:
 
        - openscap/scanning/oval
 
 
 
= Implementation =
 
 
 
A Python module and a simple command line tool implementing the
 
features described in this draft are available for experimenting:
 
  
 
* GitHub: https://github.com/psss/fmf
 
* GitHub: https://github.com/psss/fmf
 
* Docs: https://fmf.readthedocs.io/
 
* Docs: https://fmf.readthedocs.io/
 +
* Fedora: https://apps.fedoraproject.org/packages/fmf/
 
* Copr: https://copr.fedorainfracloud.org/coprs/psss/fmf/
 
* Copr: https://copr.fedorainfracloud.org/coprs/psss/fmf/
 
* PIP: https://pypi.python.org/pypi/fmf
 
* PIP: https://pypi.python.org/pypi/fmf
 +
 +
We are currently working on a Metadata specification which
 +
describes in detail individual attributes to be used for selecting
 +
and executing tests in the Fedora CI:
 +
 +
* Metadata: https://fedoraproject.org/wiki/CI/Metadata

Revision as of 12:02, 18 September 2018

Introduction

In order to keep test execution efficient when number of test cases grows, it is crucial to maintain corresponding metadata, which define some aspects of how the test coverage is executed.

This concept proposes a flexible format for defining metadata in plain text files which can be stored close to the test code and structured in a hiearchical way with support for inheritance.

Contact: Petr Šplíchal

Features

Here are some of the key features of the Flexible Metadata Format:

  • Simple ... common use cases super simple to read & write
  • Hiearchy ... metadata structured in a hierarchy
  • Inheritance ... data inheritance prevent duplication
  • Elasticity ... single file or data scattered across hierarchy
  • Virtual ... support for maintaining virtual test cases

Links

A Python module and a simple command line tool are available for experimenting. Detailed documentation is included as well:

We are currently working on a Metadata specification which describes in detail individual attributes to be used for selecting and executing tests in the Fedora CI: