From Fedora Project Wiki

(Initial draft)
 
(Incorporate quick introduction feedback)
Line 24: Line 24:
* As a tester I want to select a subset of test cases for execution by specifying a tag.
* As a tester I want to select a subset of test cases for execution by specifying a tag.
* As a tester I want to specify which environment is relevant for testing.
* As a tester I want to specify which environment is relevant for testing.
* As a user I want to easily define metadata for multiple cases to simplify maintenance.
* As a user I want to easily define common metadata for multiple cases to simplify maintenance.
* As a user I want to provide specific definition for selected tests to complement generic metadata.
* As a user I want to provide specific metadata for selected tests to complement common metadata.
* As individual tester and test contributor I want to execute specific single test case.
* As an individual tester and test contributor I want to execute specific single test case.
* As an automation tool I need a metadata storage with good api, extensible, quick for reading.
* As an automation tool I need a metadata storage with good api, extensible, quick for reading.


Line 36: Line 36:
== Naming ==
== Naming ==


A dedicated file name extension <code>fmf</code> to easily find all metadata files:
A dedicated file name extension <code>fmf</code> as an
abbreviation of Flexible Metadata Format to easily find all
metadata files:


* main.fmf
* main.fmf
* smoke.fmf
* smoke.fmf
The format does not define attribute naming in any way. This is up
to individual projects. The only exception is the special name
<code>main</code> which is reserved for main directory index.
Attribute namespacing can be introduced as needed to prevent
collisions between similar attributes. For example:
* test-description, requirement-description
* test:description, requirement:description
* test_description, requirement_description


== Objects ==
== Objects ==
Line 83: Line 96:
  tester: Petr Šplíchal <psplicha@redhat.com>
  tester: Petr Šplíchal <psplicha@redhat.com>
  tags: [Tier2, TierSecurity]
  tags: [Tier2, TierSecurity]
  requires: [httpd]
  test: runtest.sh
  time: 3 min
  time: 3 min


Line 89: Line 102:


Hiearchy is defined by directory structure (see example above) and
Hiearchy is defined by directory structure (see example above) and
explicit nesting using attributes starting with "/". Defining
explicit nesting using attributes starting with <code>/</code>.
metadata for several tests in a single file is straightforward:
Defining metadata for several tests in a single file is
straightforward:


  /download:
  /download:
Line 96: Line 110:
     tester: Petr Šplíchal <psplicha@redhat.com>
     tester: Petr Šplíchal <psplicha@redhat.com>
     tags: [Tier2, TierSecurity]
     tags: [Tier2, TierSecurity]
     requires: [httpd]
     test: runtest.sh
     time: 3 min
     time: 3 min
  /recursion:
  /recursion:
Line 102: Line 116:
     tester: Petr Šplíchal <psplicha@redhat.com>
     tester: Petr Šplíchal <psplicha@redhat.com>
     tags: [Tier2, TierSecurity]
     tags: [Tier2, TierSecurity]
     requires: [httpd]
     test: runtest.sh
     time: 20 min
     time: 20 min


Line 113: Line 127:
  tester: Petr Šplíchal <psplicha@redhat.com>
  tester: Petr Šplíchal <psplicha@redhat.com>
  tags: [Tier2, TierSecurity]
  tags: [Tier2, TierSecurity]
  requires: [httpd]
  test: runtest.sh
   
   
  /download:
  /download:
Line 129: Line 143:
whatever is more desired for the project.
whatever is more desired for the project.


<code>wget/main.fmf</code>
File <code>wget/main.fmf</code>:


  tester: Petr Šplíchal <psplicha@redhat.com>
  tester: Petr Šplíchal <psplicha@redhat.com>
  tags: [Tier2, TierSecurity]
  tags: [Tier2, TierSecurity]
  requires: [httpd]
  test: runtest.sh


<code>wget/download/main.fmf</code>
File <code>wget/download/main.fmf</code>:


  description: Check basic download options
  description: Check basic download options
  time: 3 min
  time: 3 min


<code>wget/recursion/main.fmf</code>
File: <code>wget/recursion/main.fmf</code>:


  description: Check recursive download options
  description: Check recursive download options
Line 147: Line 161:
This allows reasonable structure for both small and large
This allows reasonable structure for both small and large
projects.
projects.
== Scatter ==
Thanks to elasticity, metadata can be scattered across several
files. For example <code>wget/download</code> metadata can be
defined in the following three files:
File <code>wget/main.fmf</code>:
/download:
    description: Check basic download options
    test: runtest.sh
File <code>wget/download.fmf</code>:
description: Check basic download options
test: runtest.sh
File <code>wget/download/main.fmf</code>:
description: Check basic download options
test: runtest.sh
Parsing is done from top to bottom (in the order of examples
above). Later/lower defined attributes replace values defined
earlier/higher in the structure.


== Leaves ==
== Leaves ==
Line 152: Line 192:
When searching, '''key content''' is used to define which leaves
When searching, '''key content''' is used to define which leaves
from the metadata tree will be selected. For example, every test
from the metadata tree will be selected. For example, every test
case to be executed must have the "test" attribute defined, every
case to be executed must have the <code>test</code> attribute
requirement to be considered for test coverage evaluation must
defined, every requirement to be considered for test coverage
have the "requirement" attribute defined. Otherwise object data is
evaluation must have the <code>requirement</code> attribute
used for inheritance only.
defined. Otherwise object data is used for inheritance only.


  description: Check basic download options
  description: Check basic download options
Line 171: Line 211:


  description: Check basic download options
  description: Check basic download options
tags: [Tier2, TierSecurity]
requires: [httpd]
  test: runtest.sh
  test: runtest.sh
time: 3 min
   
   
  /nofips:
  /fast:
  /fips:
    description: Check basic download options (quick smoke test)
     description: Check basic download options (in fips mode)
    environment: MODE=fast
     extra wow options: --fips
    tags: [Tier1]
    time: 1 min
  /full:
     description: Check basic download options (full test set)
     environment: MODE=full
    tags: [Tier2]
    time: 3 min


In this way we can efficiently create virtual test cases.
In this way we can efficiently create virtual test cases.
== API ==
A python module is planned to be implemented which would take care
of parsing and merging the metadata files and providing an easy
API for automation scripts. A corresponding simple command line
tool should allow to easily investigate metadata and could be used
for example for adhoc test execution.
fmf --key test --search tag=Tier1
fmf --key requirement --search priority=high


= Examples =
= Examples =
Line 257: Line 312:
  description: Check basic download options
  description: Check basic download options
  tags: [Tier2, TierSecurity]
  tags: [Tier2, TierSecurity]
  requires: [httpd]
  test: runtest.sh
  time: 3 min
  time: 3 min
   
   

Revision as of 10:06, 8 January 2018

Introduction

Handle test metadata in an efficient way.

Contact: Petr Šplíchal

Stones

These are essential corner stones for the design:

  • Text files under version control
  • Keep common uses cases simple
  • Use hiearchy to organize content
  • Prevent duplication where possible
  • Metadata close to the test code
  • Solution should be open source
  • Focus on essential use cases

Stories

Important user stories to be covered:

  • As a tester or developer I want to easy read and modify metadata and see history.
  • As a tester I want to select a subset of test cases for execution by specifying a tag.
  • As a tester I want to specify which environment is relevant for testing.
  • As a user I want to easily define common metadata for multiple cases to simplify maintenance.
  • As a user I want to provide specific metadata for selected tests to complement common metadata.
  • As an individual tester and test contributor I want to execute specific single test case.
  • As an automation tool I need a metadata storage with good api, extensible, quick for reading.

Choices

  • Use git for version control and history of changes.
  • Yaml format easily readable for both machines and humans.

Naming

A dedicated file name extension fmf as an abbreviation of Flexible Metadata Format to easily find all metadata files:

  • main.fmf
  • smoke.fmf

The format does not define attribute naming in any way. This is up to individual projects. The only exception is the special name main which is reserved for main directory index.

Attribute namespacing can be introduced as needed to prevent collisions between similar attributes. For example:

  • test-description, requirement-description
  • test:description, requirement:description
  • test_description, requirement_description

Objects

Objects are identified by path from the git root directory. Special file name main.fmf works similarly as index.html:

Location Identifier
wget/main.fmf wget
wget/download/main.fmf wget/download
wget/download/smoke.fmf wget/download/smoke

Features

Let's demonstrate the features on a simple wget example with the following directory structure:

wget
├── download
├── protocols
│   ├── ftp
│   ├── http
│   └── https
├── recursion
└── smoke

Simple

The most common use cases super simple to read & write. Test metadata for a single test look like this:

description: Check basic download options
tester: Petr Šplíchal <psplicha@redhat.com>
tags: [Tier2, TierSecurity]
test: runtest.sh
time: 3 min

Hiearchy

Hiearchy is defined by directory structure (see example above) and explicit nesting using attributes starting with /. Defining metadata for several tests in a single file is straightforward:

/download:
    description: Check basic download options
    tester: Petr Šplíchal <psplicha@redhat.com>
    tags: [Tier2, TierSecurity]
    test: runtest.sh
    time: 3 min
/recursion:
    description: Check recursive download options
    tester: Petr Šplíchal <psplicha@redhat.com>
    tags: [Tier2, TierSecurity]
    test: runtest.sh
    time: 20 min

Content above would be stored in wget/main.fmf file.

Inheritance

Metadata is inherited from parent objects:

tester: Petr Šplíchal <psplicha@redhat.com>
tags: [Tier2, TierSecurity]
test: runtest.sh

/download:
    description: Check basic download options
    time: 3 min
/recursion:
    description: Check recursive download options
    time: 20 min

This nicely prevents unnecessary duplication.

Elasticity

Use a single file or scatter metadata across the hiearchy, whatever is more desired for the project.

File wget/main.fmf:

tester: Petr Šplíchal <psplicha@redhat.com>
tags: [Tier2, TierSecurity]
test: runtest.sh

File wget/download/main.fmf:

description: Check basic download options
time: 3 min

File: wget/recursion/main.fmf:

description: Check recursive download options
time: 20 min

This allows reasonable structure for both small and large projects.

Scatter

Thanks to elasticity, metadata can be scattered across several files. For example wget/download metadata can be defined in the following three files:

File wget/main.fmf:

/download:
    description: Check basic download options
    test: runtest.sh

File wget/download.fmf:

description: Check basic download options
test: runtest.sh

File wget/download/main.fmf:

description: Check basic download options
test: runtest.sh

Parsing is done from top to bottom (in the order of examples above). Later/lower defined attributes replace values defined earlier/higher in the structure.

Leaves

When searching, key content is used to define which leaves from the metadata tree will be selected. For example, every test case to be executed must have the test attribute defined, every requirement to be considered for test coverage evaluation must have the requirement attribute defined. Otherwise object data is used for inheritance only.

description: Check basic download options
test: runtest.sh
time: 3 min

The key content attributes are not supposed to be hard-coded in the Flexible Metadata Format but freely configurable. Multiple key content attributes (e.g. script & backend) could be used as well.

Virtual

Using a single test code for testing multiple scenarios can be easily implemented using leaves inheriting from the same parent:

description: Check basic download options
test: runtest.sh

/fast:
    description: Check basic download options (quick smoke test)
    environment: MODE=fast
    tags: [Tier1]
    time: 1 min
/full:
    description: Check basic download options (full test set)
    environment: MODE=full
    tags: [Tier2]
    time: 3 min

In this way we can efficiently create virtual test cases.

API

A python module is planned to be implemented which would take care of parsing and merging the metadata files and providing an easy API for automation scripts. A corresponding simple command line tool should allow to easily investigate metadata and could be used for example for adhoc test execution.

fmf --key test --search tag=Tier1
fmf --key requirement --search priority=high


Examples

Relevancy

Test Case Relevancy can be naturaly integrated:

description: Check basic download options
tags: [Tier2, TierSecurity]
relevancy:
- "distro < rhel-7: False"
- "arch = s390x: False"

Note that, because of YAML parsing, relevancy rules have to be enclosed in quotes. Another option is to use text format:

description: Check basic download options
tags: [Tier2, TierSecurity]
relevancy: |
    distro < rhel-7: False
    arch = s390x: False

Which seems a bit more clear.

Coverage

Test coverage information can be stored in a single file, for example wget/requirements.fmf:

/protocols:
    priority: high
    /ftp:
        requirement: Download a file using the ftp protocol.
        coverage: wget/protocols/ftp
    /http:
        requirement: Download a file using the http protocol.
        coverage: wget/protocols/http
    /https:
        requirement: Download a file using the https protocol.
        coverage: wget/protocols/https

/download:
    priority: medium
    /output-document-pipe:
        requirement: Save content to pipe.
        coverage: wget/download
    /output-document-file:
        requirement: Save content to a file.
        coverage: wget/download

/upload:
    priority: medium
    /post-file:
        requirement: Upload a file to the server
        coverage: wget/protocols/http
    /post-data:
        requirement: Upload a string to the server
        coverage: wget/protocols/http

Or split by functionality area into separate files as desired, for example wget/download/requirements.fmf:

priority: medium
/output-document-pipe:
    requirement: Save content to pipe.
    coverage: wget/download
/output-document-file:
    requirement: Save content to a file.
    coverage: wget/download

Or integrated with test case metadata, e.g. wget/download/main.fmf:

description: Check basic download options
tags: [Tier2, TierSecurity]
test: runtest.sh
time: 3 min

/requirements
    requirement: Various download options working correctly
    priority: low
    /get-file:
        coverage: wget/download
    /output-document:
        coverage: wget/download
    /continue:
    /timestamping:
    /tries:
    /no-clobber:
        coverage: wget/download
    /progress:
    /quota:
    /server-response:
    /bind-address:
    /spider:

In the example above three requirements are already covered, the rest still await for test coverage (attributes value is null).

Strategist

Here's an example implementation of test-strategist data for openscap using the Flexible Metadata Format:

/probes:
    description: Probes
    /offline:
        description: Offline scanning
    /online:
        description: Online scanning
/scanning:
    description: Reading and understanding source datastreams
    /oval:
        influencers:
        - openscap/probes/offline
        - openscap/probes/online
    /ds:
        influencers:
        - openscap/scanning/oval
        - openscap/scanning/cpe
    /cpe:
        influencers:
        - openscap/scanning/oval