mNo edit summary |
(Minor adjustments to path and time.) |
||
| Line 103: | Line 103: | ||
User stories: | User stories: | ||
* As a test writer I define two virtual test cases, both using the same script for executing. | * As a test writer I define two virtual test cases, both using the same script for executing. See also the [[#Virtual_Tests|Virtual Tests]] example. | ||
Example: | Example: | ||
| Line 113: | Line 113: | ||
In order to prevent stuck tests consuming resources we should be able to define a maximum time for test execution. | In order to prevent stuck tests consuming resources we should be able to define a maximum time for test execution. | ||
* Name: | * Name: time | ||
* Type: string | * Type: string | ||
* Purpose: maximum time for test execution after which a running test is killed by the test harness | * Purpose: maximum time for test execution after which a running test is killed by the test harness | ||
| Line 119: | Line 119: | ||
Notes: | Notes: | ||
* | * Let's use the same format as the ``sleep`` command. For example: 3m, 2h, 1d. | ||
User stories: | User stories: | ||
Revision as of 07:20, 27 April 2018
Requirements
In order to use the Flexible Metadata Format effectively for the CI testing we need to agree on the essential set of attributes to be used. For each attribute we need to standardize:
- Name ... unique, well chosen, possibly with a prefix
- Type ... expected content type: string, number, list, dictionary
- Purpose ... description of the attribute purpose
For names we should probably consider using namespace prefix (such as test-description, requirement-description) to prevent future collisions with other attributes. Each attribute definition should contain at least one apt example of the usage. Or better, a set of user stories to be covered.
Attributes
In this section there are attributes proposed so far. Material for discussion. Nothing final for now.
Summary
In order to efficiently collaborate on test maintenance it's crucial to have a short summary of what the test does.
- Name ... summary
- Type ... string (one line, up to 50 characters)
- Purpose ... concise summary of what the test does
User stories:
- As a developer reviewing 10 failed tests I would like to get quickly an idea of what my change broke.
Notes:
- Shall we recommend 50 characters or less? Like there is for commits? Yes.
Example:
summary: Test wget recursive download options
Description
For complex tests it makes sense to provide more detailed description to better clarify what is covered by the test.
- Name ... description
- Type ... string (multi line, plain text)
- Purpose ... detailed description of what the test does
User stories:
- As a tester I come to a test code I wrote 10 years ago (so I have absolutely no idea about it) and would like to quickly understand what it does.
- As a developer I review existing test coverage for my component and would like to get an overall idea what is covered without having to read the whole test code.
Example:
description: | This test checks all available wget options related to downloading files recursively. First a tree directory structure is created for testing. Then a file download is performed for different recursion depth specified by the "--level=depth" option.
Tags
Throughout the years, free-form tags proved to be useful for many, many scenarios. Primarily to provide an easy way how to select a subset of objects.
- Name: tags
- Type: list
- Purpose: free-form tags for easy filtering
Notes:
- Tags are case-sensitive.
- Using lowercase is recommended.
User stories:
- A a developer/tester I would like to run only a subset of available tests.
Example:
tags: [Tier1, fast]
Test
This is the key content attribute defining how the test is to be executed.
- Name: test
- Type: string
- Purpose: shell command which executes the test
User stories:
- As a developer/tester I want to easily execute all available tests with just one command.
- As a test writer I want to run a single test script in multiple ways (e.g. providing different parameters)
Example:
test: ./runtest.sh
Path
As the object hierarchy does not need to copy the filesystem structure (e.g. virtual test cases) we need a way how to define where the test is located.
- Name: path
- Type: string
- Purpose: filesystem directory to be entered before executing the test
User stories:
- As a test writer I define two virtual test cases, both using the same script for executing. See also the Virtual Tests example.
Example:
path: wget/recursion
Time
In order to prevent stuck tests consuming resources we should be able to define a maximum time for test execution.
- Name: time
- Type: string
- Purpose: maximum time for test execution after which a running test is killed by the test harness
Notes:
- Let's use the same format as the
sleepcommand. For example: 3m, 2h, 1d.
User stories:
- As a deverloper/tester I want to prevent resource wasting by stuck tests.
- As a test harness I need to know after how long time I should kill test if it is still running.
Example:
time: 5m
Examples
BeakerLib Tests
Three beakerlib tests, each in it's own directory:
main.fmf
test: ./runtest.sh
/one:
path: tests/one
/two:
path: tests/two
/three:
path: tests/three
fmf
tests/one path: tests/one test: ./runtest.sh
tests/two path: tests/two test: ./runtest.sh
tests/three path: tests/three test: ./runtest.sh
Three Scripts
Three different script residing in a single directory:
main.fmf
path: tests
/one:
test: ./one
/two:
test: ./two
/three:
test: ./three
fmf
tests/one path: tests test: ./one
tests/two path: tests test: ./two
tests/three path: tests test: ./three
Virtual Tests
Thre virtual test cases based on a single test script:
main.fmf
path: tests/virtual
/one:
test: ./script --one
/two:
test: ./script --two
/three:
test: ./script --three
fmf
tests/one path: tests test: ./script --one
tests/two path: tests test: ./script --two
tests/three path: tests test: ./script --three