|
|
| (17 intermediate revisions by the same user not shown) |
| Line 1: |
Line 1: |
| = Requirements =
| | Fedora CI metadata specification moved to https://pagure.io/fedora-ci/metadata. |
| | |
| In order to use the [[Flexible_Metadata_Format|Flexible Metadata Format]] effectively for the CI testing we need to agree on the essential set of attributes to be used. For each attribute we need to standardize:
| |
| | |
| * '''Name''' ... unique, well chosen, possibly with a prefix
| |
| * '''Type''' ... expected content type: string, number, list, dictionary
| |
| * '''Purpose''' ... description of the attribute purpose
| |
| | |
| For names we should probably consider using namespace prefix (such as test-description, requirement-description) to prevent future collisions with other attributes. Each attribute definition should contain at least one apt example of the usage. Or better, a set of user stories to be covered.
| |
| | |
| = Attributes =
| |
| | |
| In this section there are attributes proposed so far. Material for discussion. Nothing final for now.
| |
| | |
| == Summary ==
| |
| | |
| In order to efficiently collaborate on test maintenance it's crucial to have a short summary of what the test does.
| |
| | |
| * Name ... summary, test-summary
| |
| * Type ... string (one line)
| |
| * Purpose ... concise summary of what the test does
| |
| | |
| User stories:
| |
| | |
| * As a developer reviewing 10 failed tests I would like to get quickly an idea of what my change broke.
| |
| | |
| Notes:
| |
| | |
| * Shall we recommend 50 characters or less? Like there is for [https://stackoverflow.com/questions/2290016/git-commit-messages-50-72-formatting commits]?
| |
| | |
| Example:
| |
| | |
| summary: Test wget recursive download options
| |
| | |
| == Description ==
| |
| | |
| For complex tests it makes sense to provide more detailed description to better clarify what is covered by the test.
| |
| | |
| * Name ... description, test-description
| |
| * Type ... string (multi line)
| |
| * Purpose ... detailed description of what the test does
| |
| | |
| User stories:
| |
| | |
| * As a tester I come to a test code I wrote 10 years ago (so I have absolutely no idea about it) and would like to quickly understand what it does.
| |
| * As a developer I review existing test coverage for my component a would like to get an overall idea what is covered without having to read the whole test code.
| |
| | |
| Example:
| |
| | |
| description: |
| |
| This test checks all available wget options related to
| |
| downloading files recursively. First a tree directory
| |
| structure is created for testing. Then a file download
| |
| is performed for different recursion depth specified by
| |
| the "--level=depth" option.
| |
| | |
| == Tags ==
| |
| | |
| Throughout the years, free-form tags proved to be useful for many, many scenarios. Primarily to provide an easy way how to select a subset of objects.
| |
| | |
| * Name: tags
| |
| * Type: list
| |
| * Purpose: free-form tags for easy filtering
| |
| | |
| Notes:
| |
| | |
| * Tags should be case-sensitive. Or not?
| |
| | |
| User stories:
| |
| | |
| * A a developer/tester I would like to run only a subset of available tests.
| |
| | |
| Example:
| |
| | |
| tags: [Tier1, fast]
| |
| | |
| == Test ==
| |
| | |
| This is the [http://fmf.readthedocs.io/en/latest/features.html#leaves key content] attribute defining how the test is to be executed.
| |
| | |
| * Name: test
| |
| * Type: string
| |
| * Purpose: shell command which executes the test
| |
| | |
| User stories:
| |
| | |
| * As a developer/tester I want to easily execute all available tests with just one command.
| |
| * As a test writer I want to run a single test script in multiple ways (e.g. providing different parameters)
| |
| | |
| Example:
| |
| | |
| test: ./runtest.sh
| |
| | |
| == Path ==
| |
| | |
| As the object hierarchy does not need to copy the filesystem structure (e.g. [http://fmf.readthedocs.io/en/latest/features.html#virtual virtual] test cases) we need a way how to define where the test is located.
| |
| | |
| * Name: path
| |
| * Type: string
| |
| * Purpose: filesystem directory to be entered before executing the test
| |
| | |
| User stories:
| |
| | |
| * As a test writer I define two virtual test cases, both using the same script for executing.
| |
| | |
| Example:
| |
| | |
| path: wget/recursion
| |
| | |
| | |
| = Examples =
| |
| | |
| == BeakerLib Tests ==
| |
| | |
| Three beakerlib tests, each in it's own directory:
| |
| | |
| main.fmf
| |
| | |
| test: ./runtest.sh
| |
|
| |
| /one:
| |
| path: tests/one
| |
| /two:
| |
| path: tests/two
| |
| /three:
| |
| path: tests/three
| |
| | |
| fmf
| |
| | |
| tests/one
| |
| path: tests/one
| |
| test: ./runtest.sh
| |
| | |
| tests/two
| |
| path: tests/two
| |
| test: ./runtest.sh
| |
| | |
| tests/three
| |
| path: tests/three
| |
| test: ./runtest.sh
| |
| | |
| == Three Scripts ==
| |
| | |
| Three different script residing in a single directory:
| |
| | |
| main.fmf
| |
| | |
| path: tests
| |
|
| |
| /one:
| |
| test: ./one
| |
| /two:
| |
| test: ./two
| |
| /three:
| |
| test: ./three
| |
| | |
| fmf
| |
| | |
| tests/one
| |
| path: tests
| |
| test: ./one
| |
| | |
| tests/two
| |
| path: tests
| |
| test: ./two
| |
| | |
| tests/three
| |
| path: tests
| |
| test: ./three
| |
| | |
| == Virtual Tests ==
| |
| | |
| Thre virtual test cases based on a single test script:
| |
| | |
| main.fmf
| |
| | |
| path: tests/virtual
| |
|
| |
| /one:
| |
| test: ./script --one
| |
| /two:
| |
| test: ./script --two
| |
| /three:
| |
| test: ./script --three
| |
| | |
| fmf
| |
| | |
| tests/one
| |
| path: tests
| |
| test: ./script --one
| |
| | |
| tests/two
| |
| path: tests
| |
| test: ./script --two
| |
| | |
| tests/three
| |
| path: tests
| |
| test: ./script --three
| |
