From Fedora Project Wiki

Revision as of 13:20, 23 March 2017 by Stefw (talk | contribs) (Initial page)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

Standard Dicsovery, Packaging, Invocation of Integration Tests

This is incomplete
This file is incomplete.

Reason

Many Fedora packages have unit tests. These tests are typically run during a %check RPM build step. These tests run in a BuildRoot.

On the other hand, integration testing happens against a composed system. system. Several upstream projects have integration tests, but these have typically not been used or packaged in Fedora. Both Fedora QA and the Atomic Host team would like to create more integration tests, stored in dist-git.

Summary

What follows is a standard way to discover, package and invoke integration tests for a package stored in a Fedora dist-git repo.

The integration tests are packaged and delivered through Fedora as %{name}-tests subpackages of the package they are associated with.

This change requires no changes to Fedora infrastructure itself. It is limited to changes in dist-git repos. However certain key infrastructure changes could mitigate usability or side-effects of this change.

This is a proposal
Feedback is more than welcome. There's a discussion tab above.

Packaging

Each dist-git repo that has integration tests should package those tests in a %{name}-tests subpackage. This is similar to the %{name}-debuginfo or %{name}-docs subpackages we have today.

The spec file for a dist-git repo may include upstream integration tests into its %{name}-tests subpackage. The spec file may also include tests directly from files in the dist-git repo itself.

Staging

The %{name}-test subpackage should depend on all other packages that the test needs to run.

Some integration tests may choose to test in-situ, on the system on which the %{name}-tests is installed. In these cases the %{name}-tests subpackage should depend on the %{name} or other subpackages that are being tested.

More rigourous integration tests test an integrated system from the outside. It is the responsibility of the %{name}-tests subpackages to provision virtual machines or containers necessary to do such testing. In almost all cases this will happen by way of a provisioning framework such as Avocado, Ansible, Module Testing Framework, linch-pin, etc.

Invocation

TODO: Expand this section.

* Any executable file directly under /usr/tests/ is a test suite to be executed
* Zero exit code from the executable indicates testsuite success, a non-zero exit code indicates test suite failure
* The stdout/stderr of a test suite is the test log
* Additional output artifacts are placed in /run/tests/
* The testsuite is invoked as root, and may drop privileges as desired

Discovery

In a dist-git repo the list of integration tests that should be invoked on a given package are placed in a file called tests in that dist-git repo. The format of this file has not yet been defined, but a simple text file similar to sources listing the package names that include integration tests to use when validating the dist-git repo.

Examples

TODO: Build out this section

Notes

* ...
* ...