From Fedora Project Wiki
(run-fedtest → run-installed-test)
(Clarified added links to example test suites and testing systems)
Line 44: Line 44:
 
This standard interface describes how to discover, stage and invoke tests. It is important to cleanly separate implementation details of the ''testing system'' from the ''test suite'' and its framework. It is also important to allow packagers to locally and manually invoke a ''test suite''.
 
This standard interface describes how to discover, stage and invoke tests. It is important to cleanly separate implementation details of the ''testing system'' from the ''test suite'' and its framework. It is also important to allow packagers to locally and manually invoke a ''test suite''.
  
'''Testing CI System''' responsibilities:
+
'''Testing System''' responsibilities:
 
* Builds or otherwise acquires a ''test subject'', such as a package, container image, OSTree ...
 
* Builds or otherwise acquires a ''test subject'', such as a package, container image, OSTree ...
 
* Schedules and orchestrates a test job on appropriate compute, storage, etc.
 
* Schedules and orchestrates a test job on appropriate compute, storage, etc.
Line 51: Line 51:
 
* Gathers the ''test results'' and ''test artifacts'' (using standard interface)
 
* Gathers the ''test results'' and ''test artifacts'' (using standard interface)
 
* Announces and relays those results for gating, archival, etc.
 
* Announces and relays those results for gating, archival, etc.
 +
 +
'''Testing System''' examples: [https://jenkins.io/ Jenkins], [https://taskotron.fedoraproject.org/ Taskotron], [https://docs.openstack.org/infra/zuul/ ZUUL], [https://ci.centos.org/ CentOS CI], Red Hat CI, [https://travis-ci.org/ Travis], [https://semaphoreci.com/ Semaphore], [https://developers.openshift.com/managing-your-applications/continuous-integration.html Openshift CI/CD]  ...
  
 
[[File:Invoking-tests-standard-interface.png|800px]]
 
[[File:Invoking-tests-standard-interface.png|800px]]
  
 
'''Testsuite''' responsibilities:
 
'''Testsuite''' responsibilities:
* Includes the testing framework (declared as a dependency for staging)
+
* Includes the ''test framework' (declared as a dependency for staging)
 
* May provision necessary container or VM based on the ''test subject''
 
* May provision necessary container or VM based on the ''test subject''
 
* Provide ''test results'' and ''test artifacts'' (as described by standard interface)
 
* Provide ''test results'' and ''test artifacts'' (as described by standard interface)
 +
 +
'''Test Suite and Framework''' examples: [https://avocado-framework.github.io/ Avocado], [https://wiki.gnome.org/Initiatives/GnomeGoals/InstalledTests GNOME Installed Tests], [https://pagure.io/modularity-testing-framework/ Modularity Testing Framework], [https://github.com/projectatomic/atomic-host-tests Ansible tests in Atomic Host], [https://tunir.readthedocs.io/en/latest/ Tunir tests], docker test images, ...
  
 
The format of the textual logs and ''test artifacts'' that come out of a test suite is not prescribed by this document. Nor is it envisioned to be standardized across all possible ''test suites''.
 
The format of the textual logs and ''test artifacts'' that come out of a test suite is not prescribed by this document. Nor is it envisioned to be standardized across all possible ''test suites''.
Line 63: Line 67:
 
=== Packaging ===
 
=== Packaging ===
  
The integration tests are packaged and delivered through Fedora as
+
The integration tests are packaged and delivered through Fedora as packages.
subpackages of the package they are associated with.
 
  
 
Each dist-git repo that has integration tests should package those tests in one or more subpackages
 
Each dist-git repo that has integration tests should package those tests in one or more subpackages
Line 71: Line 74:
 
today.
 
today.
  
The spec file for a dist-git repo may include upstream integration tests into
+
The spec file for a dist-git repo may install upstream integration tests as files in
its <code>%{name}-tests</code> subpackage. The spec file may also include tests
+
its <code>%{name}-tests</code> package. The spec file may also include tests
 
directly from files in <code>tests/</code> subdirectory of the dist-git repo itself.
 
directly from files in <code>tests/</code> subdirectory of the dist-git repo itself.
 +
 +
The tests package should use <code>Requires:</code> to require any other package, testing framework, or dependency necessary to run the tests. In ''in-situ'' testing cases, the tests package will directly <code>Requires:</code> the package of the ''test subject''.
  
 
=== Invocation ===
 
=== Invocation ===
  
To invoke the test suite, the subpackage is installed. Each test of the suite
+
To invoke the test suite, the test package that contains it is installed. Each test of the suite
 
installs an executable in the path <code>/usr/tests/</code>''sourcepackage''<code>/</code> (this will avoid name collisions between packages).
 
installs an executable in the path <code>/usr/tests/</code>''sourcepackage''<code>/</code> (this will avoid name collisions between packages).
  
 
To invoke the test suite, one would:
 
To invoke the test suite, one would:
  
# Create a temporary directory, called `$TESTDIR` here.
+
# Create a temporary directory, referred to as: <code>$TESTDIR</code>
# Place the subjects being tested in <code>$TESTDIR/subjects/</code>
+
# Place the ''test subject(s)'' being tested in <code>$TESTDIR/subjects/</code>
# Execute all executable files in <code>/usr/tests/*/</code> one at a time.
+
# Execute all executable files in <code>/usr/tests/*/</code> directories one at a time.
## The test is invoked with a working directory of <code>$TESTDIR</code>
+
## Each executable test is invoked with a working directory of <code>$TESTDIR</code>
## The test is invoked as root, and may drop privileges as desired
+
## Each executable test is invoked as root, and may drop privileges as desired.
## Treat the stdout/stderr of each tests process as the test log. This is a standard artifact and written to <code>$TESTDIR/artifacts/testname.log</code>.
+
## Treat the stdout/stderr of the test process as the test log. This is a standard ''test artifact'' and written to <code>$TESTDIR/artifacts/testname.log</code>.
## Examine the exit code of each test process. Zero exit code is success, non-zero is failure.
+
## Examine the exit code of each test process. Zero exit code is a successful ''test result'', non-zero is failure.
# Tests can put any additional artifacts like screenshots into <code>$TESTDIR/artifacts/</code>.
+
# Tests can put any additional ''test artifacts'' like screenshots into <code>$TESTDIR/artifacts/</code>.
  
 
This ensures that tests can be run on a production system without accidentally clobbering permanent directories,
 
This ensures that tests can be run on a production system without accidentally clobbering permanent directories,
Line 96: Line 101:
 
as these would usually get stored for a longer time period.
 
as these would usually get stored for a longer time period.
  
These steps would usually be done through a standard test driver tool (particularly for sensible stdout/err teeing and log capturing),
+
These steps would usually be done through a standard test driver tool (particularly for sensible stdout/stderr teeing and log capturing), but its usage is not mandatory for developing and calling tests manually.
but its usage is not mandatory for developing and calling tests manually.
 
 
 
Multiple subpackages may be installed as long as their dependencies do not conflict.
 
  
 
=== Staging ===
 
=== Staging ===
  
The <code>%{name}-test</code> subpackage should <code>Requires:</code> all other packages
+
The <code>%{name}-test</code> package should <code>Requires:</code> all other packages
 
that the testsuite executable needs in order to run. This includes libraries or frameworks,
 
that the testsuite executable needs in order to run. This includes libraries or frameworks,
 
or subsystems like <code>libvirt</code>.
 
or subsystems like <code>libvirt</code>.
  
Some integration tests may choose to test in-situ, on the system on which the test suite
+
Some integration tests may choose to test ''in-situ'', on the system on which the test suite
is installed. In these cases the <code>%{name}-tests</code> subpackage should directly
+
is installed. In these cases the <code>%{name}-tests</code> package should directly
 
depend on the package being tested.
 
depend on the package being tested.
  
More rigourous integration tests test an integrated system from the outside. It is the
+
More rigorous integration tests are ''outside-in''. They test an integrated system without affecting its contents. It is the responsibility of the <code>%{name}-tests</code> packages to provision virtual machines or containers necessary to do such testing. In almost all cases this will happen by way of a provisioning framework such as [http://avocado-framework.readthedocs.org/ Avocado], [https://www.ansible.com/ Ansible], [https://pagure.io/modularity-testing-framework/ Module Testing Framework], [https://linch-pin.readthedocs.io/en/latest/ linch-pin], etc.
responsibility of the <code>%{name}-tests</code> subpackages to provision virtual
 
machines or containers necessary to do such testing. In almost all cases this will happen
 
by way of a provisioning framework such as Avocado, Ansible, Module Testing Framework,
 
linch-pin, etc.
 
  
 +
Multiple tests packages may be installed as long as their dependencies do not conflict.
  
 
=== Discovery ===
 
=== Discovery ===
A testing system needs to be able to efficiently answer the question "does this subject have any tests packages, and if so, what are their names".
 
This should be automatically discoverable to the extent possible.
 
  
Particularly for any RPM test subject this process requires no additional metadata and can be fully automatic:
+
A testing system needs to be able to efficiently answer the question "does this subject have any tests packages, and if so, what are their names". This should be automatically discoverable to the extent possible.
It is possible to map a RPM to its SRPM source package (<code>&lt;rpm:sourcerpm&gt;</code> in the package index <code>*-primary.xml.gz</code>),
 
and map an SRPM to all the RPMs that it builds (from the same index), and using the <code>*-filelists.xml.gz</code> index one can mechanically tell which
 
of the RPMs are of this "test package" kind described here.
 
  
For other types of test subject cases such as docker images or distribution ISO files this discovery still needs to be discussed.
+
For any RPM ''test subject'' this process requires no additional metadata and can be fully automatic:
E. g. a <code>Dockerfile</code> might grow a reference to a test package RPM, or at least initially there is a manually maintained map of subject
+
 
to test package in the testing system.
+
* It is possible to map a RPM to its SRPM source package (<code>&lt;rpm:sourcerpm&gt;</code> in the package index <code>*-primary.xml.gz</code>).
 +
* One can map an SRPM to all the RPMs that it builds (from the same index), and using the <code>*-filelists.xml.gz</code> index one can mechanically tell which of the RPMs are of this test package kind described here.
 +
 
 +
'''TODO''': For other types of test subject cases such as docker images or distribution ISO files this discovery still needs to be discussed.
 +
* E. g. a <code>Dockerfile</code> might grow a reference to a test package RPM, or at least initially there is a manually maintained map of subject to test package in the testing system.
  
 
== Scope ==
 
== Scope ==
  
This change requires no initial changes to Fedora infrastructure itself, but is limited
+
This change requires no initial changes to Fedora infrastructure itself. The change only affects contents spec files in dist-git repos.  
to changes in dist-git repos. However certain key infrastructure changes could
 
mitigate usability or side-effects of this change.
 
  
In particular, once this grows beyond the experimental phase, these test packages need to be put into a separate archive, similar to <code>-debuginfo</code>.
+
'''TODO:''' However certain key infrastructure changes could mitigate usability or side-effects of this change. In particular, once this grows beyond the experimental phase, these test packages need to be put into a separate archive, similar to <code>-debuginfo</code>.
  
'''TODO:'''
 
 
* How much effort is that to set up?
 
* How much effort is that to set up?
 
* Does this require any additional tags, keywords, or other explicit declaration in the spec file, other than "this RPM ships something in <code>/usr/tests/*</code>"?
 
* Does this require any additional tags, keywords, or other explicit declaration in the spec file, other than "this RPM ships something in <code>/usr/tests/*</code>"?
 
  
 
== Benefit to Fedora ==
 
== Benefit to Fedora ==
  
Developers benefit by having a consistent target for how to describe tests,
+
Developers benefit by having a consistent target for how to describe tests, while also being able to execute them locally while debugging issues or iterating on tests.
while also being able to execute them locally while debugging issues or
 
iterating on tests.
 
  
By packaging, staging and invoking tests consistently in Fedora we
+
By packaging, staging and invoking tests consistently in Fedora we create an eco-system for the tests that allows varied test frameworks as well as CI system infrastructure to interoperate. The integration tests outlast the implementation details of either the frameworks they're written in or the CI systems running them.
create an eco-system for the tests that allows varied test frameworks as
 
well as CI system infrastructure to interoperate. The integration tests
 
outlast the implementation details of either the frameworks they're written
 
in or the CI systems running them.
 
  
 
== User Experience ==
 
== User Experience ==
Line 160: Line 148:
 
A standard way to package tests benefits Fedora stability, and makes Fedora better for users.
 
A standard way to package tests benefits Fedora stability, and makes Fedora better for users.
  
Users could also benefit by having tests that they can reproduce on their own systems.
+
Users could also benefit by having tests that they can reproduce on their own systems. They could install the similar to how they consume <code>%{name}-doc</code> or <code>%{name}-debuginfo</code> subpackages today.
They could install the similar to how they consume <code>%{name}-doc</code>
 
or <code>%{name}-debuginfo</code> subpackages today.
 
  
We may choose to avoid having such packages available in the standard repositories.
+
We may choose to avoid having such packages available in the standard repositories. We may choose to only have them in <code>updates-testing</code> or an arrangement similar to <code>debuginfo</code>. These choices will require some
We may choose to only have them in <code>updates-testing</code> or an
 
arrangement similar to <code>debuginfo</code>. These choices will require some
 
 
markup and/or change to infrastructure.
 
markup and/or change to infrastructure.
  
 
== Upgrade/compatibility impact ==
 
== Upgrade/compatibility impact ==
  
Although there may already be subpackages that are named
+
Although there may already be packages that are named <code>%{name}-tests</code> this is merely a convention, and such packages will not affect the behavior of this proposal.
<code>%{name}-tests</code> this is merely a convention, and such packages will
 
not affect the behavior of this proposal.
 
  
 
== Examples ==
 
== Examples ==

Revision as of 11:59, 30 March 2017

Standard Discovery, Packaging, Invocation of Integration Tests

Warning.png
This is incomplete
This file is incomplete.
Warning.png
This is a proposal
Feedback is more than welcome. There's a discussion tab above.

Summary

What follows is a standard way to discover, package and invoke integration tests for a package stored in a Fedora dist-git repo.

Many Fedora packages have unit tests. These tests are typically run during a %check RPM build step and run in a build root. On the other hand, integration testing should happen against a composed system. Upstream projects have integration tests, both Fedora QA and the Atomic Host team would like to create more integration tests, Red Hat would like to bring integration tests upstream.

Goals:

  • A standard way to to discover, stage and invoke integration tests.
  • Implementation details of the test suite or its framework should not leak into or rely on the CI system invoking it.
  • CI system implementation details should not leak into the test suite or its metadata.
  • It should be possible to change the CI system that runs a test suite.
  • It should be possible for a dist-git packager to run a test suite locally.
  • It must be possible to package upstream tests.
  • Both in-situ tests, and more rigorous outside-in tests must be possible.

Owner

Terminology

  • Test Subject: The items that are to be tested. Typically this is a set of RPMs (updating a package to a new version), a container, or a distribution install ISO/VM image.
  • Test: A callable/runnable piece of code and corresponding test data and mocks which exercises and evaluates a subject.
  • Test Suite: The collection of all tests that apply to a subject. It's common to split the testing of different aspects into different tests for easier result evaluation, code maintenance, or parallelization.
  • Test Result: A boolean output of a test which decides whether the test passed. This is being used for automatic result processing (gating in Continuous Integration).
  • Test Artifact: Any additional output of the test such as the stdout/err output, log files, screen shots, core dumps, or TAP/Junit/subunit streams. These are mostly for human consumption (evaluating test failures by developers), but in case of machine readable files like JUnit they can also be used in result browsers for presenting test results.
  • Testing System: A continuous integration or other testing system that would like to discover, stage and invoke tests for a test subject.

Detailed Description

This standard interface describes how to discover, stage and invoke tests. It is important to cleanly separate implementation details of the testing system from the test suite and its framework. It is also important to allow packagers to locally and manually invoke a test suite.

Testing System responsibilities:

  • Builds or otherwise acquires a test subject, such as a package, container image, OSTree ...
  • Schedules and orchestrates a test job on appropriate compute, storage, etc.
  • Stages the test suite (using standard interface)
  • Invokes the test suite (using standard interface)
  • Gathers the test results and test artifacts (using standard interface)
  • Announces and relays those results for gating, archival, etc.

Testing System examples: Jenkins, Taskotron, ZUUL, CentOS CI, Red Hat CI, Travis, Semaphore, Openshift CI/CD ...

Invoking-tests-standard-interface.png

Testsuite responsibilities:

  • Includes the test framework' (declared as a dependency for staging)
  • May provision necessary container or VM based on the test subject
  • Provide test results and test artifacts (as described by standard interface)

Test Suite and Framework examples: Avocado, GNOME Installed Tests, Modularity Testing Framework, Ansible tests in Atomic Host, Tunir tests, docker test images, ...

The format of the textual logs and test artifacts that come out of a test suite is not prescribed by this document. Nor is it envisioned to be standardized across all possible test suites.

Packaging

The integration tests are packaged and delivered through Fedora as packages.

Each dist-git repo that has integration tests should package those tests in one or more subpackages like %{name}-tests. This is similar to the %{name}-debuginfo or %{name}-docs subpackages we have today.

The spec file for a dist-git repo may install upstream integration tests as files in its %{name}-tests package. The spec file may also include tests directly from files in tests/ subdirectory of the dist-git repo itself.

The tests package should use Requires: to require any other package, testing framework, or dependency necessary to run the tests. In in-situ testing cases, the tests package will directly Requires: the package of the test subject.

Invocation

To invoke the test suite, the test package that contains it is installed. Each test of the suite installs an executable in the path /usr/tests/sourcepackage/ (this will avoid name collisions between packages).

To invoke the test suite, one would:

  1. Create a temporary directory, referred to as: $TESTDIR
  2. Place the test subject(s) being tested in $TESTDIR/subjects/
  3. Execute all executable files in /usr/tests/*/ directories one at a time.
    1. Each executable test is invoked with a working directory of $TESTDIR
    2. Each executable test is invoked as root, and may drop privileges as desired.
    3. Treat the stdout/stderr of the test process as the test log. This is a standard test artifact and written to $TESTDIR/artifacts/testname.log.
    4. Examine the exit code of each test process. Zero exit code is a successful test result, non-zero is failure.
  4. Tests can put any additional test artifacts like screenshots into $TESTDIR/artifacts/.

This ensures that tests can be run on a production system without accidentally clobbering permanent directories, don't require root privileges (simplifies test development), and that CI systems have one unique place from where to collect artifacts. It also avoids collecting temporary files such as downloaded container or VM images as artifacts, as these would usually get stored for a longer time period.

These steps would usually be done through a standard test driver tool (particularly for sensible stdout/stderr teeing and log capturing), but its usage is not mandatory for developing and calling tests manually.

Staging

The %{name}-test package should Requires: all other packages that the testsuite executable needs in order to run. This includes libraries or frameworks, or subsystems like libvirt.

Some integration tests may choose to test in-situ, on the system on which the test suite is installed. In these cases the %{name}-tests package should directly depend on the package being tested.

More rigorous integration tests are outside-in. They test an integrated system without affecting its contents. It is the responsibility of the %{name}-tests packages to provision virtual machines or containers necessary to do such testing. In almost all cases this will happen by way of a provisioning framework such as Avocado, Ansible, Module Testing Framework, linch-pin, etc.

Multiple tests packages may be installed as long as their dependencies do not conflict.

Discovery

A testing system needs to be able to efficiently answer the question "does this subject have any tests packages, and if so, what are their names". This should be automatically discoverable to the extent possible.

For any RPM test subject this process requires no additional metadata and can be fully automatic:

* It is possible to map a RPM to its SRPM source package (<rpm:sourcerpm> in the package index *-primary.xml.gz).
* One can map an SRPM to all the RPMs that it builds (from the same index), and using the *-filelists.xml.gz index one can mechanically tell which of the RPMs are of this test package kind described here.

TODO: For other types of test subject cases such as docker images or distribution ISO files this discovery still needs to be discussed.

* E. g. a Dockerfile might grow a reference to a test package RPM, or at least initially there is a manually maintained map of subject to test package in the testing system.

Scope

This change requires no initial changes to Fedora infrastructure itself. The change only affects contents spec files in dist-git repos.

TODO: However certain key infrastructure changes could mitigate usability or side-effects of this change. In particular, once this grows beyond the experimental phase, these test packages need to be put into a separate archive, similar to -debuginfo.

  • How much effort is that to set up?
  • Does this require any additional tags, keywords, or other explicit declaration in the spec file, other than "this RPM ships something in /usr/tests/*"?

Benefit to Fedora

Developers benefit by having a consistent target for how to describe tests, while also being able to execute them locally while debugging issues or iterating on tests.

By packaging, staging and invoking tests consistently in Fedora we create an eco-system for the tests that allows varied test frameworks as well as CI system infrastructure to interoperate. The integration tests outlast the implementation details of either the frameworks they're written in or the CI systems running them.

User Experience

A standard way to package tests benefits Fedora stability, and makes Fedora better for users.

Users could also benefit by having tests that they can reproduce on their own systems. They could install the similar to how they consume %{name}-doc or %{name}-debuginfo subpackages today.

We may choose to avoid having such packages available in the standard repositories. We may choose to only have them in updates-testing or an arrangement similar to debuginfo. These choices will require some markup and/or change to infrastructure.

Upgrade/compatibility impact

Although there may already be packages that are named %{name}-tests this is merely a convention, and such packages will not affect the behavior of this proposal.

Examples

With this you can install test RPM from above gzip repo:

 $ sudo rpm -i results_gzip/1.8/2.fc27/gzip-tests-1.8-2.fc25.x86_64.rpm

and run the gzip tests on the already installed package (as user) with

 $ ~/run-installed-test
 Subjects/artifacts directory: /tmp/fedtest.vsR
 -----------------------------------------
 Running /usr/tests/gzip/test-simple
 -----------------------------------------
 ++ ls 'subjects/*.rpm'
 + echo Bla
 + cp bla.file bla.file.orig
 + gzip bla.file
 + gunzip bla.file.gz
 + cmp bla.file bla.file.orig
 + rm bla.file bla.file.orig
 PASS: /usr/tests/gzip/test-simple
 $ ls -l /tmp/fedtest.vsR/artifacts/
 -rw-r--r-- 1 martin martin 156 Mar 28 16:49 test-simple.log

or run them as root (as officially specified) with a subject (locally built gzip RPM):

 $ sudo ~/run-installed-test results_gzip/1.8/2.fc27/gzip-1.8-2.fc25.x86_64.rpm
 Installing subject results_gzip/1.8/2.fc27/gzip-1.8-2.fc25.x86_64.rpm
 Subjects/artifacts directory: /tmp/fedtest.Cck
 -----------------------------------------
 Running /usr/tests/gzip/test-simple
 -----------------------------------------
 ++ ls subjects/gzip-1.8-2.fc25.x86_64.rpm
 + '[' -w / ']'
 + rpm --verbose --force -U subjects/gzip-1.8-2.fc25.x86_64.rpm
 Preparing packages...
 gzip-1.8-2.fc25.x86_64
 + echo Bla
 + cp bla.file bla.file.orig
 + gzip bla.file
 + gunzip bla.file.gz
 + cmp bla.file bla.file.orig
 + rm bla.file bla.file.orig
 PASS: /usr/tests/gzip/test-simple

Similarly, the glib tests can be invoked with

 sudo rpm -i results_glib2/2.52.0/1.fc27/glib2-distro-tests-2.52.0-1.fc25.x86_64.rpm
 sudo ~/run-installed-test 
 Subjects/artifacts directory: /tmp/fedtest.40K
 -----------------------------------------
 Running /usr/tests/glib2/test-docker-rawhide
 -----------------------------------------
 + dnf install -y glib2-tests gnome-desktop-testing dbus-x11 xorg-x11-server-Xvfb which
 [...]
 + dbus-run-session -- xvfb-run -a gnome-desktop-testing-runner glib/utils
 Running test: glib/utils.test
 1..25
 # Start of utils tests
 [..]
 PASS: glib/utils.test
 SUMMARY: total=1; passed=1; skipped=0; failed=0; user=0.0s; system=0.0s; maxrss=3336
 PASS: /usr/tests/glib2/test-docker-rawhide

Passing in additional RPMs and generated log artifact are similar.

Notes

  • ...
  • ...