From Fedora Project Wiki
(Initial page)
 
(Updates building this out)
Line 3: Line 3:
 
{{admon/warning|This is incomplete|This file is incomplete.}}
 
{{admon/warning|This is incomplete|This file is incomplete.}}
  
== Reason ==
+
{{admon/warning|This is a proposal|Feedback is more than welcome.
 +
There's a ''discussion'' tab above.}}
 +
 
 +
== Summary ==
 +
 
 +
What follows is a standard way to discover, package and invoke integration
 +
tests for a package stored in a Fedora dist-git repo.
 +
 
 +
== Owner ==
 +
 
 +
* Name: [[User:Stefw| Stef Walter]]
 +
* Email: stfw@fedoraproject.org
 +
* Name: ''Looking for a co-owner''
 +
<!-- * Release notes owner: To be assigned by docs team [[User:FASAccountName| Release notes owner name]] <email address> -->
 +
 
 +
== Detailed Description ==
  
 
Many Fedora packages have unit tests. These tests are typically run during a
 
Many Fedora packages have unit tests. These tests are typically run during a
<code>%check</code> RPM build step. These tests run in a BuildRoot.
+
<code>%check</code> RPM build step. These tests run in the build root in which
 +
the package is built and may involve the <code>BuildRequire</code> dependencies.
  
 
On the other hand, integration testing happens against a composed system.
 
On the other hand, integration testing happens against a composed system.
system. Several upstream projects have integration tests, but these have
+
Several upstream projects have integration tests, but these integration tests have
typically not been used or packaged in Fedora. Both Fedora QA and the
+
typically not been used or packaged in Fedora. Both Fedora QA and the Atomic Host
Atomic Host team would like to create more integration tests, stored in
+
team would like to create more integration tests, stored in dist-git, and use a CI
dist-git.
+
system to run them on Fedora packages.
 +
 
 +
What follows is a standard way to discover, package and invoke integration
 +
tests for a package stored in a Fedora dist-git repo.
 +
 
 +
It is important that the CI system which runs the test is independent from the
 +
layout of the test themselves.
 +
 
 +
'''Goals:'''
  
== Summary ==
+
* Implementation details of the integration testsuite or its framework should not leak into or rely on the CI system invoking the test.
 +
* CI system implementation details should not leak into the testsuite or its metadata.
 +
* It should be possible to change the CI system that runs a testsuite.
 +
* It should be possible for a dist-git packager to run a testsuite locally.
  
 
What follows is a standard way to discover, package and invoke integration
 
What follows is a standard way to discover, package and invoke integration
Line 22: Line 49:
 
<code>%{name}-tests</code> subpackages of the package they are associated with.
 
<code>%{name}-tests</code> subpackages of the package they are associated with.
  
This change requires no changes to Fedora infrastructure itself. It is limited
+
=== Packaging ===
to changes in dist-git repos. However certain key infrastructure changes could
 
mitigate usability or side-effects of this change.
 
 
 
{{admon/warning|This is a proposal|Feedback is more than welcome.
 
There's a ''discussion'' tab above.}}
 
 
 
== Packaging ==
 
  
 
Each dist-git repo that has integration tests should package those tests in a
 
Each dist-git repo that has integration tests should package those tests in a
Line 38: Line 58:
 
The spec file for a dist-git repo may include upstream integration tests into
 
The spec file for a dist-git repo may include upstream integration tests into
 
its <code>%{name}-tests</code> subpackage. The spec file may also include tests
 
its <code>%{name}-tests</code> subpackage. The spec file may also include tests
directly from files in the dist-git repo itself.
+
directly from files in <code>tests/</code> subdirectory of the dist-git repo itself.
 +
 
 +
=== Invocation ===
 +
 
 +
To invoke the testsuite, the subpackage is installed. Each integration testsuite
 +
that is included in installs an executable in the path <code>/usr/tests/</code>.
 +
 
 +
To invoke the integration test suites, one would:
  
== Staging ==
+
# Place the artifact being tested, usually RPM files, in <code>/var/cache/tests/</code>
 +
# Execute all executable files in <code>/usr/tests/</code> one at a time.
 +
## The testsuite is invoked with a working directory of <code>/</code>
 +
## The testsuite is invoked as root, and may drop privileges as desired
 +
## Treat the stdout/stderr of each testsuite process as the testsuite result log.
 +
## Examine the exit code of each testsuite process. Zero exit code is success, non-zero is failure.
 +
# Gather any additional result artifacts like screenshots which test suites place in a <code>/var/log/tests/</code> directory
  
The <code>%{name}-test</code> subpackage should depend on all other packages that the
+
Multiple subpackages may be installed as long as their dependencies do not conflict.
test needs to run.
 
  
Some integration tests may choose to test in-situ, on the system on which the <code>%{name}-tests</code>
+
=== Staging ===
is installed. In these cases the <code>%{name}-tests</code> subpackage should depend on
+
 
the <code>%{name}</code> or other subpackages that are being tested.
+
The <code>%{name}-test</code> subpackage should <code>Requires:</code> all other packages
 +
that the testsuite executable needs in order to run. This includes libraries or frameworks,
 +
or subsystems like <code>libvirt</code>.
 +
 
 +
Some integration tests may choose to test in-situ, on the system on which the testsuite
 +
is installed. In these cases the <code>%{name}-tests</code> subpackage should directly
 +
depend on the package being tested.
  
 
More rigourous integration tests test an integrated system from the outside. It is the
 
More rigourous integration tests test an integrated system from the outside. It is the
Line 55: Line 93:
 
linch-pin, etc.
 
linch-pin, etc.
  
== Invocation ==
 
  
TODO: Expand this section.
+
=== Discovery ===
 +
 
 +
A <code>check</code> file in a dist-git repo should list the various names of
 +
packages that should be installed in order to run the integration tests for that
 +
package. If this file is empty then the list of packages will default to
 +
<code>%{name}-tests</code>.
 +
 
 +
The format of this file has not yet been defined, but a simple text file similar
 +
to sources listing package N[VR]'s may suffice.
 +
 
 +
== Scope ==
 +
 
 +
This change requires no changes to Fedora infrastructure itself. It is limited
 +
to changes in dist-git repos. However certain key infrastructure changes could
 +
mitigate usability or side-effects of this change.
 +
 
 +
== Benefit to Fedora ==
 +
 
 +
Developers benefit by having a consistent target for how to describe tests,
 +
while also being able to execute them locally while debugging issues or
 +
iterating on tests.
 +
 
 +
By packaging, staging and invoking tests consistently in Fedora we create an
 +
create an eco-system for the tests that allows varied testsuite frameworks as
 +
well as CI system infrastructure to interoperate. The integration tests
 +
outlast the implementation details of either the frameworks they're written
 +
in or the CI systems running them.
 +
 
 +
== User Experience ==
 +
 
 +
Users benefit by having tests that they can reproduce on their own systems.
 +
They can install the similar to how they consume <code>%{name}-doc</code>
 +
or <code>%{name}-debuginfo</code> subpackages today.
  
* Any executable file directly under /usr/tests/ is a test suite to be executed
+
We may choose to avoid having such packages available in the standard repositories.
* Zero exit code from the executable indicates testsuite success, a non-zero exit code indicates test suite failure
+
We may choose to only have them in <code>updates-testing</code> or an
* The stdout/stderr of a test suite is the test log
+
arrangement similar to <code>debuginfo</code>. These choices will require some
* Additional output artifacts are placed in /run/tests/
+
markup and/or change to infrastructure.
* The testsuite is invoked as root, and may drop privileges as desired
 
  
== Discovery ==
+
== Upgrade/compatibility impact ==
  
In a dist-git repo the list of integration tests that should be invoked on a given package
+
Although there may already be subpackages that are named
are placed in a file called <code>tests</code> in that dist-git repo. The format of this
+
<code>%{name}-tests</code> this is merely a convention, and such packages will
file has not yet been defined, but a simple text file similar to sources listing the
+
not affect the behavior of this proposal.
package names that include integration tests to use when validating the dist-git repo.
 
  
== Examples ==  
+
== Examples ==
  
 
TODO: Build out this section
 
TODO: Build out this section
Line 78: Line 145:
 
== Notes ==
 
== Notes ==
  
* ...
+
* ...
* ...
+
* ...

Revision as of 14:06, 23 March 2017

Standard Dicsovery, Packaging, Invocation of Integration Tests

Warning.png
This is incomplete
This file is incomplete.
Warning.png
This is a proposal
Feedback is more than welcome. There's a discussion tab above.

Summary

What follows is a standard way to discover, package and invoke integration tests for a package stored in a Fedora dist-git repo.

Owner

  • Name: Stef Walter
  • Email: stfw@fedoraproject.org
  • Name: Looking for a co-owner

Detailed Description

Many Fedora packages have unit tests. These tests are typically run during a %check RPM build step. These tests run in the build root in which the package is built and may involve the BuildRequire dependencies.

On the other hand, integration testing happens against a composed system. Several upstream projects have integration tests, but these integration tests have typically not been used or packaged in Fedora. Both Fedora QA and the Atomic Host team would like to create more integration tests, stored in dist-git, and use a CI system to run them on Fedora packages.

What follows is a standard way to discover, package and invoke integration tests for a package stored in a Fedora dist-git repo.

It is important that the CI system which runs the test is independent from the layout of the test themselves.

Goals:

  • Implementation details of the integration testsuite or its framework should not leak into or rely on the CI system invoking the test.
  • CI system implementation details should not leak into the testsuite or its metadata.
  • It should be possible to change the CI system that runs a testsuite.
  • It should be possible for a dist-git packager to run a testsuite locally.

What follows is a standard way to discover, package and invoke integration tests for a package stored in a Fedora dist-git repo.

The integration tests are packaged and delivered through Fedora as %{name}-tests subpackages of the package they are associated with.

Packaging

Each dist-git repo that has integration tests should package those tests in a %{name}-tests subpackage. This is similar to the %{name}-debuginfo or %{name}-docs subpackages we have today.

The spec file for a dist-git repo may include upstream integration tests into its %{name}-tests subpackage. The spec file may also include tests directly from files in tests/ subdirectory of the dist-git repo itself.

Invocation

To invoke the testsuite, the subpackage is installed. Each integration testsuite that is included in installs an executable in the path /usr/tests/.

To invoke the integration test suites, one would:

  1. Place the artifact being tested, usually RPM files, in /var/cache/tests/
  2. Execute all executable files in /usr/tests/ one at a time.
    1. The testsuite is invoked with a working directory of /
    2. The testsuite is invoked as root, and may drop privileges as desired
    3. Treat the stdout/stderr of each testsuite process as the testsuite result log.
    4. Examine the exit code of each testsuite process. Zero exit code is success, non-zero is failure.
  3. Gather any additional result artifacts like screenshots which test suites place in a /var/log/tests/ directory

Multiple subpackages may be installed as long as their dependencies do not conflict.

Staging

The %{name}-test subpackage should Requires: all other packages that the testsuite executable needs in order to run. This includes libraries or frameworks, or subsystems like libvirt.

Some integration tests may choose to test in-situ, on the system on which the testsuite is installed. In these cases the %{name}-tests subpackage should directly depend on the package being tested.

More rigourous integration tests test an integrated system from the outside. It is the responsibility of the %{name}-tests subpackages to provision virtual machines or containers necessary to do such testing. In almost all cases this will happen by way of a provisioning framework such as Avocado, Ansible, Module Testing Framework, linch-pin, etc.


Discovery

A check file in a dist-git repo should list the various names of packages that should be installed in order to run the integration tests for that package. If this file is empty then the list of packages will default to %{name}-tests.

The format of this file has not yet been defined, but a simple text file similar to sources listing package N[VR]'s may suffice.

Scope

This change requires no changes to Fedora infrastructure itself. It is limited to changes in dist-git repos. However certain key infrastructure changes could mitigate usability or side-effects of this change.

Benefit to Fedora

Developers benefit by having a consistent target for how to describe tests, while also being able to execute them locally while debugging issues or iterating on tests.

By packaging, staging and invoking tests consistently in Fedora we create an create an eco-system for the tests that allows varied testsuite frameworks as well as CI system infrastructure to interoperate. The integration tests outlast the implementation details of either the frameworks they're written in or the CI systems running them.

User Experience

Users benefit by having tests that they can reproduce on their own systems. They can install the similar to how they consume %{name}-doc or %{name}-debuginfo subpackages today.

We may choose to avoid having such packages available in the standard repositories. We may choose to only have them in updates-testing or an arrangement similar to debuginfo. These choices will require some markup and/or change to infrastructure.

Upgrade/compatibility impact

Although there may already be subpackages that are named %{name}-tests this is merely a convention, and such packages will not affect the behavior of this proposal.

Examples

TODO: Build out this section

Notes

  • ...
  • ...