From Fedora Project Wiki
(add comparison with autopkgtest)
("test runner" "test environment")
 
(61 intermediate revisions by 23 users not shown)
Line 1: Line 1:
= Standard Discovery, Packaging, Invocation of Integration Tests =
+
= Standard Discovery, Staging and Invocation of Integration Tests =
 
 
{{admon/warning|This is incomplete|This file is incomplete.}}
 
 
 
{{admon/warning|This is a proposal|Feedback is more than welcome.
 
There's a ''discussion'' tab above.}}
 
  
 
== Summary ==
 
== Summary ==
  
What follows is a standard way to discover, package and invoke integration
+
Let's define a clear delineation of between a ''test suite'' (including framework) and the CI system that is running the test suite. This is the standard interface.
tests for a package stored in a Fedora dist-git repo.
 
  
Many Fedora packages have unit tests. These tests are typically run during a <code>%check</code> RPM build step and run in a build root. On the other hand, integration testing should happen against a composed system. Upstream projects have integration tests, both Fedora QA and the Atomic Host team would like to create more integration tests, Red Hat would like to bring integration tests upstream.
+
[[File:Invoking-tests-standard-interface.png|800px]]
  
'''Goals:'''
+
What follows is a standard way to discover, package and invoke integration tests for a package stored in a Fedora dist-git repo.
  
* A standard way to to discover, stage and invoke integration tests.
+
Many Fedora packages have unit tests. These tests are typically run during a <code>%check</code> RPM build step and run in a build root. On the other hand, integration testing should happen against a composed system. Upstream projects have integration tests, both Fedora QA and the Atomic Host team would like to create more integration tests, Red Hat would like to bring integration tests upstream.
* Implementation details of the test suite or its framework should not leak into or rely on the CI system invoking it.
 
* CI system implementation details should not leak into the test suite or its metadata.
 
* It should be possible to change the CI system that runs a test suite.
 
* It should be possible for a dist-git packager to run a test suite locally.
 
* It must be possible to package upstream tests.
 
* Both ''in-situ'' tests, and more rigorous ''outside-in'' tests must be possible.
 
  
 
== Owner ==
 
== Owner ==
Line 27: Line 15:
 
* Name: [[User:Stefw| Stef Walter]]
 
* Name: [[User:Stefw| Stef Walter]]
 
* Email: stefw@fedoraproject.org
 
* Email: stefw@fedoraproject.org
* Name: [[User:Pitti| Martin Pitt]]
+
 
* Email: martin@piware.de
+
* Name: [[User:pingou|Pierre-Yves Chibon]]
<!-- * Release notes owner: To be assigned by docs team [[User:FASAccountName| Release notes owner name]] <email address> -->
+
* Email: pingou@fedoraproject.org
 +
 
 +
* Name: [[User:astepano|Andrei Stepanov]]
 +
 
 +
* Name: [[User:sturivny|Serhii Turivnyi]]
 +
* Email: sturivny@fedoraproject.org
  
 
== Terminology ==
 
== Terminology ==
  
* '''Test Subject''': The items that are to be tested. Typically this is a set of RPMs (updating a package to a new version), a container, or a distribution install ISO/VM image.
+
* '''Test Subject''': The items that are to be tested.  
* '''Test''': A callable/runnable piece of code and corresponding test data and mocks which exercises and evaluates a ''subject''.
+
** Examples: RPMs, OCI image, ISO, QCow2, Module repository ...
* '''Test Suite''': The collection of all tests that apply to a ''subject''. It's common to split the testing of different aspects into different tests for easier result evaluation, code maintenance, or parallelization.
+
* '''Test''': A callable/runnable piece of code and corresponding test data and mocks which exercises and evaluates a ''test subject''.
 +
** '''Test environment''': environment where actual test run takes place. Test has direct impact on test environment.
 +
* '''Test Suite''': The collection of all tests that apply to a ''test subject''.  
 
* '''Test Framework''': A library or component that the ''test suite'' and ''tests'' use to accomplish their job.
 
* '''Test Framework''': A library or component that the ''test suite'' and ''tests'' use to accomplish their job.
** Examples: [https://avocado-framework.github.io/ Avocado], [https://wiki.gnome.org/Initiatives/GnomeGoals/InstalledTests GNOME Installed Tests], [https://pagure.io/modularity-testing-framework/ Modularity Testing Framework], [https://github.com/projectatomic/atomic-host-tests Ansible tests in Atomic Host], [https://tunir.readthedocs.io/en/latest/ Tunir tests], docker test images, ...
+
** Examples: [https://avocado-framework.github.io/ Avocado], [https://wiki.gnome.org/Initiatives/GnomeGoals/InstalledTests GNOME Installed Tests], [https://github.com/fedora-modularity/meta-test-family/ Meta Test Family], [https://github.com/projectatomic/atomic-host-tests Ansible tests in Atomic Host], [https://tunir.readthedocs.io/en/latest/ Tunir tests], docker test images, ...
* '''Test Result''': A boolean output of a ''test'' which decides whether the test passed. This is being used for automatic result processing (gating in Continuous Integration).
+
* '''Test Result''': A boolean pass/fail output of a ''test suite''.  
* '''Test Artifact''': Any additional output of the ''test'' such as the stdout/err output, log files, screen shots, core dumps, or TAP/Junit/subunit streams. These are mostly for human consumption (evaluating test failures by developers), but in case of machine readable files like JUnit they can also be used in result browsers for presenting test results.
+
** ''Test results'' are for consumption by automated aspects of a ''testing systems''.
* '''Testing System''': A continuous integration or other testing system that would like to discover, stage and invoke tests for a test ''subject''.
+
* '''Test Artifact''': Any additional output of the test suite such as the stdout/stderr output, log files, screenshots, core dumps, or TAP/Junit/subunit streams.  
** Examples: [https://jenkins.io/ Jenkins], [https://taskotron.fedoraproject.org/ Taskotron], [https://docs.openstack.org/infra/zuul/ ZUUL], [https://ci.centos.org/ CentOS CI], Red Hat CI, [https://travis-ci.org/ Travis], [https://semaphoreci.com/ Semaphore], [https://developers.openshift.com/managing-your-applications/continuous-integration.html Openshift CI/CD], [https://wiki.ubuntu.com/ProposedMigration/AutopkgtestInfrastructure Ubuntu CI], ...
+
** ''Test artifacts'' are for consumption by humans, archival or big data analysis.
 +
* '''Testing System''': A CI or other ''testing system'' that would like to discover, stage and invoke tests for a ''test subject''.
 +
** Examples: [https://jenkins.io/ Jenkins], [https://taskotron.fedoraproject.org/ Taskotron], [https://docs.openstack.org/infra/zuul/ ZUUL], [https://ci.centos.org/ CentOS CI], [https://github.com/projectatomic/papr Papr], [https://travis-ci.org/ Travis], [https://semaphoreci.com/ Semaphore], [https://developers.openshift.com/managing-your-applications/continuous-integration.html Openshift CI/CD], [https://wiki.ubuntu.com/ProposedMigration/AutopkgtestInfrastructure Ubuntu CI], ...
 +
** Testing system uses '''test runner''' as a place for running tests. Testing system delegates test-running to test runner. Test runner examples: local machine, VM, ...
  
== Detailed Description ==
+
== Responsibilities ==
  
This standard interface describes how to discover, stage and invoke tests. It is important to cleanly separate implementation details of the ''testing system'' from the ''test suite'' and its framework. It is also important to allow packagers to locally and manually invoke a ''test suite''.
+
The '''testing system''' is responsible to:
 +
* Build or otherwise acquire the ''test subject'', such as a package, container image, tree …
 +
* Decide which ''test suite'' to run, often by using the standard interface to discover appropriate ''tests'' for the dist-git repo that a test subject originated in.
 +
* Schedule, provision or orchestrate a job to run the ''test suite'' on appropriate compute, storage, ...
 +
* Stage the ''test suite'' as described by the ''standard interface''.
 +
* Invoke the ''test suite'' as described by the ''standard interface''.
 +
* Gather the ''test results'' and ''test artifacts'' as described by the ''standard interface''.
 +
* Announce and relay the ''test results'' and ''test artifacts'' for gating, archival ...
  
'''Testing System''' responsibilities:
+
The '''standard interface''' describes how to:
* Builds or otherwise acquires a ''test subject'', such as a package, container image, OSTree ...
+
* Discover a ''test suite'' for a given dist-git repo.
* Schedules and orchestrates a test job on appropriate compute, storage, etc.
+
* Uniquely identify a ''test suite''.
* Stages the ''test suite'' (using standard interface)
+
* Stage a ''test suite'' and its dependencies such as ''test frameworks''.
* Invokes the ''test suite'' (using standard interface)
+
* Provide the ''test subject'' to the ''test suite''.
* Gathers the ''test results'' and ''test artifacts'' (using standard interface)
+
* Invoke a ''test suite'' in a consistent way.
* Announces and relays those results for gating, archival, etc.
+
* Gather ''test results'' and ''test artifacts'' from the invoked ''test suite''.
  
[[File:Invoking-tests-standard-interface.png|800px]]
+
The '''test suite''' is responsible to:
 
+
* Declare its dependencies such as a ''test framework'' via the ''standard interface''.
'''Testsuite''' responsibilities:
+
* Execute the ''test framework'' as necessary.
* Includes the ''test framework' (declared as a dependency for staging)
+
* Provision (usually locally) any containers or virtual machines necessary for testing the ''test subject''.
* May provision necessary container or VM based on the ''test subject''
+
* Provide ''test results'' and ''test subjects'' back according to the standard
* Provide ''test results'' and ''test artifacts'' (as described by standard interface)
 
  
 
The format of the textual logs and ''test artifacts'' that come out of a test suite is not prescribed by this document. Nor is it envisioned to be standardized across all possible ''test suites''.
 
The format of the textual logs and ''test artifacts'' that come out of a test suite is not prescribed by this document. Nor is it envisioned to be standardized across all possible ''test suites''.
  
=== Packaging ===
+
== Requirements ==
 
 
The integration tests are packaged and delivered through Fedora as packages.
 
 
 
Each dist-git repo that has integration tests should package those tests in one or more subpackages
 
like <code>%{name}-tests</code>. This is similar to the
 
<code>%{name}-debuginfo</code> or <code>%{name}-docs</code> subpackages we have
 
today.
 
 
 
The spec file for a dist-git repo may install upstream integration tests as files in
 
its <code>%{name}-tests</code> package. The spec file may also include tests
 
directly from files in <code>tests/</code> subdirectory of the dist-git repo itself.
 
 
 
The tests package should use <code>Requires:</code> to require any other package, testing framework, or dependency necessary to run the tests. In ''in-situ'' testing cases, the tests package will directly <code>Requires:</code> the package of the ''test subject''.
 
 
 
=== Invocation ===
 
 
 
To invoke the test suite, the test package that contains it is installed. Each test of the suite
 
installs an executable in the path <code>/usr/tests/</code>''sourcepackage''<code>/</code> (this will avoid name collisions between packages).
 
 
 
To invoke the test suite, one would:
 
 
 
# Create a temporary directory, referred to as: <code>$TESTDIR</code>
 
# Place the ''test subject(s)'' being tested in <code>$TESTDIR/subjects/</code>
 
# Execute all executable files in <code>/usr/tests/*/</code> directories one at a time.
 
## Each executable test is invoked with a working directory of <code>$TESTDIR</code>
 
## Each executable test is invoked as root, and may drop privileges as desired.
 
## Treat the stdout/stderr of the test process as the test log. This is a standard ''test artifact'' and written to <code>$TESTDIR/artifacts/testname.log</code>.
 
## Examine the exit code of each test process. Zero exit code is a successful ''test result'', non-zero is failure.
 
# Tests can put any additional ''test artifacts'' like screenshots into <code>$TESTDIR/artifacts/</code>.
 
 
 
This ensures that tests can be run on a production system without accidentally clobbering permanent directories,
 
don't require root privileges (simplifies test development), and that CI systems have one unique place from where
 
to collect artifacts. It also avoids collecting temporary files such as downloaded container or VM images as artifacts,
 
as these would usually get stored for a longer time period.
 
 
 
These steps would usually be done through a standard test driver tool (particularly for sensible stdout/stderr teeing and log capturing), but its usage is not mandatory for developing and calling tests manually.
 
 
 
=== Staging ===
 
 
 
The <code>%{name}-test</code> package should <code>Requires:</code> all other packages
 
that the testsuite executable needs in order to run. This includes libraries or frameworks,
 
or subsystems like <code>libvirt</code>.
 
  
Some integration tests may choose to test ''in-situ'', on the system on which the test suite
+
* The ''test suite'' and ''test framework'' SHOULD NOT leak its implementation details into the testing system, other than via the ''standard interface''.
is installed. In these cases the <code>%{name}-tests</code> package should directly
+
* The ''test suite'' and ''test framework'' SHOULD NOT rely on the behavior of the testing system other than the ''standard interface''.
depend on the package being tested.
+
* The ''standard interface'' MUST enable a dist-git packager to run a ''test suite'' locally.
 
+
** ''Test suites'' or ''test frameworks'' MAY call out to the network for certain tasks.
More rigorous integration tests are ''outside-in''. They test an integrated system without affecting its contents. It is the responsibility of the <code>%{name}-tests</code> packages to provision virtual machines or containers necessary to do such testing. In almost all cases this will happen by way of a provisioning framework such as [http://avocado-framework.readthedocs.org/ Avocado], [https://www.ansible.com/ Ansible], [https://pagure.io/modularity-testing-framework/ Module Testing Framework], [https://linch-pin.readthedocs.io/en/latest/ linch-pin], etc.
+
* It MUST be possible to stage an upstream ''test suite'' using the ''standard interface''.
 
+
* Both ''in-situ tests'', and more rigorous ''outside-in tests'' MUST be possible with the ''standard interface''.
Multiple tests packages may be installed as long as their dependencies do not conflict.
+
** For ''in-situ tests'' the ''test suite'' is in the same file system tree and process space as the ''test subject''.
 
+
** For ''outside-in tests'' the ''test suite'' is outside of the file system tree and process space of the ''test subject''.
=== Discovery ===
+
* The ''test suite'' and ''test framework'' SHOULD be able to provision containers and virtual machines necessary for its testing without requesting them from the ''testing system''.
 
+
* The ''standard interface'' SHOULD describe how to uniquely identify a ''test suite'',
A testing system needs to be able to efficiently answer the question "does this subject have any tests packages, and if so, what are their names". This should be automatically discoverable to the extent possible.
 
 
 
For any RPM ''test subject'' this process requires no additional metadata and can be fully automatic:
 
 
 
* It is possible to map a RPM to its SRPM source package (<code>&lt;rpm:sourcerpm&gt;</code> in the package index <code>*-primary.xml.gz</code>).
 
* One can map an SRPM to all the RPMs that it builds (from the same index), and using the <code>*-filelists.xml.gz</code> index one can mechanically tell which of the RPMs are of this test package kind described here.
 
 
 
'''TODO''': For other types of test subject cases such as docker images or distribution ISO files this discovery still needs to be discussed.
 
* E. g. a <code>Dockerfile</code> might grow a reference to a test package RPM, or at least initially there is a manually maintained map of subject to test package in the testing system.
 
 
 
== Scope ==
 
 
 
This change requires no initial changes to Fedora infrastructure itself. The change only affects contents spec files in  dist-git repos.
 
 
 
'''TODO:''' However certain key infrastructure changes could mitigate usability or side-effects of this change. In particular, once this grows beyond the experimental phase, these test packages need to be put into a separate archive, similar to <code>-debuginfo</code>.
 
 
 
* How much effort is that to set up?
 
* Does this require any additional tags, keywords, or other explicit declaration in the spec file, other than "this RPM ships something in <code>/usr/tests/*</code>"?
 
  
 
== Benefit to Fedora ==
 
== Benefit to Fedora ==
Line 141: Line 85:
 
Developers benefit by having a consistent target for how to describe tests, while also being able to execute them locally while debugging issues or iterating on tests.
 
Developers benefit by having a consistent target for how to describe tests, while also being able to execute them locally while debugging issues or iterating on tests.
  
By packaging, staging and invoking tests consistently in Fedora we create an eco-system for the tests that allows varied test frameworks as well as CI system infrastructure to interoperate. The integration tests outlast the implementation details of either the frameworks they're written in or the CI systems running them.
+
By staging and invoking tests consistently in Fedora we create an eco-system for the tests that allows varied test frameworks as well as CI system infrastructure to interoperate. The integration tests outlast the implementation details of either the frameworks they're written in or the CI systems running them.
  
== User Experience ==
+
== Detailed Description ==
  
A standard way to package tests benefits Fedora stability, and makes Fedora better for users.
+
This standard interface describes how to discover, stage and invoke tests. It is important to cleanly separate implementation details of the ''testing system'' from the ''test suite'' and its framework. It is also important to allow Fedora packagers to locally and manually invoke a ''test suite''.
  
Users could also benefit by having tests that they can reproduce on their own systems. They could install the similar to how they consume <code>%{name}-doc</code> or <code>%{name}-debuginfo</code> subpackages today.
+
'''First see the [https://fedoraproject.org/wiki/Changes/InvokingTests#Terminology Terminogy] division of [https://fedoraproject.org/wiki/Changes/InvokingTests#Responsibilities Responsibilities] and [https://fedoraproject.org/wiki/Changes/InvokingTests#Requirements Requirements]'''
  
We may choose to avoid having such packages available in the standard repositories. We may choose to only have them in <code>updates-testing</code> or an arrangement similar to <code>debuginfo</code>. These choices will require some
+
=== Staging ===
markup and/or change to infrastructure.
 
 
 
== Upgrade/compatibility impact ==
 
  
Although there may already be packages that are named <code>%{name}-tests</code> this is merely a convention, and such packages will not affect the behavior of this proposal.
+
Tests files will be added into the <code>tests/</code> folder of a dist-git repository branch. The structure of the files and folders is left to the liberty of the packagers but there are one or more playbooks in the <code>tests/</code> folder that can be invoked to run the test suites.
  
== Comparison with Debian's autopkgtest ==
+
# The ''testing system'' SHOULD stage the tests on target (eg: Fedora) operating system appropriate for the branch name of the dist-git repository containing the tests.
 
+
# The ''testing system'' SHOULD stage a clean a system for each set of tests it runs.
Debian/Ubuntu have used CI with packaged tests (called "autopkgtests") for many years, with
+
# The ''testing system'' MUST stage the following packages:
[https://ci.debian.net/status/ over 7.000 tests]. These are good candidates or at least bases for taking into Fedora
+
## <code>ansible python2-dnf libselinux-python standard-test-roles</code>
packages. This compares the structure of autopkgtest with this proposal to learn from autopkgtest's experiences and take
+
# The ''testing system'' MUST clone the dist-git repository for the test, and checks out the appropriate branch.
what works, and justifies the differences. See the
+
# The contents of <code>/etc/yum.repos.d</code> on the staged system SHOULD be replaced with repository information that reflects the known good Fedora packages corresponding to the branch of the dist-git repository.
[https://people.debian.org/~mpitt/autopkgtest/README.package-tests.html format definition] for details.
+
## The ''testing system'' MAY use multiple repositories, including ''updates'' or ''updates-testing'' to ensure this.
 
 
=== Packaging ===
 
 
 
'''Similarities:''' Both specifications use an existing test metatada format (RPM spec files with <code>Requires:</code> here, Debian RFC822 control files with <code>Depends:</code>in autopkgtest).
 
 
 
'''Differences:'''
 
* This specification requires packaging tests as binary RPM packages, whereas autopkgtest opted for keeping the test in the source package (equivalent of dist-git) only. The latter avoids the overhead of packaging the tests and having to create a separate archive for them.
 
* As autopkgtest uses a separate control file (<code>debian/tests/control</code> instead of <code>debian/control</code> which describes the binary packages), it offers a much richer set of test metadata which cannot be expressed with <code>debian/control</code> or RPM spec files.
 
  
 
=== Invocation ===
 
=== Invocation ===
  
'''Simimlarities''': The test interface is very similar: In both specifications, a test is an executable
+
The testing system MUST run each playbook matching the glob <code>tests/tests*.yml</code> in the dist-git repo. Each of these files constitutes a
(of any script or compiled language), the exit code is the primary indicator of pass/fail, the executable's stdout/err
+
test suite. Each test suite is invoked independently by the testing system as follows.
is a standard test artifact ("test log"), and tests can write additional artifacts into the
 
<code>$AUTOPKGTEST_ARTIFACTS</code> dir (like <code>./artifacts/</code> here).
 
  
'''Differences:'''
+
The ''test subjects'' are passed to the playbook and inventory as operating system environment and ansible environment. Often only one ''test subject'' is passed in. However multiple subjects may be concatenated together in a shell escaped string. The playbooks and/or inventory script split the string. The extensions as follows are used to determine the type of subject:
* By default, autopkgtest considers a test as failed if it produces anything on stderr, for catching unexpected new warnings. This can be disabled with adding <code>Restrictions: allow-stderr</code> to the test metadata. However, this turned out to be not overly useful, and tests which want to intercept warnings should better do that themselves.
 
* autopkgtest has no concept of passing test subjects to the test. Tests expect that their subjects are already available/installed, i. e. they get called in a testbed of the desired kind and state. It is the responsibility of the autopkgtest command line tool (the "test driver/executor") to install proposed new package(s) into the testbed (due to its origin of being primarily focussed on testing packages). For testing desktop/cloud images, upgrades, or other non-package subjects, it is instead the testing system's responsibility to produce the desired testbed and call <code>autopkgtest</code> on it. As the scope of this specification puts the staging into the hand of the test instead of the testing system, passing the test subject is a necessary consequence.
 
  
=== Staging ===
+
{|
 +
! Identifier !! Test subject
 +
|-
 +
| *.rpm    || Absolute path to an RPM file
 +
|-
 +
| *.repo    || Absolute repo filenames appropriate for <code>/etc/yum.repos.d</code>
 +
|-
 +
| *.qcow2, *.qcow2c || Absolute path to one virtual machine disk image bootable with cloud-init
 +
|-
 +
| *.oci    || Absolute path of one OCI container image filesystem bundle
 +
|-
 +
| docker:*  || Fully qualified path to a docker image in a registry
 +
|-
 +
| ...      || Other ''test subject'' identifiers may be added later.
 +
|}
  
'''Similarities:''' Both specifications use standard dpkg/rpm package dependencies (<code>Depends:</code> for dpkg, <code>Requires:</code> for rpm) to pull in test dependencies, and both can opt into doing their own provisioning of containers/VMs etc. for doing outside-in tests instead of in-situ. However, of all the ~ 7.000 autopkgtests, only a small handful is actually doing that (known cases are systemd and open-iscsi), as the vast majority of package/upgrade/image tests can (because it's sufficient) and should (because it's magnitudes faster) be run in-situ.
 
  
'''Differences:''' Here the test itself is responsible for installing the test subjects, while in autopkgtest it's the testing system's responsibility (see above).
+
Various ''tests'' in a playbook constitute a ''test suite''. Some parts of these ''test suites'' will run in certain contexts, against certain deliverable artifacts. Certain tests will run against Atomic Host deliverables, while others will not. Certain tests will run against Docker deliverables while others will not. This is related to, but does not exactly overlap with the ''test subject'' identifiers above. Ansible tags are used to denote these contexts.
  
=== Discovery ===
+
{|
 +
! Tag      !! Test context
 +
|-
 +
| atomic    || Atomic Host
 +
|-
 +
| container || A Docker or OCI container
 +
|-
 +
| classic  || Tested against a classic installed YUM/DNF installed system.
 +
|-
 +
| ...      || Other ''test subject'' identifiers may be added later.
 +
|}
  
'''Similarities:''' The idea is the same in both specifications. Here, as soon as there is a binary package that ships <code>/usr/tests/*</code> it can be discovered through file lists. In autopkgtest, as soon as there is a <code>debian/tests/control</code>, the source package index entry will automatically get a <code>Testsuite: autopkgtest</code> tag. So in both cases the developer does not need to explicitly do anything other than adding the tests.
 
  
'''Differences:''' None concerning the interface, just technical implementation details due to how rpm/dpkg work.
+
To invoke the tests, the ''testing system'' must perform the following tasks for each ''test suite'' playbook:
  
== Examples ==
+
# MUST execute the playbook with the following operating system environment variables:
 +
## <code>TEST_SUBJECTS</code>: The ''test subjects'' string as described above
 +
## <code>TEST_ARTIFACTS</code>: The full path of an empty folder for ''test artifacts''
 +
# MUST execute the playbook with the following Ansible variables.
 +
## <code>subjects</code>: The ''test subjects'' string as described above
 +
## <code>artifacts</code>: The full path of an empty folder for ''test artifacts''
 +
# SHOULD execute the playbook with all Ansible tags best represent the intended ''test context''.
 +
## The choice of ''test context'' tags is related to the ''test subject'' being tested
 +
# MUST execute Ansible with inventory set to the full path of the file or directory <code>tests/inventory</code> if it exists.
 +
## If the <code>tests/inventory</code> file doesn't exist, then the following inventory SHOULD be used as a default:<br> <code>/usr/share/ansible/inventory</code>
 +
# MUST execute the playbook as root.
 +
# MUST examine the exit code of the playbook. A zero exit code is successful ''test result'', non-zero is failure.
 +
# MUST treat the file <code>test.log</code> in the <code>artifacts</code> folder as the main readable output of the test.
 +
# SHOULD place the textual stdout/stderr of the <code>ansible-playbook</code> command in the <code>ansible.log</code> file in the <code>artifacts</code> folder.
 +
# SHOULD treat the contents of the <code>artifacts</code> folder as the ''test artifacts''.
  
What follows are examples of writing and/or packaging existing tests to this standard.
+
Each ''test suite'' playbook or ''test framework'' contained therein:
  
There is a mock ''test system''' which is a simple shell script: [http://piware.de/tmp/run-installed-test run-installed-test]. It runs all <code>/usr/tests/*</code>, can pass arbitrary subjects to them, and report/capture the results/logs. This is purely to study what a CI system would do and whether the standard interface works.
+
# SHOULD drop privileges appropriately if the ''test suite'' should be run as non-root.
 +
# MUST install any requirements of its ''test suite'' or ''test framework'' and MUST fail if this is not possible.
 +
# MUST provision the ''test subject'' listed in the <code>subjects</code> variable appropriately for its playbook name (described above) and MUST fail if this is not possible.
 +
# MUST place the main readable output of the ''test suite'' into a <code>test.log</code> file in the <code>artifacts</code> variable folder. This MUST happen even if some of the test suites fail.
 +
# SHOULD place additional ''test artifacts'' in the folder defined in the <code>artifacts</code> variable.
 +
# MUST return a zero exit code of the playbook if the ''test result'' is a pass, or a non-zero exit code if the ''test result'' is a fail.
  
=== Example: Simple in-situ test ===
+
If an inventory file or script exists, it:
  
Add simple downstream integration test for gzip:
+
# MUST describe where to invoke the playbook and how to connect to that target.
 +
# SHOULD launch or install any supported <code>$TEST_SUBJECTS</code> so that the playbook can be invoked against them.
 +
# SHOULD put relevant logs in the <code>$TEST_ARTIFACTS</code> directory.
  
* Package: '''gzip-tests'''
+
=== Discovery ===
* dist-git: https://github.com/martinpitt/fedora-gzip-test
 
* Reference: https://patches.ubuntu.com/g/gzip/ (taken verbatim from Ubuntu's autopkgtest)
 
  
With this you can install test RPM from above gzip repo:
+
Test discovery is done via dist-git. Both packages and modules may have tests in this format. To list which ''test context'' a given dist-git directory or playbook is relevant for, use a command like the following:
  
  $ sudo rpm -i results_gzip/1.8/2.fc27/gzip-tests-1.8-2.fc25.x86_64.rpm
+
<pre>
 +
# ansible-playbook --list-tags tests.yml
 +
</pre>
  
and run the gzip tests on the already installed package (as user) with
+
== Scope ==
  
  $ ~/run-installed-test
+
Since the tests are added in a sub-folder of the dist-git repo, there are no changes required to the Fedora infrastructure and will have no impact on the packagers' workflow and tooling.
  Subjects/artifacts directory: /tmp/test.vsR
 
  -----------------------------------------
 
  Running /usr/tests/gzip/test-simple
 
  -----------------------------------------
 
  ++ ls 'subjects/*.rpm'
 
  + echo Bla
 
  + cp bla.file bla.file.orig
 
  + gzip bla.file
 
  + gunzip bla.file.gz
 
  + cmp bla.file bla.file.orig
 
  + rm bla.file bla.file.orig
 
  PASS: /usr/tests/gzip/test-simple
 
  
  $ ls -l /tmp/test.vsR/artifacts/
+
Only the testing system will need to be taught to install the requirements and run the playbooks.
  -rw-r--r-- 1 martin martin 156 Mar 28 16:49 test-simple.log
 
  
or run them as root (as officially specified) with a subject (locally built gzip RPM):
+
== User Experience ==
  
  $ sudo ~/run-installed-test results_gzip/1.8/2.fc27/gzip-1.8-2.fc25.x86_64.rpm
+
A standard way to package, store and run tests benefits Fedora stability, and makes Fedora better for users.
  Installing subject results_gzip/1.8/2.fc27/gzip-1.8-2.fc25.x86_64.rpm
 
  Subjects/artifacts directory: /tmp/test.Cck
 
  -----------------------------------------
 
  Running /usr/tests/gzip/test-simple
 
  -----------------------------------------
 
  ++ ls subjects/gzip-1.8-2.fc25.x86_64.rpm
 
  + '[' -w / ']'
 
  + rpm --verbose --force -U subjects/gzip-1.8-2.fc25.x86_64.rpm
 
  Preparing packages...
 
  gzip-1.8-2.fc25.x86_64
 
  + echo Bla
 
  + cp bla.file bla.file.orig
 
  + gzip bla.file
 
  + gunzip bla.file.gz
 
  + cmp bla.file bla.file.orig
 
  + rm bla.file bla.file.orig
 
  PASS: /usr/tests/gzip/test-simple
 
  
=== Example: GNOME style "Installed Tests" ===
+
* This structure makes it easy to run locally thus potentially reproducing an error triggered on the test system.
 +
* Ansible is being more and more popular, thus making it easier for people to contribute new tests
 +
* Used by a lot of sys-admin, ansible could help sys-admin to bring test-cases to the packagers and developers about situation where something failed for them.
  
Add downstream integration test running in gnome installed tests.
+
== Upgrade/compatibility impact ==
 
 
* Package: '''glib2-tests'''
 
* dist-git: https://github.com/martinpitt/fedora-glib2-test
 
* Reference: https://wiki.gnome.org/Initiatives/GnomeGoals/InstalledTests
 
 
 
=== Example: Tests run in Docker Container ===
 
 
 
Add integration test running glib2 installed tests in a docker container. This is also an example of having two different tests packages being created by the same dist-git repo.
 
 
 
* Package: '''glib2-container-tests'''
 
* dist-git: https://github.com/martinpitt/fedora-glib2-test
 
* Based on Debian/Ubuntu's autopkgtest glib2 related script
 
 
 
=== Example: Modularity testing Framework ===
 
 
 
TODO: Port [https://pagure.io/modularity-testing-framework/blob/master/f/examples an example]
 
 
 
=== Example: Ansible with Atomic Host ===
 
 
 
TODO: Port [https://github.com/projectatomic/atomic-host-tests an existing test]
 
  
=== Example: Beakerlib based test ===
+
There are no real upgrade or compatibility impact. The tests will be branched per release as spec files are branched dist-git is now.
  
TODO: Port and shim a beakerlib test
+
== Documentation ==
  
=== Example: Cockpit upstream test ===
+
* [[CI|CI Landing page]]
 +
* [[CI/Tests|Documentation page]]
  
TODO: Include these tests in '''cockpit-tests'''
+
== Proposals and Evaluation ==
  
== Notes ==
+
During the selection process for a standard test invocation and and layout format for Fedora, [[Changes/InvokingTestsProposals|several proposals]] were examined.
  
* ...
+
[[Category:FedoraAtomicCi]]
* ...
+
[[Category:FedoraCi]]

Latest revision as of 16:20, 30 January 2018

Standard Discovery, Staging and Invocation of Integration Tests

Summary

Let's define a clear delineation of between a test suite (including framework) and the CI system that is running the test suite. This is the standard interface.

Invoking-tests-standard-interface.png

What follows is a standard way to discover, package and invoke integration tests for a package stored in a Fedora dist-git repo.

Many Fedora packages have unit tests. These tests are typically run during a %check RPM build step and run in a build root. On the other hand, integration testing should happen against a composed system. Upstream projects have integration tests, both Fedora QA and the Atomic Host team would like to create more integration tests, Red Hat would like to bring integration tests upstream.

Owner

Terminology

  • Test Subject: The items that are to be tested.
    • Examples: RPMs, OCI image, ISO, QCow2, Module repository ...
  • Test: A callable/runnable piece of code and corresponding test data and mocks which exercises and evaluates a test subject.
    • Test environment: environment where actual test run takes place. Test has direct impact on test environment.
  • Test Suite: The collection of all tests that apply to a test subject.
  • Test Framework: A library or component that the test suite and tests use to accomplish their job.
  • Test Result: A boolean pass/fail output of a test suite.
    • Test results are for consumption by automated aspects of a testing systems.
  • Test Artifact: Any additional output of the test suite such as the stdout/stderr output, log files, screenshots, core dumps, or TAP/Junit/subunit streams.
    • Test artifacts are for consumption by humans, archival or big data analysis.
  • Testing System: A CI or other testing system that would like to discover, stage and invoke tests for a test subject.

Responsibilities

The testing system is responsible to:

  • Build or otherwise acquire the test subject, such as a package, container image, tree …
  • Decide which test suite to run, often by using the standard interface to discover appropriate tests for the dist-git repo that a test subject originated in.
  • Schedule, provision or orchestrate a job to run the test suite on appropriate compute, storage, ...
  • Stage the test suite as described by the standard interface.
  • Invoke the test suite as described by the standard interface.
  • Gather the test results and test artifacts as described by the standard interface.
  • Announce and relay the test results and test artifacts for gating, archival ...

The standard interface describes how to:

  • Discover a test suite for a given dist-git repo.
  • Uniquely identify a test suite.
  • Stage a test suite and its dependencies such as test frameworks.
  • Provide the test subject to the test suite.
  • Invoke a test suite in a consistent way.
  • Gather test results and test artifacts from the invoked test suite.

The test suite is responsible to:

  • Declare its dependencies such as a test framework via the standard interface.
  • Execute the test framework as necessary.
  • Provision (usually locally) any containers or virtual machines necessary for testing the test subject.
  • Provide test results and test subjects back according to the standard

The format of the textual logs and test artifacts that come out of a test suite is not prescribed by this document. Nor is it envisioned to be standardized across all possible test suites.

Requirements

  • The test suite and test framework SHOULD NOT leak its implementation details into the testing system, other than via the standard interface.
  • The test suite and test framework SHOULD NOT rely on the behavior of the testing system other than the standard interface.
  • The standard interface MUST enable a dist-git packager to run a test suite locally.
    • Test suites or test frameworks MAY call out to the network for certain tasks.
  • It MUST be possible to stage an upstream test suite using the standard interface.
  • Both in-situ tests, and more rigorous outside-in tests MUST be possible with the standard interface.
    • For in-situ tests the test suite is in the same file system tree and process space as the test subject.
    • For outside-in tests the test suite is outside of the file system tree and process space of the test subject.
  • The test suite and test framework SHOULD be able to provision containers and virtual machines necessary for its testing without requesting them from the testing system.
  • The standard interface SHOULD describe how to uniquely identify a test suite,

Benefit to Fedora

Developers benefit by having a consistent target for how to describe tests, while also being able to execute them locally while debugging issues or iterating on tests.

By staging and invoking tests consistently in Fedora we create an eco-system for the tests that allows varied test frameworks as well as CI system infrastructure to interoperate. The integration tests outlast the implementation details of either the frameworks they're written in or the CI systems running them.

Detailed Description

This standard interface describes how to discover, stage and invoke tests. It is important to cleanly separate implementation details of the testing system from the test suite and its framework. It is also important to allow Fedora packagers to locally and manually invoke a test suite.

First see the Terminogy division of Responsibilities and Requirements

Staging

Tests files will be added into the tests/ folder of a dist-git repository branch. The structure of the files and folders is left to the liberty of the packagers but there are one or more playbooks in the tests/ folder that can be invoked to run the test suites.

  1. The testing system SHOULD stage the tests on target (eg: Fedora) operating system appropriate for the branch name of the dist-git repository containing the tests.
  2. The testing system SHOULD stage a clean a system for each set of tests it runs.
  3. The testing system MUST stage the following packages:
    1. ansible python2-dnf libselinux-python standard-test-roles
  4. The testing system MUST clone the dist-git repository for the test, and checks out the appropriate branch.
  5. The contents of /etc/yum.repos.d on the staged system SHOULD be replaced with repository information that reflects the known good Fedora packages corresponding to the branch of the dist-git repository.
    1. The testing system MAY use multiple repositories, including updates or updates-testing to ensure this.

Invocation

The testing system MUST run each playbook matching the glob tests/tests*.yml in the dist-git repo. Each of these files constitutes a test suite. Each test suite is invoked independently by the testing system as follows.

The test subjects are passed to the playbook and inventory as operating system environment and ansible environment. Often only one test subject is passed in. However multiple subjects may be concatenated together in a shell escaped string. The playbooks and/or inventory script split the string. The extensions as follows are used to determine the type of subject:

Identifier Test subject
*.rpm Absolute path to an RPM file
*.repo Absolute repo filenames appropriate for /etc/yum.repos.d
*.qcow2, *.qcow2c Absolute path to one virtual machine disk image bootable with cloud-init
*.oci Absolute path of one OCI container image filesystem bundle
docker:* Fully qualified path to a docker image in a registry
... Other test subject identifiers may be added later.


Various tests in a playbook constitute a test suite. Some parts of these test suites will run in certain contexts, against certain deliverable artifacts. Certain tests will run against Atomic Host deliverables, while others will not. Certain tests will run against Docker deliverables while others will not. This is related to, but does not exactly overlap with the test subject identifiers above. Ansible tags are used to denote these contexts.

Tag Test context
atomic Atomic Host
container A Docker or OCI container
classic Tested against a classic installed YUM/DNF installed system.
... Other test subject identifiers may be added later.


To invoke the tests, the testing system must perform the following tasks for each test suite playbook:

  1. MUST execute the playbook with the following operating system environment variables:
    1. TEST_SUBJECTS: The test subjects string as described above
    2. TEST_ARTIFACTS: The full path of an empty folder for test artifacts
  2. MUST execute the playbook with the following Ansible variables.
    1. subjects: The test subjects string as described above
    2. artifacts: The full path of an empty folder for test artifacts
  3. SHOULD execute the playbook with all Ansible tags best represent the intended test context.
    1. The choice of test context tags is related to the test subject being tested
  4. MUST execute Ansible with inventory set to the full path of the file or directory tests/inventory if it exists.
    1. If the tests/inventory file doesn't exist, then the following inventory SHOULD be used as a default:
      /usr/share/ansible/inventory
  5. MUST execute the playbook as root.
  6. MUST examine the exit code of the playbook. A zero exit code is successful test result, non-zero is failure.
  7. MUST treat the file test.log in the artifacts folder as the main readable output of the test.
  8. SHOULD place the textual stdout/stderr of the ansible-playbook command in the ansible.log file in the artifacts folder.
  9. SHOULD treat the contents of the artifacts folder as the test artifacts.

Each test suite playbook or test framework contained therein:

  1. SHOULD drop privileges appropriately if the test suite should be run as non-root.
  2. MUST install any requirements of its test suite or test framework and MUST fail if this is not possible.
  3. MUST provision the test subject listed in the subjects variable appropriately for its playbook name (described above) and MUST fail if this is not possible.
  4. MUST place the main readable output of the test suite into a test.log file in the artifacts variable folder. This MUST happen even if some of the test suites fail.
  5. SHOULD place additional test artifacts in the folder defined in the artifacts variable.
  6. MUST return a zero exit code of the playbook if the test result is a pass, or a non-zero exit code if the test result is a fail.

If an inventory file or script exists, it:

  1. MUST describe where to invoke the playbook and how to connect to that target.
  2. SHOULD launch or install any supported $TEST_SUBJECTS so that the playbook can be invoked against them.
  3. SHOULD put relevant logs in the $TEST_ARTIFACTS directory.

Discovery

Test discovery is done via dist-git. Both packages and modules may have tests in this format. To list which test context a given dist-git directory or playbook is relevant for, use a command like the following:

# ansible-playbook --list-tags tests.yml

Scope

Since the tests are added in a sub-folder of the dist-git repo, there are no changes required to the Fedora infrastructure and will have no impact on the packagers' workflow and tooling.

Only the testing system will need to be taught to install the requirements and run the playbooks.

User Experience

A standard way to package, store and run tests benefits Fedora stability, and makes Fedora better for users.

  • This structure makes it easy to run locally thus potentially reproducing an error triggered on the test system.
  • Ansible is being more and more popular, thus making it easier for people to contribute new tests
  • Used by a lot of sys-admin, ansible could help sys-admin to bring test-cases to the packagers and developers about situation where something failed for them.

Upgrade/compatibility impact

There are no real upgrade or compatibility impact. The tests will be branched per release as spec files are branched dist-git is now.

Documentation

Proposals and Evaluation

During the selection process for a standard test invocation and and layout format for Fedora, several proposals were examined.