From Fedora Project Wiki
(Voting: adding my vote)
("test runner" "test environment")
 
(27 intermediate revisions by 13 users not shown)
Line 1: Line 1:
= Standard Discovery, Packaging, Invocation of Integration Tests =
+
= Standard Discovery, Staging and Invocation of Integration Tests =
 
 
{{admon/warning|This Page is being reworked|This message will be removed when refactoring is complete}}
 
  
 
== Summary ==
 
== Summary ==
  
Lets define a clear delineation of between a ''test suite'' (including framework) and the CI system that is running the test suite. This is the standard interface.
+
Let's define a clear delineation of between a ''test suite'' (including framework) and the CI system that is running the test suite. This is the standard interface.
  
 
[[File:Invoking-tests-standard-interface.png|800px]]
 
[[File:Invoking-tests-standard-interface.png|800px]]
Line 11: Line 9:
 
What follows is a standard way to discover, package and invoke integration tests for a package stored in a Fedora dist-git repo.
 
What follows is a standard way to discover, package and invoke integration tests for a package stored in a Fedora dist-git repo.
  
Many Fedora packages have unit tests. These tests are typically run during a <code>%check</code> RPM build step and run in a build root. On the other hand, integration testing should happen against a composed system. Upstream projects have integration tests, both Fedora QA and the Atomic Host team would like to create more integration tests, Red Hat would like to bring integration tests upstream.  
+
Many Fedora packages have unit tests. These tests are typically run during a <code>%check</code> RPM build step and run in a build root. On the other hand, integration testing should happen against a composed system. Upstream projects have integration tests, both Fedora QA and the Atomic Host team would like to create more integration tests, Red Hat would like to bring integration tests upstream.
  
 
== Owner ==
 
== Owner ==
Line 21: Line 19:
 
* Email: pingou@fedoraproject.org
 
* Email: pingou@fedoraproject.org
  
= Current Proposals =
+
* Name: [[User:astepano|Andrei Stepanov]]
  
There are currently two proposals for how to implement this change and a final decision has not yet been made as to which is the final proposal. The two current proposals are:
+
* Name: [[User:sturivny|Serhii Turivnyi]]
 
+
* Email: sturivny@fedoraproject.org
* [[Changes/InvokingTestsPackaged|Tests invoked via RPM packages]]
 
* [[Changes/InvokingTestsAnsible|Tests invoked via Ansible]]
 
* [[Changes/InvokingTestsControl|Tests described by a control file]]
 
  
 
== Terminology ==
 
== Terminology ==
Line 34: Line 29:
 
** Examples: RPMs, OCI image, ISO, QCow2, Module repository ...
 
** Examples: RPMs, OCI image, ISO, QCow2, Module repository ...
 
* '''Test''': A callable/runnable piece of code and corresponding test data and mocks which exercises and evaluates a ''test subject''.
 
* '''Test''': A callable/runnable piece of code and corresponding test data and mocks which exercises and evaluates a ''test subject''.
 +
** '''Test environment''': environment where actual test run takes place. Test has direct impact on test environment.
 
* '''Test Suite''': The collection of all tests that apply to a ''test subject''.  
 
* '''Test Suite''': The collection of all tests that apply to a ''test subject''.  
 
* '''Test Framework''': A library or component that the ''test suite'' and ''tests'' use to accomplish their job.
 
* '''Test Framework''': A library or component that the ''test suite'' and ''tests'' use to accomplish their job.
** Examples: [https://avocado-framework.github.io/ Avocado], [https://wiki.gnome.org/Initiatives/GnomeGoals/InstalledTests GNOME Installed Tests], [https://pagure.io/modularity-testing-framework/ Modularity Testing Framework], [https://github.com/projectatomic/atomic-host-tests Ansible tests in Atomic Host], [https://tunir.readthedocs.io/en/latest/ Tunir tests], docker test images, ...
+
** Examples: [https://avocado-framework.github.io/ Avocado], [https://wiki.gnome.org/Initiatives/GnomeGoals/InstalledTests GNOME Installed Tests], [https://github.com/fedora-modularity/meta-test-family/ Meta Test Family], [https://github.com/projectatomic/atomic-host-tests Ansible tests in Atomic Host], [https://tunir.readthedocs.io/en/latest/ Tunir tests], docker test images, ...
 
* '''Test Result''': A boolean pass/fail output of a ''test suite''.  
 
* '''Test Result''': A boolean pass/fail output of a ''test suite''.  
 
** ''Test results'' are for consumption by automated aspects of a ''testing systems''.
 
** ''Test results'' are for consumption by automated aspects of a ''testing systems''.
Line 42: Line 38:
 
** ''Test artifacts'' are for consumption by humans, archival or big data analysis.
 
** ''Test artifacts'' are for consumption by humans, archival or big data analysis.
 
* '''Testing System''': A CI or other ''testing system'' that would like to discover, stage and invoke tests for a ''test subject''.
 
* '''Testing System''': A CI or other ''testing system'' that would like to discover, stage and invoke tests for a ''test subject''.
** Examples: [https://jenkins.io/ Jenkins], [https://taskotron.fedoraproject.org/ Taskotron], [https://docs.openstack.org/infra/zuul/ ZUUL], [https://ci.centos.org/ CentOS CI], Red Hat CI, [https://travis-ci.org/ Travis], [https://semaphoreci.com/ Semaphore], [https://developers.openshift.com/managing-your-applications/continuous-integration.html Openshift CI/CD], [https://wiki.ubuntu.com/ProposedMigration/AutopkgtestInfrastructure Ubuntu CI], ...
+
** Examples: [https://jenkins.io/ Jenkins], [https://taskotron.fedoraproject.org/ Taskotron], [https://docs.openstack.org/infra/zuul/ ZUUL], [https://ci.centos.org/ CentOS CI], [https://github.com/projectatomic/papr Papr], [https://travis-ci.org/ Travis], [https://semaphoreci.com/ Semaphore], [https://developers.openshift.com/managing-your-applications/continuous-integration.html Openshift CI/CD], [https://wiki.ubuntu.com/ProposedMigration/AutopkgtestInfrastructure Ubuntu CI], ...
 +
** Testing system uses '''test runner''' as a place for running tests. Testing system delegates test-running to test runner. Test runner examples: local machine, VM, ...
  
 
== Responsibilities ==
 
== Responsibilities ==
  
 
The '''testing system''' is responsible to:
 
The '''testing system''' is responsible to:
* Build or otherwise acquire the ''test subject'', such as package, container image, tree …
+
* Build or otherwise acquire the ''test subject'', such as a package, container image, tree …
 
* Decide which ''test suite'' to run, often by using the standard interface to discover appropriate ''tests'' for the dist-git repo that a test subject originated in.
 
* Decide which ''test suite'' to run, often by using the standard interface to discover appropriate ''tests'' for the dist-git repo that a test subject originated in.
 
* Schedule, provision or orchestrate a job to run the ''test suite'' on appropriate compute, storage, ...
 
* Schedule, provision or orchestrate a job to run the ''test suite'' on appropriate compute, storage, ...
Line 74: Line 71:
  
 
* The ''test suite'' and ''test framework'' SHOULD NOT leak its implementation details into the testing system, other than via the ''standard interface''.
 
* The ''test suite'' and ''test framework'' SHOULD NOT leak its implementation details into the testing system, other than via the ''standard interface''.
* The ''test suite'' and ''test framework'' SHOULD NOT rely on behavior of the testing system other than the ''standard interface''.
+
* The ''test suite'' and ''test framework'' SHOULD NOT rely on the behavior of the testing system other than the ''standard interface''.
 
* The ''standard interface'' MUST enable a dist-git packager to run a ''test suite'' locally.
 
* The ''standard interface'' MUST enable a dist-git packager to run a ''test suite'' locally.
 
** ''Test suites'' or ''test frameworks'' MAY call out to the network for certain tasks.
 
** ''Test suites'' or ''test frameworks'' MAY call out to the network for certain tasks.
Line 82: Line 79:
 
** For ''outside-in tests'' the ''test suite'' is outside of the file system tree and process space of the ''test subject''.
 
** For ''outside-in tests'' the ''test suite'' is outside of the file system tree and process space of the ''test subject''.
 
* The ''test suite'' and ''test framework'' SHOULD be able to provision containers and virtual machines necessary for its testing without requesting them from the ''testing system''.
 
* The ''test suite'' and ''test framework'' SHOULD be able to provision containers and virtual machines necessary for its testing without requesting them from the ''testing system''.
* The ''standard interface'' SHOULD describe how to uniquely identify a ''test suite'',  
+
* The ''standard interface'' SHOULD describe how to uniquely identify a ''test suite'',
 
 
 
 
  
 
== Benefit to Fedora ==
 
== Benefit to Fedora ==
Line 92: Line 87:
 
By staging and invoking tests consistently in Fedora we create an eco-system for the tests that allows varied test frameworks as well as CI system infrastructure to interoperate. The integration tests outlast the implementation details of either the frameworks they're written in or the CI systems running them.
 
By staging and invoking tests consistently in Fedora we create an eco-system for the tests that allows varied test frameworks as well as CI system infrastructure to interoperate. The integration tests outlast the implementation details of either the frameworks they're written in or the CI systems running them.
  
== Evaluations ==
+
== Detailed Description ==
  
''Instructions:'' In depth evaluations should be done in the sub-proposal pages. Read the proposals, you'll find the evaluation sections. Indicate vote below in the [[Changes/InvokingTests#voting|voting section]]
+
This standard interface describes how to discover, stage and invoke tests. It is important to cleanly separate implementation details of the ''testing system'' from the ''test suite'' and its framework. It is also important to allow Fedora packagers to locally and manually invoke a ''test suite''.
  
{{anchor|voting}}
+
'''First see the [https://fedoraproject.org/wiki/Changes/InvokingTests#Terminology Terminogy] division of [https://fedoraproject.org/wiki/Changes/InvokingTests#Responsibilities Responsibilities] and [https://fedoraproject.org/wiki/Changes/InvokingTests#Requirements Requirements]'''
=== Voting ===
+
 
 +
=== Staging ===
 +
 
 +
Tests files will be added into the <code>tests/</code> folder of a dist-git repository branch. The structure of the files and folders is left to the liberty of the packagers but there are one or more playbooks in the <code>tests/</code> folder that can be invoked to run the test suites.
 +
 
 +
# The ''testing system'' SHOULD stage the tests on target (eg: Fedora) operating system appropriate for the branch name of the dist-git repository containing the tests.
 +
# The ''testing system'' SHOULD stage a clean a system for each set of tests it runs.
 +
# The ''testing system'' MUST stage the following packages:
 +
## <code>ansible python2-dnf libselinux-python standard-test-roles</code>
 +
# The ''testing system'' MUST clone the dist-git repository for the test, and checks out the appropriate branch.
 +
# The contents of <code>/etc/yum.repos.d</code> on the staged system SHOULD be replaced with repository information that reflects the known good Fedora packages corresponding to the branch of the dist-git repository.
 +
## The ''testing system'' MAY use multiple repositories, including ''updates'' or ''updates-testing'' to ensure this.
 +
 
 +
=== Invocation ===
 +
 
 +
The testing system MUST run each playbook matching the glob <code>tests/tests*.yml</code> in the dist-git repo. Each of these files constitutes a
 +
test suite. Each test suite is invoked independently by the testing system as follows.
 +
 
 +
The ''test subjects'' are passed to the playbook and inventory as operating system environment and ansible environment. Often only one ''test subject'' is passed in. However multiple subjects may be concatenated together in a shell escaped string. The playbooks and/or inventory script split the string. The extensions as follows are used to determine the type of subject:
  
Every single vote requires an evaluation.
 
 
{|
 
{|
! Contributor !! Packaged Tests !! Ansible Tests !! Control Tests!! Notes
+
! Identifier !! Test subject
 
|-
 
|-
| [[User:YourUserName|YourUserName]] || || +1 || || This is just an example, please vote for one of the options
+
| *.rpm    || Absolute path to an RPM file
 
|-
 
|-
| [[User:Roshi|Roshi]] || || +1 || || 
+
| *.repo    || Absolute repo filenames appropriate for <code>/etc/yum.repos.d</code>
 
|-
 
|-
| [[User:Ausil|Ausil]] || || +1 || || Ansible is a bit more work but I think will give better results and options 
+
| *.qcow2, *.qcow2c || Absolute path to one virtual machine disk image bootable with cloud-init
 
|-
 
|-
| [[User:pingou|pingou]] || || +1 || || Ansible clearly has some down side but I do think it is simpler and can be more powerful than the RPM approach
+
| *.oci    || Absolute path of one OCI container image filesystem bundle
 
|-
 
|-
| [[User:stefw|Stef]] || +1 || || || Tests should be a core part of the distro, hence preference for packaging
+
| docker:|| Fully qualified path to a docker image in a registry
 
|-
 
|-
| [[User:jgalipea|Jenny]] || || +1 || || Ansible would be a dependency but it meets the needs better for configuring a system for the tests it is invoking. Packing tests into rpms is an added layer of complexity that is overhead and time consuming. Do not want to go down that path again.
+
| ...      || Other ''test subject'' identifiers may be added later.
 +
|}
 +
 
 +
 
 +
Various ''tests'' in a playbook constitute a ''test suite''. Some parts of these ''test suites'' will run in certain contexts, against certain deliverable artifacts. Certain tests will run against Atomic Host deliverables, while others will not. Certain tests will run against Docker deliverables while others will not. This is related to, but does not exactly overlap with the ''test subject'' identifiers above. Ansible tags are used to denote these contexts.
 +
 
 +
{|
 +
! Tag      !! Test context
 
|-
 
|-
| [[User:jscotka|jscotka]] || +0.5  || || || Bigger preference for packaging, because of combining upstream/downstream testing (need proper version of tests for installed packages). But I also like idea of ansible tooling. Not decided yet. There is also third possibility ( I hope it will be added asap)
+
| atomic    || Atomic Host
 
|-
 
|-
| [[User:miabbott|miabbott]] || || +1 || ||  The ease of use and amount of available documentation for Ansible are some of it's strongest points for the proposal.  Ansible should have better support for provisioning hosts for different kinds of tests.
+
| container || A Docker or OCI container
 
|-
 
|-
| [[User:alivigni|alivigni]] || || +1 || ||  The ease of use and amount of available documentation for Ansible are some of it's strongest points for the proposal.  Ansible is a general tool used inside and outside of Fedora and is constantly being enhanced with new features.  It also allows a common way to drive testing that any CI tool can use.
+
| classic  || Tested against a classic installed YUM/DNF installed system.
|-
 
| [[User:dustymabe|dustymabe]] || || +1 || ||  I think ansible gives a balance of simple & sophisticated tooling to enable us to write simple tests or write complex tests. If a user is not familiar with ansible then they can use an example yaml file to just execute a shell script. More advanced users can ramp up to ansible's potential.
 
|-
 
| [[User:Ncoghlan|Nick Coghlan]] || +1 || || || I started this evaluation expecting to vote for the Ansible option, but changed my mind when I asked myself the question: "Given this approach, how hard would it to be to bootstrap the other?". Given that Fedora and its derivatives are inherently based on RPM, I think the winner on that basis is a packaging based approach, with a helper module and spec file boilerplate to bootstrap Ansible based test environments in libvirt and/or docker for components that can't run their integration tests non-intrusively on the host. This does imply some assumed capabilities for the bootstrapped Ansible environment (1. "Give me a matching local VM"; 2. "Give me a matching container"; 3. "Give me a matching OpenStack VM", 4. "Give me a matching Beaker machine"), but that would be the case even with the Ansible-as-the-baseline option.
 
|-
 
| [[User:gdk|gdk]] || || +1 || ||  Reason one: easier to onboard new test authors because the Ansible approach will be far easier to learn. Reason two: multiple CI options will be easier to unlock down the road with a suite of Ansible tests ready to go. Reason three: you will have a ton of Ansible knowledge right at your fingertips. :)
 
|-
 
| [[User:psabata|contyk]] || +1 || || || The RPM approach has so many benefits I don't even know where to start.  Every test package states exactly what it requires, along with the exact build it's testing; we get this information automatically and in most cases for free when building the package.  Packages from large, standardized ecosystems such as Perl, Python, Ruby and similar make up a large part of our distribution; for most of those, generating the test subpackages could be almost entirely automated; their tests are in a known location, they're invoked in a known way and we have dependency generators for them.  Some already have or had macros for exactly this purpose.  Currently everyone (and I think this is unlikely to change, and not because of what tools we use) who would be involved in working on tests knows RPM packaging to at least some degree and wouldn't have to learn anything new.  I'm sure the package maintainer would be more than willing to help additional interested contributors in extending the suite.  Not running tests during the package build would actually simplify packaging as the package author doesn't need to list all the test build dependencies in the main package.  The builds run faster since the buildroots are smaller and they don't block builders for the entire duration of the test suite.  You could do that with the other two approaches as well but then you also need to modify the RPM package which is what you're trying to avoid.  Also, the packager can switch from testing during build to async testing whenever they feel like it without the tests running twice or not at all at any point in time.  It just feels natural to me.
 
|-
 
| [[User:Tflink|tflink]] || || +1 || || I think that there are down-sides to all of these proposals but considering our constraints, I think that starting with ansible is the best option right now.
 
 
|-
 
|-
 +
| ...      || Other ''test subject'' identifiers may be added later.
 
|}
 
|}
 +
 +
 +
To invoke the tests, the ''testing system'' must perform the following tasks for each ''test suite'' playbook:
 +
 +
# MUST execute the playbook with the following operating system environment variables:
 +
## <code>TEST_SUBJECTS</code>: The ''test subjects'' string as described above
 +
## <code>TEST_ARTIFACTS</code>: The full path of an empty folder for ''test artifacts''
 +
# MUST execute the playbook with the following Ansible variables.
 +
## <code>subjects</code>: The ''test subjects'' string as described above
 +
## <code>artifacts</code>: The full path of an empty folder for ''test artifacts''
 +
# SHOULD execute the playbook with all Ansible tags best represent the intended ''test context''.
 +
## The choice of ''test context'' tags is related to the ''test subject'' being tested
 +
# MUST execute Ansible with inventory set to the full path of the file or directory <code>tests/inventory</code> if it exists.
 +
## If the <code>tests/inventory</code> file doesn't exist, then the following inventory SHOULD be used as a default:<br> <code>/usr/share/ansible/inventory</code>
 +
# MUST execute the playbook as root.
 +
# MUST examine the exit code of the playbook. A zero exit code is successful ''test result'', non-zero is failure.
 +
# MUST treat the file <code>test.log</code> in the <code>artifacts</code> folder as the main readable output of the test.
 +
# SHOULD place the textual stdout/stderr of the <code>ansible-playbook</code> command in the <code>ansible.log</code> file in the <code>artifacts</code> folder.
 +
# SHOULD treat the contents of the <code>artifacts</code> folder as the ''test artifacts''.
 +
 +
Each ''test suite'' playbook or ''test framework'' contained therein:
 +
 +
# SHOULD drop privileges appropriately if the ''test suite'' should be run as non-root.
 +
# MUST install any requirements of its ''test suite'' or ''test framework'' and MUST fail if this is not possible.
 +
# MUST provision the ''test subject'' listed in the <code>subjects</code> variable appropriately for its playbook name (described above) and MUST fail if this is not possible.
 +
# MUST place the main readable output of the ''test suite'' into a <code>test.log</code> file in the <code>artifacts</code> variable folder. This MUST happen even if some of the test suites fail.
 +
# SHOULD place additional ''test artifacts'' in the folder defined in the <code>artifacts</code> variable.
 +
# MUST return a zero exit code of the playbook if the ''test result'' is a pass, or a non-zero exit code if the ''test result'' is a fail.
 +
 +
If an inventory file or script exists, it:
 +
 +
# MUST describe where to invoke the playbook and how to connect to that target.
 +
# SHOULD launch or install any supported <code>$TEST_SUBJECTS</code> so that the playbook can be invoked against them.
 +
# SHOULD put relevant logs in the <code>$TEST_ARTIFACTS</code> directory.
 +
 +
=== Discovery ===
 +
 +
Test discovery is done via dist-git. Both packages and modules may have tests in this format. To list which ''test context'' a given dist-git directory or playbook is relevant for, use a command like the following:
 +
 +
<pre>
 +
# ansible-playbook --list-tags tests.yml
 +
</pre>
 +
 +
== Scope ==
 +
 +
Since the tests are added in a sub-folder of the dist-git repo, there are no changes required to the Fedora infrastructure and will have no impact on the packagers' workflow and tooling.
 +
 +
Only the testing system will need to be taught to install the requirements and run the playbooks.
 +
 +
== User Experience ==
 +
 +
A standard way to package, store and run tests benefits Fedora stability, and makes Fedora better for users.
 +
 +
* This structure makes it easy to run locally thus potentially reproducing an error triggered on the test system.
 +
* Ansible is being more and more popular, thus making it easier for people to contribute new tests
 +
* Used by a lot of sys-admin, ansible could help sys-admin to bring test-cases to the packagers and developers about situation where something failed for them.
 +
 +
== Upgrade/compatibility impact ==
 +
 +
There are no real upgrade or compatibility impact. The tests will be branched per release as spec files are branched dist-git is now.
 +
 +
== Documentation ==
 +
 +
* [[CI|CI Landing page]]
 +
* [[CI/Tests|Documentation page]]
 +
 +
== Proposals and Evaluation ==
 +
 +
During the selection process for a standard test invocation and and layout format for Fedora, [[Changes/InvokingTestsProposals|several proposals]] were examined.
 +
 +
[[Category:FedoraAtomicCi]]
 +
[[Category:FedoraCi]]

Latest revision as of 16:20, 30 January 2018

Standard Discovery, Staging and Invocation of Integration Tests

Summary

Let's define a clear delineation of between a test suite (including framework) and the CI system that is running the test suite. This is the standard interface.

Invoking-tests-standard-interface.png

What follows is a standard way to discover, package and invoke integration tests for a package stored in a Fedora dist-git repo.

Many Fedora packages have unit tests. These tests are typically run during a %check RPM build step and run in a build root. On the other hand, integration testing should happen against a composed system. Upstream projects have integration tests, both Fedora QA and the Atomic Host team would like to create more integration tests, Red Hat would like to bring integration tests upstream.

Owner

Terminology

  • Test Subject: The items that are to be tested.
    • Examples: RPMs, OCI image, ISO, QCow2, Module repository ...
  • Test: A callable/runnable piece of code and corresponding test data and mocks which exercises and evaluates a test subject.
    • Test environment: environment where actual test run takes place. Test has direct impact on test environment.
  • Test Suite: The collection of all tests that apply to a test subject.
  • Test Framework: A library or component that the test suite and tests use to accomplish their job.
  • Test Result: A boolean pass/fail output of a test suite.
    • Test results are for consumption by automated aspects of a testing systems.
  • Test Artifact: Any additional output of the test suite such as the stdout/stderr output, log files, screenshots, core dumps, or TAP/Junit/subunit streams.
    • Test artifacts are for consumption by humans, archival or big data analysis.
  • Testing System: A CI or other testing system that would like to discover, stage and invoke tests for a test subject.

Responsibilities

The testing system is responsible to:

  • Build or otherwise acquire the test subject, such as a package, container image, tree …
  • Decide which test suite to run, often by using the standard interface to discover appropriate tests for the dist-git repo that a test subject originated in.
  • Schedule, provision or orchestrate a job to run the test suite on appropriate compute, storage, ...
  • Stage the test suite as described by the standard interface.
  • Invoke the test suite as described by the standard interface.
  • Gather the test results and test artifacts as described by the standard interface.
  • Announce and relay the test results and test artifacts for gating, archival ...

The standard interface describes how to:

  • Discover a test suite for a given dist-git repo.
  • Uniquely identify a test suite.
  • Stage a test suite and its dependencies such as test frameworks.
  • Provide the test subject to the test suite.
  • Invoke a test suite in a consistent way.
  • Gather test results and test artifacts from the invoked test suite.

The test suite is responsible to:

  • Declare its dependencies such as a test framework via the standard interface.
  • Execute the test framework as necessary.
  • Provision (usually locally) any containers or virtual machines necessary for testing the test subject.
  • Provide test results and test subjects back according to the standard

The format of the textual logs and test artifacts that come out of a test suite is not prescribed by this document. Nor is it envisioned to be standardized across all possible test suites.

Requirements

  • The test suite and test framework SHOULD NOT leak its implementation details into the testing system, other than via the standard interface.
  • The test suite and test framework SHOULD NOT rely on the behavior of the testing system other than the standard interface.
  • The standard interface MUST enable a dist-git packager to run a test suite locally.
    • Test suites or test frameworks MAY call out to the network for certain tasks.
  • It MUST be possible to stage an upstream test suite using the standard interface.
  • Both in-situ tests, and more rigorous outside-in tests MUST be possible with the standard interface.
    • For in-situ tests the test suite is in the same file system tree and process space as the test subject.
    • For outside-in tests the test suite is outside of the file system tree and process space of the test subject.
  • The test suite and test framework SHOULD be able to provision containers and virtual machines necessary for its testing without requesting them from the testing system.
  • The standard interface SHOULD describe how to uniquely identify a test suite,

Benefit to Fedora

Developers benefit by having a consistent target for how to describe tests, while also being able to execute them locally while debugging issues or iterating on tests.

By staging and invoking tests consistently in Fedora we create an eco-system for the tests that allows varied test frameworks as well as CI system infrastructure to interoperate. The integration tests outlast the implementation details of either the frameworks they're written in or the CI systems running them.

Detailed Description

This standard interface describes how to discover, stage and invoke tests. It is important to cleanly separate implementation details of the testing system from the test suite and its framework. It is also important to allow Fedora packagers to locally and manually invoke a test suite.

First see the Terminogy division of Responsibilities and Requirements

Staging

Tests files will be added into the tests/ folder of a dist-git repository branch. The structure of the files and folders is left to the liberty of the packagers but there are one or more playbooks in the tests/ folder that can be invoked to run the test suites.

  1. The testing system SHOULD stage the tests on target (eg: Fedora) operating system appropriate for the branch name of the dist-git repository containing the tests.
  2. The testing system SHOULD stage a clean a system for each set of tests it runs.
  3. The testing system MUST stage the following packages:
    1. ansible python2-dnf libselinux-python standard-test-roles
  4. The testing system MUST clone the dist-git repository for the test, and checks out the appropriate branch.
  5. The contents of /etc/yum.repos.d on the staged system SHOULD be replaced with repository information that reflects the known good Fedora packages corresponding to the branch of the dist-git repository.
    1. The testing system MAY use multiple repositories, including updates or updates-testing to ensure this.

Invocation

The testing system MUST run each playbook matching the glob tests/tests*.yml in the dist-git repo. Each of these files constitutes a test suite. Each test suite is invoked independently by the testing system as follows.

The test subjects are passed to the playbook and inventory as operating system environment and ansible environment. Often only one test subject is passed in. However multiple subjects may be concatenated together in a shell escaped string. The playbooks and/or inventory script split the string. The extensions as follows are used to determine the type of subject:

Identifier Test subject
*.rpm Absolute path to an RPM file
*.repo Absolute repo filenames appropriate for /etc/yum.repos.d
*.qcow2, *.qcow2c Absolute path to one virtual machine disk image bootable with cloud-init
*.oci Absolute path of one OCI container image filesystem bundle
docker:* Fully qualified path to a docker image in a registry
... Other test subject identifiers may be added later.


Various tests in a playbook constitute a test suite. Some parts of these test suites will run in certain contexts, against certain deliverable artifacts. Certain tests will run against Atomic Host deliverables, while others will not. Certain tests will run against Docker deliverables while others will not. This is related to, but does not exactly overlap with the test subject identifiers above. Ansible tags are used to denote these contexts.

Tag Test context
atomic Atomic Host
container A Docker or OCI container
classic Tested against a classic installed YUM/DNF installed system.
... Other test subject identifiers may be added later.


To invoke the tests, the testing system must perform the following tasks for each test suite playbook:

  1. MUST execute the playbook with the following operating system environment variables:
    1. TEST_SUBJECTS: The test subjects string as described above
    2. TEST_ARTIFACTS: The full path of an empty folder for test artifacts
  2. MUST execute the playbook with the following Ansible variables.
    1. subjects: The test subjects string as described above
    2. artifacts: The full path of an empty folder for test artifacts
  3. SHOULD execute the playbook with all Ansible tags best represent the intended test context.
    1. The choice of test context tags is related to the test subject being tested
  4. MUST execute Ansible with inventory set to the full path of the file or directory tests/inventory if it exists.
    1. If the tests/inventory file doesn't exist, then the following inventory SHOULD be used as a default:
      /usr/share/ansible/inventory
  5. MUST execute the playbook as root.
  6. MUST examine the exit code of the playbook. A zero exit code is successful test result, non-zero is failure.
  7. MUST treat the file test.log in the artifacts folder as the main readable output of the test.
  8. SHOULD place the textual stdout/stderr of the ansible-playbook command in the ansible.log file in the artifacts folder.
  9. SHOULD treat the contents of the artifacts folder as the test artifacts.

Each test suite playbook or test framework contained therein:

  1. SHOULD drop privileges appropriately if the test suite should be run as non-root.
  2. MUST install any requirements of its test suite or test framework and MUST fail if this is not possible.
  3. MUST provision the test subject listed in the subjects variable appropriately for its playbook name (described above) and MUST fail if this is not possible.
  4. MUST place the main readable output of the test suite into a test.log file in the artifacts variable folder. This MUST happen even if some of the test suites fail.
  5. SHOULD place additional test artifacts in the folder defined in the artifacts variable.
  6. MUST return a zero exit code of the playbook if the test result is a pass, or a non-zero exit code if the test result is a fail.

If an inventory file or script exists, it:

  1. MUST describe where to invoke the playbook and how to connect to that target.
  2. SHOULD launch or install any supported $TEST_SUBJECTS so that the playbook can be invoked against them.
  3. SHOULD put relevant logs in the $TEST_ARTIFACTS directory.

Discovery

Test discovery is done via dist-git. Both packages and modules may have tests in this format. To list which test context a given dist-git directory or playbook is relevant for, use a command like the following:

# ansible-playbook --list-tags tests.yml

Scope

Since the tests are added in a sub-folder of the dist-git repo, there are no changes required to the Fedora infrastructure and will have no impact on the packagers' workflow and tooling.

Only the testing system will need to be taught to install the requirements and run the playbooks.

User Experience

A standard way to package, store and run tests benefits Fedora stability, and makes Fedora better for users.

  • This structure makes it easy to run locally thus potentially reproducing an error triggered on the test system.
  • Ansible is being more and more popular, thus making it easier for people to contribute new tests
  • Used by a lot of sys-admin, ansible could help sys-admin to bring test-cases to the packagers and developers about situation where something failed for them.

Upgrade/compatibility impact

There are no real upgrade or compatibility impact. The tests will be branched per release as spec files are branched dist-git is now.

Documentation

Proposals and Evaluation

During the selection process for a standard test invocation and and layout format for Fedora, several proposals were examined.