From Fedora Project Wiki

< Changes

Revision as of 21:14, 19 June 2017 by Merlinm (talk | contribs)

Standard Discovery, Packaging, Invocation of Integration Tests

Summary

Lets define a clear delineation of between a test suite (including framework) and the CI system that is running the test suite. This is the standard interface.

Invoking-tests-standard-interface.png

What follows is a standard way to discover, package and invoke integration tests for a package stored in a Fedora dist-git repo.

Many Fedora packages have unit tests. These tests are typically run during a %check RPM build step and run in a build root. On the other hand, integration testing should happen against a composed system. Upstream projects have integration tests, both Fedora QA and the Atomic Host team would like to create more integration tests, Red Hat would like to bring integration tests upstream.

Owner

Terminology

  • Test Subject: The items that are to be tested.
    • Examples: RPMs, OCI image, ISO, QCow2, Module repository ...
  • Test: A callable/runnable piece of code and corresponding test data and mocks which exercises and evaluates a test subject.
  • Test Suite: The collection of all tests that apply to a test subject.
  • Test Framework: A library or component that the test suite and tests use to accomplish their job.
  • Test Result: A boolean pass/fail output of a test suite.
    • Test results are for consumption by automated aspects of a testing systems.
  • Test Artifact: Any additional output of the test suite such as the stdout/stderr output, log files, screenshots, core dumps, or TAP/Junit/subunit streams.
    • Test artifacts are for consumption by humans, archival or big data analysis.
  • Testing System: A CI or other testing system that would like to discover, stage and invoke tests for a test subject.

Responsibilities

The testing system is responsible to:

  • Build or otherwise acquire the test subject, such as package, container image, tree …
  • Decide which test suite to run, often by using the standard interface to discover appropriate tests for the dist-git repo that a test subject originated in.
  • Schedule, provision or orchestrate a job to run the test suite on appropriate compute, storage, ...
  • Stage the test suite as described by the standard interface.
  • Invoke the test suite as described by the standard interface.
  • Gather the test results and test artifacts as described by the standard interface.
  • Announce and relay the test results and test artifacts for gating, archival ...

The standard interface describes how to:

  • Discover a test suite for a given dist-git repo.
  • Uniquely identify a test suite.
  • Stage a test suite and its dependencies such as test frameworks.
  • Provide the test subject to the test suite.
  • Invoke a test suite in a consistent way.
  • Gather test results and test artifacts from the invoked test suite.

The test suite is responsible to:

  • Declare its dependencies such as a test framework via the standard interface.
  • Execute the test framework as necessary.
  • Provision (usually locally) any containers or virtual machines necessary for testing the test subject.
  • Provide test results and test subjects back according to the standard

The format of the textual logs and test artifacts that come out of a test suite is not prescribed by this document. Nor is it envisioned to be standardized across all possible test suites.

Requirements

  • The test suite and test framework SHOULD NOT leak its implementation details into the testing system, other than via the standard interface.
  • The test suite and test framework SHOULD NOT rely on behavior of the testing system other than the standard interface.
  • The standard interface MUST enable a dist-git packager to run a test suite locally.
    • Test suites or test frameworks MAY call out to the network for certain tasks.
  • It MUST be possible to stage an upstream test suite using the standard interface.
  • Both in-situ tests, and more rigorous outside-in tests MUST be possible with the standard interface.
    • For in-situ tests the test suite is in the same file system tree and process space as the test subject.
    • For outside-in tests the test suite is outside of the file system tree and process space of the test subject.
  • The test suite and test framework SHOULD be able to provision containers and virtual machines necessary for its testing without requesting them from the testing system.
  • The standard interface SHOULD describe how to uniquely identify a test suite,

Benefit to Fedora

Developers benefit by having a consistent target for how to describe tests, while also being able to execute them locally while debugging issues or iterating on tests.

By staging and invoking tests consistently in Fedora we create an eco-system for the tests that allows varied test frameworks as well as CI system infrastructure to interoperate. The integration tests outlast the implementation details of either the frameworks they're written in or the CI systems running them.

Detailed Description

This standard interface describes how to discover, stage and invoke tests. It is important to cleanly separate implementation details of the testing system from the test suite and its framework. It is also important to allow Fedora packagers to locally and manually invoke a test suite.

First see the Terminogy division of Responsibilities and Requirements

Staging

Tests files will be added into the tests/ folder of a dist-git repository branch. The structure of the files and folders is left to the liberty of the packagers but there are one or more playbooks in the tests/ folder that can be invoked to run the test suites.

  1. The testing system SHOULD stage the tests on Fedora operating system appropriate for the branch name of the dist-git repository containing the tests.
  2. The testing system SHOULD stage a clean a system for each set of tests it runs.
  3. The testing system MUST stage the following packages:
    1. ansible python2-dnf libselinux-python
  4. The testing system MUST clone the dist-git repository for the test, and checks out the appropriate branch.
  5. The contents of /etc/yum.repos.d on the staged system SHOULD be replaced with repository information that reflects the known good Fedora packages corresponding to the branch of the dist-git repository.
    1. The testing system MAY use multiple repositories, including updates or updates-testing to ensure this.

Invocation

The testing system MUST select a playbooks in the tests/ folder depending on the type of test subject it would like to test. The filename of each of these playbooks start with the test_ prefix and ends with a .yml extension. The following well known playbooks correspond to common test subjects. Additional playbooks will be added to this list as additional test subjects become common:

Playbook invoked Test subject
test_rpm.yml A string containing a space separated list of rpm filenames
test_repo.yml A string containing a space separated list of repo filenames appropriate for /etc/yum.repos.d
test_cloud.yml A string containing the filename of one virtual machine disk image bootable with cloud-init
test_docker.yml A string containing the name of one docker image
test_oci.yml A string containing the filename of one OCI container image filesystem bundle
test_local.yml An empty string. No test subject or installation.

If a playbook for a given test subject is not present in a dist-git repository, the testing system SHOULD treat the test as having been "skipped". That is, the invocation SHOULD neither pass nor fail.

The test_local.yml SHOULD test a booted system where the test suite, its framework, and test subject are already installed. This playbook is usually invoked by the other playbooks. Additional playbooks may be present in the tests/ folder, and these MAY represent multiple test suites. The testing system is not expected to be aware of these additional playbooks.

To invoke the selected playbook, the testing system:

  1. MUST execute the playbook locally with ansible_connection=local and host localhost
  2. MUST execute the playbook with the following variables.
    1. subjects: The test subjects string as described above
    2. artifacts: The full path of an empty folder for test artifacts
  3. MUST execute the playbook as root.
  4. MUST examine the exit code of the playbook. A zero exit code is successful test result, non-zero is failure.
  5. MUST treat the file test.log in the artifacts folder as the main readable output of the test.
  6. SHOULD place the textual stdout/stderr of the ansible-playbook command in the ansible.log file in the artifacts folder.
  7. SHOULD treat the contents of the artifacts folder as the test artifacts.

The playbook and its test suite or test framework:

  1. SHOULD drop privileges appropriately if the test suite should be run as non-root.
  2. MUST install any requirements of its test suite or test framework and MUST fail if this is not possible.
  3. MUST provision the test subject listed in the subjects variable appropriately for its playbook name (described above) and MUST fail if this is not possible.
  4. MUST place the main readable output of the test suite into a test.log file in the artifacts variable folder. This MUST happen even if some of the test suites fail.
  5. SHOULD place additional test artifacts in the folder defined in the artifacts variable.

Discovery

A testing system needs to be able to efficiently answer the question "does this subject have any tests packages, and if so, what are their names". This should be automatically discoverable to the extent possible.

Use repoquery, basically I propose we rely on the dependency chain of the RPMs itself instead of trying to replicate it differently.

repoquery --whatrequires or an equivalent relying on mdapi: https://apps.fedoraproject.org/mdapi/ (which I need to adjust to support back walking (ie find which packages requires "foo" instead of what packages "foo" requires which we currently have) and we should be able to build a list of dependencies.

In addition, a test suite can be uniquely identified using the git hash of the commit of the git repo.

Scope

Since the tests are added in a sub-folder of the dist-git repo, there are no changes required to the Fedora infrastructure and will have no impact on the packagers' workflow and tooling.

Only the testing system will need to be taught to install the requirements and run the playbooks.

User Experience

A standard way to package, store and run tests benefits Fedora stability, and makes Fedora better for users.

  • This structure makes it easy to run locally thus potentially reproducing an error triggered on the test system.
  • Ansible is being more and more popular, thus making it easier for people to contribute new tests
  • Used by a lot of sys-admin, ansible could help sys-admin to bring test-cases to the packagers and developers about situation where something failed for them.

Upgrade/compatibility impact

There are no real upgrade or compatibility impact. The tests will be branched per release as spec files are branched dist-git is now.


Examples

What follows are examples of writing and/or packaging existing tests to this standard. This is how to run the various examples:


  • test_rpm.yml
 $ fedpkg local
 $ mkdir -p ./artifacts
 $ sudo ansible-playbook tests/test_rpm.yml -e artifacts=$PWD/artifacts -e subjects=$PWD/x86_64/sed-4*.x86_64.rpm
  • test_local.yml
 $ mkdir -p ./artifacts
 $ sudo ansible-playbook tests/test_local.yml -e artifacts=$PWD/artifacts -e subjects=
  • test_cloud.yml
 $ mkdir -p ./artifacts
 $ curl -o cloud.qcow2 https://s3.amazonaws.com/fedora-atomic-s3/Fedora-26-20170331.n.0/Fedora-Atomic-26-20170331.n.0.x86_64.qcow2
 $ sudo ansible-playbook tests/test_cloud.yml -e artifacts=$PWD/artifacts -e subjects=$PWD/cloud.qcow2
  • test_docker.yml
 $ mkdir -p ./artifacts
 $ docker pull docker.io/fedora:latest
 $ sudo ansible-playbook tests/test_docker.yml -e artifacts=$PWD/artifacts -e subjects=fedora:latest
  • test_oci.yml
    • No examples here yet
  • test_repo.yml
 $ mkdir -p ./artifacts
 ... get a repo file ...
 $ sudo ansible-playbook tests/test_repo.yml -e artifacts=$PWD/artifacts -e subjects=$PWD/haproxy.repo

Example: Simple in-situ test

Copy of Debian 'gzip' test:

Example: GNOME style "Installed Tests"

Upstream glib2-tests being executed according to this standard interface:

Example: Tests run in Docker Container

Example: Modularity testing Framework

Module testing framework tests wrapped in this standard interface: 

 [haproxy-repo-test-subject]
 name=Example haproxy repo test subject
 baseurl=http://kojipkgs.fedoraproject.org/repos/module-8e83a5f6f6ed55ca/latest/x86_64/
 gpgcheck=0
 enabled=1

Example: Ansible with Atomic Host

TODO: Port an existing test

Example: Beakerlib based test

Beakerlib tests of sed package:

Beakerlib test of 'setup' package:

Beakerlib test of 'coreutils' package:


Proposals and Evaluation

During the selection process for a standard test invocation and and layout format for Fedora, several proposals were examined.

Retrieved from "https://fedoraproject.org/w/index.php?title=Changes/InvokingTests&oldid=495084"