From Fedora Project Wiki

< CI

(Move BeakerLib and RHTS roles to Standard Test Roles)
(Remove duplicated content, reorganize)
Line 1: Line 1:
Tests are stored in [[Package_maintenance_guide|package or module git repositories]] along with the packages and modules that they test. The tests are updated together with the software.
 
  
== Setting up ==
+
== Enabling Tests ==
  
See [[/HowToStart/]] to get a quick introduction into adding new tests. You can use the <code>fedpkg</code> tool to checkout a package git repository. If a <code>tests/</code> subdirectory exists, then the repository contains tests. The files ending in <code>.yml</code> in the <code>tests/</code> subdirectory each represent a test or a part of a test.
+
Tests may be written in different ways, but are exposed and invoked in a standard way as defined by the [[CI/Standard_Test_Interface|Standard Test Interface]] directly in the package [https://src.fedoraproject.org/projects/rpms/%2A git repository]. To start working on tests you can clone a package repo directly:
  
The tests are wrapped or written as [http://docs.ansible.com/ansible/playbooks.html Ansible playbooks] and
+
git clone https://src.fedoraproject.org/rpms/qrencode.git
[https://fedoraproject.org/wiki/Changes/InvokingTests invoked according to this specification]. To invoke the tests you need the following dependencies (as described by the specification) installed on a modern Fedora system:
 
  
<pre>
+
You can also use the <code>fedpkg</code> to clone the repo. See the [[Package_maintenance_guide|Package Maintenance Guide]] for more info about the tool:
$ sudo dnf install ansible python2-dnf libselinux-python standard-test-roles
 
</pre>
 
 
 
For the following documentation, we'll checkout the https://upstreamfirst.fedorainfracloud.org/gzip.git tests
 
 
 
<pre>
 
$ git clone https://upstreamfirst.fedorainfracloud.org/gzip.git
 
$ cd gzip/
 
$ sudo -s
 
</pre>
 
 
 
Although some playbooks may function without sudo, tests are always invoked as root. The test
 
itself may set up users and/or drop permissions if a part of that test. But in general be sure
 
to be root when invoking tests.
 
 
 
For the following documentation we'll checkout the <code>gzip</code> tests
 
 
 
{{admon/important|Work in progress|The upstreaming of tests is currently ongoing.
 
Use the following command.}}
 
Make sure you have installed <code>dnf install fedora-packager</code> package.
 
 
 
<pre>
 
$ fedpkg clone -a gzip
 
$ cd gzip/tests/
 
$ sudo -s
 
</pre>
 
 
 
== Running tests ==
 
 
 
You can always invoke the tests locally. Many tests modify or change the system they are run against, so take that into account when looking at how to invoke tests. The following examples invoke tests against the same system that the package git repository is checked out on. Below there are further options for invoking tests against another fully formed and integrated systems, such as an Atomic Host or container image ''test subject''.
 
 
 
{{admon/important|Work in progress|Before running tests make sure that all logs /tmp/artifacts/test.* are deleted.}}
 
 
 
 
 
There may be more than one test present in a package git repository, but the file <code>tests.yml</code> is the main entry point. To run it use the following command:
 
 
 
<pre>
 
# ansible-playbook tests.yml
 
</pre>
 
 
 
You can find output artifacts of the tests in an <code>artifacts/</code> or specify a specific directory like this:
 
 
 
<pre>
 
# ansible-playbook -e artifacts=/tmp/output tests.yml
 
</pre>
 
 
 
You can filter which kinds of tests are run by providing a <code>--tags</code> argument. To only run tests that are suited for classic systems installed by <code>yum</code> or <code>dnf</code> you can use a command like:
 
 
 
<pre>
 
# ansible-playbook --tags=classic tests.yml
 
</pre>
 
 
 
When run by a CI System the tests are [[Changes/InvokingTests|invoked according to a specification]]. Look here for more details and standard invocation variables.
 
 
 
=== Testing specific RPMs ===
 
 
 
When you run the tests as above, the tests assume that the system to be tested is the same as the system invoking the tests. In particular, the test assumes that the thing to be tested is already installed.
 
 
 
A ''test subject'' is what we call the thing to be tested. RPMs are a particular kind of ''test subject''. To turn a test subject into a launched, installed system to be tested, we use [http://docs.ansible.com/ansible/intro_dynamic_inventory.html Ansible dynamic inventory]. Lets invoke the tests with an inventory and a specific version of gzip:
 
 
 
<pre>
 
# export ANSIBLE_INVENTORY=$(test -e inventory && echo inventory || echo /usr/share/ansible/inventory)
 
# curl -o gzip.rpm https://kojipkgs.fedoraproject.org//packages/gzip/1.8/2.fc26/x86_64/gzip-1.8-2.fc26.x86_64.rpm
 
# export TEST_SUBJECTS=$PWD/gzip.rpm
 
# ansible-playbook tests.yml
 
</pre>
 
 
 
You'll notice that the RPM is installed into the testable system before invoking the tests. Some tests contain their own inventory, that is their own instructions for turning a ''test subject'' into one or more testable systems. But in this case we use the default <code>standard-test-roles</code> inventory in <code>/usr/share/ansible/inventory</code> to do this.
 
 
 
=== Testing an Atomic Host ===
 
 
 
The former example may seem a bit contrived, but the concept of a ''test subject'' starts to make more sense when you want to test a fully formed and integrated deliverable, such as Atomic Host. The ''test subject'' again represents the thing to be tested. The ''test subject'' in this case is a QCow2 image. To turn a test subject into a launched system ready to be tested, we use [http://docs.ansible.com/ansible/intro_dynamic_inventory.html Ansible dynamic inventory].
 
 
 
<pre>
 
# export ANSIBLE_INVENTORY=$(test -e inventory && echo inventory || echo /usr/share/ansible/inventory)
 
# curl -Lo fedora-atomic-host.qcow2 https://getfedora.org/atomic_qcow2_latest
 
# export TEST_SUBJECTS=$PWD/fedora-atomic-host.qcow2
 
# ansible-playbook --tags=atomic tests.yml
 
</pre>
 
 
 
If you watch closely you'll see that the Atomic Host image is booted, and the tests run against the launched image. Not all tests are able to function in the somewhat different environment of Atomic Host, in fact, for certain cases, the software to be tested may not be included in the Atomic Host ''test subject''. But most of the tests in core packages should work here.
 
 
 
Some tests contain their own inventory, that is their own instructions for turning a ''test subject'' into one or more testable systems. But in this case we use the default <code>standard-test-roles</code> inventory to do this.
 
 
 
The <code>--tags</code> argument filters out tests that are not suitable for running on an Atomic Host, either because the system functions differently, or the correct packages are not available on that system.
 
 
 
To diagnose why the tests failed, and log into the running Atomic Host, you can specify the following environment variable. After the playbook runs, you'll see diagnosis information with a helpful <code>ssh</code> command to log into the host:
 
 
 
<pre>
 
# export TEST_DEBUG=1
 
</pre>
 
 
 
To increase verbosity you can also use keys  <code>-v</code> or <code>-vvv</code> for full debugging, for example:
 
 
 
<pre>
 
# ansible-playbook --tags=atomic tests.yml -v
 
</pre>
 
  
or for full verbosity:
+
fedpkg clone -a qrencode
  
<pre>
+
Tests are enabled by including the <code>tests.yml</code> file under the <code>tests</code> directory:
# ansible-playbook --tags=atomic tests.yml -vvv
 
</pre>
 
  
{{admon/warning|Required Packages|Please note that if '''required_packages''' are specified in '''tests.yml''' for Atomic Host, additional packages will be installed using the '''rpm-ostree''' command which is affecting the test subject (it's similar as rebuilding an rpm package to be tested) so this should be used with caution and only when necessary. Also be aware that there are certain limitations for this approach (e.g. it's not possible to install different version of packages that are already part of the tree).}}
+
cd qrencode/tests
 +
cat tests.yml
  
{{admon/warning|Required Packages| Atomic Host is shipped as a base ostree image, however you can install additional packages with a help of <code>rpm-ostree install</code> command. Currently (10.01.2018 )  repo with additional packages is actual only for the latest base-ostree image. Consequence: tests that install additional packages for Atomic Host can fail sometimes with: <code>error: The following base packages would be replaced: xxx</code> Solution: make sure you have the latest Atomic Host image. Additional information you can find [https://github.com/projectatomic/rpm-ostree/issues/415 rpm-ostree issue 415] and a possible solution in the feature using <code>rpm-ostree jigdo</code> [https://github.com/projectatomic/rpm-ostree/issues/1081 rpm-ostree issue 1081] }}
+
Tests are wrapped or written as [http://docs.ansible.com/ansible/playbooks.html Ansible playbooks]. Here is an example of a simple playbok which enables a single <code>smoke</code> test of the <code>qrencode</code> package:
  
=== Debugging Atomic Host tests ===
+
- hosts: localhost
 +
  roles:
 +
  - role: standard-test-beakerlib
 +
    tags:
 +
    - classic
 +
    - container
 +
    - atomic
 +
    tests:
 +
    - smoke
 +
    required_packages:
 +
    - qrencode
 +
    - file
  
For debugging tests option ''TEST_DEBUG'' should be set to ''1''.
+
Let's now briefly look at the playbook to see which variables are defined in order to enable the smoke test:
To do that, execute command:
 
<pre>
 
# export TEST_DEBUG=1
 
</pre>
 
 
 
 
 
After that you can run tests i.e. execute command:
 
 
 
<pre>
 
# ansible-playbook --tags=atomic tests.yml
 
</pre>
 
 
 
You'll see output like this:
 
 
 
<pre>
 
PLAY RECAP ****************************************************************************************************************************************
 
/home/sturi/Programs/Images/atomic.qcow2 : ok=1    changed=0    unreachable=0    failed=1 
 
  
(.ansible) [root@kh226 docker]#
+
* '''role''' — this test uses role <code>standard-test-beakerlib</code> from [[CI/Standard_Test_Roles|Standard Test Roles]] to run a BeakerLib test
DIAGNOSE: ssh -p 2222 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root@127.0.0.3 # password: foobar
+
* '''tags''' — all three test subjects ([[CI/Standard_Test_Roles#Atomic|classic]] rpm, docker [[CI/Standard_Test_Roles#Container|container]] and [[CI/Standard_Test_Roles#Atomic|atomic]] host) are relevant for this test
DIAGNOSE: kill 13376 # when finished
+
* '''tests''' — list of tests to be executed (here we have just a single smoke test)
</pre>
+
* '''required_packages''' — list of rpm packages required for test execution
  
Look for ''DIAGNOSE'' there creds for the Virtual machine, IP address, username, password, and port.
+
There may by multiple files ending in <code>.yml</code> in the <code>tests/</code> subdirectory and each of them can represent a test or a part of a test. All of them need to be included in the main <code>tests.yaml</code> file. Let's have a look at the <code>gzip</code> example:
In our case:
 
  
* IP address: 127.0.0.3
+
> fedpkg clone -a gzip
* username/password: root/foobar
+
Cloning into 'gzip'...
* SSH port: 2222
 
  
Now you can connect to the VM using those creds. Open a new terminal and execute:
+
> cd gzip/tests/
 +
> ls
 +
test-simple  test_simple.yml  tests.yml
  
<pre>
+
> cat tests.yml
# ssh root@127.0.0.3 -p 2222
+
- include: test_simple.yml
</pre>
 
  
The output:
+
== Executing Tests ==
  
<pre>
+
Before running tests make sure you have the following dependencies installed on your system:
[root@kh226 docker]# ssh root@127.0.0.3 -p 2222
 
  
Are you sure you want to continue connecting (yes/no)? yes
+
dnf install ansible python2-dnf libselinux-python standard-test-roles
Warning: Permanently added '[127.0.0.3]:2222' (ECDSA) to the list of known hosts.
 
root@127.0.0.3's password:
 
[root@localhost ~]#
 
</pre>
 
  
=== Testing a Container Image ===
+
Although some playbooks may function without sudo, tests are always invoked as root. The test itself may set up users and/or drop permissions if a part of that test. But in general be sure to be root when invoking tests.
  
Another example is to use a ''test subject'' of a container image. This is also a fully formed and integrated deliverable. The ''test subject'' again represents the thing to be tested. The container image is pulled from a registry and launched using docker by an [http://docs.ansible.com/ansible/intro_dynamic_inventory.html Ansible dynamic inventory].
+
{{admon/important|Tests may modify or destroy your environment|It's recommended to use a virtual machine for testing to prevent any unwated changes performed by the test to your system.}}
  
<pre>
+
Running a test directly on the current system is easy:
# export ANSIBLE_INVENTORY=$(test -e inventory && echo inventory || echo /usr/share/ansible/inventory)
 
# export TEST_SUBJECTS=docker:docker.io/library/fedora:26
 
# ansible-playbook --tags=container tests.yml
 
</pre>
 
  
If you watch closely you'll notice the image is pulled if not already local, launched as a container, and then prepared for the tests to run on. The first time this may take a little longer. Not all tests are able to function in the somewhat different environment of a container. In fact, for certain tests, the software to be tested may not be included in the container. But many of the tests for core packages should work here.
+
ansible-playbook tests.yml
  
The <code>--tags</code> argument filters out tests that are not suitable for running in a container, either because the system functions differently, or the correct packages are not installable.
+
To only run tests that are suited for classic systems installed by <code>yum</code> or <code>dnf</code> use the <code>--tags</code> argument:
  
To diagnose why the tests failed, and log into the running container, you can specify the following environment variable. After the playbook runs, you'll see diagnosis information with a helpful <code>docker exec</code> command to log into the container:
+
ansible-playbook --tags=classic tests.yml
  
<pre>
+
See [[CI/Standard_Test_Roles|Standard Test Roles]] documentation for detailed instructions how to run tests for a specific [[CI/Standard_Test_Roles#Package|Rpm Package]], [[CI/Standard_Test_Roles#Container|Docker Container]] or [[CI/Standard_Test_Roles#Atomic|Atomic Host]].
# export TEST_DEBUG=1
 
</pre>
 
  
To increase verbosity you can also use keys  <code>-v</code> or <code>-vvv</code> for full debugging, for example:
+
== Adding Tests ==
<pre>
 
# ansible-playbook --tags=container tests.yml -v
 
</pre>
 
or for full verbosity:
 
<pre>
 
# ansible-playbook --tags=container tests.yml -vvv
 
</pre>
 
  
==== Additional arguments for Docker ====
+
Test code itself can be stored directly in the dist-git (recommended as default) or fetched from another repository hosted in the Fedora infrastructure. [[CI/Share_Test_Code|Test Namespace]] can be used for storing test code relevant for multiple packages. The simplest way to add a new test is by using one of the existing [[CI/Standard_Test_Roles|Standard Test Roles]] which take care of many implementatin details. If you want to create a custom test follow instructions below.
  
Tests for containers are run with a help of Docker. Containers are run within default security context.
+
Once you've identified a dist-git repository you will be adding new tests to (above), you can start to write a new Ansible test. Create an [http://docs.ansible.com/ansible/latest/playbooks.html Ansible playbook] with a new name. Make sure the extension is <code>.yml</code>. Lets place the following example in <code>test_pid_1.yml</code> file.
For more info see [https://docs.docker.com/engine/security/seccomp/ Seccomp security profiles for Docker].
 
It is possible that some tests require additional privileges. In this case specify necessary arguments for Docker using an environment variable '''TEST_DOCKER_EXTRA_ARGS'''. For this create a file '''inventory''' file in '''tests''' directory with the following content:
 
 
 
<pre>
 
#!/bin/bash
 
export TEST_DOCKER_EXTRA_ARGS="--security-opt seccomp:unconfined"
 
exec merge-standard-inventory "$@"
 
</pre>
 
 
 
or
 
 
 
<pre>
 
#!/bin/bash
 
export TEST_DOCKER_EXTRA_ARGS="--privileged"
 
exec merge-standard-inventory "$@"
 
</pre>
 
 
 
You can find nformation about '''merge-standard-inventory''' script you can find at: [https://pagure.io/standard-test-roles/blob/master/f/scripts/README.md README file for standard-test-roles]
 
 
== Adding tests ==
 
 
 
Tests are stored in package or module git repositories along with the packages and modules that they test. The tests are updated together with the software. If you're not a package maintainer be sure to talk with the maintainers or team and come to such a common understanding. See [[CI/Pull_Requests|Pull Requests]] for details about creating pull requests.
 
 
 
=== Untracked Files ===
 
 
 
A number of entries should be added to <code>.gitignore</code> file to specify intentionally untracked files that Git should ignore. Such files are created during tests run. Create a <code>tests/.gitignore</code> file with next contents:
 
 
 
<pre>
 
# Ignore tests runs/artefacts.
 
artifacts/**
 
**/*.retry
 
</pre>
 
 
 
=== Writing a new test ===
 
 
 
Once you've identified a dist-git repository you will be adding new tests to (above), you can start to write a new Ansible test. If you wish to wrap an existing test, see the sections below.
 
 
 
Create an [http://docs.ansible.com/ansible/latest/playbooks.html Ansible playbook] with a new name. Make sure the extension is <code>.yml</code>. Lets place the following example in <code>test_pid_1.yml</code> file.
 
  
 
<pre>
 
<pre>
Line 267: Line 104:
 
The <code>block</code> is the section that runs the actual test. In this example, we use a rather convoluted way of checking that PID 1 exists. However, by doing so, we place an extra test artifact in the artifacts directory.
 
The <code>block</code> is the section that runs the actual test. In this example, we use a rather convoluted way of checking that PID 1 exists. However, by doing so, we place an extra test artifact in the artifacts directory.
  
Lastly, we download the artifacts. Remember that the test is not always running on the same system that it was invoked on. Try running this example test against an Atomic Host or Docker image, using the examples above. It should pass. Try changing the <code>/proc/1</code> argument to another value, and the test should fail.
+
Lastly, we download the artifacts. Remember that the test is not always running on the same system that it was invoked on. Try running this example test against an [[CI/Standard_Test_Roles#Atomic|Atomic Host]] or [[CI/Standard_Test_Roles#Container|Docker Container]]. It should pass. Try changing the <code>/proc/1</code> argument to another value, and the test should fail.
  
 
You can use most of the Ansible techniques in your playbooks. And take a look at the [https://pagure.io/standard-test-roles standard-test-roles] for Ansible roles to make writing your tests easier.
 
You can use most of the Ansible techniques in your playbooks. And take a look at the [https://pagure.io/standard-test-roles standard-test-roles] for Ansible roles to make writing your tests easier.
  
=== Marking the test to be run ===
+
See [[CI/Pull_Requests|Pull Requests]] for details about creating pull requests.
  
Just having a <code>.yml</code> file in the right directory doesn't yet mean it will be invoked. Make sure to reference or add it from a <code>tests.yml</code> playbook. This is the entry point that the testing or CI system will use to invoke all the tests for a given package.
+
'''Marking the test to be run'''
 +
 
 +
Just having a <code>.yml</code> file in the right directory doesn't yet mean it will be invoked. Make sure to reference or add it from a <code>tests.yml</code> playbook. This is the entry point that the testing or CI system will use to invoke all the tests for a given package.
  
 
If the <code>tests.yml</code> file doesn't yet exist, create it. Lets continue with our above example and create a <code>tests.yml</code> with the following content:
 
If the <code>tests.yml</code> file doesn't yet exist, create it. Lets continue with our above example and create a <code>tests.yml</code> with the following content:
  
<pre>
+
- import_playbook: test_pid_1.yml
- import_playbook: test_pid_1.yml
 
</pre>
 
  
 
You can now run this test with the standard commands above.
 
You can now run this test with the standard commands above.
  
{{admon/important|Work in progress|Work is being done to define how to tag Ansible playbooks in such a way as to indicate whether a test is able to be run in an Atomic Host, a Docker image, and so on.}}
+
== Wrapping Tests ==
 
 
=== Wrapping Tests ===
 
  
 
Let's say you have a script that runs a test. Its stdout and stderr is the test output, and an exit status of zero indicates success. Here's how we would wrap that test to be invoked. Lets say we have a simple script like in a file called <code>test-simple</code>
 
Let's say you have a script that runs a test. Its stdout and stderr is the test output, and an exit status of zero indicates success. Here's how we would wrap that test to be invoked. Lets say we have a simple script like in a file called <code>test-simple</code>
  
<pre>
+
#!/bin/sh
#!/bin/sh
+
set -ex
set -ex
+
# exercise installed gzip/gunzip programs
# exercise installed gzip/gunzip programs
+
echo "Bla" > bla.file
echo "Bla" > bla.file
+
cp bla.file bla.file.orig
cp bla.file bla.file.orig
+
gzip bla.file
gzip bla.file
+
gunzip bla.file.gz
gunzip bla.file.gz
+
cmp bla.file bla.file.orig
cmp bla.file bla.file.orig
+
rm bla.file bla.file.orig
rm bla.file bla.file.orig
 
</pre>
 
  
 
We can write an Ansible wrapper for this script like this in <code>test_simple.yml</code>:
 
We can write an Ansible wrapper for this script like this in <code>test_simple.yml</code>:
Line 340: Line 173:
 
If the <code>tests.yml</code> file doesn't yet exist, create it. Lets continue with our above example and create a <code>tests.yml</code> with the following content:
 
If the <code>tests.yml</code> file doesn't yet exist, create it. Lets continue with our above example and create a <code>tests.yml</code> with the following content:
  
<pre>
+
- import_playbook: test_simple.yml
- import_playbook: test_simple.yml
 
</pre>
 
  
Try running this example test against an Atomic Host or Docker image, using the examples above. It should pass.
+
Try running this example test against an [[CI/Standard_Test_Roles#Atomic|Atomic Host]] or [[CI/Standard_Test_Roles#Container|Docker Container]]. It should pass.
  
See [[CI/Standard_Test_Roles|Standard Test Roles]] documentation
+
See [[CI/Standard_Test_Roles|Standard Test Roles]] documentation for instructions how to wrap a [[CI/Standard_Test_Roles#BeakerLib|BeakerLib]] and [[CI/Standard_Test_Roles#RHTS|RHTS]] tests.
for instructions how to wrap a
 
[[CI/Standard_Test_Roles#BeakerLib|BeakerLib]] and
 
[[CI/Standard_Test_Roles#RHTS|RHTS]] tests.
 
  
 
[[Category:FedoraAtomicCi]]
 
[[Category:FedoraAtomicCi]]

Revision as of 16:59, 7 March 2018

Enabling Tests

Tests may be written in different ways, but are exposed and invoked in a standard way as defined by the Standard Test Interface directly in the package git repository. To start working on tests you can clone a package repo directly:

git clone https://src.fedoraproject.org/rpms/qrencode.git

You can also use the fedpkg to clone the repo. See the Package Maintenance Guide for more info about the tool:

fedpkg clone -a qrencode

Tests are enabled by including the tests.yml file under the tests directory:

cd qrencode/tests
cat tests.yml

Tests are wrapped or written as Ansible playbooks. Here is an example of a simple playbok which enables a single smoke test of the qrencode package:

- hosts: localhost
  roles:
  - role: standard-test-beakerlib
    tags:
    - classic
    - container
    - atomic
    tests:
    - smoke
    required_packages:
    - qrencode
    - file

Let's now briefly look at the playbook to see which variables are defined in order to enable the smoke test:

  • role — this test uses role standard-test-beakerlib from Standard Test Roles to run a BeakerLib test
  • tags — all three test subjects (classic rpm, docker container and atomic host) are relevant for this test
  • tests — list of tests to be executed (here we have just a single smoke test)
  • required_packages — list of rpm packages required for test execution

There may by multiple files ending in .yml in the tests/ subdirectory and each of them can represent a test or a part of a test. All of them need to be included in the main tests.yaml file. Let's have a look at the gzip example:

> fedpkg clone -a gzip
Cloning into 'gzip'...
> cd gzip/tests/
> ls
test-simple  test_simple.yml  tests.yml
> cat tests.yml
- include: test_simple.yml

Executing Tests

Before running tests make sure you have the following dependencies installed on your system:

dnf install ansible python2-dnf libselinux-python standard-test-roles

Although some playbooks may function without sudo, tests are always invoked as root. The test itself may set up users and/or drop permissions if a part of that test. But in general be sure to be root when invoking tests.

Important.png
Tests may modify or destroy your environment
It's recommended to use a virtual machine for testing to prevent any unwated changes performed by the test to your system.

Running a test directly on the current system is easy:

ansible-playbook tests.yml

To only run tests that are suited for classic systems installed by yum or dnf use the --tags argument:

ansible-playbook --tags=classic tests.yml

See Standard Test Roles documentation for detailed instructions how to run tests for a specific Rpm Package, Docker Container or Atomic Host.

Adding Tests

Test code itself can be stored directly in the dist-git (recommended as default) or fetched from another repository hosted in the Fedora infrastructure. Test Namespace can be used for storing test code relevant for multiple packages. The simplest way to add a new test is by using one of the existing Standard Test Roles which take care of many implementatin details. If you want to create a custom test follow instructions below.

Once you've identified a dist-git repository you will be adding new tests to (above), you can start to write a new Ansible test. Create an Ansible playbook with a new name. Make sure the extension is .yml. Lets place the following example in test_pid_1.yml file.

---
- hosts: localhost
  vars:
  - artifacts: ./artifacts
  tags:
  - atomic
  - classic
  - container
  tasks:
  - name: Test block
    block:
      - name: Test that /proc/1 exists
        shell: ls /proc > /tmp/test.log && grep -qw 1 /tmp/test.log

    always:
      - name: Pull out the artifacts
        fetch:
          dest: "{{ artifacts }}/"
          src: "/tmp/test.log"
          flat: yes

All tests have an artifacts directory where they place their output. The testing or CI system that invokes the test will fill in this variable with a directory that it will archive. We ensure this directory exists in the test.

By use of tags we note what kind of systems this test is suitable to run on.

The block is the section that runs the actual test. In this example, we use a rather convoluted way of checking that PID 1 exists. However, by doing so, we place an extra test artifact in the artifacts directory.

Lastly, we download the artifacts. Remember that the test is not always running on the same system that it was invoked on. Try running this example test against an Atomic Host or Docker Container. It should pass. Try changing the /proc/1 argument to another value, and the test should fail.

You can use most of the Ansible techniques in your playbooks. And take a look at the standard-test-roles for Ansible roles to make writing your tests easier.

See Pull Requests for details about creating pull requests.

Marking the test to be run

Just having a .yml file in the right directory doesn't yet mean it will be invoked. Make sure to reference or add it from a tests.yml playbook. This is the entry point that the testing or CI system will use to invoke all the tests for a given package.

If the tests.yml file doesn't yet exist, create it. Lets continue with our above example and create a tests.yml with the following content:

- import_playbook: test_pid_1.yml

You can now run this test with the standard commands above.

Wrapping Tests

Let's say you have a script that runs a test. Its stdout and stderr is the test output, and an exit status of zero indicates success. Here's how we would wrap that test to be invoked. Lets say we have a simple script like in a file called test-simple

#!/bin/sh
set -ex
# exercise installed gzip/gunzip programs
echo "Bla" > bla.file
cp bla.file bla.file.orig
gzip bla.file
gunzip bla.file.gz
cmp bla.file bla.file.orig
rm bla.file bla.file.orig

We can write an Ansible wrapper for this script like this in test_simple.yml:

---
- hosts: localhost
  vars:
  - artifacts: ./artifacts
  tags:
  - atomic
  - classic
  - container
  remote_user: root
  tasks:
  - name: Install the test files
    copy: src={{ item.file }} dest=/usr/local/bin/{{ item.dest }} mode=0755
    with_items:
    - {file: test-simple, dest: test-simple }

  - name: Test block
    block:
      - name: Execute the tests
        shell: exec > /tmp/test.log 2>&1 && /usr/local/bin/test-simple

    always:
      - name: Pull out the logs
        fetch:
          dest: "{{ artifacts }}/"
          src: "/tmp/test.log"
          flat: yes

All tests have an artifacts directory where they place their output. The testing or CI system that invokes the test will fill in this variable with a directory that it will archive. We create ensure this directory exists in the test.

The block is the section that runs the actual test.

Lastly, we download the artifacts. Remember that the test is not always running on the same system that it was invoked on.

If the tests.yml file doesn't yet exist, create it. Lets continue with our above example and create a tests.yml with the following content:

- import_playbook: test_simple.yml

Try running this example test against an Atomic Host or Docker Container. It should pass.

See Standard Test Roles documentation for instructions how to wrap a BeakerLib and RHTS tests.