From Fedora Project Wiki

Tests are stored in package or module git repositories along with the packages and modules that they test. The tests are updated together with the software.

Setting up

Important.png
How to start contributing tests for Fedora Atomic Host CI
https://pagure.io/fork/psss/fedora-ci/AtomicCi/blob/start/f/start.md


You can use the fedpkg tool to checkout a package git repository. If a tests/ subdirectory exists, then the repository contains tests. The files ending in .yml in the tests/ subdirectory each represent a test or a part of a test.

The tests are wrapped or written as Ansible playbooks and invoked according to this specification. To invoke the tests you need the following dependencies (as described by the specification) installed on a modern Fedora system:

$ sudo dnf install ansible python2-dnf libselinux-python standard-test-roles

For the following documentation, we'll checkout the https://upstreamfirst.fedorainfracloud.org/gzip.git tests

$ git clone https://upstreamfirst.fedorainfracloud.org/gzip.git
$ cd gzip/
$ sudo -s

Although some playbooks may function without sudo, tests are always invoked as root. The test itself may set up users and/or drop permissions if a part of that test. But in general be sure to be root when invoking tests.

For the following documentation we'll checkout the gzip tests

Important.png
Work in progress
The upstreaming of tests is currently ongoing. Use the following command.
$ fedpkg clone gzip
$ cd gzip/tests/
$ sudo -s

Running tests

You can always invoke the tests locally. Many tests modify or change the system they are run against, so take that into account when looking at how to invoke tests. The following tests invoke tests against the same system that the package git repository is checked out on. Below there are further options for invoking tests against another fully formed and integrated systems, such as an Atomic Host or container image test subject.

There may be more than one test present in a package git repository, but the file tests.yml is the main entry point. To run it use the following command:

# ansible-playbook tests.yml

You can find output artifacts of the tests in an artifacts/ or specify a specific directory like this:

# ansible-playbook -e artifacts=/tmp/output tests.yml

You can filter which kinds of tests are run by providing a --tags argument. To only run tests that are suited for classic systems installed by yum or dnf you can use a command like:

# ansible-playbook --tags=classic tests.yml

When run by a CI System the tests are invoked according to a specification. Look here for more details and standard invocation variables.

Testing specific RPMs

When you run the tests as above, the tests assume that the system to be tested is the same as the system invoking the tests. In particular, the test assumes that the thing to be tested is already installed.

A test subject is what we call the thing to be tested. RPMs are a particular kind of test subject. To turn a test subject into a launched, installed system to be tested, we use Ansible dynamic inventory. Lets invoke the tests with an inventory and a specific version of gzip:

# export ANSIBLE_INVENTORY=$(test -e inventory && echo inventory || echo /usr/share/ansible/inventory)
# curl -o gzip.rpm https://kojipkgs.fedoraproject.org//packages/gzip/1.8/2.fc26/x86_64/gzip-1.8-2.fc26.x86_64.rpm
# export TEST_SUBJECTS=$PWD/gzip.rpm
# ansible-playbook tests.yml

You'll notice that the RPM is installed into the testable system before invoking the tests. Some tests contain their own inventory, that is their own instructions for turning a test subject into one or more testable systems. But in this case we use the default standard-test-roles inventory in /usr/share/ansible/inventory to do this.

Testing an Atomic Host

The former example may seem a bit contrived, but the concept of a test subject starts to make more sense when you want to test a fully formed and integrated deliverable, such as Atomic Host. The test subject again represents the thing to be tested. The test subject in this case is a QCow2 image. To turn a test subject into a launched system ready to be tested, we use Ansible dynamic inventory.

# export ANSIBLE_INVENTORY=$(test -e inventory && echo inventory || echo /usr/share/ansible/inventory)
# curl -Lo atomic.qcow2 https://ftp-stud.hs-esslingen.de/pub/Mirrors/alt.fedoraproject.org/atomic/stable/Fedora-Atomic-26-20170707.1/CloudImages/x86_64/images/Fedora-Atomic-26-20170707.1.x86_64.qcow2
# export TEST_SUBJECTS=$PWD/atomic.qcow2
# ansible-playbook --tags=atomic tests.yml

If you watch closely you'll see that the Atomic Host image is booted, and the tests run against the launched image. Not all tests are able to function in the somewhat different environment of Atomic Host, in fact, for certain cases, the software to be tested may not be included in the Atomic Host test subject. But most of the tests in core packages should work here.

Some tests contain their own inventory, that is their own instructions for turning a test subject into one or more testable systems. But in this case we use the default standard-test-roles inventory to do this.

The --tags argument filters out tests that are not suitable for running on an Atomic Host, either because the system functions differently, or the correct packages are not available on that system.

To diagnose why the tests failed, and log into the running Atomic Host, you can specify the following environment variable. After the playbook runs, you'll see diagnosis information with a helpful ssh command to log into the host:

# export TEST_DEBUG=1

To increase verbosity you can also use keys -v or -vvv for full debugging, for example:

# ansible-playbook --tags=atomic tests.yml -v

or for full verbosity:

# ansible-playbook --tags=atomic tests.yml -vvv

Debugging Atomic Host tests

For debugging tests option TEST_DEBUG should be set to 1. To do that, execute command:

# export TEST_DEBUG=1


After that you can run tests i.e. execute command:

# ansible-playbook --tags=atomic tests.yml

You'll see output like this:

PLAY RECAP ****************************************************************************************************************************************
/home/sturi/Programs/Images/atomic.qcow2 : ok=1    changed=0    unreachable=0    failed=1   

(.ansible) [root@kh226 docker]# 
DIAGNOSE: ssh -p 2222 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root@127.0.0.3 # password: foobar
DIAGNOSE: kill 13376 # when finished

Look for DIAGNOSE there creds for the Virtual machine, IP address, username, password, and port. In our case:

  • IP address: 127.0.0.3
  • username/password: root/foobar
  • SSH port: 2222

Now you can connect to the VM using those creds. Open a new terminal and execute:

# ssh root@127.0.0.3 -p 2222

The output:

[root@kh226 docker]# ssh root@127.0.0.3 -p 2222

Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[127.0.0.3]:2222' (ECDSA) to the list of known hosts.
root@127.0.0.3's password: 
[root@localhost ~]# 

Testing a Container Image

Another example is to use a test subject of a container image. This is also a fully formed and integrated deliverable. The test subject again represents the thing to be tested. The container image is pulled from a registry and launched using docker by an Ansible dynamic inventory.

# export ANSIBLE_INVENTORY=$(test -e inventory && echo inventory || echo /usr/share/ansible/inventory)
# export TEST_SUBJECTS=docker:docker.io/library/fedora:26
# ansible-playbook --tags=container tests.yml

If you watch closely you'll notice the image is pulled if not already local, launched as a container, and then prepared for the tests to run on. The first time this may take a little longer. Not all tests are able to function in the somewhat different environment of a container. In fact, for certain tests, the software to be tested may not be included in the container. But many of the tests for core packages should work here.

The --tags argument filters out tests that are not suitable for running in a container, either because the system functions differently, or the correct packages are not installable.

To diagnose why the tests failed, and log into the running container, you can specify the following environment variable. After the playbook runs, you'll see diagnosis information with a helpful docker exec command to log into the container:

# export TEST_DEBUG=1

To increase verbosity you can also use keys -v or -vvv for full debugging, for example:

# ansible-playbook --tags=container tests.yml -v

or for full verbosity:

# ansible-playbook --tags=container tests.yml -vvv

Adding tests

Tests are stored in package or module git repositories along with the packages and modules that they test. The tests are updated together with the software. If you're not a package maintainer be sure to talk with the maintainers or team and come to such a common understanding.

Repository for test

Important.png
Work in progress
One can contribute to dist-git packages without being the maintainer by using a Pagure interface. It is not yet ready. Place tests in the following location. src.fedoraproject.org

At the current time, you need to find or create a repository for the tests in the src.fedoraproject.org repositories. The name should be identical to the dist-git repository of the package or module that you are targeting.

  1. Use the Browse button to find existing repos.
  2. If a repository with the name of the target dist-git repository does exist, use the [+] create a button to make a new one.
  3. Use the Project name that is identical to the name of the dist-git repository.
  4. Leave the Create README option unchecked

These repositories will soon be moved into the real package dist-git repositories. The contents of each repository will go into the tests/ folder of the target dist-git repository.

Creating pull request

Create a repository on the fedorapeople.org. How to create a repository on the fedorapeople.org After that, you will be able to connect by SSH, for example:

$ ssh -v <user name>@fedorapeople.org

There are should be *public_git* folder created. Inside the *public_git* folder new git repositories can be created. To check that everything works and ready for pull request by cloning created repository to your local machine. For example, for docker repository:

[username@local_pc Temp]$ git clone git://fedorapeople.org/~username/docker.git
Cloning into 'docker'...
.
.
.
[username@local_pc Temp]$ ls -la
total 16
drwxrwxr-x.  3 username username 4096 Sep 28 15:46 .
drwx--x---+ 29 username username 4096 Sep 28 07:40 ..
drwxrwxr-x.  4 username username 4096 Sep 28 15:46 docker
[username@local_pc Temp]$ 

Now we are ready to make pull request. Navigate to the src.fedoraproject.org repositories and create a new pull request.

Idea.png
Working with Pull Requests
You may encounter a situation where you want to include changes from the master branch that were made after you created your pull request. Follow the article Working with Pull Requests

Writing a new test

Once you've identified a dist-git repository you will be adding new tests to (above), you can start to write a new Ansible test. If you wish to wrap an existing test, see the sections below.

Create an Ansible playbook with a new name. Make sure the extension is .yml. Lets place the following example in test_pid_1.yml file.

---
- hosts: localhost
  vars:
  - artifacts: ./artifacts
  tags:
  - atomic
  - classic
  - container
  tasks:
  - name: Test block
    block:
      - name: Test that /proc/1 exists
        shell: ls /proc > /tmp/test.log && grep -qw 1 /tmp/test.log

    always:
      - name: Pull out the artifacts
        fetch:
          dest: "{{ artifacts }}/"
          src: "/tmp/test.log"
          flat: yes

All tests have an artifacts directory where they place their output. The testing or CI system that invokes the test will fill in this variable with a directory that it will archive. We ensure this directory exists in the test.

By use of tags we note what kind of systems this test is suitable to run on.

The block is the section that runs the actual test. In this example, we use a rather convoluted way of checking that PID 1 exists. However, by doing so, we place an extra test artifact in the artifacts directory.

Lastly, we download the artifacts. Remember that the test is not always running on the same system that it was invoked on. Try running this example test against an Atomic Host or Docker image, using the examples above. It should pass. Try changing the /proc/1 argument to another value, and the test should fail.

You can use most of the Ansible techniques in your playbooks. And take a look at the standard-test-roles for Ansible roles to make writing your tests easier.

Marking the test to be run

Just having a .yml file in the right directory doesn't yet mean it will be invoked. Make sure to reference or add it from a tests.yml playbook. This is the entry point that the testing or CI system will use to invoke all the tests for a given package.

If the tests.yml file doesn't yet exist, create it. Lets continue with our above example and create a tests.yml with the following content:

- include: test_pid_1.yml

You can now run this test with the standard commands above.

Important.png
Work in progress
Work is being done to define how to tag Ansible playbooks in such a way as to indicate whether a test is able to be run in an Atomic Host, a Docker image, and so on.

Wrapping a script test

Let's say you have a script that runs a test. Its stdout and stderr is the test output, and an exit status of zero indicates success. Here's how we would wrap that test to be invoked. Lets say we have a simple script like in a file called test-simple

#!/bin/sh
set -ex
# exercise installed gzip/gunzip programs
echo "Bla" > bla.file
cp bla.file bla.file.orig
gzip bla.file
gunzip bla.file.gz
cmp bla.file bla.file.orig
rm bla.file bla.file.orig

We can write an Ansible wrapper for this script like this in test-simple.yml:

---
- hosts: localhost
  vars:
  - artifacts: ./artifacts
  tags:
  - atomic
  - classic
  - container
  remote_user: root
  tasks:
  - name: Install the test files
    copy: src={{ item.file }} dest=/usr/local/bin/{{ item.dest }} mode=0755
    with_items:
    - {file: test-simple, dest: test-simple }

  - name: Test block
    block:
      - name: Execute the tests
        shell: exec > /tmp/test.log 2>&1 && /usr/local/bin/test-simple

    always:
      - name: Pull out the logs
        fetch:
          dest: "{{ artifacts }}/"
          src: "/tmp/test.log"
          flat: yes

All tests have an artifacts directory where they place their output. The testing or CI system that invokes the test will fill in this variable with a directory that it will archive. We create ensure this directory exists in the test.

The block is the section that runs the actual test. In this example, we use a rather convoluted way of checking that PID 1 exists. However, by doing so, we place an extra test artifact in the artifacts directory.

Lastly, we download the artifacts. Remember that the test is not always running on the same system that it was invoked on.

If the tests.yml file doesn't yet exist, create it. Lets continue with our above example and create a tests.yml with the following content:

- include: test_pid_1.yml

Try running this example test against an Atomic Host or Docker image, using the examples above. It should pass. Try changing the /proc/1 argument to another value, and the test should fail.

Wrapping an installed test

TODO

Wrapping a beakerlib test

If you have a set of beakerlib tests, it is recommended to place each test in its own subdirectory.

Then, create tests.yml file with contents similar to the following which runs tests using the 'standard-test-beakerlib' role included in the standard-test-roles package. The list provided for the 'tests' parameter should be the list of names of your per-test subdirectories, and the 'required_packages' parameter should contain a list of additional packages that need to be installed to run the tests.

---
- hosts: localhost
  tags:
  - atomic
  - classic
  - container
  roles:
  - role: standard-test-beakerlib
    tests:
    - cmd-line-options
    required_packages:
    - which         # which package required for cmd-line-options
    - rpm-build     # upstream-testsuite requires rpmbuild command
    - libtool       # upstream-testsuite requires libtool
    - gettext       # upstream-testsuite requires gettext

Note: The 'required_packages' parameter is ignored when running on Atomic Host--since there is no way to install additional packages in that environment.

Wrapping a Restraint Test

If you have a set of tests you want to run using restraint, each test must be placed in its own subdirectory.

Then, create tests.yml file with contents similar to the following which runs tests using the 'standard-test-rhts' role included in the standard-test-roles package. The list provided for the 'tests' parameter should be the list of names of your per-test subdirectories, and the 'required_packages' parameter should contain a list of additional packages that need to be installed to run the tests.

---
- hosts: localhost
  tags:
  - classic
  - docker
  roles:
  - role: standard-test-rhts
    tests:
    - cmd-line-options
    required_packages:
    - which         # which package required for cmd-line-options
    - rpm-build     # upstream-testsuite requires rpmbuild command
    - libtool       # upstream-testsuite requires libtool
    - gettext       # upstream-testsuite requires gettext

Note: Tests using the 'standard-test-rhts' role are not compatible with Atomic Host--since it requires the installation of additional packages and there is no way to do so in that environment. We reflect that by omitting atomic from the <tags> section.