๐ Introduction
Are you eager to try out how the Fedora CI tests work? Do you want to get a quick hands-on experience without having to read too much documentation? This quick introduction for the impatient will show you a minimal set of steps to execute existing tests as well as provide useful links to resources where you can learn more.
๐ First Steps
Install the following essential packages on your system:
sudo dnf install fedpkg libselinux-python standard-test-roles
Use fedpkg
to clone the package git repository. See the Package Maintenance Guide for more info about the tool.
fedpkg clone -a bash
Tests are defined according to the Standard Test Interface in the tests
directory:
cd bash/tests/
Test coverage to be executed together with the basic set of metadata is described in the tests.yml playbook. Use ansible-playbook
to run all available tests for the classic environment on the local host:
ansible-playbook --tags=classic tests.yml
From the ansible output you can directly see an overall summary of the testing. If you see failed=0
at the end of the log then all tests passed:
localhost: ok=29 changed=11 unreachable=0 failed=0
For more detailed test results check the test.log
and other files in the artifacts
directory:
vim artifacts/test.log
That's it! You just executed test coverage for the Bash package :)
๐ Test Subjects
To execute tests against different test subjects we need to prepare the environment. Let's store the detailed test results in /tmp/artifacts
, use dynamic inventory as defined by the Standard Test Roles and download the latest Atomic Host image.
export TEST_ARTIFACTS=/tmp/artifacts export ANSIBLE_INVENTORY=/usr/share/ansible/inventory curl -Lo /tmp/atomic.qcow2 https://getfedora.org/atomic_qcow2_latest
Now let's try to run tests against all supported test subjects.
๐ Classic
Run tests against classic rpms installed on the system:
export TEST_SUBJECTS= ansible-playbook --tags=classic tests.yml
See Classic for detailed docs.
๐ Container
Run tests in a docker container:
export TEST_SUBJECTS=docker:docker.io/library/fedora:latest ansible-playbook --tags=container tests.yml
See Container for detailed docs.
๐ Atomic
Run tests against the Atomic Host:
export TEST_SUBJECTS=/tmp/atomic.qcow2 ansible-playbook --tags=atomic tests.yml
See Atomic for detailed docs.
๐ Hints
๐ Debug
Would you like to investigate why a test failed? Enable debugging to easilly connect to running Atomic or Container to investigate:
export TEST_DEBUG=1 ansible-playbook --tags=atomic tests.yml
See Debug for details about debugging.
๐ Ignore
Use .gitignore
to specify files that Git should ignore. Such files are created during tests run. Create a tests/.gitignore
file with the following contents:
# Ignore tests runs/artefacts. artifacts/** **/*.retry
๐ Contributing
Are you interested in contributing a new test coverage? You are most welcome! As you have seen Executing a test is quite easy. Writing a new test or Wrapping an existing one is quite simple as well. Here's a few recommendations for creating a new pull request.
Unless you are maintainer of the package, who has direct commit access, create a fork of the package git repository using the Fork button in Pagure web interface and add your private fork as a new remote. Create a branch for your new tests. For example:
git remote add fork ssh://psss@pkgs.fedoraproject.org/forks/psss/rpms/bash.git git checkout -b tests
Create new test coverage under the tests
directory, update the tests.yml
file accorgingly or create a new one. Run tests and verify they are stable and working fine in all supported environments. Add files to git, commit and push:
git add tests.yml test1 test2 test3 git commit -m "Add CI tests using the Standard Test Interface" git push fork tests:tests
It is a good idea to include more details and links in the commit message to make the pull request easier for review:
Add CI tests using the Standard Test Interface Adding initial set of basic functionality tests for bash according to the Standard Test Interface [1]. See Quick Start Guide for brief introduction about how to run these tests [2]. [1] https://fedoraproject.org/wiki/CI/Standard_Test_Interface [2] https://fedoraproject.org/wiki/CI/Quick_Start_Guide
Create a new pull request from your tests
branch against the master branch in the Pagure web interface. You might want to include an additional info about the tests such as:
There are three tests available: smoke and func have been tested across all environments (classic, container, atomic), login is relevant for classic only (because of a missing dependency). Please, merge the tests into all currently supported branches.