From Fedora Project Wiki

< CI

(Pull Request workaround warning)
(Contributing, long lines)
Line 1: Line 1:
 
= Introduction =
 
= Introduction =
  
Are you eager to try out how the Fedora CI tests work? Do you want
+
Are you eager to try out how the Fedora CI tests work? Do you want to get a quick hands-on experience without having to read too much documentation? This quick introduction for the impatient will show you a minimal set of steps to execute existing tests as well as provide useful links to resources where you can learn more.
to get a quick hands-on experience without having to read too much
 
documentation? This quick introduction for the impatient will show
 
you a minimal set of steps to execute existing tests as well as
 
provide useful links to resources where you can learn more.
 
  
 
= First Steps =
 
= First Steps =
Line 13: Line 9:
 
  sudo dnf install fedpkg libselinux-python standard-test-roles
 
  sudo dnf install fedpkg libselinux-python standard-test-roles
  
Use <code>fedpkg</code> to clone the package git repository. See
+
Use <code>fedpkg</code> to clone the package git repository. See the [[Package_maintenance_guide|Package Maintenance Guide]] for more info about the tool.
the [[Package_maintenance_guide|Package Maintenance Guide]] for
 
more info about the tool.
 
  
 
  fedpkg clone -a bash
 
  fedpkg clone -a bash
  
Tests are defined according to the
+
Tests are defined according to the [[CI/Standard_Test_Interface|Standard Test Interface]] in the <code>tests</code> directory:
[[CI/Standard_Test_Interface|Standard Test Interface]] in the
 
<code>tests</code> directory:
 
  
 
  cd bash/tests/
 
  cd bash/tests/
  
Test coverage to be executed together with the basic set of
+
Test coverage to be executed together with the basic set of metadata is described in the [https://src.fedoraproject.org/rpms/bash/blob/master/f/tests/tests.yml tests.yml] playbook. Use <code>ansible-playbook</code> to run all available tests for the classic environment on the local host:
metadata is described in the
 
[https://src.fedoraproject.org/rpms/bash/blob/master/f/tests/tests.yml tests.yml]
 
playbook. Use <code>ansible-playbook</code> to run all available
 
tests for the classic environment on the local host:
 
  
 
  ansible-playbook --tags=classic tests.yml
 
  ansible-playbook --tags=classic tests.yml
  
From the ansible output you can directly see an overall summary of
+
From the ansible output you can directly see an overall summary of the testing. If you see <code>failed=0</code> at the end of the log then all tests passed:
the testing. If you see <code>failed=0</code> at the end of the
 
log then all tests passed:
 
  
 
  localhost: ok=29 changed=11 unreachable=0 failed=0
 
  localhost: ok=29 changed=11 unreachable=0 failed=0
  
For more detailed test results check the <code>test.log</code> and
+
For more detailed test results check the <code>test.log</code> and other files in the <code>artifacts</code> directory:
other files in the <code>artifacts</code> directory:
 
  
 
  vim artifacts/test.log
 
  vim artifacts/test.log
Line 48: Line 33:
 
= Test Subjects =
 
= Test Subjects =
  
To execute tests against different test subjects we need to
+
To execute tests against different test subjects we need to prepare the environment. Let's store the detailed test results in <code>/tmp/artifacts</code>, use dynamic inventory as defined by the [[CI/Standard_Test_Roles|Standard Test Roles]] and download the latest Atomic Host image.
prepare the environment. Let's store the detailed test results in
 
<code>/tmp/artifacts</code>, use dynamic inventory as defined by
 
the [[CI/Standard_Test_Roles|Standard Test Roles]] and download
 
the latest Atomic Host image.
 
  
 
  export TEST_ARTIFACTS=/tmp/artifacts
 
  export TEST_ARTIFACTS=/tmp/artifacts
Line 91: Line 72:
 
== Debug ==
 
== Debug ==
  
Would you like to investigate why a test failed? Enable debugging
+
Would you like to investigate why a test failed? Enable debugging to easilly connect to running Atomic or Container to investigate:
to easilly connect to running Atomic or Container to investigate:
 
  
 
  export TEST_DEBUG=1
 
  export TEST_DEBUG=1
 
  ansible-playbook --tags=atomic tests.yml
 
  ansible-playbook --tags=atomic tests.yml
  
See [[CI/Standard_Test_Roles#Debug|Debug]] for details about
+
See [[CI/Standard_Test_Roles#Debug|Debug]] for details about debugging.
debugging.
 
  
 
== Ignore ==
 
== Ignore ==
  
Use <code>.gitignore</code> to specify files that Git should
+
Use <code>.gitignore</code> to specify files that Git should ignore. Such files are created during tests run. Create a <code>tests/.gitignore</code> file with the following contents:
ignore. Such files are created during tests run. Create a
 
<code>tests/.gitignore</code> file with the following contents:
 
  
 
  # Ignore tests runs/artefacts.
 
  # Ignore tests runs/artefacts.
Line 110: Line 87:
 
  **/*.retry
 
  **/*.retry
  
= Adding Tests =
+
= Contributing =
  
Unless you are maintainer of the package, who has direct commit
+
Are you interested in contributing a new test coverage? You are most welcome! As you have seen [[CI/Tests#Executing|Executing]] a test is quite easy. [[CI/Tests#Writing|Writing]] a new test or [[CI/Tests#Wrapping|Wrapping]] an existing one is quite simple as well. Here's a few recommendations for creating a new pull request.
access, create a fork of the package git repository using the Fork
+
 
button in [https://src.fedoraproject.org/rpms/bash Pagure] web
+
Unless you are maintainer of the package, who has direct commit access, create a fork of the package git repository using the Fork button in [https://src.fedoraproject.org/rpms/bash Pagure] web interface and add your private fork as a new remote. Create a branch for your new tests. For example:
interface and add your private fork as a new remote. Create a
 
branch for your new tests. For example:
 
  
 
  git remote add fork ssh://psss@pkgs.fedoraproject.org/forks/psss/rpms/bash.git
 
  git remote add fork ssh://psss@pkgs.fedoraproject.org/forks/psss/rpms/bash.git
 
  git checkout -b tests
 
  git checkout -b tests
  
Create new test coverage under the <code>tests</code> directory,
+
Create new test coverage under the <code>tests</code> directory, update the <code>tests.yml</code> file accorgingly or create a new one. Run tests and verify they are stable and working fine in all supported environments. Add files to git, commit and push:
update the <code>tests.yml</code> file accorgingly or create a new
 
one. Run tests and verify they are stable and working fine in all
 
supported environments. Add files to git, commit and push:
 
  
 
  git add tests.yml test1 test2 test3
 
  git add tests.yml test1 test2 test3
Line 130: Line 102:
 
  git push fork tests:tests
 
  git push fork tests:tests
  
It is a good idea to include more details and links in the commit
+
It is a good idea to include more details and links in the commit message to make the pull request easier for review:
message to make the pull request easier for review:
 
  
 
  Add CI tests using the Standard Test Interface
 
  Add CI tests using the Standard Test Interface
Line 142: Line 113:
 
  [2] https://fedoraproject.org/wiki/CI/Quick_Start_Guide
 
  [2] https://fedoraproject.org/wiki/CI/Quick_Start_Guide
  
Create a new pull request from your <code>tests</code> branch
+
Create a new pull request from your <code>tests</code> branch against the master branch in the [https://src.fedoraproject.org/fork/psss/rpms/bash Pagure] web interface. You might want to include an additional info about the tests such as:
against the master branch in the
 
[https://src.fedoraproject.org/fork/psss/rpms/bash Pagure] web
 
interface. You might want to include an additional info about the
 
tests such as:
 
  
 
  There are three tests available: smoke and func have been tested
 
  There are three tests available: smoke and func have been tested

Revision as of 10:37, 8 March 2018

Introduction

Are you eager to try out how the Fedora CI tests work? Do you want to get a quick hands-on experience without having to read too much documentation? This quick introduction for the impatient will show you a minimal set of steps to execute existing tests as well as provide useful links to resources where you can learn more.

First Steps

Install the following essential packages on your system:

sudo dnf install fedpkg libselinux-python standard-test-roles

Use fedpkg to clone the package git repository. See the Package Maintenance Guide for more info about the tool.

fedpkg clone -a bash

Tests are defined according to the Standard Test Interface in the tests directory:

cd bash/tests/

Test coverage to be executed together with the basic set of metadata is described in the tests.yml playbook. Use ansible-playbook to run all available tests for the classic environment on the local host:

ansible-playbook --tags=classic tests.yml

From the ansible output you can directly see an overall summary of the testing. If you see failed=0 at the end of the log then all tests passed:

localhost: ok=29 changed=11 unreachable=0 failed=0

For more detailed test results check the test.log and other files in the artifacts directory:

vim artifacts/test.log

That's it! You just executed test coverage for the Bash package :)

Test Subjects

To execute tests against different test subjects we need to prepare the environment. Let's store the detailed test results in /tmp/artifacts, use dynamic inventory as defined by the Standard Test Roles and download the latest Atomic Host image.

export TEST_ARTIFACTS=/tmp/artifacts
export ANSIBLE_INVENTORY=/usr/share/ansible/inventory
curl -Lo /tmp/atomic.qcow2 https://getfedora.org/atomic_qcow2_latest

Now let's try to run tests against all supported test subjects.

Classic

Run tests against classic rpms installed on the system:

export TEST_SUBJECTS=
ansible-playbook --tags=classic tests.yml

See Classic for detailed docs.

Container

Run tests in a docker container:

export TEST_SUBJECTS=docker:docker.io/library/fedora:latest
ansible-playbook --tags=container tests.yml

See Container for detailed docs.

Atomic

Run tests against the Atomic Host:

export TEST_SUBJECTS=/tmp/atomic.qcow2
ansible-playbook --tags=atomic tests.yml

See Atomic for detailed docs.

Hints

Debug

Would you like to investigate why a test failed? Enable debugging to easilly connect to running Atomic or Container to investigate:

export TEST_DEBUG=1
ansible-playbook --tags=atomic tests.yml

See Debug for details about debugging.

Ignore

Use .gitignore to specify files that Git should ignore. Such files are created during tests run. Create a tests/.gitignore file with the following contents:

# Ignore tests runs/artefacts.
artifacts/**
**/*.retry

Contributing

Are you interested in contributing a new test coverage? You are most welcome! As you have seen Executing a test is quite easy. Writing a new test or Wrapping an existing one is quite simple as well. Here's a few recommendations for creating a new pull request.

Unless you are maintainer of the package, who has direct commit access, create a fork of the package git repository using the Fork button in Pagure web interface and add your private fork as a new remote. Create a branch for your new tests. For example:

git remote add fork ssh://psss@pkgs.fedoraproject.org/forks/psss/rpms/bash.git
git checkout -b tests

Create new test coverage under the tests directory, update the tests.yml file accorgingly or create a new one. Run tests and verify they are stable and working fine in all supported environments. Add files to git, commit and push:

git add tests.yml test1 test2 test3
git commit -m "Add CI tests using the Standard Test Interface"
git push fork tests:tests

It is a good idea to include more details and links in the commit message to make the pull request easier for review:

Add CI tests using the Standard Test Interface

Adding initial set of basic functionality tests for bash
according to the Standard Test Interface [1]. See Quick Start
Guide for brief introduction about how to run these tests [2].

[1] https://fedoraproject.org/wiki/CI/Standard_Test_Interface
[2] https://fedoraproject.org/wiki/CI/Quick_Start_Guide

Create a new pull request from your tests branch against the master branch in the Pagure web interface. You might want to include an additional info about the tests such as:

There are three tests available: smoke and func have been tested
across all environments (classic, container, atomic), login is
relevant for classic only (because of a missing dependency).
Please, merge the tests into all currently supported branches.
Important.png
Pull Requests Workaround
If you are not a member of the Fedora packager group see the Pull Requests page for a temporary workaround to create a pull request from a remote repository.