From Fedora Project Wiki

< CI

(Added links to tests/ruby repository)
(Add "Testing Tests" section)
 
(7 intermediate revisions by 2 users not shown)
Line 26: Line 26:
  
 
* https://src.fedoraproject.org/projects/tests/*
 
* https://src.fedoraproject.org/projects/tests/*
 +
 +
Use <code>fedpkg</code> to quickly clone repositories from the
 +
tests namespace:
 +
 +
fedpkg clone tests/shell
  
 
Some of the [[CI/Standard_Test_Roles|Standard Test Roles]]
 
Some of the [[CI/Standard_Test_Roles|Standard Test Roles]]
(currently beakerlib and rhts) support fetching test code from
+
(currently basic and beakerlib) support fetching test code from
 
remote repositories directly in their config in this way:
 
remote repositories directly in their config in this way:
  
Line 35: Line 40:
 
   - repo: "https://src.fedoraproject.org/tests/shell.git"
 
   - repo: "https://src.fedoraproject.org/tests/shell.git"
 
     dest: "shell"
 
     dest: "shell"
 +
 +
It is also possible to specify version (branch, commit hash) which should
 +
be fetched from the remote repository:
 +
 +
- role: standard-test-beakerlib
 +
  repositories:
 +
  - repo: "https://src.fedoraproject.org/tests/shell.git"
 +
    dest: "shell"
 +
    version: "devel"
 +
 +
= Testing Tests =
 +
 +
It is a good idea to ensure that updating tests in the shared
 +
repository does not negatively impact packages which they are
 +
testing. To enable pull request pipeline for tests stored in the
 +
Fedora dist git tests namespace simply include `tests.yml` file in
 +
the root of the test repository.
 +
 +
- hosts: localhost
 +
  roles:
 +
  - role: standard-test-basic
 +
    tags:
 +
    - classic
 +
    tests:
 +
    - smoke27:
 +
        dir: smoke
 +
        run: VERSION=2.7 METHOD=virtualenv ./venv.sh
 +
    - smoke36:
 +
        dir: smoke
 +
        run: VERSION=3.6 METHOD=virtualenv ./venv.sh
 +
    - smoke37:
 +
        dir: smoke
 +
        run: VERSION=3.7 METHOD=virtualenv ./venv.sh
 +
    required_packages:
 +
    - python27
 +
    - python36
 +
    - python37
 +
 +
The example above is a simplified version of the
 +
[https://src.fedoraproject.org/tests/python/blob/master/f/tests.yml tests.yml]
 +
file from the Python shared test repo and shows how to enable
 +
`smoke` test to be executed against three versions of the Python
 +
interpreter.
  
 
= Examples =
 
= Examples =
Line 101: Line 149:
 
maintaining those tests in 80 repos would be a tedious task.
 
maintaining those tests in 80 repos would be a tedious task.
  
Currently the shared [[https://src.fedoraproject.org/tests/ruby|tests/ruby]] repository hosts these three
+
Currently the shared [https://src.fedoraproject.org/tests/ruby tests/ruby]
ruby integration tests:
+
repository hosts these three ruby integration tests:
  
 
Available integration tests:
 
Available integration tests:
Line 122: Line 170:
 
= Start =
 
= Start =
  
In order to create a new repository in the tests namespace file a
+
In order to create a new repository in the tests namespace use the fedpkg's request-tests-repo command. For example to create a shared test repository with the name foo, which will be available at [https://src.fedoraproject.org/tests/foo.git]  
[https://pagure.io/fedora-infrastructure/new_issue new issue] in
+
 
the fedora infrastructure queue. Specify desired repository name,
+
* Setup authentication to pagure according to the help in request-repo command
main admin and a short description/justification. For example:
 
  
* https://pagure.io/fedora-infrastructure/issue/6699
+
fedpkg request-repo -h
  
Creating new test repositories will be soon supported directly in
+
* Request a new repository with a sensible decription
the <code>fedpkg</code> tool as well:
 
  
* https://pagure.io/fedpkg/issue/176
+
fedpkg --module-name foo request-tests-repo "Description of the foo repository"

Latest revision as of 13:12, 27 November 2018

Motivation

In order to make the CI workflow reliable and efficient it is crucial to keep the test coverage in a good shape at all times. Sharing test code between several packages (even within multiple branches of the same package) may significantly help to:

  • Prevent test code duplication
  • Minimize test maintenance
  • Catch incompatibilities early

In general, tests define how the software works and the basic functionality of many packages doesn’t change that often. We try hard to keep the backward compatibility where possible. Thus it seems natural that, for such components, tests guarding the spec could change at a slower pace than the distribution branches.

See the whole ci-list discussion for some more context.

Implementation

Store test code in your preferred repository and reference the tests from the dist-git yaml file. There is also a special tests namespace dedicated for storing Fedora CI integration tests:

Use fedpkg to quickly clone repositories from the tests namespace:

fedpkg clone tests/shell

Some of the Standard Test Roles (currently basic and beakerlib) support fetching test code from remote repositories directly in their config in this way:

- role: standard-test-beakerlib
  repositories:
  - repo: "https://src.fedoraproject.org/tests/shell.git"
    dest: "shell"

It is also possible to specify version (branch, commit hash) which should be fetched from the remote repository:

- role: standard-test-beakerlib
  repositories:
  - repo: "https://src.fedoraproject.org/tests/shell.git"
    dest: "shell"
    version: "devel"

Testing Tests

It is a good idea to ensure that updating tests in the shared repository does not negatively impact packages which they are testing. To enable pull request pipeline for tests stored in the Fedora dist git tests namespace simply include tests.yml file in the root of the test repository.

- hosts: localhost
  roles:
  - role: standard-test-basic
    tags:
    - classic
    tests:
    - smoke27:
        dir: smoke
        run: VERSION=2.7 METHOD=virtualenv ./venv.sh
    - smoke36:
        dir: smoke
        run: VERSION=3.6 METHOD=virtualenv ./venv.sh
    - smoke37:
        dir: smoke
        run: VERSION=3.7 METHOD=virtualenv ./venv.sh
    required_packages:
    - python27
    - python36
    - python37

The example above is a simplified version of the tests.yml file from the Python shared test repo and shows how to enable smoke test to be executed against three versions of the Python interpreter.

Examples

Here are some real-life examples where sharing test code can increase long-term efficiency.

Shell

There are several shells which implement the POSIX specification: bash, ksh, mksh, zsh, dash. All of them share a significant amount of test coverage and it does not make sense to commit & maintain identical tests in five different repositories (+ possible branches).

Shell tests repository:

Bash tests.yml:

- hosts: localhost
  roles:
  - role: standard-test-beakerlib
    tags:
    - classic
    repositories:
    - repo: "https://src.fedoraproject.org/tests/shell.git"
      dest: "shell"
    tests:
    - shell/func
    - shell/login
    - shell/smoke
    required_packages:
    - expect            # login requires expect
    - which             # smoke requires which

Ksh tests.yml:

- hosts: localhost
  roles:
  - role: standard-test-beakerlib
    tags:
    - classic
    repositories:
    - repo: "https://src.fedoraproject.org/tests/shell.git"
      dest: "shell"
    tests:
    - shell/func
    - shell/login
    - shell/smoke
    environment:
      PACKAGES: ksh
      SH_BIN: ksh
    required_packages:
    - ksh
    - expect            # login requires expect
    - which             # smoke requires which

Ruby

Another example is Ruby: With about 80 packages related to Ruby on Rails it would be useful and efficient to have a single place for integration tests which verify that the framework is correctly working after updating any of these packages. Conversely, maintaining those tests in 80 repos would be a tedious task.

Currently the shared tests/ruby repository hosts these three ruby integration tests:

Available integration tests:

  • systemtap-static-probes-in-ruby - exercising ruby's systemtap api
  • bundler-unit-test - run bundler's unit tests
  • run-basic-rails-application - run a simple rails application

SELinux

Several SELinux user space components are sharing test coverage in a single selinux test repository:

Start

In order to create a new repository in the tests namespace use the fedpkg's request-tests-repo command. For example to create a shared test repository with the name foo, which will be available at [1]

  • Setup authentication to pagure according to the help in request-repo command
fedpkg request-repo -h
  • Request a new repository with a sensible decription
fedpkg --module-name foo request-tests-repo "Description of the foo repository"