From Fedora Project Wiki

Fedora uses the openQA automated testing system as a significant part of the release validation testing process, and for testing updates. On this page you can find more information about openQA, how Fedora uses it, and how to install your own instance of openQA so you can try it out and contribute to test development.

General information

openQA is an automated test system designed around operating system-level testing. Its key feature is that it interacts with the system under test much like a human would, by sending input using a virtual keyboard and mouse (or serial console input), and monitoring the screen, serial console and audio outputs for output. This makes it ideally suited to running operating system-level tests without making any modifications to the software being tested.

For much more information on openQA, see the upstream documentation.

How Fedora uses openQA

There are currently two official Fedora openQA deployments: production and staging.

At present, Fedora uses openQA for release validation type testing and for update testing. A set of tests oriented around testing a complete Fedora distribution compose are run:

A subset of tests is run on:

A set of tests are also run on some of the updates under test for Fedora Rawhide, Branched and stable releases. This set contains a subset of the release validation tests plus some update-specific tests, including tests that build live, installer and rpm-ostree (Silverblue) images with the update included and test that they work. The tests are run on all critical path updates, and for some other specific 'allowlisted' packages (where the testing seems specifically relevant and valuable) - they are not run on all updates as we do not have sufficient testing resources for this. The results of these tests are visible on the Automated Tests tab of the update page, for updates which are tested. Critical path updates for all releases (including Rawhide) are 'gated' on some of these test results (the update cannot be pushed stable if a test fails, unless the failure is waived). 'Allowlisted' packages can opt into gating if they desire.

The results of all openQA tests are forwarded to ResultsDB. When there is a release validation test event for a compose, openQA results are mapped to Wikitcms test cases and reported to the appropriate wiki pages: results you see from the user coconut are results derived from openQA testing.

An email summary of most of the compose runs is produced and mailed to test and devel by the check-compose tool.

The fedora_nightlies tool which produces the nightly compose finder page considers openQA results in determining the test pass/fail status of the images it tracks.

Fedora openQA tests and tools

The Fedora openQA tests can be found in the os-autoinst-distri-fedora repository. The library, CLI and fedora-messaging consumers that handle scheduling openQA jobs and forwarding results to ResultsDB and Wikitcms can be found in the fedora_openqa repository. The createhdds tool used to create disk image files required by some of the tests can be found in the createhdds repository.

Each of these repositories includes documentation relating to its specific role in the overall system. The fedora_openqa repository includes instructions on installing and using the scheduler and report forwarding features, and the os-autoinst-distri-fedora repository includes Fedora-specific instructions for test development and documentation of Fedora test conventions, functions and settings.

The official Fedora openQA deployments are managed via Ansible playbooks in the Fedora Infrastructure ansible repository. The roles can be found in roles/openqa, and we attempt to maintain them in such a way that you can use them to deploy openQA instances outside of Fedora infrastructure, though this is not tested very often and may bitrot from time to time.

Installing openQA

Running tests directly

To run the standard set of Fedora tests on a regular Fedora (or Cloud, IoT, CoreOS or respin) compose, a Fedora update, or a scratch build, you can use the fedora-openqa tool in the fedora_openqa repository. A typical run of the tool will download all testable image files from the compose or all packages from the update or scratch build, and run all the tests for each image or all the update tests for all arches for which the openQA instance is configured. The tool also has options to limit execution to specific arches and/or 'flavors' (for composes, these mostly map to specific images; for updates, to subsets of the update tests like Workstation, KDE, Server etc.)

For unusual scheduling requests not handled by fedora-openqa (e.g. scheduling just one specific test), you can send a request to the openQA REST API using a client tool included in openQA. An example invocation can be found in the upstream documentation. Note that the settings required may vary depending on the compose and image being tested and the test being run; you may have to look into the behaviour of fedora_openqa to determine the necessary settings.

There are also several tips on using another script to take shortcuts during test development in the upstream test developer guide.

openQA and other automated test systems

People quite often wonder how Fedora came to start using openQA, and how use of openQA relates to other automated test projects, particularly Taskotron and Fedora CI.

Taskotron (and its predecessor AutoQA) were initially envisioned as comprehensive automated test systems for Fedora, and their design goals did encompass much of the testing that is currently performed by openQA. However, around late 2014, we came to a point where Taskotron was not yet capable of implementing most of the release validation testing required for Fedora releases, but performing all of this testing manually was starting to overwhelm the QA team.

After SUSE's Richard Brown implemented basic openQA testing of Fedora as a SUSE 'hack week' project, Fedora QA decided to see if we could use openQA to automate at least some of the release validation tests immediately. We found we were able to do so quite quickly and that the project did substantially reduce the burden of manual testing.

We quite quickly progressed from a manually-deployed openQA instance running on openSUSE to the production and staging Ansible-ized deployments of openQA-on-Fedora we have now, and found it easy and valuable to extend openQA testing beyond the original plan to cover quite a large proportion of the release validation tests, and run on nightly composes as well as candidates; it was similarly fairly easy and valuable to extend the system to update testing.

Fedora CI was then conceived with broadly similar goals to Taskotron, and Taskotron was retired in April 2020. Fedora CI is particularly aimed at enabling CI workflows for Fedora packagers. It is focusing on enabling integration of package-level tests with package pull request, build and update workflows. It is not necessarily expected that all automated testing related to Fedora will run through Fedora CI. Fedora CI works differently to openQA, and so each system is capable of some forms of testing that are not possible or would be difficult or inefficient in the other; openQA's strengths are particularly in installation and desktop testing, and other forms of testing where it is useful to have control over an entire virtual machine life cycle.

As part of the broader Fedora CI / Factory 2.0 effort we have built some systems and standards to support integration of multiple automated test systems. For instance, openQA and Fedora CI both submit results to Fedora ResultsDB, and we have configured Fedora's Greenwave to be able to gather and evaluate results from both systems. Both systems also publish fedora-messaging messages, and work is ongoing to ensure openQA, Fedora CI and possibly other systems publish messages conforming to the CI Messages spec, which would enable other workflows built on fedora-messaging to use information from all of the systems.

We believe openQA will be used permanently for the types of testing for which it is particularly well-suited. There are some tests currently implemented in openQA not because they are particularly well-suited to its abilities but simply because it was available at the time, and these may be moved into more appropriate systems in future as they become available. There are currently no plans to re-implement the most unique and useful openQA features - mouse/keyboard input, screen matching, and so forth - as part of Fedora CI development.