Fedora uses the openQA automated testing system as a significant part of the release validation testing process, and for testing updates. On this page you can find more information about openQA, how Fedora uses it, and how to install your own instance of openQA so you can try it out and contribute to test development.
openQA is an automated test system designed around operating system-level testing. Its key feature is that it interacts with the system under test much like a human would, by sending input using a virtual keyboard and mouse (or serial console input), and monitoring the screen, serial console and audio outputs for output. This makes it ideally suited to running operating system-level tests without making any modifications to the software being tested.
For much more information on openQA, see the upstream documentation.
How Fedora uses openQA
At present, Fedora uses openQA for release validation type testing. We have developed a set of tests oriented around testing a complete Fedora distribution compose. These tests are run:
- On every nightly compose of Rawhide or Branched
- On every 'candidate' compose (including release candidates)
We also run a subset of the compose tests, plus some other tests, on some of the updates under test for Fedora stable and Branched releases. The tests are run on all critical path updates, and for some other specific 'whitelisted' packages (where the testing seems specifically relevant and valuable) - they are not run on all updates as we do not have sufficient testing resources for this. The results of these tests are visible on the Automated Tests tab of the update page, for updates which are tested. Updates can optionally be 'gated' on these results.
The results of all openQA tests are forwarded to ResultsDB. When there is a release validation test event for a compose, openQA results are mapped to Wikitcms test cases and reported to the appropriate wiki pages: results you see from the user coconut are results derived from openQA testing.
Future plans include 'gating' Rawhide composes on openQA test results (i.e. not pushing a Rawhide compose into the mirror system if it fails the openQA tests).
Fedora openQA tests and tools
The Fedora openQA tests can be found in the os-autoinst-distri-fedora repository. The library, CLI and Fedmsg consumers that handle scheduling openQA jobs and forwarding results to ResultsDB and Wikitcms can be found in the fedora_openqa repository. The tool used to create disk image files required by some of the tests can be found in the createhdds repository.
Each of these repositories includes documentation relating to its specific role in the overall system. Therepository includes instructions on installing and using the scheduler and report forwarding features, and the repository includes Fedora-specific instructions for test development and documentation of Fedora test conventions, functions and settings.
The official Fedora openQA deployments are managed via Ansible playbooks in the Fedora Infrastructure ansible repository. The roles can be found in , and we attempt to maintain them in such a way that you can use them to deploy openQA instances outside of Fedora infrastructure, though this is not tested very often and may bitrot from time to time.
- General instructions (from upstream)
- Fedora-specific direct system install guide
- Docker container install guide
- Advanced networking instructions (from upstream, for tests requiring interaction between workers)
- Fedora-specific advanced networking guide
Running tests directly
To run the standard set of Fedora tests on a regular Fedora compose, you can use the fedora_openqa repository. This will cause the server to download all testable image files from the compose and run all the tests for each image.tool in the
To run tests only for a specific image, including images not from regular composes, you can send a request to the openQA REST API using a client tool included in openQA. An example invocation can be found in the upstream documentation. Note that the settings required may vary depending on the compose and image being tested and the test being run; you may have to look into the behaviour of to determine the necessary settings.
There are also several tips on using another script to take shortcuts during test development in the upstream test developer guide.
openQA and other automated test systems
People quite often wonder how Fedora came to start using openQA, and how use of openQA relates to other automated test projects, particularly Taskotron.
Taskotron (and its predecessor AutoQA) were initially envisioned as comprehensive automated test systems for Fedora, and their design goals do encompass much of the testing that is currently performed by openQA. However, around late 2014, we came to a point where Taskotron was not yet capable of implementing most of the release validation testing required for Fedora releases, but performing all of this testing manually was starting to overwhelm the QA team.
After SUSE's Richard Brown implemented basic openQA testing of Fedora as a SUSE 'hack week' project, Fedora QA decided to see if we could use openQA to automate at least some of the release validation tests immediately. We found we were able to do so quite quickly and that the project did substantially reduce the burden of manual testing.
We quite quickly progressed from a manually-deployed openQA instance running on openSUSE to the production and staging Ansible-ized deployments of openQA-on-Fedora we have now, and found it easy and valuable to extend openQA testing beyond the original plan to cover quite a large proportion of the release validation tests, and run on nightly composes as well as candidates.
We now conceive of Taskotron as being a set of modular components that can be used in automated testing and CI workflows, and no longer necessarily expect absolutely all Fedora-related automated testing to run through Taskotron's native test execution components. openQA result forwarding to ResultsDB can already be considered a form of openQA/Taskotron integration, as ResultsDB began as the 'Taskotron results store'; the move to consider it a more generic system for storing test results is part of the overall adjustment to thinking of Taskotron as a modular system.
We believe openQA will be used permanently for the types of testing for which it is particularly well-suited. There are some tests currently implemented in openQA not because they are particularly well-suited to its abilities but simply because it was available at the time, and these may be moved into more appropriate systems in future as they become available. There are currently no plans to re-implement the most unique and useful openQA features - mouse/keyboard input, screen matching, and so forth - as part of Taskotron development.
As mentioned above, openQA is already quite well integrated into the rest of the Fedora infrastructure through the use of Fedmsg and the forwarding of results into ResultsDB. We also envisage potentially using Taskotron triggers to schedule openQA jobs in the future, rather than using the current standalone scheduler code.