User:Kparal/Verifying AutoQA tests

This page is a draft only
It is still under construction and content may change. Do not rely on the information on this page.

Introduction

This guide presumes that you have a new AutoQA test written (preferably according to Writing AutoQA Tests) and you want to verify that it works ok. This article will show you how to do it.

Prerequisites

You have to have autotest installed and AutoQA installed. You can choose not to install and configure autotest-server (so you will use autotest-client + autoqa only) if you are willing to execute the test on your local machine. It is easier to run the test locally but you must be sure it is not destructive and there are no harmful commands that could be run by mistake (or you will be running the test on a machine that you don't have to worry about).

Examine the watcher

When you have the test ready, you have already chosen the right hook (meaning event type) for your test and added your test name to the testlist file of that hook. Now we need to simulate running the hook's watcher on AutoQA server to see what commands would be run. We can do that by adding --dry-run (use --help to see more of useful options).

Let's say our test uses post-koji-build hook, which announces every package built and tagged with dist-fX-updates-candidate tag in Koji. So we would run:

# /usr/share/autoqa/post-koji-build/watch-koji-builds.py --dry-run
No previous run - checking builds in the past 3 hours
autoqa post-koji-build --name espeak --kojitag dist-f12-updates-candidate --arch x86_64 --arch i686 espeak-1.42.04-1.fc12
autoqa post-koji-build --name kdemultimedia --kojitag dist-f11-updates-candidate --arch x86_64 kdemultimedia-4.3.4-1.fc11
autoqa post-koji-build --name kdeplasma-addons --kojitag dist-f11-updates-candidate --arch x86_64 kdeplasma-addons-4.3.4-1.fc11
autoqa post-koji-build --name cryptopp --kojitag dist-f12-updates-candidate --arch x86_64 --arch i686 cryptopp-5.6.1-0.1.svn479.fc12
autoqa post-koji-build --name drupal --kojitag dist-f12-updates-candidate --arch x86_64 drupal-6.15-1.fc12
autoqa post-koji-build --name seamonkey --kojitag dist-f12-updates-candidate --arch x86_64 --arch i686 seamonkey-2.0.1-1.fc12
... output trimmed ...

For every line, all tests from post-koji-build hook (specified in testlist file) would be run on all the architectures specified by --arch option. For our purposes we will pick one command, let's say the first one.

Examine the control file

We will now try what would happen if the chosen command would be actually run. By appending --dry-run option to the command the autoqa harness will prepare everything needed for autotest harness and print what would be run, but not execute it. Let's see what happens:

# autoqa post-koji-build --name espeak --kojitag dist-f12-updates-candidate --arch x86_64 --arch i686 espeak-1.42.04-1.fc12 --dry-run
/usr/share/autotest/client/bin/autotest --verbose -t post-koji-build:rpmlint.x86_64 /tmp/autoqa-control.y_ODy7
keeping /tmp/autoqa-control.y_ODy7 at user request
/usr/share/autotest/client/bin/autotest --verbose -t post-koji-build:rpmguard.x86_64 /tmp/autoqa-control.rjzqJ_
keeping /tmp/autoqa-control.rjzqJ_ at user request

There are two lines saying that autotest would be run with a particular control file. There are two of them because we asked for testing on two different architectures. Those control files were kept on disk for our examination. Pick one of them and display it. You should see something like this:

# -*- coding: utf-8 -*-

autoqa_conf = '''
... output trimmed ...
'''

kojitag='dist-f12-updates-candidate'
nvr='espeak-1.42.04-1.fc12'
name='espeak'

... output trimmed ...

job.run_test('rpmlint', name=name, nvr=nvr, kojitag=kojitag, config=autoqa_conf)

It's almost the same config file that you created, but on top some more data are added.

Run just your test

From the previous output we have picked the first line. But we don't want to run all tests of that post-koji-build hook on it, just our one. Suppose we are writing test named rpmlint (already present in the AutoQA by the way). We will modify the command to look like this:

autoqa post-koji-build --name espeak --kojitag dist-f12-updates-candidate --arch x86_64 --arch i686 espeak-1.42.04-1.fc12 --test rpmlint

If you don't have autotest-server installed and configured