From Fedora Project Wiki

m (Add post-koji-build link)
(improve page structure)
 
(5 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 
{{header|qa}}
 
{{header|qa}}
 
{{draft}}
 
{{draft}}
 +
 
== Overview ==
 
== Overview ==
The <code>hooks/</code> directory in the <code>autoqa</code> source tree contains the hooks that AutoQA knows about. A hook has five main parts:
+
The {{filename|events/}} directory in the <code>autoqa</code> source tree contains the events that AutoQA knows about. Watchers for these events are stored within {{filename|watchers/}} directory.
  
# <code>README</code>
+
An event has these parts:
 +
# {{filename|README}}
 
#* describes the event itself and the required (and optional) arguments that will be passed along to the tests.
 
#* describes the event itself and the required (and optional) arguments that will be passed along to the tests.
# <code>hook.py</code>
+
# {{filename|event.py}}
#* python code that is used to parse the test arguments, as described in the <code>README</code> file. This is the formal definition of the test arguments.
+
#* python code that is used to parse the test arguments, as described in the {{filename|README}} file. This is the formal definition of the test arguments.
# <code>testlist</code>
+
 
#* contains the list of test names that will be launched when this hook is triggered.
+
A watcher consists of:
# <code>control.template</code> and <code>test_class_template.py</code>
+
# {{filename|README}}
#* generic templates for creating new tests that use this hook. See below for more information on writing new tests.
+
#* describes the general area that the watcher monitors and lists all events that are triggered by it.
# Watcher
+
# {{filename|watcher.py}}
#* This is the code that watches for the event and launches the <code>autoqa</code> harness with the arguments described in <code>README</code> and <code>hook.py</code>.
+
#* Is the code that watches for the event and launches the <code>autoqa</code> harness with the arguments described in the event's {{filename|README}}.
#* Currently, all existing watchers are scripts that get run periodically by <code>crond</code> to check to see if the event has occurred since the last time it was run. If so, it launches <code>autoqa</code>.
+
#* Currently, most of the existing watchers are scripts that get run periodically by <code>crond</code> to check to see if the event has occurred since the last time it was run. If so, it launches <code>autoqa</code>.
 
#* In the future this will change to a daemon that waits for notifications about the event - see the [[Messaging SIG]]'s [[Publish Subscribe Notification Proposal]] for further info about that.
 
#* In the future this will change to a daemon that waits for notifications about the event - see the [[Messaging SIG]]'s [[Publish Subscribe Notification Proposal]] for further info about that.
  
== README ==
+
== Event ==
 +
 
 +
=== README ===
  
This is the human-readable description for this hook. It describes the event, the required and optional arguments that will be passed to autoqa and the tests, and any special argument processing or test filtering that might happen.  
+
This is the human-readable description for this event. It describes its purpose, the required and optional arguments that will be passed to autoqa and the tests, and any special argument processing or test filtering that might happen.  
  
 
Here's an example, the <code>post-repo-update</code> README:
 
Here's an example, the <code>post-repo-update</code> README:
 
<pre>
 
<pre>
This hook is for tests that run after a repo has changed. A repo is considered
+
This event is for tests that run after a repo has changed. A repo is considered
 
"changed" if its package contents and metadata have been changed in some way.
 
"changed" if its package contents and metadata have been changed in some way.
  
Line 29: Line 33:
 
points to the changed repo.
 
points to the changed repo.
  
Some tests (e.g. repoclosure) need a list of "parent" repos to run properly.
+
Optional arguments:
You can specify these by doing '--parent URL1 --parent URL2 ...'
+
    --name: human readable name for the repo under test
 +
    --parent: a repo URL that is "parent" to the the repo under test; may be
 +
              specified multiple times
  
Any instances of '%a' in the URLs will be replaced by one of the listed
+
AutoQA tests can expect the following variables from post-repo-update event:
arches when the tests are actually run.
+
  baseurl: url of repo that changed
 +
  parents: list of urls for 'parent' repos of the given repo (may be empty)
 +
  name: name for repo that changed (may be empty)
 
</pre>
 
</pre>
  
Every hook has at least one required argument - usually a URL that points to the new package/tree/repo/whatever. It can also define one or more optional arguments, which will be handled by autoqa commandline arguments. Those get defined in hook.py.
+
Every event has at least one required argument - usually a URL that points to the new package/tree/repo/whatever. It can also define one or more optional arguments, which will be handled by autoqa commandline arguments. Those get defined in event.py.
  
== hook.py ==
+
=== event.py ===
  
This contains python code that will be loaded by autoqa when it is launched by the watcher. This code handles parsing the autoqa arguments, generating the data to be passed to the test, and filtering the list of tests, if needed. It must contain three functions: <code>extend_parser</code>, <code>process_testdata</code>, and <code>process_testlist</code>.
+
This contains python code that will be loaded by autoqa when it is launched by the watcher. This code handles parsing the autoqa arguments and generating the data to be passed to the test. It must contain two methods: <code>extend_parser</code> and <code>process_testdata</code>.
  
=== extend_parser() ===
+
==== extend_parser() ====
  
This function adds hook-specific options to the given [http://docs.python.org/library/optparse.html OptionParser object], so autoqa can properly parse the arguments given by the watcher. Here's the <code>extend_parser()</code> from <code>post-repo-update</code>:
+
This function adds event-specific options to the given [http://docs.python.org/library/optparse.html OptionParser object], so autoqa can properly parse the arguments given by the watcher. Here's the <code>extend_parser()</code> from <code>post-repo-update</code>:
  
 
<pre>
 
<pre>
Line 50: Line 58:
  
 
def extend_parser(parser):
 
def extend_parser(parser):
     '''Extend the given OptionParser object with settings for this hook.'''
+
     '''Extend the given OptionParser object with settings for this event.'''
 
     parser.set_usage('%%prog %s [options] REPOURL' % name)
 
     parser.set_usage('%%prog %s [options] REPOURL' % name)
 
     group = optparse.OptionGroup(parser, '%s options' % name)
 
     group = optparse.OptionGroup(parser, '%s options' % name)
Line 63: Line 71:
 
The new options are generally used to handle the ''optional'' arguments to the test. This is where we've defined the <code>--parent</code> argument that the README mentioned.  
 
The new options are generally used to handle the ''optional'' arguments to the test. This is where we've defined the <code>--parent</code> argument that the README mentioned.  
  
The required argument(s) are handled in <code>process_testdata()</code>.
+
The arguments are then handled in <code>process_testdata()</code>.
  
=== process_testdata() ===
+
==== process_testdata() ====
  
 
This function generates and returns a dict of testdata - the key=value data that will be passed along to the test.
 
This function generates and returns a dict of testdata - the key=value data that will be passed along to the test.
  
It uses the results of <code>parser.parse_args()</code> - <code>opts</code> contains the options, and <code>args</code> contains the list of unhandled (usually required) args. It also gets the requested test <code>arch</code>. (In the future it may get some extra keyword arguments, so it is usually defined with an **extra parameter.)
+
It uses the results of <code>parser.parse_args()</code> - <code>opts</code> contains the options, and <code>args</code> contains the list of unhandled (usually required) args. (In the future it may get some extra keyword arguments, so it is usually defined with an **extra parameter.)
  
 
Here's the one from <code>post-repo-update</code>:
 
Here's the one from <code>post-repo-update</code>:
  
 
<pre>
 
<pre>
def process_testdata(opts, args, arch, **extra):
+
def process_testdata(parser, opts, args, **extra):
     testdata = {'url': args[0].replace('%a', arch),
+
     '''Given an optparse.Values object and a list of args (as returned from
                'parents': ' '.join(opts.parent).replace('%a', arch)}
+
    OptionParser.parse_args()), return a dict containing the appropriate key=val
     if opts.name:
+
     pairs required by test's control files and test object. The event can also
        testdata['reponame'] = '%s-%s' % (opts.name, arch)
+
     call parser.error here if it find out that not all options are correctly
     else:
+
    populated.'''
        testdata['reponame'] = testdata['url']
 
    return testdata
 
</pre>
 
  
As you can see, it sets three values - The required argument is a URL, so we set <code>url</code> to the first non-option argument. <code>parents</code> is set to a space-separated list of the given <code>--parent</code> items. And <code>reponame</code> is set to the <code>--name</code> argument if given - otherwise we just use the URL, which is a nice unique identifier.
+
    if not args:
 +
        parser.error('No repo URL was specified as a test argument!')
  
=== process_testlist() ===
+
    testdata = {'baseurl': args[0],
 
+
                'parents': opts.parent,
Finally we have <code>process_testlist()</code>. This function takes the list of known tests for this hook (see testlist below) and filters out anything that might not be appropriate for the given arguments. It returns the modified list of tests.
+
                'name': opts.name or ''}
 
+
     return testdata
In its simplest form, this function can just be defined as follows:
 
 
 
<pre>
 
def process_testlist(opts, args, testlist):
 
     return testlist
 
 
</pre>
 
</pre>
  
Here's the version used by <code>post-repo-update</code>:
+
As you can see, it sets three values - The required argument is a URL, so we set <code>baseurl</code> to the first non-option argument. <code>parents</code> is set to a list of the given <code>--parent</code> items. And <code>name</code> is set to the <code>--name</code> argument if given - otherwise we just use empty string.
 
 
<pre>
 
def process_testlist(opts, args, testlist):
 
    if not opts.name.lower().startswith('rawhide'):
 
        if 'rats_sanity' in testlist:
 
            testlist.remove('rats_sanity')
 
    return testlist
 
</pre>
 
 
 
The <code>rats_sanity</code> test is only appropriate to run on [[Rawhide]] repos, so if the repo isn't named <code>rawhide</code>-something, we remove it from the list.
 
 
 
== templates ==
 
 
 
Each hook should provide two template files - <code>control.template</code>, which is an example <code>control</code> file, and <code>test_class_template.py</code>, which is an example test object. For more info about control files and test objects, see [[Writing AutoQA Tests]] which explains the components of AutoQA tests in great detail.
 
 
 
Writing the test file templates is pretty simple. You can mostly just copy the existing template files from another hook. You'll need to edit the parts that explain the hook-specific arguments - the things that get handled and passed along by autoqa, as described above.
 
 
 
=== control.template ===
 
 
 
Once again, an example from <code>post-repo-update</code>:
 
 
 
<pre>
 
TIME="SHORT"
 
AUTHOR = "YOUR NAME <you@youremail.biz>"
 
DOC = """
 
This is the long description of the purpose of this test.
 
"""
 
NAME = 'short_testname'
 
TEST_TYPE = 'CLIENT' # SERVER can be used for tests that need multiple machines
 
TEST_CLASS = 'General'
 
TEST_CATEGORY = 'Functional'
 
 
 
# post-repo-update tests can expect the following variables from autoqa:
 
#
 
#url - url of repo that changed
 
#parents - space-separated list of urls for 'parent' repos of the given repo
 
#reponame - name for repo that changed
 
#autoqa_conf - contents of /etc/autoqa.conf on the server
 
 
 
job.run_test('testclassname', baseurl=url,
 
                              parents=parents,
 
                              reponame=reponame,
 
                              config=autoqa_conf)
 
</pre>
 
 
 
The comment in the middle of the file explains the arguments passed from autoqa, and the final <code>run_test</code> line gives an example of how those arguments should be passed to the test object.
 
 
 
The variable names are the keys from the <code>testdata</code> dict returned by <code>process_testdata</code>, above. <code>autoqa_conf</code> is a special variable that holds the contents of the server's <code>/etc/autoqa.conf</code>. This is used for passing system-wide config stuff along to the tests.
 
 
 
You can name the keyword arguments to the <code>run_test</code> function pretty much anything ''except'' "url". That's reserved by the autotest system itself.
 
 
 
=== test_class_template.py ===
 
 
 
This is an example test class.
 
 
 
<pre>
 
from autotest_lib.client.bin import test, utils
 
from autotest_lib.client.bin.test_config import config_loader
 
 
 
class post_repo_update_test_class_name(test.test):
 
    version = 1 # increment if setup() changes
 
 
 
    def initialize(self, config):
 
        self.config = config_loader(config, self.tmpdir)
 
 
 
    def setup(self):
 
        # this is where you can install required packages and such, e.g.:
 
        #utils.system('yum -y install yum-utils')
 
        pass
 
 
 
    def run_once(self, baseurl, parents, reponame):
 
        parentlist = ' '.split(parents)
 
        cmd = 'test_binary --url %s' % baseurl
 
        # You can get stuff from the [test] section of autoqa.conf like this
 
        email = self.config.get('test','result_email')
 
        if email:
 
            cmd += ' --email %s --subject %s' % (email, reponame)
 
        self.results = utils.system_output(cmd, retain_output=True)
 
</pre>
 
 
 
Again, this is mostly just boilerplate example code, ''except'' the names of the arguments to <code>initialize()</code> and <code>run_once()</code>. The arguments to <code>run_once()</code> must have the same names as the keyword arguments listed in <code>control.template</code>. This is how we get the autoqa arguments passed along to the test code itself.
 
 
 
This applies to <code>initialize()</code> as well. Note that you don't need ''all'' the keyword arguments - in this example, <code>initialize()</code> only pays attention to <code>config</code> and <code>run_once()</code> handles <code>baseurl</code>, <code>parents</code>, and <code>reponame</code>.
 
 
 
<code>run_once()</code> has some silly example code that shows how some of its arguments might conceivably be used; probably most users will throw this away, so don't worry too much about it being useful or correct.
 
 
 
== testlist ==
 
 
 
This file simply lists the names of tests that should be run for this hook. These names must correspond to directories in <code>tests/</code> that contain <code>control</code> files.
 
  
 
== Watcher ==
 
== Watcher ==
  
This is the heart of the hook: the part that actually watches for the event and triggers autoqa when it happens.  
+
The watcher is a program that actually watches for the event and triggers autoqa when it happens.  
  
The watcher is responsible for constructing the correct <code>autoqa</code> commandline (as defined in <code>README</code> and <code>hook.py</code>, above) and executing <code>autoqa</code>. The rest of the design depends on the nature of the event.   
+
The watcher is responsible for constructing the correct <code>autoqa</code> commandline (as defined in event [[#README]], above) and executing <code>autoqa</code>. The rest of the design depends on the nature of the event.   
  
The current watchers for <code>post-repo-update</code> and <code>post-tree-compose</code> are run at regular intervals by <code>cron</code>. They check a certain file in each known repo (<code>repomd.xml</code> or <code>.treeinfo</code>, respectively) to see if it's different from the cached copy. If it's changed, they save the new file and launch <code>autoqa</code> for the repo/tree that has updated.
+
The current watchers for <code>post-repo-update</code> and <code>post-tree-compose</code> are run at regular intervals by <code>cron</code>. They check a certain file in each known repo ({{filename|repomd.xml}} or {{filename|.treeinfo}}, respectively) to see if it's different from the cached copy. If it's changed, they save the new file and launch <code>autoqa</code> for the repo/tree that has updated.
  
Yours could work that way, or it could be a daemon that waits for a signal, or any other design that makes sense. A full discussion about how to write a watcher is outside the scope of this page, but you might get some inspiration by looking at the current watchers:
+
Yours could work that way, or it could be a daemon that waits for a signal, or any other design that makes sense. When writing a new watcher you might get some inspiration by looking at the current watchers:
  
* [https://fedorahosted.org/autoqa/browser/hooks/post-repo-update/watch-repos.py watch-repos.py]
+
* yum-repo [https://fedorahosted.org/autoqa/browser/watchers/yum-repo/watcher.py watcher.py]
* [https://fedorahosted.org/autoqa/browser/hooks/post-tree-compose/watch-composes.py watch-composes.py]
+
* compose [https://fedorahosted.org/autoqa/browser/watchers/compose/watcher.py watcher.py]
* [https://fedorahosted.org/autoqa/browser/hooks/post-koji-build/watch-koji-builds.py watch-koji-builds.py]
+
* koji-bodhi [https://fedorahosted.org/autoqa/browser/watchers/koji-bodhi/watcher.py watcher.py]
  
 
[[Category:AutoQA]]
 
[[Category:AutoQA]]

Latest revision as of 12:12, 30 March 2011

QA.png


Warning.png
This page is a draft only
It is still under construction and content may change. Do not rely on the information on this page.

Overview

The events/ directory in the autoqa source tree contains the events that AutoQA knows about. Watchers for these events are stored within watchers/ directory.

An event has these parts:

  1. README
    • describes the event itself and the required (and optional) arguments that will be passed along to the tests.
  2. event.py
    • python code that is used to parse the test arguments, as described in the README file. This is the formal definition of the test arguments.

A watcher consists of:

  1. README
    • describes the general area that the watcher monitors and lists all events that are triggered by it.
  2. watcher.py
    • Is the code that watches for the event and launches the autoqa harness with the arguments described in the event's README.
    • Currently, most of the existing watchers are scripts that get run periodically by crond to check to see if the event has occurred since the last time it was run. If so, it launches autoqa.
    • In the future this will change to a daemon that waits for notifications about the event - see the Messaging SIG's Publish Subscribe Notification Proposal for further info about that.

Event

README

This is the human-readable description for this event. It describes its purpose, the required and optional arguments that will be passed to autoqa and the tests, and any special argument processing or test filtering that might happen.

Here's an example, the post-repo-update README:

This event is for tests that run after a repo has changed. A repo is considered
"changed" if its package contents and metadata have been changed in some way.

The required argument for autoqa is a yum-compatible URL (probably http) that
points to the changed repo.

Optional arguments:
    --name: human readable name for the repo under test
    --parent: a repo URL that is "parent" to the the repo under test; may be
              specified multiple times

AutoQA tests can expect the following variables from post-repo-update event:
  baseurl: url of repo that changed
  parents: list of urls for 'parent' repos of the given repo (may be empty)
  name: name for repo that changed (may be empty)

Every event has at least one required argument - usually a URL that points to the new package/tree/repo/whatever. It can also define one or more optional arguments, which will be handled by autoqa commandline arguments. Those get defined in event.py.

event.py

This contains python code that will be loaded by autoqa when it is launched by the watcher. This code handles parsing the autoqa arguments and generating the data to be passed to the test. It must contain two methods: extend_parser and process_testdata.

extend_parser()

This function adds event-specific options to the given OptionParser object, so autoqa can properly parse the arguments given by the watcher. Here's the extend_parser() from post-repo-update:

import optparse

def extend_parser(parser):
    '''Extend the given OptionParser object with settings for this event.'''
    parser.set_usage('%%prog %s [options] REPOURL' % name)
    group = optparse.OptionGroup(parser, '%s options' % name)
    group.add_option('-n', '--name', default='',
        help='Short, human-readable name for the repo under test')
    group.add_option('-p', '--parent', action='append', default=[],
        help='URL of a "parent" repo that this repo depends on for closure')
    parser.add_option_group(group)
    return parser

The new options are generally used to handle the optional arguments to the test. This is where we've defined the --parent argument that the README mentioned.

The arguments are then handled in process_testdata().

process_testdata()

This function generates and returns a dict of testdata - the key=value data that will be passed along to the test.

It uses the results of parser.parse_args() - opts contains the options, and args contains the list of unhandled (usually required) args. (In the future it may get some extra keyword arguments, so it is usually defined with an **extra parameter.)

Here's the one from post-repo-update:

def process_testdata(parser, opts, args, **extra):
    '''Given an optparse.Values object and a list of args (as returned from
    OptionParser.parse_args()), return a dict containing the appropriate key=val
    pairs required by test's control files and test object. The event can also
    call parser.error here if it find out that not all options are correctly
    populated.'''

    if not args:
        parser.error('No repo URL was specified as a test argument!')

    testdata = {'baseurl': args[0],
                'parents': opts.parent,
                'name': opts.name or ''}
    return testdata

As you can see, it sets three values - The required argument is a URL, so we set baseurl to the first non-option argument. parents is set to a list of the given --parent items. And name is set to the --name argument if given - otherwise we just use empty string.

Watcher

The watcher is a program that actually watches for the event and triggers autoqa when it happens.

The watcher is responsible for constructing the correct autoqa commandline (as defined in event #README, above) and executing autoqa. The rest of the design depends on the nature of the event.

The current watchers for post-repo-update and post-tree-compose are run at regular intervals by cron. They check a certain file in each known repo (repomd.xml or .treeinfo, respectively) to see if it's different from the cached copy. If it's changed, they save the new file and launch autoqa for the repo/tree that has updated.

Yours could work that way, or it could be a daemon that waits for a signal, or any other design that makes sense. When writing a new watcher you might get some inspiration by looking at the current watchers: