Talk:Autotest

Will's notes (to be integrated into main page)

Control Files

The control file is actually interpreted as a Python script. So you can do any of the normal pythonic things you might want to do.

Before it reads the control file, Autotest imports all the symbols from the autotest_lib.client.bin.util module. This means the control files can use any function defined in common_lib.utils or bin.base_utils. This lets you do things like:

arch = get_arch()
baseurl = '%s/development/%s/os/' % (mirror_baseurl, arch)
job.run_test('some_rawhide_test', arch=arch, baseurl=baseurl)

since get_arch is defined in common_lib.utils.

Test Objects: Getting test results

First, the basic rule for test results: If your run_once() method does not raise an exception, the test result is PASS. If it raises error.TestFail or error.TestWarn the test result is FAIL or WARN. Any other exception yields an ERROR result.

For simple tests you can just run the test binary like this:

self.results = utils.system_output(cmd, retain_output=True)

If cmd is successful (i.e. it returns an exit status of 0) then utils.system_output() will return the output of the command. Otherwise it will raise error.CmdError, which will immediately end the test.

Some tests always exit successfully, so you'll need to inspect their output to decide whether they passed or failed. That would look more like this:

output = utils.system_output(cmd, retain_output=True)
if 'FAILED' in output:
    raise error.TestFail
elif 'WARNING' in output:
    raise error.TestWarn

Test Objects: Returning extra data

Further test-level info can be returned by using test.write_test_keyval(dict):

extrainfo = dict()
for line in self.results.stdout:
    if line.startswith("kernel version "):
        extrainfo['kernelver'] = line.split()[3]
    ...
self.write_test_keyval(extrainfo)

• For per-iteration data (performance numbers, etc) there are three methods:
  • Just attr: test.write_attr_keyval(attr_dict)
  • Just perf: test.write_perf_keyval(perf_dict)
  • Both: test.write_iteration_keyval(attr_dict, perf_dict)

Test Objects: Attributes for directories

test objects have the following attributes available:

outputdir       eg. results/<job>/<testname.tag>
resultsdir      eg. results/<job>/<testname.tag>/results
profdir         eg. results/<job>/<testname.tag>/profiling
debugdir        eg. results/<job>/<testname.tag>/debug
bindir          eg. tests/<test>
src             eg. tests/<test>/src
tmpdir          eg. tmp/<tempname>_<testname.tag>