From Fedora Project Wiki

< QA

Line 86: Line 86:
= Stuff To Do =  
= Stuff To Do =  
== Litmus test: provide link -> do something productive without getting lost ==
== Litmus test: provide link -> do something productive without getting lost ==
   - You will need to create an account.  This is still a prototype; do not reuse a valuable password for this.
   - You will need to create an account.   
 
    - This is still a prototype; do not reuse a valuable password for this.
     - Get yourself added to the Testers group (wwoods can do this; who else?)
     - Get yourself added to the Testers group (wwoods can do this; who else?)
  - some nice css so it doesn't look like bugzilla, and hack up as necessary
   - HOWTO: installation testing: how to report results?
   - HOWTO: installation testing: how to report results?
   - HOWTO: write a test plan/case for one of your packages
   - HOWTO: write a test plan/case for one of your packages
   - wording of announcement:
   - wording of announcement:
       - to fedora-test-list?  devel-list?  announce-list?     
       - to fedora-test-list?  devel-list?  announce-list?     
Line 99: Line 98:
   - some docs on the wiki (screencast)
   - some docs on the wiki (screencast)
   - disable bug entry
   - disable bug entry
  - some nice css so it doesn't look like bugzilla, and hack up as necessary
   - have f10 testcases ready to go
   - have f10 testcases ready to go
   - XML-RPC to create each day's rawhide as a "build"
   - XML-RPC to create each day's rawhide as a "build"

Revision as of 19:49, 20 June 2008

Summary

The Fedora QA team is interested in evaluating Testopia as a test planning and results recording solution.

Background

During the Fedora9 campaign, the fedora wiki was used to planning, recording and presenting test execution status. Examples can be found:

While the information gathered was very helpful, and in the case of swfdec, was used to support dropping the proposed feature from F9, manually editing the wiki was not optimal. Additionally, tracking which test cases to include in a particular test run was easy to mis-type. Manually adding results to a wiki-style table entry was also easy to mis-type.

Pro's

  • Recorded test cases provided manual steps needed for execution and verification
  • The test plan outlined a plan of attack and provided a prioritization and organization for the proposed test cases
  • Wiki worked well for test plan and case content
  • Low barrier to entry ... a wiki is straight forward and uncomplex

Con's

  • Gather alternative views of test metrics from the wiki is not possible
  • Wiki not so great for hard metrics (date, tester, pass/fail execution metrics)

Tasks

STATUS OWNER(s) TASK
DEFERRED Create testopia.fedorahosted.org site (trac, tickets, git, mailing list etc...)
COMPLETE User:dmalcolm Rough packaging of testopia so that we can install and test on publictest2
INPROGRESS User:dmalcolm, User:jstanley Package review for testopia RPM

See bug#450013

COMPLETE User:jkeating Apply for fedora infrastructure sponsorship. Jesse has volunteered to be our sponsor
COMPLETE User:dmalcolm Apply for shared "publictest environment" to house proof of concept

Available at https://publictest2.fedoraproject.org/bugzilla

INPROGRESS User:jlaska Migrate existing F9 data into testopia for sample data set

See Testopia Plan#2

COMPLETE User:dmalcolm Craft a client script for communicating to testopia via XMLRPC using mozilla cookies for authentication (see QA/Testopia/XmlRpc

Open issues

REPORTER ISSUE
User:jlaska Testopia Component list

Should the testopia instance just re-use bugzilla product and components?

2008-06-16 update: User:jstanley has updated the components on private testopia instance to use output from the pkgdb (see https://palladium.jds2001.org/bugzilla/)

User:jlaska Attaching bugs

Testopia supports linking bugs to a test run. It appears to be looking in the current db for bugs, does it support looking into another bugzilla db for a valid bug#? While in the future our bugzilla and testpia would be one ... presently they are note. Perhaps an upstream patch is in order to support alternative bug repos (or perhaps a URL) in that field?

User:jlaska Searching

Searching for test plans/cases/builds/results is somewhat odd in Testopia. Unless you're logged in, you can't peruse the information. The posted pdf documentation outlines how to use the quick search field, but this seems odd. There appears to be no advanced search page?

  • For example, how can I search for test cases matching a specific component?
  • Or, show me all test runs against the Beta build

SOLVED - the Search link in the Testopia footer takes you to a Testopia specific search page. I haven't yet figured out who to save searches and pass them around via a url ...

User:alindebe Must be logged in for viewing

We'd like our content to be read-only for all users ... login should not be required.

Lessons Learned

The following list outlines important issues uncovered during the testopia proof of concept:

FUDCon Boston 2008 Hackfest

Stuff To Do

Litmus test: provide link -> do something productive without getting lost

 - You will need to create an account.  
    - This is still a prototype; do not reuse a valuable password for this.
    - Get yourself added to the Testers group (wwoods can do this; who else?)
 - some nice css so it doesn't look like bugzilla, and hack up as necessary
 - HOWTO: installation testing: how to report results?
 - HOWTO: write a test plan/case for one of your packages
 - wording of announcement:
     - to fedora-test-list?  devel-list?  announce-list?     
 - database backup
 - git repo for scripts
 - some docs on the wiki (screencast)
 - disable bug entry
 - have f10 testcases ready to go
 - XML-RPC to create each day's rawhide as a "build"
 - BarCamp talk: who to give?  what to say? etc
 ???? etc
 - profit!!!

What was done:

Worked around error when creating new account: when creating account, upon following the link in the confirmation email and signing in for the first time as a new user: (this is upstream bug https://bugzilla.mozilla.org/show_bug.cgi?id=378633 )

https://publictest2.fedoraproject.org/bugzilla/token.cgi


  Software error:

DBD::mysql::st execute failed: Table 'test_plan_permissions' was locked with a READ lock and can't be updated [for Statement "INSERT INTO test_plan_permissions
                                       (userid, plan_id, permissions, grant_type)
                                       VALUES (?, ?, ?, ?)" with ParamValues: 1='2', 3=2, 0='10', 2='1'] at Bugzilla/User.pm line 944
        Bugzilla::User::derive_regexp_groups('Bugzilla::User=HASH(0x7ed4ba0)') called at Bugzilla/User.pm line 1619
        Bugzilla::User::create('Bugzilla::User', 'HASH(0x7d2cca0)') called at /usr/share/bugzilla/token.cgi line 372
        main::confirm_create_account() called at /usr/share/bugzilla/token.cgi line 158

For help, please send mail to the webmaster (root@localhost 
<mailto:root@localhost>), giving this error message and the time and 
date of the error.

Turned out to be the regexp in jlaska's installation plan; removing it for now.

mysql> select * from test_plan_permissions_regexp;
+---------+-------------+-------------+
| plan_id | user_regexp | permissions |
+---------+-------------+-------------+
|       2 | .*          |           1 | 
+---------+-------------+-------------+

XML-RPC stuff (dmalcolm, bpeck; QA/Testopia/XmlRpc )