QA/Testopia Evaluation

= Summary =

The Fedora QA team is interested in evaluating Testopia as a test planning and results recording solution.

= FAQ =

Where is the Fedora Testopia instance?
See https://publictest2.fedoraproject.org/testopia

Is Testopia packaged for Fedora?
It is packaged, and currently undergoing package review (see bug#450013)

What user account should I use?
At this time, Testopia does not support integration with existing FAS2 or bugzilla.redhat.com user accounts

How can I sign up for an account?
Please use the built-in Testopia account system for registering a new account. To sign up for a new account, please visit https://publictest2.fedoraproject.org/testopia/createaccount.cgi.

How will Testopia be used?
During the F10 campaign, several areas will be identified for testing reporting using Testopia. The first of which will be Installation testing. Additionally, all upcoming F10 features will also be planned and tracked using Testopia. Depending on feedback, additional subsystems may be added to Testopia for test planning and results collection.

What happens after F10?
The Fedora QA community will be asked to test using Testopia for F10 community test efforts. After F10 is released, the Fedora QA team will need to make a recommendation on the future use of Testopia for planning and reporting on community testing.

How can I help contribute?
Any assistance is appreciated. Depending on your background/skills, there are several ways you can help:


 * 1) File bugs against upstream testopia (see https://bugzilla.mozilla.org/enter_bug.cgi?product=Testopia)
 * 2) Discuss any experiences or issues encountered with fedora QA community (fedora-test-list or IRC at #fedora-qa) ... we welcome your feedback
 * 3) Have a favorite component that you don't see a test plan for?  Go ahead and propose a test plan for F10 (see  an example)
 * 4) Good with HTML and/or CSS?  We'll need a fedora friendly look and feel.  If you are interested please stop by the mailing list or irc (#fedora-qa)

= Background =

During the Fedora9 campaign, the fedora wiki was used for planning, recording and presenting test execution status. Examples can be found:


 * Fedora 9 Installation Test Plan
 * Alpha
 * Beta
 * Preview Release
 * RC


 * Additional test results recorded for:
 * Swfdec (rawhide)
 * PackageKit (rawhide)
 * LiveCD (FinalRelease)

While the information gathered was very helpful, and in the case of swfdec, was used to support dropping the proposed feature from F9, manually editing the wiki was not optimal. Additionally, tracking which test cases to include in a particular test run was easy to mis-type. Manually adding results to a wiki-style table entry was also easy to mis-type.

Pro's

 * Recorded test cases provided manual steps needed for execution and verification
 * The test plan outlined a plan of attack and provided a prioritization and organization for the proposed test cases
 * Wiki worked well for test plan and case content
 * Low barrier to entry ... a wiki is straight forward and uncomplex

Con's

 * Gather alternative views of test metrics from the wiki is not possible
 * Wiki not so great for hard metrics (date, tester, pass/fail execution metrics)

= Tasks =

= Open issues =

= FUDCon Boston 2008 Hackfest =

Stuff To Do

 * 1) You will need to create an account.
 * 2) Go to https://publictest2.fedoraproject.org/testopia
 * 3) Click on "New Account" and supply your email address.
 * 4) This is still a prototype; do not reuse a valuable password for this.
 * 5) Get yourself added to the Testers group (wwoods can do this; who else?)
 * 6) Look'n'feel - some nice css so it doesn't look like bugzilla, and hack up as necessary
 * 7) some docs on the wiki (screencast)
 * 8) HOWTO: installation testing: how to report results?
 * 9) HOWTO: write a test plan/case for one of your packages
 * 10) wording of announcement:
 * 11) to fedora-test-list?  devel-list?  announce-list?
 * 12) database backup
 * 13) git repo for scripts
 * 14) disable bug entry
 * 15) have f10 install testcases ready to go
 * 16) XML-RPC to create each day's rawhide as a "build"
 * 17) BarCamp talk: who to give?  what to say? etc


 * 1) profit!!!

Fit and Finish

 * Worked around error when creating new account

when creating account, upon following the link in the confirmation email and signing in for the first time as a new user: (this is upstream bug https://bugzilla.mozilla.org/show_bug.cgi?id=378633 ) https://publictest2.fedoraproject.org/testopia/token.cgi

Software error:

DBD::mysql::st execute failed: Table 'test_plan_permissions' was locked with a READ lock and can't be updated [for Statement "INSERT INTO test_plan_permissions                                      (userid, plan_id, permissions, grant_type)                                       VALUES (?, ?, ?, ?)" with ParamValues: 1='2', 3=2, 0='10', 2='1'] at Bugzilla/User.pm line 944 Bugzilla::User::derive_regexp_groups('Bugzilla::User=HASH(0x7ed4ba0)') called at Bugzilla/User.pm line 1619 Bugzilla::User::create('Bugzilla::User', 'HASH(0x7d2cca0)') called at /usr/share/bugzilla/token.cgi line 372 main::confirm_create_account called at /usr/share/bugzilla/token.cgi line 158

For help, please send mail to the webmaster (root@localhost ), giving this error message and the time and date of the error.

Turned out to be the regexp in jlaska's installation plan; removing it for now. mysql> select * from test_plan_permissions_regexp; +-+-+-+ +-+-+-+ +-+-+-+
 * plan_id | user_regexp | permissions |
 * 2 | .*         |           1 |

Update 2008-06-30: Saw again on an account creation: DBD::mysql::st execute failed: Table 'test_plan_permissions' was locked with a READ lock and can't be updated [for Statement "INSERT INTO test_plan_permissions                                      (userid, plan_id, permissions, grant_type)                                       VALUES (?, ?, ?, ?)" with ParamValues: 1='11', 3=2, 0='24', 2='0'] at Bugzilla/User.pm line 944 Bugzilla::User::derive_regexp_groups('Bugzilla::User=HASH(0x2c69c60)') called at Bugzilla/User.pm line 1619 Bugzilla::User::create('Bugzilla::User', 'HASH(0x2ac1da0)') called at /usr/share/bugzilla/token.cgi line 372 main::confirm_create_account called at /usr/share/bugzilla/token.cgi line 158

mysql> select * from test_plan_permissions_regexp; +-+-+-+ +-+-+-+ +-+-+-+ 1 row in set (0.00 sec)
 * plan_id | user_regexp | permissions |
 * 11 | .*         |           0 |


 * Changed "mailfrom" from bugzilla-daemon@fedoraproject.org to testopia@fedoraproject.org


 * Various template hacks, see git log on /usr/share/bugzilla


 * Tried setting "expires" on static content

XML-RPC stuff

 * dmalcolm updated the python XML-RPC helper to handles login cookies sanely, working on getting this upstream; see QA/Testopia/XmlRpc
 * bpeck worked on reporting results from automated testing directly via XML-RPC; see QA/Testopia/XmlRpcReporting