No edit summary |
(rejig 'new build available') |
||
| (7 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
== Introduction == | == Introduction == | ||
The QA group co-ordinates Rawhide Acceptance Test(RATs) events before each Fedora release and pre-release is made, to ensure they reach [[Fedora_Release_Criteria]]. Testing covers the [[QA:Installation_validation_testing|installation process]] and key [[QA:Desktop_validation_testing|desktop functionality]]. There are also some tests for specific spins. This page provides guidance on arranging and announcing such a Rawhide Acceptance Test event. For any concerns, please contact the [[QA]] group. | The QA group co-ordinates Rawhide Acceptance Test(RATs) events before each Fedora release and pre-release is made, to ensure they reach [[Fedora_Release_Criteria]]. Testing covers the [[QA:Installation_validation_testing|installation process]] and key [[QA:Desktop_validation_testing|desktop functionality]]. There are also some tests for specific spins. This page provides guidance on arranging and announcing such a Rawhide Acceptance Test event. For any concerns, please contact the [[QA]] group. | ||
=== Differs from release validation events === | |||
RATs are automated test events which happen before the [[QA/SOP_Release_Validation_Test_Event|manual release validation test events]] for each release phase (Alpha, Beta, Final). | |||
== Look up the date == | == Look up the date == | ||
| Line 9: | Line 12: | ||
== Create test result pages and categories == | == Create test result pages and categories == | ||
Test result pages are needed to gather test results of installation and desktop against Fedora candidate builds. Refer to [[QA:Create_Install_Test_Result_Page]] for guidance to create an installation result page using [[QA: | Test result pages are needed to gather test results of installation and desktop against Fedora candidate builds. Refer to [[QA:Create_Install_Test_Result_Page]] for guidance to create an installation result page using [[QA:Fedora_Rawhide_Acceptance_Test_Template]]. | ||
Often you will need to deal with categories. For each release, there should be a category named '''Fedora (Release) Test Results''', which should be a member of the category '''Test Results'''. For each milestone, there should be at least one categories - '''Fedora_(Release)_(Milestone)_Rawhide_Acceptance_Test | Often you will need to deal with categories. For each release, there should be a category named '''Fedora (Release) Test Results''', which should be a member of the category '''Test Results'''. For each milestone, there should be at least one categories - '''Fedora_(Release)_(Milestone)_Rawhide_Acceptance_Test''' - which should be members of the category '''Fedora (Release) Test Results'''. For the very first candidate build for each Fedora release - usually Alpha RATs1 - you will need to edit the '''Fedora (Release) Test Results''' category page and make it a member of the '''Test Results''' category, and edit the '''Fedora_(Release)_(Milestone)_Rawhide_Acceptance_Test''' category page and make it a member of the '''Fedora (Release) Test Results''' category. Then ensure that the results pages you create are made members of the '''Fedora (Release) (Milestone) Test Results''' category. For the following candidate builds, follow this general procedure to add category pages to the higher-level categories and categorize results pages as appropriate. | ||
Please note that you can make some special changes to the result pages according to the practical situation, such as copying parts of previous results to the new result page, highlighting critical cases, adding important notes etc. | Please note that you can make some special changes to the result pages according to the practical situation, such as copying parts of previous results to the new result page, highlighting critical cases, adding important notes etc. | ||
| Line 17: | Line 20: | ||
== Change current links and IRC topic == | == Change current links and IRC topic == | ||
One or two days before test event day, update the following redirect links and make sure they point to the pages you created: | One or two days before test event day, update the following redirect links and make sure they point to the pages you created: | ||
* {{noredirect| | * {{noredirect|Test Results:Current Rawhide Acceptance Installation Test}} | ||
Also change the topic of #fedora-qa (freenode) to current test event (if you have the permission). | Also change the topic of #fedora-qa (freenode) to current test event (if you have the permission). | ||
== Make announcements == | == Make announcements == | ||
Currently, we announce validation test events to the [https://admin.fedoraproject.org/mailman/listinfo/test-announce test-announce] mailing list. The announcement mail should provide enough information related to the test focus areas. See an [ | Currently, we announce validation test events to the [https://admin.fedoraproject.org/mailman/listinfo/test-announce test-announce] mailing list. The announcement mail should provide enough information related to the test focus areas. See an [https://fedoraproject.org/wiki/QA:Rawhide_Acceptance_Test_Announce_Example example] which generally includes: | ||
* Introduction of this test event (date, what to test, which release criteria to meet, etc) | * Introduction of this test event (date, what to test, which release criteria to meet, etc) | ||
* Others to be emphasized | * Others to be emphasized | ||
== Perform testing == | |||
When the needed files has been prepared, please refer to the [[QA:Rawhide_Acceptance_Test_Plan|rawhide acceptance test plan]] which can help you finish the testing. Following but not limited with these steps， try to find and file as more bugs as earlier which you suspect will block or impact Alpha TC1 testing. | |||
== | |||
== Report and Summary == | == Report and Summary == | ||
After testing has been completed， send a test summary to the [https://admin.fedoraproject.org/mailman/listinfo/test-announce test-announce] mailing list. The test summary is intended to keep testers informed as to what was accomplished by their testing, whether there are any remaining tasks and to recognize key contributors. A sample report is available at http://lists.fedoraproject.org/pipermail/test | After testing has been completed， send a test summary to the [https://admin.fedoraproject.org/mailman/listinfo/test-announce test-announce] mailing list. The test summary is intended to keep testers informed as to what was accomplished by their testing, whether there are any remaining tasks and to recognize key contributors. A sample report is available at http://lists.fedoraproject.org/pipermail/test/2011-July/101376.html. | ||
The test summary should include the following information: | The test summary should include the following information: | ||
| Line 64: | Line 48: | ||
| xargs bugzilla query --outputformat="%{bug_id} %{bug_status} %{resolution} - %{short_desc} - %{blocked}" -b</pre> | | xargs bugzilla query --outputformat="%{bug_id} %{bug_status} %{resolution} - %{short_desc} - %{blocked}" -b</pre> | ||
== When a new build is available == | |||
If a new build is made available for testing, such as Pre-Alpha RATs2, Pre-Alpha RATs3 etc., you should re-do the entire process: create new test result pages for it, update the redirect pages, send out a new announcement, do the testing again, and send out a new results summary. The announcement should highlight the specific changes from the previous candidate build, as in this [http://lists.fedoraproject.org/pipermail/test-announce/2010-March/000052.html example]. | |||
[[Category:QA SOPs]] | [[Category:QA SOPs]] | ||
Latest revision as of 22:27, 28 July 2011
Introduction
The QA group co-ordinates Rawhide Acceptance Test(RATs) events before each Fedora release and pre-release is made, to ensure they reach Fedora_Release_Criteria. Testing covers the installation process and key desktop functionality. There are also some tests for specific spins. This page provides guidance on arranging and announcing such a Rawhide Acceptance Test event. For any concerns, please contact the QA group.
Differs from release validation events
RATs are automated test events which happen before the manual release validation test events for each release phase (Alpha, Beta, Final).
Look up the date
Validation test events are held for each candidate build. Several of these builds are made in the period one or two weeks before the release date for any given milestone (Alpha, Beta and Final). See quality task schedule for specific dates. Then make the following preparations with reference to those dates.
Track image creation tickets
Find the image creation ticket from this list. There are separate live image and installer image tickets for the RATs builds for each milestone of each release. Add yourself to the CC list. Then keep tracking this ticket until the images are available. If the images are not posted on time as scheduled or have a critical bug, please inform the rel-eng team about this by adding comments to the ticket.
Create test result pages and categories
Test result pages are needed to gather test results of installation and desktop against Fedora candidate builds. Refer to QA:Create_Install_Test_Result_Page for guidance to create an installation result page using QA:Fedora_Rawhide_Acceptance_Test_Template.
Often you will need to deal with categories. For each release, there should be a category named Fedora (Release) Test Results, which should be a member of the category Test Results. For each milestone, there should be at least one categories - Fedora_(Release)_(Milestone)_Rawhide_Acceptance_Test - which should be members of the category Fedora (Release) Test Results. For the very first candidate build for each Fedora release - usually Alpha RATs1 - you will need to edit the Fedora (Release) Test Results category page and make it a member of the Test Results category, and edit the Fedora_(Release)_(Milestone)_Rawhide_Acceptance_Test category page and make it a member of the Fedora (Release) Test Results category. Then ensure that the results pages you create are made members of the Fedora (Release) (Milestone) Test Results category. For the following candidate builds, follow this general procedure to add category pages to the higher-level categories and categorize results pages as appropriate.
Please note that you can make some special changes to the result pages according to the practical situation, such as copying parts of previous results to the new result page, highlighting critical cases, adding important notes etc.
Change current links and IRC topic
One or two days before test event day, update the following redirect links and make sure they point to the pages you created:
Also change the topic of #fedora-qa (freenode) to current test event (if you have the permission).
Make announcements
Currently, we announce validation test events to the test-announce mailing list. The announcement mail should provide enough information related to the test focus areas. See an example which generally includes:
- Introduction of this test event (date, what to test, which release criteria to meet, etc)
- Others to be emphasized
Perform testing
When the needed files has been prepared, please refer to the rawhide acceptance test plan which can help you finish the testing. Following but not limited with these steps， try to find and file as more bugs as earlier which you suspect will block or impact Alpha TC1 testing.
Report and Summary
After testing has been completed， send a test summary to the test-announce mailing list. The test summary is intended to keep testers informed as to what was accomplished by their testing, whether there are any remaining tasks and to recognize key contributors. A sample report is available at http://lists.fedoraproject.org/pipermail/test/2011-July/101376.html.
The test summary should include the following information:
- Test status
- A summary about what has been tested, what's remaining to accomplish, what issues were faced during test and whether we have achieved testing aim
- Bugs reported
- List filed bugs and reported issues which were found during test. Bug list including bug id, status, summary, etc...
- The following command can be used to generate a list of bugs:
curl --stderr /dev/null "https://fedoraproject.org/wiki/Test_Results:Current_Installation_Test" \
| grep -o "bugzilla\.redhat\.com.*[=\/][0-9]\{6\}" \
| grep -o "[0-9]\{6\}" | tr '\n' ',' | sed 's|,$||' \
| xargs bugzilla query --outputformat="%{bug_id} %{bug_status} %{resolution} - %{short_desc} - %{blocked}" -b
When a new build is available
If a new build is made available for testing, such as Pre-Alpha RATs2, Pre-Alpha RATs3 etc., you should re-do the entire process: create new test result pages for it, update the redirect pages, send out a new announcement, do the testing again, and send out a new results summary. The announcement should highlight the specific changes from the previous candidate build, as in this example.