QA/SOP Release Validation Test Event

From FedoraProject

< QA(Difference between revisions)
Jump to: navigation, search
(re-word a bit to note that the security lab matrix is conditional on security lab spin being built)
m
(10 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 
== Introduction ==
 
== Introduction ==
The QA group co-ordinates release validation test events before each Fedora release and pre-release is made, to ensure they reach [[Fedora_Release_Criteria]]. Testing covers [[QA:Installation_validation_testing|installation process]][[QA:Desktop_validation_testing|desktop functionality]] and custom spins. This page provides the guidance on arranging such a release validation test event. For any concerns, please contact the [[QA]] group.
+
The QA group co-ordinates release validation test events before each Fedora release and pre-release is made, to ensure they reach [[Fedora_Release_Criteria]]. Testing covers the [[QA:Installation_validation_testing|installation process]] and key [[QA:Desktop_validation_testing|desktop functionality]]. There are also some tests for specific spins. This page provides guidance on arranging and announcing such a release validation test event. For any concerns, please contact the [[QA]] group.
  
 
== Look up the date ==
 
== Look up the date ==
The validation test event is held one or two weeks before the release date for any given milestone. See [http://rbergero.fedorapeople.org/schedules/f-{{FedoraVersion||next}}/f-{{FedoraVersion||next}}-quality-tasks.html quality task schedule] for specific dates as ''Test 'version' Test Compose'' or ''Test 'Version' Candidate''. Then make the following preparations of that.  
+
Validation test events are held for each candidate build: test compose or release candidate. Several of these builds are made in the period one or two weeks before the release date for any given milestone (Alpha, Beta and Final). See [http://jreznik.fedorapeople.org/schedules/f-{{FedoraVersion||next}}/f-{{FedoraVersion||next}}-quality-tasks.html quality task schedule] for specific dates, listed as ''Test 'version' Test Compose'' or ''Test 'version' Candidate''. Then make the following preparations with reference to those dates.  
  
 
== Track image creation tickets==
 
== Track image creation tickets==
Find relative image creation ticket from [https://fedorahosted.org/rel-eng/report/2 tickets list] and add yourself to the cc. Then keep tracking this ticket until the images are available. If the images are not posted on time as scheduled or have a critical bug, please inform the rel-eng team about this by adding comments to the ticket.   
+
Find the image creation ticket from [https://fedorahosted.org/rel-eng/report/2 this list]. There are separate live image and installer image tickets for the TC and RC builds for each milestone of each release: so four tickets per milestone, twelve tickets per release. Add yourself to the CC list. Then keep tracking this ticket until the images are available. If the images are not posted on time as scheduled or have a critical bug, please inform the rel-eng team about this by adding comments to the ticket.   
 
   
 
   
== Create test result page ==
+
== Create test result pages and categories ==
Test result pages are needed to gather test results of installation and desktop against Fedora Release Candidate builds. Refer to [[QA:Create_Install_Test_Result_Page]] for guidance to create an installation result page using [[QA:{{FedoraVersion|long|next}}_Install_Results_Template]]. In a similar way, create a desktop result page using [[QA:Desktop_validation_results_template]]. If the candidate build you are announcing includes a [[Security_Lab | Security Lab spin]] build, create a security lab result page using the [[QA:Security_Lab_validation_results_template]].
+
Test result pages are needed to gather test results of installation and desktop against Fedora candidate builds. Refer to [[QA:Create_Install_Test_Result_Page]] for guidance to create an installation result page using [[QA:{{FedoraVersion|long|next}}_Install_Results_Template]]. In a similar way, create a base result page using [[QA:Base_validation_results_template]] and a desktop result page using [[QA:Desktop_validation_results_template]]. If the candidate build you are announcing includes a [[Security_Lab | Security Lab spin]] build, create a security lab result page using the [[QA:Security_Lab_validation_results_template]].
  
Please note that you can make some special changes to the result pages according to the practical situation, such as immigrating parts of previous results to the new result page, highlighting critical cases, adding important notes etc.
+
Often you will need to deal with categories. For each release, there should be a category named '''Fedora (Release) Test Results''', which should be a member of the category '''Test Results'''. For each milestone, there should be two categories - '''Fedora (Release) (Milestone) TC Test Results''' and '''Fedora (Release) (Milestone) RC Test Results''' - which should be members of the category '''Fedora (Release) Test Results'''. For the very first candidate build for each Fedora release - usually Alpha TC1 - you will need to edit the '''Fedora (Release) Test Results''' category page and make it a member of the '''Test Results''' category, and edit the '''Fedora (Release) (Milestone) TC Test Results''' category page and make it a member of the '''Fedora (Release) Test Results''' category. Then ensure that the results pages you create are made members of the '''Fedora (Release) (Milestone) TC Test Results''' category. For the following candidate builds, follow this general procedure to add category pages to the higher-level categories and categorize results pages as appropriate.
 +
 
 +
Please note that you can make some special changes to the result pages according to the practical situation, such as copying parts of previous results to the new result page, highlighting critical cases, adding important notes etc.
  
 
== Change current links and IRC topic ==  
 
== Change current links and IRC topic ==  
 
One or two days before test event day, update the following redirect links and make sure they point to the pages you created:
 
One or two days before test event day, update the following redirect links and make sure they point to the pages you created:
 
* {{noredirect|Test_Results:Current_Installation_Test}}
 
* {{noredirect|Test_Results:Current_Installation_Test}}
 +
* {{noredirect|Test_Results:Current_Base_Test}}
 
* {{noredirect|Test_Results:Current_Desktop_Test}}
 
* {{noredirect|Test_Results:Current_Desktop_Test}}
 
* {{noredirect|Test_Results:Current_Security_Lab_Test}}
 
* {{noredirect|Test_Results:Current_Security_Lab_Test}}
Line 21: Line 24:
 
Also change the topic of #fedora-qa (freenode) to current test event (if you have the permission).
 
Also change the topic of #fedora-qa (freenode) to current test event (if you have the permission).
  
== Make announcement ==
+
== Make announcements ==
Currently, we announce validation test events to the [https://admin.fedoraproject.org/mailman/listinfo/test-announce test-announce]. The announcement mail should provide enough information related to the test focus areas. See an [http://lists.fedoraproject.org/pipermail/test-announce/2010-March/000040.html example] which generally includes:
+
Currently, we announce validation test events to the [https://admin.fedoraproject.org/mailman/listinfo/test-announce test-announce] mailing list. The announcement mail should provide enough information related to the test focus areas. See an [https://lists.fedoraproject.org/pipermail/test-announce/2010-March/000040.html example] which generally includes:
  
 
* Introduction of this test event (date, what to test, which release criteria to meet, etc)
 
* Introduction of this test event (date, what to test, which release criteria to meet, etc)
 +
* For composes after TC1, a note of changes from the last compose or a link to the compose request trac ticket which provides this info
 
* How and where to add test results
 
* How and where to add test results
* Contact information of QA members who are available on test event and can help testers who encounter problems.
+
* Contact information of QA members who are available on test event and can help testers who encounter problems
* Others to be emphasized.
+
* Others to be emphasized
  
Download links should ''not'' be contained in the announcement, since in general there are several download options, and these are already documented on the Install and Desktop test pages.  The announcement should direct people to go to these pages for download instructions.
+
Download links should ''not'' be contained in the announcement, since in general there are several download options, and these are already documented on the result pages.  The announcement should direct people to go to these pages for download instructions.
  
 
Please announce this event on the mailing list at least one or two days in advance.  This should give testers sufficient time to arrange their calendars and prepare a test environment.  It is also a good idea to send a reminder e-mail the day before the test. Try to take timezones into account, to maximize convenience for testers from different regions or countries.
 
Please announce this event on the mailing list at least one or two days in advance.  This should give testers sufficient time to arrange their calendars and prepare a test environment.  It is also a good idea to send a reminder e-mail the day before the test. Try to take timezones into account, to maximize convenience for testers from different regions or countries.
 +
 +
=== Supplemental announcements ===
 +
It can be helpful to send supplemental announcements to other interested groups. It is helpful to notify the major desktop groups - [https://lists.fedoraproject.org/pipermail/desktop/ desktop], [https://lists.fedoraproject.org/pipermail/kde/ KDE], [https://lists.fedoraproject.org/pipermail/xfce/ Xfce] and [https://lists.fedoraproject.org/pipermail/lxde/ LXDE] - of each candidate build, and refer them to the desktop testing matrix. Here is an [https://lists.fedoraproject.org/pipermail/lxde/2011-February/000062.html example announcement]. If possible, it is a good idea to do a 'smoke test' on each desktop live image - ensure it boots successfully to a working desktop - before sending the announcement, to avoid wasting the time and bandwidth of the desktop group members downloading un-testable images.
  
 
== Provide help during test event ==
 
== Provide help during test event ==
Line 40: Line 47:
 
* Reference of ways to communicate at [[Communicate]]
 
* Reference of ways to communicate at [[Communicate]]
  
== When new candidate available ==
+
== When a new candidate build available ==
If a new candidate is available for testing, such as TC2, TC3, RC3, you need send a new announcement for this, like the [http://lists.fedoraproject.org/pipermail/test-announce/2010-March/000052.html example] and better to explain the differences/fixes between this version and the last one. Then create installation and desktop test result pages for it and repeat the above steps.  
+
If a new candidate build is made available for testing, such as TC2, TC3, RC3, you should re-do the entire process: create new test result pages for it, send out a new announcement, and update the redirects. The announcement should highlight the specific changes from the previous candidate build, as in this [https://lists.fedoraproject.org/pipermail/test-announce/2010-March/000052.html example].
  
 
== Report and Summary ==
 
== Report and Summary ==
After testing has completed, send a test summary to the [https://admin.fedoraproject.org/mailman/listinfo/test-announce test-announce] mailing list.  The test summary is intended to keep testers informed as to what was accomplished by their testing, whether there are any remaining tasks and to recognize key contributors.  A sample report is available at http://lists.fedoraproject.org/pipermail/test-announce/2010-October/000164.html.
+
After testing has been completed, send a test summary to the [https://admin.fedoraproject.org/mailman/listinfo/test-announce test-announce] mailing list.  The test summary is intended to keep testers informed as to what was accomplished by their testing, whether there are any remaining tasks and to recognize key contributors.  A sample report is available at https://lists.fedoraproject.org/pipermail/test-announce/2010-October/000164.html.
  
 
The test summary should include the following information:
 
The test summary should include the following information:
Line 72: Line 79:
  
 
[[Category:QA SOPs]]
 
[[Category:QA SOPs]]
[[Category:Install Test]]
+
[[Category:Installation_validation_testing]]
 +
[[Category:Desktop validation testing]]
 +
[[Category:Base validation testing]]

Revision as of 09:19, 22 October 2012

Contents

Introduction

The QA group co-ordinates release validation test events before each Fedora release and pre-release is made, to ensure they reach Fedora_Release_Criteria. Testing covers the installation process and key desktop functionality. There are also some tests for specific spins. This page provides guidance on arranging and announcing such a release validation test event. For any concerns, please contact the QA group.

Look up the date

Validation test events are held for each candidate build: test compose or release candidate. Several of these builds are made in the period one or two weeks before the release date for any given milestone (Alpha, Beta and Final). See quality task schedule for specific dates, listed as Test 'version' Test Compose or Test 'version' Candidate. Then make the following preparations with reference to those dates.

Track image creation tickets

Find the image creation ticket from this list. There are separate live image and installer image tickets for the TC and RC builds for each milestone of each release: so four tickets per milestone, twelve tickets per release. Add yourself to the CC list. Then keep tracking this ticket until the images are available. If the images are not posted on time as scheduled or have a critical bug, please inform the rel-eng team about this by adding comments to the ticket.

Create test result pages and categories

Test result pages are needed to gather test results of installation and desktop against Fedora candidate builds. Refer to QA:Create_Install_Test_Result_Page for guidance to create an installation result page using QA:Fedora 21_Install_Results_Template. In a similar way, create a base result page using QA:Base_validation_results_template and a desktop result page using QA:Desktop_validation_results_template. If the candidate build you are announcing includes a Security Lab spin build, create a security lab result page using the QA:Security_Lab_validation_results_template.

Often you will need to deal with categories. For each release, there should be a category named Fedora (Release) Test Results, which should be a member of the category Test Results. For each milestone, there should be two categories - Fedora (Release) (Milestone) TC Test Results and Fedora (Release) (Milestone) RC Test Results - which should be members of the category Fedora (Release) Test Results. For the very first candidate build for each Fedora release - usually Alpha TC1 - you will need to edit the Fedora (Release) Test Results category page and make it a member of the Test Results category, and edit the Fedora (Release) (Milestone) TC Test Results category page and make it a member of the Fedora (Release) Test Results category. Then ensure that the results pages you create are made members of the Fedora (Release) (Milestone) TC Test Results category. For the following candidate builds, follow this general procedure to add category pages to the higher-level categories and categorize results pages as appropriate.

Please note that you can make some special changes to the result pages according to the practical situation, such as copying parts of previous results to the new result page, highlighting critical cases, adding important notes etc.

Change current links and IRC topic

One or two days before test event day, update the following redirect links and make sure they point to the pages you created:

Also change the topic of #fedora-qa (freenode) to current test event (if you have the permission).

Make announcements

Currently, we announce validation test events to the test-announce mailing list. The announcement mail should provide enough information related to the test focus areas. See an example which generally includes:

  • Introduction of this test event (date, what to test, which release criteria to meet, etc)
  • For composes after TC1, a note of changes from the last compose or a link to the compose request trac ticket which provides this info
  • How and where to add test results
  • Contact information of QA members who are available on test event and can help testers who encounter problems
  • Others to be emphasized

Download links should not be contained in the announcement, since in general there are several download options, and these are already documented on the result pages. The announcement should direct people to go to these pages for download instructions.

Please announce this event on the mailing list at least one or two days in advance. This should give testers sufficient time to arrange their calendars and prepare a test environment. It is also a good idea to send a reminder e-mail the day before the test. Try to take timezones into account, to maximize convenience for testers from different regions or countries.

Supplemental announcements

It can be helpful to send supplemental announcements to other interested groups. It is helpful to notify the major desktop groups - desktop, KDE, Xfce and LXDE - of each candidate build, and refer them to the desktop testing matrix. Here is an example announcement. If possible, it is a good idea to do a 'smoke test' on each desktop live image - ensure it boots successfully to a working desktop - before sending the announcement, to avoid wasting the time and bandwidth of the desktop group members downloading un-testable images.

Provide help during test event

During the test event, many people will participate, including experienced users and new comers. Make sure the QA folks whose contact information was announced to mailing list in this test event are available during the testing period. They will provide assistance to those who encounter issues. QA people should be available at:

When a new candidate build available

If a new candidate build is made available for testing, such as TC2, TC3, RC3, you should re-do the entire process: create new test result pages for it, send out a new announcement, and update the redirects. The announcement should highlight the specific changes from the previous candidate build, as in this example.

Report and Summary

After testing has been completed, send a test summary to the test-announce mailing list. The test summary is intended to keep testers informed as to what was accomplished by their testing, whether there are any remaining tasks and to recognize key contributors. A sample report is available at https://lists.fedoraproject.org/pipermail/test-announce/2010-October/000164.html.

The test summary should include the following information:

Test status
A summary about what has been tested, what's remaining to accomplish, what issues were faced during test and whether we have achieved testing aim
Bugs reported
List filed bugs and reported issues which were found during test. Bug list including bug id, status, summary, etc...
The following command can be used to generate a list of bugs:
curl --stderr /dev/null "https://fedoraproject.org/wiki/Test_Results:Current_Installation_Test" \
   | grep -o "bugzilla\.redhat\.com.*[=\/][0-9]\{6\}" \
   | grep -o "[0-9]\{6\}" | tr '\n' ',' | sed 's|,$||' \
   | xargs bugzilla query --outputformat="%{bug_id} %{bug_status} %{resolution} - %{short_desc} - %{blocked}" -b
Note.png
Got python-bugzilla?
This command requires the Package-x-generic-16.pngpython-bugzilla package be installed. You may need to weed out any 'false' references to pre-existing bugs which are mentioned on the page for some reason.
Analyze results and assess risk
Analyze the results and provide an assessment where additional risk areas may exist
Thanks to all testers
Testers have volunteered their time, it never hurts to be thankful for their contributions.
To generate a list of testers who contributed to the wiki, you can use the following command:
curl --stderr /dev/null "https://fedoraproject.org/w/index.php?title=Test_Results:Fedora_14_Final_RC1_Install&action=raw" \
   | grep -A999999 "^=.*Test Matrix.*=$" \
   | sed -e "s|{{\(result[^}]*\)}}|\n\1 |g" | grep "^result" \
   | gawk 'BEGIN{FS="[| ]"} $3{print $3} !$3{print "UNTESTED"}' \
   | sort | uniq -c | sort -bgr