From Fedora Project Wiki

 
(48 intermediate revisions by 8 users not shown)
Line 1: Line 1:
== Background ==
Largely inspired by the success [[Fedora_13_QA_Retrospective | Fedora 13 QA Retrospecitve]] and the benefit of capturing data from ideas and events when they happen, this page is for tracking Fedora 14 schedule successes and mis-steps.  It's goal is iteratively track and refine our process of [http://poelcat.wordpress.com/2010/06/30/shipping-on-time-is-a-feature-for-fedora-14/ testing and releasing Fedora 14 on time].
Any Fedora team member or community observer is welcome to add information here.
== Successes ==
{|class="wikitable"
!border="1"| Task  || Date  || What Happened? || How to Repeat It?
|-
|                  ||        ||                ||
|-
|}
== Could Have Been Better ==
== Could Have Been Better ==


{|class="wikitable"
{|class="wikitable"
!border="1"| Task  || Date  || What Happened? || What To Do Differently Next Time?
!border="1"| Task  || Date  || What Happened? || What To Do Differently Next Time? || Actions Taken
|-
|-
|   Create Installable Images for QA testing #1  || 2010-07-08    ||  
| Create Installable Images for QA testing #1  || 2010-07-08    ||  
* Not ready until 2010-07-12   
* Not ready until 2010-07-12   
* Anaconda did not land in rawhide until 2010-07-08
* Installer lead was out during week, backup built [http://koji.fedoraproject.org/koji/buildinfo?buildID=182543 anaconda-14.10-1.fc14] and [http://koji.fedoraproject.org/koji/buildinfo?buildID=182553 pykickstart-1.76-1.fc14] in rawhide on 2010-07-08
* Release Engineering lead was on out on 2010-07-09
* Release Engineering lead was on out on 2010-07-09
* Koji Outage 2010-07-10 to 2010-07-11 ||
* Koji Outage 2010-07-10 to 2010-07-11  
||
* Verify that desired anaconda version is available the day before compose
* Verify that desired anaconda version is available the day before compose
* Coordinate with installer lead several weeks prior
* Create a checklist of reminders and dependencies that should be confirmed prior to the compose date?
* Send a reminder to devel-announce to have packages in ''rawhide'' for upcoming compose
* Build up the Release Engineering team so that alternate team members can compose releases
* Build up the Release Engineering team so that alternate team members can compose releases
||
* Added specific tasks to the development schedule reminding of need for latest installter build
* Continue deepening Release Engineering SOPs
|-
|  Create Installable Images for QA testing #3  || 2010-07-22 ||
* Not ready until 2010-07-23
* glibc change for minimal kernel delayed getting rawhide composed
||
* Urge the glibc folks to communicate better when they are going to make disruptive changes like requiring a much newer kernel.
* Enhance QA and Release Engineering tools to automatically discover breakages like this in advance--prior to the actual compose.
|| Keep in mind for the future.
|-
|-
| Fedora 14 Blocker Meetings are already taking a minimum of an hour || 2010-07-24 ||
* We are averaging 10 minutes of discussion per bug--this will be a problem as the number of blockers grows
* Not getting timely feedback needed from bug assignee (package maintainer)
* In the absence of information from the maintainer the meeting is held up while various people try to research the problem and group discussion attempts to fill in information gaps--many times this works, but it is incredibly inefficient without the package maintainer present.
* IRC communication can be ''laggy'' and disorganized
||
* Review and hound maintainers for information about bugs in the comments of proposed and approved blocker bugs as soon as they come in--don't wait for Friday's meeting
* Use Fedora Talk (conferencing system) to speed up discussion and keep meeting more focused
** stream audio to the web
** people without the ability to connect to Fedora Talk can listen to web stream and respond in IRC
|| Continue to look for ways to streamline meetings.  By the end of the release cycle the current approach seemed to still be the best, though not perfect.
|-
| Branch Fedora 14 || 2010-07-27 ||
* Completed on 2010-07-29
* New boost package landed on 2010-07-27
* Python rebuild to 2.7 took longer than expected
* Change from CVS to Git 
||
* Assume that branching takes more than one day?
|| Keep an eye to see if this a reoccurring problem in Fedora 15
|-
|| Fedora 14 Alpha Test Compose || 2010-07-29 ||
* Test Compose #1 produced on 2010-08-01.  Does not boot because of [https://bugzilla.redhat.com/show_bug.cgi?id=620274 bug 602274]
* Test Compose #2 produced on 2010-08-03.  Cannot install because of [https://bugzilla.redhat.com/show_bug.cgi?id=620900 bug 620900]
* Gave up on Test Compose #3 and moved to preparing for the Release Candidate
||
* Smash less major changes together at the same time: python 2.7, changing SCM, systemd
* Could we do any sanity testing before publishing images to make sure they boot to save bandwidth and turn around time?  In the case of Test Compose #1 it fell over right way.
* Kevin Kofler suggests it may ultimately have worked out better simply to delay the Alpha freeze and leave dist-f14 tag open until the major problems were resolved
|| We don't see any specific actions that can be taken at this time.  Hopefully good lessons were learned from this experience so that they are not repeated.
|-
|| Feature Submission Deadline || 2010-07-13  ||
* It isn't obvious that spins are included in this item, and hence are due before this date.
* Seems that spin deadline could be later, there is so much other stuff eg even the main Fedora gnome/kde spins to test. (The current spins wrangler disagrees.)
||
* Concentrate on one spin successful booting (what does this mean? [[User:Poelstra|poelcat]] 20:53, 23 November 2010 (UTC))
* Set a slightly or much later spin completion date (what completion date? [[User:Poelstra|poelcat]] 20:53, 23 November 2010 (UTC))
||
* TBD
* Need more information from Spins SIG to clarify above questions and also need to know how they would like the schedule to be set for Fedora 15
|-
|| Nightly Spin composes || 2010-07-* ||
* soname bumps broke a number of the spins that made it difficult to test spins and that cascaded into discovering several bugs very close to the alpha freeze and a lot of scrambling to solve then in a short period of time.
||
* Get features that impact lots of other packages (or testing thereof) in well before the deadline.
||
* Discuss with FESCo to see if there is anything that could be pursued here.
|-
|| Nightly Spin ISOs for Alpha || 2010-08-24 ||
* Overlapping issues with broken dependencies have resulted in the Games Spin not composing since July 27th at which point the Splash Screen bug (617115) still existed in livecd-tools. So the Games Spin ISO that exists at the Alpha release will be unusable.
||
* Don't let updates with dependency issues into stable.
|| Investigate current policy around dependency issues in stable and discuss policy changes (if applicable) with FESCo.
|-
|| Notify mirrors about new release content || 2010-08-23 ||
* https://fedorahosted.org/rel-eng/ticket/3862
* No SOP exists for this particular rel-eng task.
* Mirrors may have gotten word late about the F14 Alpha content availability, compared to previous releases.  A notice was sent [https://www.redhat.com/mailman/private/mirror-list-d/2010-August/msg00003.html 24 hours ahead (subscriber only link)].  Only about 50% of the mirrors included F14 Alpha content by 1400 UTC 2010-08-24, which may have been related.  Usually the mirror list is notified the Friday before the release.
||
* Include a task on the schedule going forward to include making this announcement.
* Include the actual procedure in the automatically filed rel-eng Trac ticket for releases.
||
* Update schedule to include announcment on Friday following, Thursday "sync to mirrors."
* Coordinate writing of SOP with Release Engineering
|-
|| Web content change for F14 Alpha || 2010-08-24 ||
* Links in web content didn't match the actual mirror content (web links started with ''F14-'' whereas actual content starts with ''Fedora-''
||
* [[Fedora Websites Release SOP]] should include info on naming, preferably point to the Release Engineering SOP for creating the ISOs we ship
* Release Engineering needs an SOP for creating the content (including standard naming)
||
* Work with Websites team to clarify SOP
* Work with Release Engineering team to create SOP
|-
|| Meeting times || 2010-08-31 ||
* Blocker meetings are in the schedule without a specified time so hard to plan ahead for people in different timezones.
||
* If possible include the time of meetings in the ical schedule - so they appear at right time/date in calendars.
||
* Add UTC time to schedule task descriptions.
* Note: unfortunately there are some bugs in TaskJuggler that make it difficult/impossible to add specific times to the ical file--this is why all ical entries are "full day" events, agree this is not ideal. [[User:Poelstra|poelcat]] 20:53, 23 November 2010 (UTC)
|-
| Compose Beta RC || 2010-09-16 || RC compose was not ready until the end of 2010-09-17 because of two un-resolved blocker bugs (anaconda and kernel).  Both bugs had been open for several days. ||  Determine if one week is really enough time from the creation of the ''Test Compose'' and and the compose of the ''Release Candidate''.  During this one week period we are supposed to test the Test Compose, identify bugs, and fix all blocker bugs, so the compose of the ''Release Candidate'' can begin.
|| Pulled Test Compose in by two days--instead of starting on Thursday, have Release Engineering create two days earlier (Tuesday). The result is that testing will happen earlier too. [[User:Poelstra|poelcat]] 20:53, 23 November 2010 (UTC)
|-
| Building new spin-kickstart packages || 2010-10-12  || With the new updates policy we can't get the updated package into stable on freeze day unless we have more lead time. || Since there is already a plan to do an update if there is any change during the freeze, it isn't a big deal to do the package builds 8 days earlier. The branch before the GA freeze should stay aligned with that package build and also move 8 days earlier. (After looking more closely the spin freeze is 7 days before the final change deadline and maybe we don't need a full 8 days more.)
|| Ask for clarification on spins mailing.  It is unclear what the specific changes should be made to the schedule.
|-
| Testing live spins || 2010-10-22 || While I (bruno) didn't get some stuff done in time this time around, trying to go through test cases at the RC phase is too late. || If we do spins the same for F15, we need to have the mandatory testing to be done by spin owners sync'd up with the TC phase, so that we have time to react.
||
* Need confirmation as to where final spins creation should land on schedule
* Need confirmation/clarification as to whether spins happen for Alpha and Beta and if so what create date should be
* Add explicit task(s) for testing spins to schedule after discussing dates with Spins SIG
* Consider discussion/review at FUDCon
|-
| Final Release Readiness Meeting || 2010-10-18 || Key team leads did not attend or came very late: Design, Release Engineering, Infrastructure, and Websites. || || Touch base with attendees personally and ask they that send an alternate if they cannot attend.
|-
|-
| New FI task: Hide -Alpha and -Beta releases || 2010-11-02 || MirrorManager task to hide version-Alpha and -Beta from the mirror lists on release day || || Added an additional Release Engineering ticket to for the final release to: [[Release_Engineering_Release_Tickets#Final]]
|-
<!-- | Task  || Date  || What Happened? || What To Do Differently Next Time? -->
|}
|}
[[Category:F14]]

Latest revision as of 19:01, 24 November 2010

Could Have Been Better

Task Date What Happened? What To Do Differently Next Time? Actions Taken
Create Installable Images for QA testing #1 2010-07-08
  • Not ready until 2010-07-12
  • Installer lead was out during week, backup built anaconda-14.10-1.fc14 and pykickstart-1.76-1.fc14 in rawhide on 2010-07-08
  • Release Engineering lead was on out on 2010-07-09
  • Koji Outage 2010-07-10 to 2010-07-11
  • Verify that desired anaconda version is available the day before compose
  • Coordinate with installer lead several weeks prior
  • Create a checklist of reminders and dependencies that should be confirmed prior to the compose date?
  • Send a reminder to devel-announce to have packages in rawhide for upcoming compose
  • Build up the Release Engineering team so that alternate team members can compose releases
  • Added specific tasks to the development schedule reminding of need for latest installter build
  • Continue deepening Release Engineering SOPs
Create Installable Images for QA testing #3 2010-07-22
  • Not ready until 2010-07-23
  • glibc change for minimal kernel delayed getting rawhide composed
  • Urge the glibc folks to communicate better when they are going to make disruptive changes like requiring a much newer kernel.
  • Enhance QA and Release Engineering tools to automatically discover breakages like this in advance--prior to the actual compose.
Keep in mind for the future.
Fedora 14 Blocker Meetings are already taking a minimum of an hour 2010-07-24
  • We are averaging 10 minutes of discussion per bug--this will be a problem as the number of blockers grows
  • Not getting timely feedback needed from bug assignee (package maintainer)
  • In the absence of information from the maintainer the meeting is held up while various people try to research the problem and group discussion attempts to fill in information gaps--many times this works, but it is incredibly inefficient without the package maintainer present.
  • IRC communication can be laggy and disorganized
  • Review and hound maintainers for information about bugs in the comments of proposed and approved blocker bugs as soon as they come in--don't wait for Friday's meeting
  • Use Fedora Talk (conferencing system) to speed up discussion and keep meeting more focused
    • stream audio to the web
    • people without the ability to connect to Fedora Talk can listen to web stream and respond in IRC
Continue to look for ways to streamline meetings. By the end of the release cycle the current approach seemed to still be the best, though not perfect.
Branch Fedora 14 2010-07-27
  • Completed on 2010-07-29
  • New boost package landed on 2010-07-27
  • Python rebuild to 2.7 took longer than expected
  • Change from CVS to Git
  • Assume that branching takes more than one day?
Keep an eye to see if this a reoccurring problem in Fedora 15
Fedora 14 Alpha Test Compose 2010-07-29
  • Test Compose #1 produced on 2010-08-01. Does not boot because of bug 602274
  • Test Compose #2 produced on 2010-08-03. Cannot install because of bug 620900
  • Gave up on Test Compose #3 and moved to preparing for the Release Candidate
  • Smash less major changes together at the same time: python 2.7, changing SCM, systemd
  • Could we do any sanity testing before publishing images to make sure they boot to save bandwidth and turn around time? In the case of Test Compose #1 it fell over right way.
  • Kevin Kofler suggests it may ultimately have worked out better simply to delay the Alpha freeze and leave dist-f14 tag open until the major problems were resolved
We don't see any specific actions that can be taken at this time. Hopefully good lessons were learned from this experience so that they are not repeated.
Feature Submission Deadline 2010-07-13
  • It isn't obvious that spins are included in this item, and hence are due before this date.
  • Seems that spin deadline could be later, there is so much other stuff eg even the main Fedora gnome/kde spins to test. (The current spins wrangler disagrees.)
  • Concentrate on one spin successful booting (what does this mean? poelcat 20:53, 23 November 2010 (UTC))
  • Set a slightly or much later spin completion date (what completion date? poelcat 20:53, 23 November 2010 (UTC))
  • TBD
  • Need more information from Spins SIG to clarify above questions and also need to know how they would like the schedule to be set for Fedora 15
Nightly Spin composes 2010-07-*
  • soname bumps broke a number of the spins that made it difficult to test spins and that cascaded into discovering several bugs very close to the alpha freeze and a lot of scrambling to solve then in a short period of time.
  • Get features that impact lots of other packages (or testing thereof) in well before the deadline.
  • Discuss with FESCo to see if there is anything that could be pursued here.
Nightly Spin ISOs for Alpha 2010-08-24
  • Overlapping issues with broken dependencies have resulted in the Games Spin not composing since July 27th at which point the Splash Screen bug (617115) still existed in livecd-tools. So the Games Spin ISO that exists at the Alpha release will be unusable.
  • Don't let updates with dependency issues into stable.
Investigate current policy around dependency issues in stable and discuss policy changes (if applicable) with FESCo.
Notify mirrors about new release content 2010-08-23
  • https://fedorahosted.org/rel-eng/ticket/3862
  • No SOP exists for this particular rel-eng task.
  • Mirrors may have gotten word late about the F14 Alpha content availability, compared to previous releases. A notice was sent 24 hours ahead (subscriber only link). Only about 50% of the mirrors included F14 Alpha content by 1400 UTC 2010-08-24, which may have been related. Usually the mirror list is notified the Friday before the release.
  • Include a task on the schedule going forward to include making this announcement.
  • Include the actual procedure in the automatically filed rel-eng Trac ticket for releases.
  • Update schedule to include announcment on Friday following, Thursday "sync to mirrors."
  • Coordinate writing of SOP with Release Engineering
Web content change for F14 Alpha 2010-08-24
  • Links in web content didn't match the actual mirror content (web links started with F14- whereas actual content starts with Fedora-
  • Fedora Websites Release SOP should include info on naming, preferably point to the Release Engineering SOP for creating the ISOs we ship
  • Release Engineering needs an SOP for creating the content (including standard naming)
  • Work with Websites team to clarify SOP
  • Work with Release Engineering team to create SOP
Meeting times 2010-08-31
  • Blocker meetings are in the schedule without a specified time so hard to plan ahead for people in different timezones.
  • If possible include the time of meetings in the ical schedule - so they appear at right time/date in calendars.
  • Add UTC time to schedule task descriptions.
  • Note: unfortunately there are some bugs in TaskJuggler that make it difficult/impossible to add specific times to the ical file--this is why all ical entries are "full day" events, agree this is not ideal. poelcat 20:53, 23 November 2010 (UTC)
Compose Beta RC 2010-09-16 RC compose was not ready until the end of 2010-09-17 because of two un-resolved blocker bugs (anaconda and kernel). Both bugs had been open for several days. Determine if one week is really enough time from the creation of the Test Compose and and the compose of the Release Candidate. During this one week period we are supposed to test the Test Compose, identify bugs, and fix all blocker bugs, so the compose of the Release Candidate can begin. Pulled Test Compose in by two days--instead of starting on Thursday, have Release Engineering create two days earlier (Tuesday). The result is that testing will happen earlier too. poelcat 20:53, 23 November 2010 (UTC)
Building new spin-kickstart packages 2010-10-12 With the new updates policy we can't get the updated package into stable on freeze day unless we have more lead time. Since there is already a plan to do an update if there is any change during the freeze, it isn't a big deal to do the package builds 8 days earlier. The branch before the GA freeze should stay aligned with that package build and also move 8 days earlier. (After looking more closely the spin freeze is 7 days before the final change deadline and maybe we don't need a full 8 days more.) Ask for clarification on spins mailing. It is unclear what the specific changes should be made to the schedule.
Testing live spins 2010-10-22 While I (bruno) didn't get some stuff done in time this time around, trying to go through test cases at the RC phase is too late. If we do spins the same for F15, we need to have the mandatory testing to be done by spin owners sync'd up with the TC phase, so that we have time to react.
  • Need confirmation as to where final spins creation should land on schedule
  • Need confirmation/clarification as to whether spins happen for Alpha and Beta and if so what create date should be
  • Add explicit task(s) for testing spins to schedule after discussing dates with Spins SIG
  • Consider discussion/review at FUDCon
Final Release Readiness Meeting 2010-10-18 Key team leads did not attend or came very late: Design, Release Engineering, Infrastructure, and Websites. Touch base with attendees personally and ask they that send an alternate if they cannot attend.
New FI task: Hide -Alpha and -Beta releases 2010-11-02 MirrorManager task to hide version-Alpha and -Beta from the mirror lists on release day Added an additional Release Engineering ticket to for the final release to: Release_Engineering_Release_Tickets#Final