Updates, testing, and you - for F7 and beyond
The Old Days
Here's an overview of the pre-Merge procedures for pushing new updates to a stable distribution.
Fedora - Too little
To push a new update, the procedure is different for Extras and for Core:
Extras:
- Build new packages into the proper place
- Package automatically goes live the next day
Core:
- Build new packages into the proper place
- Use the Updates System to request an update
- Package moves to updates-testing
- Testers test while packages are in updates-testing
- Owner can push packages live at their discretion
Where's the QA on Extras? Loads of apps in F7 don't even start right now.
(See bugs
241419 ,
241450 ,
241465 ,
241055 ,
241425 ,
241416 ,
241443 , and so on)
Why isn't Core QA required? Sometimes updates go out that break stuff! (e.g. openldap)
Why doesn't something check to make sure the packages are signed?
Why doesn't anything check to make sure all the deps are solved before the packages get pushed?
RHEL - Too much
For comparison, here's a rough view of the procedure for pushing a new update for RHEL:
RHEL:
- Obtain approval from Product Management, QA, and Development teams
- Build new packages into the proper place
- File update request, including:
- Update type
- Release announcement
- Full, detailed change log
- List of bugs fixed
- Instructions on testing and reproducing bugs
- Package name and location
- You may not build further packages without permission from PM/QA
- Automated package testing happens
- package checksums, size, etc. are checked
- rpm diff program finds interesting changes in packages (new binaries, etc)
- If unexpected changes happen, developer must request waiver
- [etc]
- Tester (RH QA Associate) verifies packages
- Repeat for all arches (i386, x86_64, ppc, ia64, s390):
- Verify that each bug in the request exists in old packages (if possible)
- Verify new packages can be fetched from RHN
- Verify new packages upgrade properly
- Check each bug listed in request to test fix
- Verify function of packages - run all binaries, test suites, etc.
- Approve / reject packages
- Developer requests update approval from Docs team
- Docs team approves update text
- Check grammar and content of changes
- Check grammar and content of metadata / headers
- Approve request
- Tester readies packages for pushing live
- Request package signing from rel-eng
- Release engineering pushes update live
- Sign approved packages
- Wait for Quarterly Update time (unless Security errata)
- Push packages to live site
Yikes.
Takes a lot of effort from a lot of people
Introduces delays as requests are bounced through errata tool, bugzilla, etc.
Confusing and painful - The only way you can get people to go through this is if you're paying 'em :D
The Proposal
Phase 1 (F7)
- Build new packages into the proper place
- Request update using bodhi, including:
- Update type (security, bugfix, enhancement)
- List of bugs fixed (optional)
- Update notes (optional)
- Package moves to updates-testing
- Testers verify packages get a week to try it out and offer comments
- If feedback is OK, rel-eng signs + pushes packages live
And that's it. Packages get tested, making our users happy, but it's not too much work for developers and testers, and that makes us happy. Everybody wins!