User:Tflink/f21 delay taskotron development proposal

Fedora 21 Delay Proposal: Taskbot Development

This proposal is a DRAFT
This proposal is still an early draft and is still being written - please confirm with tflink before relying on this for anything

Summary

While progress on automated testing has been slow as of late, the changes proposed for Fedora.next will be made much more workable with better automation in the QA space. Instead of working on small improvements as time allows between Fedora releases, we propose that Fedora 21 be delayed so that we can focus on getting our next generation automation system in place to support the new direction for Fedora.

Owner

• Name: Tim Flink
• Email: tflink at fedoraproject org
• Release notes owner:

Current status

• Targeted release: Fedora 21
• Last updated: October 7, 2013
• Tracker bug: None

Assumptions

There are very few specifics on the timeframe for the proposals requested by fesco for projects which could be done during if f21 was delayed for infrastructure/tooling work in qa, releng and other areas. Therefore, I am making several assumptions to make this proposal clearer.

• Any delay would start after January 31, 2014
• Available people is still TBD - this will depend on any other proposals considered and accepted by FESCo for the possible delay period
  • I assume that there will be at least 3 full time engineers dedicated to this work. There may be additional folks from the community but that part is still TBD as I haven't asked for commitments from anyone else yet.
• Conservative estimates that under promise and over deliver are preferred.
  • There are still unknowns in this plan so I am making conservative estimates and leaving a wishlist of things to work on if time permits

Background

While AutoQA has served us well over the years, it has some issues that we need to fix in order to move forward. With this in mind, we have proposed a new automation system which we are calling Taskbot. We will be preserving as much of AutoQA as makes sense but moving forward is a higher priority than reusing old code and solutions.

• An initial description of Taskbot
• How is Taskbot different from AutoQA

The development plan for taskbot describes several phases for incremental development of automation features. The full list of Taskbot sub-projects is also available. Taskbot Phase 0 is currently in progress and unless there are significant disruptions from Fedora 20 testing, will be complete before the end of January 2014.

Detailed Description

We have been planning and starting the investigation process for a new automation system which will eventually replace AutoQA. Unfortunately, this is far from a small undertaking and most of our resources are occupied with testing during the normal Fedora release cycle. There is some time between releases but it has proven to be far shorter than we would need to get a significant project started and finished before the next release.

By delaying the release of Fedora 21, we would be able to commit more resources to automation development and the work would be done more quickly. Exactly how quickly is addressed with scope.

Benefit to Fedora

The main benefit to Fedora would be more test coverage stemming from more automated checking and freeing up human testers to do more complicated tests than currently done as part of release validation. This new automation is coming whether the delay is accepted or not but the delay would free up more developers to work on implementation, bringing these benefits to Fedora faster.

Benefit To End Users

• More stability in Fedora

Benefit To Developers

• More automated checks
• Better communication of results
• Opportunity to have checks added to automation

Benefit To Testers

• Less time spent on the validation treadmill doing repetitive tasks
• More time to participate in test days, test new features

Scope

Schedule TBD
This will be filled in once we have quorum from affected parties (mostly potential developers)

No Delay

• Completion of Taskbot Phase 0
• Completion of Taskbot Phase 1 by May 1, 2014.
• Starting later phases after completion of phase 1.

1 Month Delay

• Completion of Taskbot Phase 0
• Completion of Taskbot Phase 1 by March 1, 2014.
• Starting later phases after completion of phase 1.

2-3 Month Delay

After the completion of Taskbot Phase 1, there are multiple options for more functionality to implement (Later phases in the development plan, other subprojects) but with the exception of implementing an ABI break/change tool (less than 1 month), there are too many unknowns for us to make reasonable estimates on the time required to complete those phases.

We do think it is reasonably likely that we'd be able to complete either Taskbot Phase X or Taskbot Phase Y given an additional 2 months (making for a 3 month total delay) but we are not willing to commit to their completion in that time frame right now.

User Experience

Fedora users may notice the delayed release and will hopefully notice an increase in quality.

Dependencies

There are no dependencies that are not listed in this proposal.

Contingency Plan

We keep working on automation with more limited resources. The scope of our plans will remain the same but our timelines will stretch out without the resources we would otherwise gain from delaying Fedora 21.

Documentation

Documentation is still being written but so far, we have:

• An initial description of Taskbot
• How is Taskbot different from AutoQA
• Taskbot Subprojects
• Development plan for taskbot