From Fedora Project Wiki

Revision as of 07:01, 4 October 2013 by Tflink (talk | contribs) (initial outline of phase 0 and phase1)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

Introduction

There are a lot of moving bits and pieces to where we'd like to see qa automation go for Fedora. While it would be great to say that all of it could get done in a month or two, that's really not practical. We're aiming for a phased milestone approach to replacing and improving our existing systems in Fedora.

The list of projects that we're planning on is listed elsewhere. The aim of this document is to describe the order in which we're planning to tackle those tasks.

Phases

Phase 0: Investigation and Preparation

Phase 0 (if it can be called a phase) is mostly for investigation and preparation. There are still non-trivial details that need to be ironed out and decisions made on initial direction and preparation work that still needs to be done.

Preparation

We still need to set up and/or configure tools like a bug tracker, CI, code reviews and repositories. This doesn't mean that we're going to set up our own instances for everything - some or all of those roles will be filled by existing Fedora services.

Investigation

The large item for investigation is Task execution

  • investigate the possible strategies
  • make a decision on which one to implement
  • determine whether a central database for tasks is needed

We also need to investigate the possibility of missing messages for scheduling and verify that the current strategy is likely to work.

Deliverables

  • Decision on method and plan for task execution
  • Results of fedmsg investigation and plan for any scheduling work that will be needed for phase 1
  • Support tooling configured, setup and ready for use

Phase 1: AutoQA Replacement

In order to reduce the amount of resources (both computing and human) spent on automation, the focus of phase 1 will be to fill the roles currently filled by AutoQA

Checks Run for All Packages/Updates

  • depcheck
  • rpmlint
  • upgradepath
  • repoclosure
  • rpmguard?

Depcheck

Depcheck does not currently work with yum in Fedora 19 and newer. As Fedora 18 will be going EOL approximately 30 days after the release of Fedora 20, this needs to be addressed soon.

This can be dealt with in one of two ways:

  • replace depcheck with a similar test that is easier to maintain
  • fix depcheck so that it works with yum in Fedora 19 and later, delaying the need to replace depcheck

Result Notification

Instead of implementing new methods for notifying maintainers of results, we will continue with using bodhi comments for check result notification.

Deliverables

  • Deployment of functional system capable of the following:
    • running appropriate checks on koji builds
    • running appropriate checks on bodhi updates
    • notification of results through bodhi comments
Notification for builds
what to do about package checks that aren't appropriate for bodhi? what do we want to do about email notification?


Phase X: Beaker Integration

TBD
This is just a placeholder for now, we want to nail down phase 0 and phase 1 a bit better before getting into the order and detail of later phases

Phase Y: Installation Checks

TBD
This is just a placeholder for now, we want to nail down phase 0 and phase 1 a bit better before getting into the order and detail of later phases

Phase Z: More Sophisticated Checks

TBD
This is just a placeholder for now, we want to nail down phase 0 and phase 1 a bit better before getting into the order and detail of later phases

Later Work

This is not a complete plan for qa automation but detailed plans farther out isn't going to be productive at this time.

Some discussion of eventual goals as a high level roadmap, however, would be very helpful.