User:Roshi/QA/Cloud Docs/Test Overview

From FedoraProject

Jump to: navigation, search

Objective

This document aims to establish what testing is necessary to ensure the milestone releases of the Fedora Cloud product meet the desired levels of quality.

Scope

The Cloud Products Requirements Document - henceforth, "PRD" - defines the high-level aims of the Cloud product. Significant points include:

  • The product will be delivered as a self-contained image (AMI, qcow2, raw.xz, etc). This means testing of the product will not require any form of installation testing.
  • Aims to support the cattle aspect of the [pets vs cattle] metaphor of computing.
  • The product will be used on public and private cloud infrastructures.
  • The product will have access to Fedora package collections, this includes: Access to the standard Fedora package repositories, Standard language stacks (like Python, Ruby, NodeJS, PHP), and Modernized language and application-specific stacks (perhaps in the form of SCLs) for running user-space applications.
  • The product WG has a goal of being able to automatically generate and upload images with updated package sets out of band from normal Fedora releases. This will require a significant amount of testing and work to develop the needed infrastructure for testing generated images.
  • The product WG in concert with the Base WG, aims to increase support for Docker with a custom container image.
  • The product WG aims to provide an streamlined Atomic image for Docker containers.
  • Cloud images also need to be able to support traditional server roles, transitioning one of your "cattle" into a pet (while in the cloud).
  • Various requirements for the base system are documented in the Server TS, some of which are specific to Cloud.

Proposed test coverage

It is reasonable to assume that most of our existing Fedora_Release_Criteria, with the exception of those specifically relating to desktop functionality and installation (anaconda), will be held to apply to the Cloud product. It is therefore equally reasonable to assume our existing QA:Base_validation_testing processes will apply to the Cloud product, while a more thorough Cloud validation testing matrix will need to be created and run for along with existing test matrices for each released image of the Cloud product.

We will also require additional Cloud-specific testing to fully validate images. Developing the automated testing of generated images will form the majority of this work. Specific images (Docker and Big Data for example) will also require their own form of validation (though a large portion of these could likely be automated). All of the "featured" images will need to be tested to ensure they meet the "mandatory requirements" defined in the PRD.