From Fedora Project Wiki

(Updates to the problem space definition)
(Problem Space)
 
(37 intermediate revisions by 4 users not shown)
Line 1: Line 1:
 
{{header|qa}}
 
{{header|qa}}
{{draft}}
+
 
 
<!-- This is the FESCo proposal template.  It is not necessary that you use this template but it is encouraged that you do.  It will help FESCo focus on the issue being addressed and the parts that need discussion, as well as identify the scope of the proposal. You may remove all these comment sections from your proposal. -->
 
<!-- This is the FESCo proposal template.  It is not necessary that you use this template but it is encouraged that you do.  It will help FESCo focus on the issue being addressed and the parts that need discussion, as well as identify the scope of the proposal. You may remove all these comment sections from your proposal. -->
  
 
== Overview ==
 
== Overview ==
  
provide small auto-test system with a web-based interface which can help to customize autoinstall testing.The test system should be small enough for personal use.
+
The purpose of AutoQA installation test automation is to simplify testing and improve test efficiency. The AutoQA install test project should address the following problems:
  
== Problem Space ==
+
* The system should be easy to allow for tester and developer use
The purpose of AutoQA install test automation is to simplify testing, reduce the test execution time and improve efficiency.  The AutoQA install test should help us to solve the following problems:
+
* Have clear documentation for customizing and creating new tests
 +
* Support test execution using existing Fedora infrastructure services, but not require them
 +
* Test results are easy to verify
  
* The system should be small enough to allow for developer use
+
== Scope ==
* Have a clear method for customizing and creating new tests
+
The Fedora installation test automation project covers the whole installation from retrieving installation media to first boot. Details are below:
* Work within existing Fedora infrastructure services, but not require them
+
# retrieve installation media
* Test results are easy to verify
+
# Installation media sanity check
 +
# Install with different system specifications, installation sources and partition etc.
 +
# get the installation output and logs.
 +
 
 +
====Problem Space====
 +
The Fedora installer is a complicated software application that often requires significant setup time to properly and efficiently test.  Installer failures typically come from the following areas:
 +
 
 +
''' Image Sanity '''
 +
# ISO file size is too large
 +
# Invalid SHA256 checksum
 +
# Invalid implanted ISO md5sum
 +
# Install environment - anaconda has specific application, library and config file format requirements.
 +
# Versions check
 +
 
 +
''' Boot Methods'''
 +
# Boot media improperly built (PXE, boot.iso, DVD, efidisk.img)
 +
# Installer fails to boot as a KVM guest
 +
# Installer fails to boot as a XEN guest
 +
 
 +
'''Install Source'''
 +
# Unable to detect install.img media
 +
# Unable to transition to stage#2 installer
 +
 
 +
'''Kickstart Delivery'''
 +
# Ks.cfg could not be obtained from specified location (http, ftp, nfs, hd, initrd)
 +
# Install fails to proceed in accordance with the directives in the ks.cfg file
 +
# Install improperly sets up networking based on command-line and kickstart network parameters (boot with <code>ksdevice=eth1</code>, {{filename|ks.cfg}} contains <code>eth2</code>)
  
The installation process covers the following sections, therefore, failures typically come from the following areas:
+
'''User Interface'''
# Image Sanity
+
# X driver problems while transitioning to graphical install
## Image size too large (or small)
+
# Screen corruption during text-mode install
## Invalid SHA256 checksum
+
# VNC fails to start
## Invalid implanted ISO md5sum
+
# Serial console redirection improperly setup
## Versions check
 
# Boot Methods
 
## Boot media improperly built (PXE, boot.iso, CD/DVD, efidisk.img)
 
## Installer fails to boot as a KVM guest
 
## Installer fails to boot as a XEN guest
 
# Install Source
 
## Unable to detect install.img media
 
## Unable to transition to stage#2 installer
 
# Kickstart Delivery
 
## Ks.cfg could not be obtained from specified location (http, ftp, nfs, hd, initrd)
 
## Install fails to proceed in accordance with the directives in the ks.cfg file
 
## Install improperly sets up networking based on command-line and kickstart network parameters (boot with <code>ksdevice=eth1</code>, {{filename|ks.cfg}} contains <code>eth2</code>)
 
# User Interface
 
## X driver problems while transitioning to graphical install
 
## Screen corruption during text-mode install
 
## VNC fails to start
 
## Serial console redirection improperly setup
 
# Storage Devices
 
## Fail to detect existing storage device(s)
 
## Failure to clear stale data off of existing devices
 
## Unable to add iSCSI volumes
 
# Partitioning
 
## Failure detecting existing partition scheme (lvm, mdraid, dmraid, luks)
 
## Failure when attempting to resize existing partitions
 
## Failures while attempting to re-use existing partitions
 
## Improperly clearing stale information from disks
 
## Unable to consistently resize an existing filesystem
 
## General failures while attempting to manually partition a system
 
# Package Repository
 
## Unable to read metadata from package repositories (http, ftp, nfs, media)
 
## Failures while adding or modifying existing package repositories
 
# Package Set
 
## Network timeout while retrieving packages
 
## Dependency problems while resolving package list
 
## File conflicts during package install
 
## Package order and install errors in {{filename|install.log}}
 
## Improperly formatted {{filename|comps.xml}} data
 
# Recovery
 
## Rescue mode fails to detect existing installations (lvm, raid, luks)
 
## Rescue mode fails to establish networking
 
## Problems saving traceback information (local disk, bugzilla, remote server)
 
## Anaconda unable to download and use updates.img (from install source, local media or URL)
 
## Unable to transition to debug mode
 
# Boot loader configuration
 
## Unable to properly detect other operating systems
 
## Failure while setting up chainloader for another OS
 
# Upgrade system
 
## Failure to detect previously installed systems
 
## Errors while attempting to update bootloader configuration during upgrade
 
## Package upgrade errors in {{filename|upgrade.log}}
 
  
Auto-test will test anaconda of these media:DVD,CDROM, LiveCD,Network boot CD.If anaconda broken,typically have these issues:
+
'''Storage Devices'''
* DVD install
+
# Fail to detect existing storage device(s)
# Boot fails from DVD
+
# Failure to clear stale data off of existing devices
# Stage#2 can not start successfully
+
# Unable to add iSCSI volumes
# Partition fails
 
# Reboot to newly installed system fails
 
# Upgrade fails
 
  
* CDROM install
+
'''Partitioning'''
# Fail to boot from CDROM
+
# Failure detecting existing partition scheme (lvm, mdraid, dmraid, luks)
# Stage#2 can not start successfully
+
# Failure when attempting to resize existing partitions
# Fail to request next CDROM
+
# Failures while attempting to re-use existing partitions
# Package errors during installation
+
# Improperly clearing stale information from disks
# Reboot to newly installed system fails
+
# Unable to consistently resize an existing filesystem
 +
# General failures while attempting to manually partition a system
  
* BootCD install
+
'''Package Repository'''
# Fail to boot from BootCD
+
# Unable to read metadata from package repositories (http, ftp, nfs, media)
# kickstart install fails
+
# Failures while adding or modifying existing package repositories
# Ftp install fails
 
# NFS install fails
 
# Repo is missing
 
# Http install fails
 
  
* LiveCD install
+
'''Package Set'''
# LiveCD install fails
+
# Network timeout while retrieving packages
# Reboot to newly installed system fails
+
# Dependency problems while resolving package list
 +
# File conflicts during package install
 +
# Package order and install errors in {{filename|install.log}}
 +
# Improperly formatted {{filename|comps.xml}} data
  
==Proposed Solution ==
+
'''Boot loader configuration'''
<!-- Describe proposed solution in detail -->
+
# Unable to properly detect other operating systems
 +
# Failure while setting up chainloader for another OS
  
*  use virtual machine(libvirt+KVM) to finish install tests one by one.
+
'''Upgrade system'''
*  provide simple interface to customize testing.
+
# Failure to detect previously installed systems
*  Create scripts to do test automatically for anaconda
+
# Errors while attempting to update bootloader configuration during upgrade
*  Create test cases for the above possible breakness of anaconda(DVD,CDROM,BootCD,LiveCD)
+
# Package upgrade errors in {{filename|upgrade.log}}
*  Test result display page help to check the test result easily,the result format is good for testers to post to specified link(have no need to change a lot before post).
 
*  Write test cases From simple to complex, kickstart to graphic.
 
  
== Scope ==
+
'''Recovery'''
<!-- Describe the scope of any project work or properties that will be affected by the proposal -->
+
# Rescue mode fails to detect existing installations (lvm, raid, luks)
 +
# Rescue mode fails to establish networking
 +
# Problems saving traceback information (local disk, bugzilla, remote server)
 +
# Anaconda unable to download and use updates.img (from install source, local media or URL)
 +
# Unable to transition to debug mode
  
* Any testers from community can install the testing system and run test cases
+
== Proposed Solution ==
* After finish testing,testers will post test result to specified wiki
+
<!-- Describe proposed solution in detail -->
 +
The project will be developed based on the AutoQA framework and use the virtualization environment (libvirt,kvm-qemu). The installation automation will be deployed by the Fedora kickstart installation. virt-install will be used to create the guest.virtio is used to get the anaconda logs. The details can be found at [[Is anaconda broken  solution]]
  
 
== Active Ingredients ==
 
== Active Ingredients ==
 
<!-- A list of any prerequisites or other materials needed to implement the solution -->
 
<!-- A list of any prerequisites or other materials needed to implement the solution -->
  
* develop a system to run install system
+
* AutoQA must be packaged and available in Fedora infrastructure
* convert some of the current install test cases to cases which can be run in auto test system
+
* A valid backend test harness is packaged and available in Fedora infrastructure ([[Autotest]] currently being used by [[AutoQA]])
* Provide a test results presentation layer for manual test result submission
+
* Knowledge of TurboGears or TurboGears2
 +
* A new [[AutoQA]] test watcher may be required to initiate automated tests whenever CD and DVD ISO images are available for testing (''post-iso-build'')
 +
* Adding installer unit test execution and results display is dependent on the availability and environmental requirements of installer unit tests
 +
 
 +
 
  
 
== Roadmap ==
 
== Roadmap ==
your comments
+
* Roadmap for development of auto install at:[[Is_anaconda_broken_roadmap]]
 +
* Status for test plan/documentation changes can be seen in the [https://fedorahosted.org/fedora-qa/query?status=new&status=assigned&status=reopened&status=closed&group=status&component=Test+Review&milestone=Fedora+13&order=priority fedora-qa trac system]
 +
* Status for test automation efforts is available in the [https://fedorahosted.org/autoqa/milestone/Automate%20installation%20test%20plan autoqa TRAC system]
  
 
== Results ==
 
== Results ==
'''Version 1.0'''
+
* Meeting information and logs can be found at [[Is_anaconda_broken_meeting]]
* Goal: Create a basic system, make it simple and run successfully.
 
* Case: DVD input and support simplest test cases.
 
* Platform: Virtual Machine.
 
* Inputs and Outputs:
 
**Inputs: DVD, kickstart files, python (shell) scripts.
 
**Outputs: Logs of whole process(including anaconda logs), test results.
 
* Approach<<Fix Me>>:
 
# Create a server, and prepare DVD.iso.
 
# Create a virtual machine by the server.
 
# Run test cases using some kickstarts for install tests.
 
# Send back some logs, successful or not...
 
# Have some "results parsers" which waits for the logs from the clients and then parses them
 
* Frameworks studied so far: kvm-autotest, autotest, libvirt, dogtail.
 
 
 
'''Version 2.0'''
 
* Graphical automation testing
 
 
 
* Useful links
 
** http://libvirt.org/
 
** http://jlaska.livejournal.com/tag/cobbler
 
** GUI automation
 
# http://www.linux-kvm.org/page/KVM-Autotest
 
# The above autotest is based on this framework: http://autotest.kernel.org/
 
# The steps to work with step files: http://www.linux-kvm.org/page/KVM-Autotest/Steps
 
  
 
== Discussion Points ==
 
== Discussion Points ==
your comments
+
* List unresolved or active discussion
  
 
== Comments? ==
 
== Comments? ==
Line 158: Line 119:
  
 
== Owner ==
 
== Owner ==
Fedora auto-install test project
+
Fedora QA team
 +
 
 +
== References ==
 +
<references/>
 +
Other documentation:
 +
# General libvirt home - http://libvirt.org
 +
# Examples on using libvirt and cobbler to aid in semi-automated install testing - http://jlaska.livejournal.com/tag/cobbler
 +
# [[Autotest|General Autotest information]]
 +
# [http://www.linux-kvm.org/page/KVM-Autotestlinux|How KVM is using autotest]
 +
# GUI automation using dogtail - https://fedorahosted.org/dogtail
 +
# GUI automation using autotest step files - http://www.linux-kvm.org/page/KVM-Autotest/Steps
  
 
[[Category:Proposals]]
 
[[Category:Proposals]]
 
[[Category:FAD]]
 
[[Category:FAD]]
 
== Meeting ==
 
This project meeting is held weekly for progress. Welcome everyone to attend it and give your valuable suggestions.
 
* IRC on #fedora-meeting
 
* Each Friday at 8:30 UTC (9:30 CET and 16:30 Bejing)
 

Latest revision as of 08:51, 16 November 2011

QA.png


Overview

The purpose of AutoQA installation test automation is to simplify testing and improve test efficiency. The AutoQA install test project should address the following problems:

  • The system should be easy to allow for tester and developer use
  • Have clear documentation for customizing and creating new tests
  • Support test execution using existing Fedora infrastructure services, but not require them
  • Test results are easy to verify

Scope

The Fedora installation test automation project covers the whole installation from retrieving installation media to first boot. Details are below:

  1. retrieve installation media
  2. Installation media sanity check
  3. Install with different system specifications, installation sources and partition etc.
  4. get the installation output and logs.

Problem Space

The Fedora installer is a complicated software application that often requires significant setup time to properly and efficiently test. Installer failures typically come from the following areas:

Image Sanity

  1. ISO file size is too large
  2. Invalid SHA256 checksum
  3. Invalid implanted ISO md5sum
  4. Install environment - anaconda has specific application, library and config file format requirements.
  5. Versions check

Boot Methods

  1. Boot media improperly built (PXE, boot.iso, DVD, efidisk.img)
  2. Installer fails to boot as a KVM guest
  3. Installer fails to boot as a XEN guest

Install Source

  1. Unable to detect install.img media
  2. Unable to transition to stage#2 installer

Kickstart Delivery

  1. Ks.cfg could not be obtained from specified location (http, ftp, nfs, hd, initrd)
  2. Install fails to proceed in accordance with the directives in the ks.cfg file
  3. Install improperly sets up networking based on command-line and kickstart network parameters (boot with ksdevice=eth1, ks.cfg contains eth2)

User Interface

  1. X driver problems while transitioning to graphical install
  2. Screen corruption during text-mode install
  3. VNC fails to start
  4. Serial console redirection improperly setup

Storage Devices

  1. Fail to detect existing storage device(s)
  2. Failure to clear stale data off of existing devices
  3. Unable to add iSCSI volumes

Partitioning

  1. Failure detecting existing partition scheme (lvm, mdraid, dmraid, luks)
  2. Failure when attempting to resize existing partitions
  3. Failures while attempting to re-use existing partitions
  4. Improperly clearing stale information from disks
  5. Unable to consistently resize an existing filesystem
  6. General failures while attempting to manually partition a system

Package Repository

  1. Unable to read metadata from package repositories (http, ftp, nfs, media)
  2. Failures while adding or modifying existing package repositories

Package Set

  1. Network timeout while retrieving packages
  2. Dependency problems while resolving package list
  3. File conflicts during package install
  4. Package order and install errors in install.log
  5. Improperly formatted comps.xml data

Boot loader configuration

  1. Unable to properly detect other operating systems
  2. Failure while setting up chainloader for another OS

Upgrade system

  1. Failure to detect previously installed systems
  2. Errors while attempting to update bootloader configuration during upgrade
  3. Package upgrade errors in upgrade.log

Recovery

  1. Rescue mode fails to detect existing installations (lvm, raid, luks)
  2. Rescue mode fails to establish networking
  3. Problems saving traceback information (local disk, bugzilla, remote server)
  4. Anaconda unable to download and use updates.img (from install source, local media or URL)
  5. Unable to transition to debug mode

Proposed Solution

The project will be developed based on the AutoQA framework and use the virtualization environment (libvirt,kvm-qemu). The installation automation will be deployed by the Fedora kickstart installation. virt-install will be used to create the guest.virtio is used to get the anaconda logs. The details can be found at Is anaconda broken solution

Active Ingredients

  • AutoQA must be packaged and available in Fedora infrastructure
  • A valid backend test harness is packaged and available in Fedora infrastructure (Autotest currently being used by AutoQA)
  • Knowledge of TurboGears or TurboGears2
  • A new AutoQA test watcher may be required to initiate automated tests whenever CD and DVD ISO images are available for testing (post-iso-build)
  • Adding installer unit test execution and results display is dependent on the availability and environmental requirements of installer unit tests


Roadmap

Results

Discussion Points

  • List unresolved or active discussion

Comments?

To leave a comment, use the Talk page for this proposal.

Owner

Fedora QA team

References

Other documentation:

  1. General libvirt home - http://libvirt.org
  2. Examples on using libvirt and cobbler to aid in semi-automated install testing - http://jlaska.livejournal.com/tag/cobbler
  3. General Autotest information
  4. KVM is using autotest
  5. GUI automation using dogtail - https://fedorahosted.org/dogtail
  6. GUI automation using autotest step files - http://www.linux-kvm.org/page/KVM-Autotest/Steps