From Fedora Project Wiki

(Updates to the problem space definition)
(Minor re-organization of problem space)
Line 1: Line 1:
 
{{header|qa}}
 
{{header|qa}}
 
{{draft}}
 
{{draft}}
 +
 
<!-- This is the FESCo proposal template.  It is not necessary that you use this template but it is encouraged that you do.  It will help FESCo focus on the issue being addressed and the parts that need discussion, as well as identify the scope of the proposal. You may remove all these comment sections from your proposal. -->
 
<!-- This is the FESCo proposal template.  It is not necessary that you use this template but it is encouraged that you do.  It will help FESCo focus on the issue being addressed and the parts that need discussion, as well as identify the scope of the proposal. You may remove all these comment sections from your proposal. -->
  
Line 16: Line 17:
  
 
The installation process covers the following sections, therefore, failures typically come from the following areas:
 
The installation process covers the following sections, therefore, failures typically come from the following areas:
# Image Sanity
+
 
## Image size too large (or small)
+
''' Image Sanity '''
## Invalid SHA256 checksum
+
# Image size too large (or small)
## Invalid implanted ISO md5sum
+
# Invalid SHA256 checksum
## Versions check
+
# Invalid implanted ISO md5sum
# Boot Methods
+
# Versions check
## Boot media improperly built (PXE, boot.iso, CD/DVD, efidisk.img)
+
 
## Installer fails to boot as a KVM guest
+
''' Boot Methods'''
## Installer fails to boot as a XEN guest
+
# Boot media improperly built (PXE, boot.iso, CD/DVD, efidisk.img)
# Install Source
+
# Installer fails to boot as a KVM guest
## Unable to detect install.img media
+
# Installer fails to boot as a XEN guest
## Unable to transition to stage#2 installer
+
 
# Kickstart Delivery
+
'''Install Source'''
## Ks.cfg could not be obtained from specified location (http, ftp, nfs, hd, initrd)
+
# Unable to detect install.img media
## Install fails to proceed in accordance with the directives in the ks.cfg file
+
# Unable to transition to stage#2 installer
## Install improperly sets up networking based on command-line and kickstart network parameters (boot with <code>ksdevice=eth1</code>, {{filename|ks.cfg}} contains <code>eth2</code>)
+
 
# User Interface
+
'''Kickstart Delivery'''
## X driver problems while transitioning to graphical install
+
# Ks.cfg could not be obtained from specified location (http, ftp, nfs, hd, initrd)
## Screen corruption during text-mode install
+
# Install fails to proceed in accordance with the directives in the ks.cfg file
## VNC fails to start
+
# Install improperly sets up networking based on command-line and kickstart network parameters (boot with <code>ksdevice=eth1</code>, {{filename|ks.cfg}} contains <code>eth2</code>)
## Serial console redirection improperly setup
+
 
# Storage Devices
+
'''User Interface'''
## Fail to detect existing storage device(s)
+
# X driver problems while transitioning to graphical install
## Failure to clear stale data off of existing devices
+
# Screen corruption during text-mode install
## Unable to add iSCSI volumes
+
# VNC fails to start
# Partitioning
+
# Serial console redirection improperly setup
## Failure detecting existing partition scheme (lvm, mdraid, dmraid, luks)
+
 
## Failure when attempting to resize existing partitions
+
'''Storage Devices'''
## Failures while attempting to re-use existing partitions
+
# Fail to detect existing storage device(s)
## Improperly clearing stale information from disks
+
# Failure to clear stale data off of existing devices
## Unable to consistently resize an existing filesystem
+
# Unable to add iSCSI volumes
## General failures while attempting to manually partition a system
+
 
# Package Repository
+
'''Partitioning'''
## Unable to read metadata from package repositories (http, ftp, nfs, media)
+
# Failure detecting existing partition scheme (lvm, mdraid, dmraid, luks)
## Failures while adding or modifying existing package repositories
+
# Failure when attempting to resize existing partitions
# Package Set
+
# Failures while attempting to re-use existing partitions
## Network timeout while retrieving packages
+
# Improperly clearing stale information from disks
## Dependency problems while resolving package list
+
# Unable to consistently resize an existing filesystem
## File conflicts during package install
+
# General failures while attempting to manually partition a system
## Package order and install errors in {{filename|install.log}}
+
 
## Improperly formatted {{filename|comps.xml}} data
+
'''Package Repository'''
# Recovery
+
# Unable to read metadata from package repositories (http, ftp, nfs, media)
## Rescue mode fails to detect existing installations (lvm, raid, luks)
+
# Failures while adding or modifying existing package repositories
## Rescue mode fails to establish networking
+
 
## Problems saving traceback information (local disk, bugzilla, remote server)
+
'''Package Set'''
## Anaconda unable to download and use updates.img (from install source, local media or URL)
+
# Network timeout while retrieving packages
## Unable to transition to debug mode
+
# Dependency problems while resolving package list
# Boot loader configuration
+
# File conflicts during package install
## Unable to properly detect other operating systems
+
# Package order and install errors in {{filename|install.log}}
## Failure while setting up chainloader for another OS
+
# Improperly formatted {{filename|comps.xml}} data
# Upgrade system
+
 
## Failure to detect previously installed systems
+
'''Boot loader configuration'''
## Errors while attempting to update bootloader configuration during upgrade
+
# Unable to properly detect other operating systems
## Package upgrade errors in {{filename|upgrade.log}}
+
# Failure while setting up chainloader for another OS
 +
 
 +
'''Upgrade system'''
 +
# Failure to detect previously installed systems
 +
# Errors while attempting to update bootloader configuration during upgrade
 +
# Package upgrade errors in {{filename|upgrade.log}}
 +
 
 +
'''Recovery'''
 +
# Rescue mode fails to detect existing installations (lvm, raid, luks)
 +
# Rescue mode fails to establish networking
 +
# Problems saving traceback information (local disk, bugzilla, remote server)
 +
# Anaconda unable to download and use updates.img (from install source, local media or URL)
 +
# Unable to transition to debug mode
  
 
Auto-test will test anaconda of these media:DVD,CDROM, LiveCD,Network boot CD.If anaconda broken,typically have these issues:
 
Auto-test will test anaconda of these media:DVD,CDROM, LiveCD,Network boot CD.If anaconda broken,typically have these issues:

Revision as of 21:25, 14 December 2009

QA.png


Warning.png
This page is a draft only
It is still under construction and content may change. Do not rely on the information on this page.


Overview

provide small auto-test system with a web-based interface which can help to customize autoinstall testing.The test system should be small enough for personal use.

Problem Space

The purpose of AutoQA install test automation is to simplify testing, reduce the test execution time and improve efficiency. The AutoQA install test should help us to solve the following problems:

  • The system should be small enough to allow for developer use
  • Have a clear method for customizing and creating new tests
  • Work within existing Fedora infrastructure services, but not require them
  • Test results are easy to verify

The installation process covers the following sections, therefore, failures typically come from the following areas:

Image Sanity

  1. Image size too large (or small)
  2. Invalid SHA256 checksum
  3. Invalid implanted ISO md5sum
  4. Versions check

Boot Methods

  1. Boot media improperly built (PXE, boot.iso, CD/DVD, efidisk.img)
  2. Installer fails to boot as a KVM guest
  3. Installer fails to boot as a XEN guest

Install Source

  1. Unable to detect install.img media
  2. Unable to transition to stage#2 installer

Kickstart Delivery

  1. Ks.cfg could not be obtained from specified location (http, ftp, nfs, hd, initrd)
  2. Install fails to proceed in accordance with the directives in the ks.cfg file
  3. Install improperly sets up networking based on command-line and kickstart network parameters (boot with ksdevice=eth1, ks.cfg contains eth2)

User Interface

  1. X driver problems while transitioning to graphical install
  2. Screen corruption during text-mode install
  3. VNC fails to start
  4. Serial console redirection improperly setup

Storage Devices

  1. Fail to detect existing storage device(s)
  2. Failure to clear stale data off of existing devices
  3. Unable to add iSCSI volumes

Partitioning

  1. Failure detecting existing partition scheme (lvm, mdraid, dmraid, luks)
  2. Failure when attempting to resize existing partitions
  3. Failures while attempting to re-use existing partitions
  4. Improperly clearing stale information from disks
  5. Unable to consistently resize an existing filesystem
  6. General failures while attempting to manually partition a system

Package Repository

  1. Unable to read metadata from package repositories (http, ftp, nfs, media)
  2. Failures while adding or modifying existing package repositories

Package Set

  1. Network timeout while retrieving packages
  2. Dependency problems while resolving package list
  3. File conflicts during package install
  4. Package order and install errors in install.log
  5. Improperly formatted comps.xml data

Boot loader configuration

  1. Unable to properly detect other operating systems
  2. Failure while setting up chainloader for another OS

Upgrade system

  1. Failure to detect previously installed systems
  2. Errors while attempting to update bootloader configuration during upgrade
  3. Package upgrade errors in upgrade.log

Recovery

  1. Rescue mode fails to detect existing installations (lvm, raid, luks)
  2. Rescue mode fails to establish networking
  3. Problems saving traceback information (local disk, bugzilla, remote server)
  4. Anaconda unable to download and use updates.img (from install source, local media or URL)
  5. Unable to transition to debug mode

Auto-test will test anaconda of these media:DVD,CDROM, LiveCD,Network boot CD.If anaconda broken,typically have these issues:

  • DVD install
  1. Boot fails from DVD
  2. Stage#2 can not start successfully
  3. Partition fails
  4. Reboot to newly installed system fails
  5. Upgrade fails
  • CDROM install
  1. Fail to boot from CDROM
  2. Stage#2 can not start successfully
  3. Fail to request next CDROM
  4. Package errors during installation
  5. Reboot to newly installed system fails
  • BootCD install
  1. Fail to boot from BootCD
  2. kickstart install fails
  3. Ftp install fails
  4. NFS install fails
  5. Repo is missing
  6. Http install fails
  • LiveCD install
  1. LiveCD install fails
  2. Reboot to newly installed system fails

Proposed Solution

  • use virtual machine(libvirt+KVM) to finish install tests one by one.
  • provide simple interface to customize testing.
  • Create scripts to do test automatically for anaconda
  • Create test cases for the above possible breakness of anaconda(DVD,CDROM,BootCD,LiveCD)
  • Test result display page help to check the test result easily,the result format is good for testers to post to specified link(have no need to change a lot before post).
  • Write test cases From simple to complex, kickstart to graphic.

Scope

  • Any testers from community can install the testing system and run test cases
  • After finish testing,testers will post test result to specified wiki

Active Ingredients

  • develop a system to run install system
  • convert some of the current install test cases to cases which can be run in auto test system
  • Provide a test results presentation layer for manual test result submission

Roadmap

your comments

Results

Version 1.0

  • Goal: Create a basic system, make it simple and run successfully.
  • Case: DVD input and support simplest test cases.
  • Platform: Virtual Machine.
  • Inputs and Outputs:
    • Inputs: DVD, kickstart files, python (shell) scripts.
    • Outputs: Logs of whole process(including anaconda logs), test results.
  • Approach<<Fix Me>>:
  1. Create a server, and prepare DVD.iso.
  2. Create a virtual machine by the server.
  3. Run test cases using some kickstarts for install tests.
  4. Send back some logs, successful or not...
  5. Have some "results parsers" which waits for the logs from the clients and then parses them
  • Frameworks studied so far: kvm-autotest, autotest, libvirt, dogtail.

Version 2.0

  • Graphical automation testing
  1. http://www.linux-kvm.org/page/KVM-Autotest
  2. The above autotest is based on this framework: http://autotest.kernel.org/
  3. The steps to work with step files: http://www.linux-kvm.org/page/KVM-Autotest/Steps

Discussion Points

your comments

Comments?

To leave a comment, use the Talk page for this proposal.

Owner

Fedora auto-install test project

Meeting

This project meeting is held weekly for progress. Welcome everyone to attend it and give your valuable suggestions.

  • IRC on #fedora-meeting
  • Each Friday at 8:30 UTC (9:30 CET and 16:30 Bejing)