Is anaconda broken proposal: Difference between revisions

Latest revision as of 08:51, 16 November 2011

Overview

The purpose of AutoQA installation test automation is to simplify testing and improve test efficiency. The AutoQA install test project should address the following problems:

The system should be easy to allow for tester and developer use
Have clear documentation for customizing and creating new tests
Support test execution using existing Fedora infrastructure services, but not require them
Test results are easy to verify

Scope

The Fedora installation test automation project covers the whole installation from retrieving installation media to first boot. Details are below:

retrieve installation media
Installation media sanity check
Install with different system specifications, installation sources and partition etc.
get the installation output and logs.

Problem Space

The Fedora installer is a complicated software application that often requires significant setup time to properly and efficiently test. Installer failures typically come from the following areas:

Image Sanity

ISO file size is too large
Invalid SHA256 checksum
Invalid implanted ISO md5sum
Install environment - anaconda has specific application, library and config file format requirements.
Versions check

Boot Methods

Boot media improperly built (PXE, boot.iso, DVD, efidisk.img)
Installer fails to boot as a KVM guest
Installer fails to boot as a XEN guest

Install Source

Unable to detect install.img media
Unable to transition to stage#2 installer

Kickstart Delivery

Ks.cfg could not be obtained from specified location (http, ftp, nfs, hd, initrd)
Install fails to proceed in accordance with the directives in the ks.cfg file
Install improperly sets up networking based on command-line and kickstart network parameters (boot with ksdevice=eth1, ks.cfg contains eth2)

User Interface

X driver problems while transitioning to graphical install
Screen corruption during text-mode install
VNC fails to start
Serial console redirection improperly setup

Storage Devices

Fail to detect existing storage device(s)
Failure to clear stale data off of existing devices
Unable to add iSCSI volumes

Partitioning

Failure detecting existing partition scheme (lvm, mdraid, dmraid, luks)
Failure when attempting to resize existing partitions
Failures while attempting to re-use existing partitions
Improperly clearing stale information from disks
Unable to consistently resize an existing filesystem
General failures while attempting to manually partition a system

Package Repository

Unable to read metadata from package repositories (http, ftp, nfs, media)
Failures while adding or modifying existing package repositories

Package Set

Network timeout while retrieving packages
Dependency problems while resolving package list
File conflicts during package install
Package order and install errors in install.log
Improperly formatted comps.xml data

Boot loader configuration

Unable to properly detect other operating systems
Failure while setting up chainloader for another OS

Upgrade system

Failure to detect previously installed systems
Errors while attempting to update bootloader configuration during upgrade
Package upgrade errors in upgrade.log

Recovery

Rescue mode fails to detect existing installations (lvm, raid, luks)
Rescue mode fails to establish networking
Problems saving traceback information (local disk, bugzilla, remote server)
Anaconda unable to download and use updates.img (from install source, local media or URL)
Unable to transition to debug mode

Proposed Solution

The project will be developed based on the AutoQA framework and use the virtualization environment (libvirt,kvm-qemu). The installation automation will be deployed by the Fedora kickstart installation. virt-install will be used to create the guest.virtio is used to get the anaconda logs. The details can be found at Is anaconda broken solution

Active Ingredients

AutoQA must be packaged and available in Fedora infrastructure
A valid backend test harness is packaged and available in Fedora infrastructure (Autotest currently being used by AutoQA)
Knowledge of TurboGears or TurboGears2
A new AutoQA test watcher may be required to initiate automated tests whenever CD and DVD ISO images are available for testing (post-iso-build)
Adding installer unit test execution and results display is dependent on the availability and environmental requirements of installer unit tests

Roadmap

Roadmap for development of auto install at:Is_anaconda_broken_roadmap
Status for test plan/documentation changes can be seen in the fedora-qa trac system
Status for test automation efforts is available in the autoqa TRAC system

Results

Meeting information and logs can be found at Is_anaconda_broken_meeting

Discussion Points

List unresolved or active discussion

Comments?

To leave a comment, use the Talk page for this proposal.

Owner

Fedora QA team

@@ Line 5: / Line 5: @@
 == Overview ==
-The purpose of AutoQA install test automation is to simplify testing, reduce the test execution time and improve efficiency.  The AutoQA install test project should address the following problems:
+The purpose of AutoQA installation test automation is to simplify testing and improve test efficiency.  The AutoQA install test project should address the following problems:
 * The system should be easy to allow for tester and developer use
@@ Line 12: / Line 12: @@
 * Test results are easy to verify
-== Problem Space ==
+== Scope ==
+The Fedora installation test automation project covers the whole installation from retrieving installation media to first boot. Details are below:
+# retrieve installation media
+# Installation media sanity check
+# Install with different system specifications, installation sources and partition etc.
+# get the installation output and logs.
+====Problem Space====
 The Fedora installer is a complicated software application that often requires significant setup time to properly and efficiently test.  Installer failures typically come from the following areas:
 ''' Image Sanity '''
-# ISO file size is too large (or small)
+# ISO file size is too large
 # Invalid SHA256 checksum
 # Invalid implanted ISO md5sum
@@ Line 24: / Line 30: @@
 ''' Boot Methods'''
-# Boot media improperly built (PXE, boot.iso, CD/DVD, efidisk.img)
+# Boot media improperly built (PXE, boot.iso, DVD, efidisk.img)
 # Installer fails to boot as a KVM guest
 # Installer fails to boot as a XEN guest
@@ Line 85: / Line 91: @@
 == Proposed Solution ==
 <!-- Describe proposed solution in detail -->
+The project will be developed based on the AutoQA framework and use the virtualization environment (libvirt,kvm-qemu). The installation automation will be deployed by the Fedora kickstart installation. virt-install will be used to create the guest.virtio is used to get the anaconda logs. The details can be found at [[Is anaconda broken  solution]]
-First, in order to provide a consistent and documented test approach, the existing Fedora Install test plan <ref>[[QA:Fedora_13_Install_Test_Plan]]</ref> will be revisited.  The test plan will be adjusted to ensure proper test coverage for the failure scenarios listed above.  Existing test cases will be reviewed for accuracy.  New test cases will be created using the [[Template:QA/Test_Case]] template.  Finally, the test plan will be adjusted to match the improved Fedora Release Criteria <ref>[[Fedora Release Criteria]]</ref>.  This includes adjusting the test case priority to match milestone criteria.
-Next, in order to reduce the setup/execution time, improve efficiency and to provide test results on a more consistent basis, a subset of test cases will be chosen for automation.  Tests will be written in python and will be developed and executed on a system supporting KVM virtualization.   Test scripts will be responsible for preparing a virtual install environment, initiating a kickstart install and validating the results.  Once an initial batch of tests exist, they will be formally integrated into the [[AutoQA]] project.
-Last, a method will be developed for collecting test results into a single test result matrix.  Results may be posted to the wiki directly, or a custom turbogears application may be needed to display results <ref>For a similar project see [http://jlaska.fedorapeople.org/irb.png screenshot] of ''is rawhide broken'' and [http://git.fedorahosted.org/git/?p=autoqa.git;a=tree;f=front-ends/israwhidebroken;hb=HEAD source code] and the [[QA:Rawhide_Acceptance_Test_Plan]].</ref>.  The results will be easily accessible for testers and the installer development team.
-== Scope ==
-<!-- Describe the scope of any project work or properties that will be affected by the proposal -->
-The project will be divided into two phases.  The first phase is designed to lay the foundation for future improvements by focusing on automating the more common install scenarios using kickstart files.  The second phase will focus on expanding kickstart automation coverage, as well as investigating automation alternatives such as GUI automation and unit test integration.
-'''Phase#1'''
-* Revise Fedora Install test plan to ensure adequate test coverage exists for failure scenarios listed above
-* Select a small, but representative, subset of test cases from the install test plan to automate
-* Create python scripts to prepare KVM-based virtual environments for testing, initiate kickstart installs, and validate results
-* Integrate python scripts into the [[AutoQA]] project
-* Implement initial test result dashboard intended to eventually replace the wiki test matrix.  The dashboard will also support FAS user test result submission
-'''Phase#2'''
-* Identify and automate remainder of test cases from the install test plan
-* Identify and integrate anaconda built-in unit tests into test results dashboard
-* Increase the methods by which testing is initiated.  Initially, testing will be initiated manually, or upon creation of new rawhide install images.  Ideally, testing can be initiated when ISO images are made available, or when updated source code is contributed to anaconda
-* Identify test cases where GUI automation would be adventageous
-* Investigate methods for leveraging GUI automation to aid in automating applicable test cases
 == Active Ingredients ==
@@ Line 119: / Line 101: @@
 * A new [[AutoQA]] test watcher may be required to initiate automated tests whenever CD and DVD ISO images are available for testing (''post-iso-build'')
 * Adding installer unit test execution and results display is dependent on the availability and environmental requirements of installer unit tests
 == Roadmap ==
+* Roadmap for development of auto install at:[[Is_anaconda_broken_roadmap]]
 * Status for test plan/documentation changes can be seen in the [https://fedorahosted.org/fedora-qa/query?status=new&status=assigned&status=reopened&status=closed&group=status&component=Test+Review&milestone=Fedora+13&order=priority fedora-qa trac system]
+* Status for test automation efforts is available in the [https://fedorahosted.org/autoqa/milestone/Automate%20installation%20test%20plan autoqa TRAC system]
 == Results ==

Search