From Fedora Project Wiki

(Problem Space)
Line 15: Line 15:
 
Installation process covers the following sections, therefore Brokenness comes from these types:   
 
Installation process covers the following sections, therefore Brokenness comes from these types:   
 
* Image Sanity
 
* Image Sanity
# Error in size
+
# Error in size/Checksum  
# Error in Checksum  
+
# dependency problems/File Conflicts for all packages
# dependency errors for all packages
 
# File Conflicts
 
 
* Boot Methods  
 
* Boot Methods  
# The CD/DVD isolinux (or syslinux) menu is properly displayed
+
# The CD/DVD isolinux (or syslinux) menu displaying issue
# Anaconda boots successfully from the DVD
+
# Anaconda failed to boot
# Anaconda transitions to stage#2 without error
+
# Anaconda unable to transition to stage#2  
 
* Install Source
 
* Install Source
# Install completes successfully
+
# Install couldn't complete
# Package errors (install.log) should not occur
+
# Package errors (install.log)
# System boots successfully
 
 
* Kickstart Delivery
 
* Kickstart Delivery
# The ks.cfg is successfully obtained from the specified location
+
# Ks.cfg couldn't be obtained from specified location
# The install should proceed in accordance with the directives in the ks.cfg file  
+
# Install failed to proceed in accordance with the directives in the ks.cfg file  
 
* Package Set  
 
* Package Set  
# Anaconda completes successfully
+
# Process of package install doesn't complete
# System boots into installed environment successfully
+
# Install.log contain errors  
# Install.log does not contain errors  
+
# The specified packages aren't installed
# A (small) number of packages is installed (150~)
 
 
* Partitioning
 
* Partitioning
# The system should prompt for your passphrase only once during boot
+
# Can't prompt for an existing partition to resize
# The system unlocks the encrypted partition(s) and boots successfully
+
# Error when resizing the selected partition  
# Anaconda should prompt for an existing partition to resize
+
# The installer failed to partition the system as required
# Anaconda should successfully resize the selected partition  
+
# Encrypted partitions contain errors.
# The installer will successfully partition a bootable system using remaining free space
 
# Confirm that rootfs partition is formatted as ext4/3/2 and is not a LVM or RAID volume
 
# Partitions are created and formatted properly.  
 
 
* User Interface  
 
* User Interface  
# Graphical anaconda displays properly
+
# Unable to boot to graphical/text mode
# The installed system boots in graphical mode (runlevel 5)
+
# Graphical anaconda displays error in steps
# Anaconda starts Xvnc/telnet properly  
+
# Anaconda doesn't start Xvnc/telnet properly  
 
* Recovery
 
* Recovery
 +
# Rescue mode failed
 +
# Anaconda doesn't prepare the updates.img
 +
# Traceback errors
 
* Storage Devices  
 
* Storage Devices  
 +
# Fail to detect any storage device
 +
# Error occurred during installation
 
* Upgrade system
 
* Upgrade system
 +
# upgrade issues.
 +
# preupgrade issues.
  
 
==Proposed Solution ==
 
==Proposed Solution ==

Revision as of 08:01, 27 November 2009


Overview

provide small auto-test system with a web-based interface which can help to customize autoinstall testing.The test system should be small enough for personal use.

Problem Space

The purpose of auto test is to simplify testing,save time, improve efficiency.So the auto-test should help us to solve the following problems:

  • the system should be small enough for personal use
  • have a very simple interface for user to customize testing or configure testing
  • very easy interface for user to set up some services due to some special cases,such as ftp/nfs/http install,etc
  • test result is easy to check

Installation process covers the following sections, therefore Brokenness comes from these types:

  • Image Sanity
  1. Error in size/Checksum
  2. dependency problems/File Conflicts for all packages
  • Boot Methods
  1. The CD/DVD isolinux (or syslinux) menu displaying issue
  2. Anaconda failed to boot
  3. Anaconda unable to transition to stage#2
  • Install Source
  1. Install couldn't complete
  2. Package errors (install.log)
  • Kickstart Delivery
  1. Ks.cfg couldn't be obtained from specified location
  2. Install failed to proceed in accordance with the directives in the ks.cfg file
  • Package Set
  1. Process of package install doesn't complete
  2. Install.log contain errors
  3. The specified packages aren't installed
  • Partitioning
  1. Can't prompt for an existing partition to resize
  2. Error when resizing the selected partition
  3. The installer failed to partition the system as required
  4. Encrypted partitions contain errors.
  • User Interface
  1. Unable to boot to graphical/text mode
  2. Graphical anaconda displays error in steps
  3. Anaconda doesn't start Xvnc/telnet properly
  • Recovery
  1. Rescue mode failed
  2. Anaconda doesn't prepare the updates.img
  3. Traceback errors
  • Storage Devices
  1. Fail to detect any storage device
  2. Error occurred during installation
  • Upgrade system
  1. upgrade issues.
  2. preupgrade issues.

Proposed Solution

  • use the network install(or libvirt? does it make sense to run on virtual machines? need confirmation) to finish install tests one by one.
  • provide web-based interface to customize testing.In the web page,user can specify the path of ISO,select which cases to run or not,edit(import) kickstart file or automaticly generate kickstart file after give some selected parameters (partition,packages,install method).
  • web-based interface help to setup some needed services due to some special cases. Modify,add or delete test cases,
  • Test result display page help to check the test result easily,the result format is good for testers to post to specified link(have no need to change a lot before post).

Scope

  • Any testers from community can install the testing system and run test cases
  • After finish testing,testers will post test result to specified wiki

Active Ingredients

  • develop a system to run install system
  • convert some of the current install test cases to cases which can be run in auto test system
  • Provide a test results presentation layer for manual test result submission

Roadmap

your comments

Results

Version 1.0

  • Goal: Create a basic system, make it simple and run successfully.
  • Case: DVD input and support simplest test cases.
  • Platform: Virtual Machine.
  • Inputs and Outputs:
    • Inputs: DVD, kickstart files, python (shell) scripts.
    • Outputs: Logs of whole process(including anaconda logs), test results.
  • Approach<<Fix Me>>:
  1. Create a server, and prepare DVD.iso.
  2. Create a virtual machine by the server.
  3. Run test cases using some kickstarts for install tests.
  4. Send back some logs, successful or not...
  5. Have some "results parsers" which waits for the logs from the clients and then parses them
  • Frameworks studied so far: kvm-autotest, autotest, libvirt, dogtail.

Version 2.0

  • Graphical automation testing
  1. http://www.linux-kvm.org/page/KVM-Autotest
  2. The above autotest is based on this framework: http://autotest.kernel.org/
  3. The steps to work with step files: http://www.linux-kvm.org/page/KVM-Autotest/Steps

Discussion Points

your comments

Comments?

To leave a comment, use the Talk page for this proposal.

Owner

Fedora auto-install test project

Meeting

This project meeting is held weekly for progress. Welcome everyone to attend it and give your valuable suggestions.

  • IRC on #fedora-meeting
  • Each Friday at 8:30 UTC (9:30 CET and 16:30 Bejing)