Overview
The idea of the Package Sanity Test is to perform a set of common real-life package tasks, which will be executed every time an updated package becomes available in the Fedora repository.
In short we want to make sure the new packages can be successfully upgraded (there are no conflicts or broken dependencies which would prevent package update). The plan is to test other common package tasks such as: verify, delete, install or downgrade as well.
Successful pass of this test may become one of the package acceptance criteria in the future.
Scope
Here is the list of actions we are interested in:
- Can the package be successfully updated?
- Does the verification of the new package succeed?
- Can we delete & re-install the package?
- Does the downgrade install all the original packages?
- Can we successfully rebuild the source rpm?
Test Cases
The following information is expected to be available:
- OLD_PACKAGES: List of the old packages (including version)
- NEW_PACKAGES: List of the new packages (including version)
- UPDATES_REPO: Updates repository name
The additional PACKAGES list contains all the package names without version numbers.
Update
- use yum with enabled UPDATES_REPO to update PACKAGES
- all the NEW_PACKAGES installed on the system
Verify
- the verify test succeeds for all NEW_PACKAGES
Delete
- try to remove the NEW_PACKAGES
- skip failures caused by requested dependencies
- no package from the NEW_PACKAGES list should be installed
Install
- try to install the NEW_PACKAGES (skip if delete failed)
- all the new packages installed on the system
Downgrade
- use yum to downgrade to the original package versions
- all the OLD_PACKAGES should be installed
Rebuild
- use yumdownloader to download the source rpm package
- install packages required for building
- try to rebuild the packages
- packages built successfully
Test Environment
- The tests will be run on a fully updated Fedora release.
- The updates-testing repository is set up & disabled by default.
- There will be enough free space on disk (1GB?) for:
- downloaded packages
- rebuilding tests
Questions
- How to get the updated box?
- Provision & install? Virtual machine?
- Stable? Snapshots? Changed root?
Schedule
- Should be run for every package pushed to updates-testing repository.
Questions
- How to get the watcher notified about the update?
- Comparing repo? A message from bodhi?
- See ticket #55
Results
- Where / how to report results? (Add comment to bodhi? Adjust karma?)
- Report failing tests to developer by email?