From Fedora Project Wiki
(→‎Detailed Description: Add bugzilla refs for work items)
No edit summary
 
(8 intermediate revisions by 3 users not shown)
Line 1: Line 1:
{{autolang|base=yes}}
== Anaconda Storage Filtering ==
== Anaconda Storage Filtering ==
This is the main page for information about anaconda's further/continuing storage rewrite. The entire storage subsystem was rewritten from scratch in F11, this page tracks the cleanup and last bits of support.
This is the main page for information about anaconda's further/continuing storage rewrite. The entire storage subsystem was rewritten from scratch in F11, this page tracks the cleanup and last bits of support.
Line 6: Line 7:


== Owner ==
== Owner ==
<!--This should link to your home wiki page so we know who you are-->
* Name: [[User:Clumens| Chris Lumens]]
* Name: [[User:Dlehman| Dave Lehman]]


<!-- Include you email address that you can be reached should people want to contact you about helping with your feature, status is requested, or  technical issues need to be resolved-->
* email: clumens AT fedoraproject DOT org
* email: dlehman AT fedoraproject DOT org


== Current status ==
== Current status ==
* Targeted release: [[Releases/11 | Fedora 12 ]]  
* Targeted release: [[Releases/13 | Fedora 13 ]]  
* Last updated: 2009-07-07
* Last updated: 2010-03-24
* Percentage of completion: 25%
* Percentage of completion: 100%


== Detailed Description ==
== Summary ==
<!-- Expand on the summary, if appropriate.  A couple sentences suffices to explain the goal, but the more details you can provide the better. -->
The new storage code still needs numerous bugfixes and the addition of device filtering. This page tracks the broad categories of problem and the progress we're making.
 
This page does not track the UI cleanup for storage configuration.
 
* Storage device filtering
At the request of many folks with lots of storage available, we plan to add the ability to prune the available storage presented as an install target.
At the request of many folks with lots of storage available, we plan to add the ability to prune the available storage presented as an install target.
We want to provide users the ability to select which devices to use during installation, a way to specify disks by something other than the device node name for kickstart, and we're also curious if there is a way we can control which LUNs are scanned/activated by the kernel.
We want to provide users the ability to select which devices to use during installation, a way to specify disks by something other than the device node name for kickstart, and we're also curious if there is a way we can control which LUNs are scanned/activated by the kernel.
For device identifiers, the best we have right now is sysfs path, which is rather verbose.
This is a pervasive change. Work items include:
* Pick desired format (udev?)
* Review code for all places where device can be specified, ensure that it will accept desired format.
* Resolvedevice should handle all formats.
* Probably loader change, new class for Ignoredisk, review blacklist/whitelist support.


Additional Cleanup Work Items:
This is a pervasive change. Work items include:
* improve handling of parted's partition renumbering
* Figure out how to use udev to identify and filter various types of devices.
* improve extended partition management ([https://bugzilla.redhat.com/show_bug.cgi?id=497293  bug 497293])
* Add support to ignoredisk/clearpart/etc. kickstart commands for shell globs and non-device-node methods of referring to disks.
* improve utilization of disk space w/ growable partition requests
* Review all storage code to make sure globs work (this can probably be added right into our udev code).
* add support for optionally-partitioned devices (disks, md)
* Create UIs for selecting and filtering out devices, and make sure these work right with kickstart.
** whole-disk formatting ([https://bugzilla.redhat.com/show_bug.cgi?id=489991  bug 489991])
* Update various dialogs to work better on systems with many, many disks visibleThe goal here is to make sure you don't get 9000 popups asking for confirmation.
* improve UI handling of pre-existing devices and their formatting
* Update storage UI to show more information than just device nodes, as WWIDs or other pieces of information might be much more descriptive to the userAlso, devices nodes can change between installs.
* improve handling of unusual/broken pre-existing lvm/md setups (XXX need bz refs)
* update swap recommendation algorithm
* create md devices w/ bitmap by default
* traceback when editing locked LUKS device ([https://bugzilla.redhat.com/show_bug.cgi?id=502310  bug 502310])
* overlapping partition geometry ([https://bugzilla.redhat.com/show_bug.cgi?id=499544 bug 499544])
* can't create four primary partitions with kickstart ([https://bugzilla.redhat.com/show_bug.cgi?id=505269 bug 505269])
 
See [[Anaconda/StorageRewrite]] for further details.


== Benefit to Fedora ==
== Benefit to Fedora ==
<!-- What is the benefit to the platform?  If this is a major capability update, what has changed?  If this is a new feature, what capabilities does it bring? Why will Fedora become a better distribution or project because of this feature?-->
Advanced storage filtering will mean that people have a higher degree of confidence that anaconda is not about to destroy certain disks.  It should also mean people with very large storage configurations will have faster and easier to use installs.
We need to complete the newly redesigned storage management module and improve the stability of installation.  


== Scope ==
== Scope ==
<!-- What work do the developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
This is a major change early on in the installer, and it also affects exactly which devices anaconda's storage system will work with later.
All of anaconda's code for creating and configuring disk partitions, LVM, mdraid, dmraid, multipath, iSCSI, LUKS devices, and filesystems was rewritten. Several known problem areas remain.  


== How To Test ==
== How To Test ==
<!-- This does not need to be a full-fledged document.  Describe the dimensions of tests that this feature is expected to pass when it is done.  If it needs to be tested with different hardware or software configurations, indicate them.  The more specific you can be, the better the community testing can be.
Remember that you are writing this how to for interested testers to use to check out your feature - documenting what you do for testing is OK, but it's much better to document what *I* can do to test your feature.
A good "how to test" should answer these four questions:
0. What special hardware / data / etc. is needed (if any)?
1. How do I prepare my system to test this feature? What packages
need to be installed, config files edited, etc.?
2. What specific actions do I perform to check that the feature is
working like it's supposed to?
3. What are the expected results of those actions?
-->
Testing will need to be extensive.
Testing will need to be extensive.


We will have to verify the following areas of functionality as a starting point:
We will have to verify the following areas of functionality as a starting point:


* Ideally, find a testbed with MANY LUNs  
* Ideally, find a testbed with MANY LUNs.
* creation/configuration of filesystems
* Verify all the normal stuff in the test matrix still works.
** on new partitions
* Don't select certain devices in the filtering UI and make sure they do not show up in later storage UI.
** on existing partitions
* Verify added iscsi disks, zfcp, etc. show up automatically in the advanced filtering UI.
** on new LVM logical volumes
* On machines with advanced storage capabilities, verify those don't show up in the simple filtering UI.
** on preexisting LVM logical volumes
* On the cleardisks UI, aggregate devices (RAID, multipath) should appear as one thing, not as their component parts.
** on new software RAID devices
* Lots of kickstart testing:
** on existing software RAID devices
** Referencing disks by globs, /dev/disk/by-path, etc. works.
* creation of encrypted block devices at various levels in the device stack (partition, PV, LV, mdarray, &c)
** filter, filtertype, and cleardisks should all be skipped if clearpart and ignoredisks are used.
* detection of arbitrarily complex device configurations (luks, mdraid, lvm, &c)
** ignoredisks not being specified should still result in a fully automated install.
* management and configuration of existing encrypted block devices
** interactive should stop at every filtering screen with the UI correctly populated.
* automatic partitioning of systems
** No partitioning commands should stop at filtering UI and partitioning UI.
* clearing of existing devices/partitions
* removal of existing devices
* reconfiguration of existing devices
* detection of existing upgradable installations with widely varied storage configurations
* rescue mounting of existing systems with a wide variety of storage configurations


There is more. Unit tests are being developed, and we will plan a Fedora Test Day.
Before this code hits Rawhide it will be smoke-tested in a custom compose.


== User Experience ==
== User Experience ==
<!-- If this feature is noticeable by its target audience, how will their experiences change as a result? Describe what they will see or notice. -->
The user will potentially see several new screens early on in anaconda, as well as another screen later during partitioning. Some work still needs to be done to minimize the number of screens shown in the typical home user with one disk case, but that's polish that can be worked in later.
This should improve the user's experience with regard to the storage configuration portion of system installation.


It additionally includes the ability to filter out unwanted storage devices as potential installation targets.
This feature should improve the user's experience regarding the storage configuration portion of installation, and give them the ability to filter out unwanted storage devices as potential installation targets.


== Dependencies ==
== Dependencies ==
<!-- What other packages (RPMs) depend on this package?  Are there changes outside the developers' control on which completion of this feature depends?  In other words, completion of another feature owned by someone else and might cause you to not be able to finish on time or that you would need to coordinate?  Other upstream projects like the kernel (if this is not a kernel feature)? -->
* Anaconda's storage code uses general storage utilities such as mdadm, parted, dmraid. If liblvm becomes available in time it would be an optional work item to replace existing calls to the lvm command line with library references, but this is more likely to happen in F13.
* Anaconda's storage code uses general storage utilities such as mdadm, parted, dmraid. If liblvm becomes available in time it would be an optional work item to replace existing calls to the lvm command line with library references, but this is more likely to happen in F13.
* Storage device filtering requires udev change being driven by LVM and devicemapper guys.  
* Storage device filtering requires udev change being driven by LVM and devicemapper guys.  
* Need people with many storage devices to test as we dont have 600 luns.
* Need people with many storage devices to test as we dont have 600 luns.


== Contingency Plan ==
== Contingency Plan ==
<!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "None necessary, revert to previous release behaviour."  Or it might not.  If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy.  -->
Revert to previous codebase and selectively include bugfixes. Which will be difficult to unwedge from the new functionality, which is minor and intertwined with the bugfixes. Basically, this is a lousy option.
Revert to previous codebase and selectively include bugfixes. Which will be difficult to unwedge from the new functionality, which is minor and intertwined with the bugfixes. Basically, this is a lousy option.


== Documentation ==
== Documentation ==
See [[Anaconda/StorageRewrite]] for further details.
See [https://fedoraproject.org/wiki/Design/AnacondaStorageUI the storage design page] for further details.
 
The new kickstart features need to be documented on the wiki.


== Release Notes ==
== Release Notes ==
Line 122: Line 75:


== Comments and Discussion ==
== Comments and Discussion ==
* See [[Talk:Features/AnacondaStorageRewrite]] <!-- This adds a link to the "discussion" tab associated with your page.  This provides the ability to have ongoing comments or conversation without bogging down the main feature page -->
* See [[Talk:Features/Anaconda/StorageRewrite]]
 
[[Category:FeatureAcceptedF13]]
[[Category:FeaturePageIncomplete]]
<!-- When your feature page is completed and ready for review -->
<!-- remove Category:FeaturePageIncomplete and change it to Category:FeatureReadyForWrangler -->
<!-- After review, the feature wrangler will move your page to Category:FeatureReadyForFesco... if it still needs more work it will move back to Category:FeaturePageIncomplete-->

Latest revision as of 11:54, 17 July 2010

Anaconda Storage Filtering

This is the main page for information about anaconda's further/continuing storage rewrite. The entire storage subsystem was rewritten from scratch in F11, this page tracks the cleanup and last bits of support.

Summary

Anaconda's storage configuration code was rewritten from scratch in F11 to address several design limitations and replace outdated chunks of code. A number of outstanding problem areas remain, and this page tracks progress. It also documents the storage device filtering design and test requirements.

Owner

  • email: clumens AT fedoraproject DOT org

Current status

  • Targeted release: Fedora 13
  • Last updated: 2010-03-24
  • Percentage of completion: 100%

Summary

At the request of many folks with lots of storage available, we plan to add the ability to prune the available storage presented as an install target. We want to provide users the ability to select which devices to use during installation, a way to specify disks by something other than the device node name for kickstart, and we're also curious if there is a way we can control which LUNs are scanned/activated by the kernel.

This is a pervasive change. Work items include:

  • Figure out how to use udev to identify and filter various types of devices.
  • Add support to ignoredisk/clearpart/etc. kickstart commands for shell globs and non-device-node methods of referring to disks.
  • Review all storage code to make sure globs work (this can probably be added right into our udev code).
  • Create UIs for selecting and filtering out devices, and make sure these work right with kickstart.
  • Update various dialogs to work better on systems with many, many disks visible. The goal here is to make sure you don't get 9000 popups asking for confirmation.
  • Update storage UI to show more information than just device nodes, as WWIDs or other pieces of information might be much more descriptive to the user. Also, devices nodes can change between installs.

Benefit to Fedora

Advanced storage filtering will mean that people have a higher degree of confidence that anaconda is not about to destroy certain disks. It should also mean people with very large storage configurations will have faster and easier to use installs.

Scope

This is a major change early on in the installer, and it also affects exactly which devices anaconda's storage system will work with later.

How To Test

Testing will need to be extensive.

We will have to verify the following areas of functionality as a starting point:

  • Ideally, find a testbed with MANY LUNs.
  • Verify all the normal stuff in the test matrix still works.
  • Don't select certain devices in the filtering UI and make sure they do not show up in later storage UI.
  • Verify added iscsi disks, zfcp, etc. show up automatically in the advanced filtering UI.
  • On machines with advanced storage capabilities, verify those don't show up in the simple filtering UI.
  • On the cleardisks UI, aggregate devices (RAID, multipath) should appear as one thing, not as their component parts.
  • Lots of kickstart testing:
    • Referencing disks by globs, /dev/disk/by-path, etc. works.
    • filter, filtertype, and cleardisks should all be skipped if clearpart and ignoredisks are used.
    • ignoredisks not being specified should still result in a fully automated install.
    • interactive should stop at every filtering screen with the UI correctly populated.
    • No partitioning commands should stop at filtering UI and partitioning UI.


User Experience

The user will potentially see several new screens early on in anaconda, as well as another screen later during partitioning. Some work still needs to be done to minimize the number of screens shown in the typical home user with one disk case, but that's polish that can be worked in later.

This feature should improve the user's experience regarding the storage configuration portion of installation, and give them the ability to filter out unwanted storage devices as potential installation targets.

Dependencies

  • Anaconda's storage code uses general storage utilities such as mdadm, parted, dmraid. If liblvm becomes available in time it would be an optional work item to replace existing calls to the lvm command line with library references, but this is more likely to happen in F13.
  • Storage device filtering requires udev change being driven by LVM and devicemapper guys.
  • Need people with many storage devices to test as we dont have 600 luns.

Contingency Plan

Revert to previous codebase and selectively include bugfixes. Which will be difficult to unwedge from the new functionality, which is minor and intertwined with the bugfixes. Basically, this is a lousy option.

Documentation

See the storage design page for further details.

The new kickstart features need to be documented on the wiki.

Release Notes

  • Install Guide update to reflect ability to filter devices

Comments and Discussion