Features/ThinProvisioning

From FedoraProject

< Features(Difference between revisions)
Jump to: navigation, search
(Benefit to Fedora)
(ThinProvisioning: changed last updated date to match wiki history)
 
(24 intermediate revisions by 3 users not shown)
Line 5: Line 5:
  
 
== Owner ==
 
== Owner ==
* Name: Joe Thornber and [[User:Msnitzer|Mike Snitzer]]
+
* Name: Joe Thornber and [[User:msnitzer|Mike Snitzer]]
 
* Email:  
 
* Email:  
 
** thornber AT redhat DOT com
 
** thornber AT redhat DOT com
Line 12: Line 12:
 
== Current status ==
 
== Current status ==
 
* Targeted release: [[Releases/17 | Fedora 17 ]]  
 
* Targeted release: [[Releases/17 | Fedora 17 ]]  
* Last updated: 2011-10-12
+
* Last updated: 2012-03-23
* Percentage of completion: XX%
+
* Percentage of completion:
 
+
** kernel: 100%
<!-- CHANGE THE "FedoraVersion" TEMPLATES ABOVE TO PLAIN NUMBERS WHEN YOU COMPLETE YOUR PAGE. -->
+
** device-mapper-persistent-data tools: 100%
 +
** lvm2 thinp support: 100%
  
 
== Detailed Description ==
 
== Detailed Description ==
Line 42: Line 43:
  
 
== Scope ==
 
== Scope ==
<!-- What work do the developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
+
The bulk of the change is in the kernel (localized to the DM layer) but userspace tools for dumping, restoring and repairing the metadata are also under development.  These tools will be provided in a new 'device-mapper-persistent-data' package.  In addition the lvm2 package will be updated to ease configuration and management of thin provisioned volumes and their associated snapshots.
  
 
== How To Test ==
 
== How To Test ==
<!-- This does not need to be a full-fledged document.  Describe the dimensions of tests that this feature is expected to pass when it is done.  If it needs to be tested with different hardware or software configurations, indicate them.  The more specific you can be, the better the community testing can be.  
+
A comprehensive [https://github.com/jthornber/thinp-test-suite test suite] has been developed to verify the kernel code works as expected (depends on ruby, dt and dmsetup).
  
Remember that you are writing this how to for interested testers to use to check out your feature - documenting what you do for testing is OK, but it's much better to document what *I* can do to test your feature.
+
Any additional IO workloads (or benchmarks that model real workloads) that the community has an interest in would be welcomed tests.  Data integrity is of utmost importance so all tests that increase confidence in the feature are encouraged.  
  
A good "how to test" should answer these four questions:
+
See Documention for pointers to "how to" style usage/test guidance.
 
+
0. What special hardware / data / etc. is needed (if any)?
+
1. How do I prepare my system to test this feature? What packages
+
need to be installed, config files edited, etc.?
+
2. What specific actions do I perform to check that the feature is
+
working like it's supposed to?
+
3. What are the expected results of those actions?
+
-->
+
  
 
== User Experience ==
 
== User Experience ==
<!-- If this feature is noticeable by its target audience, how will their experiences change as a result? Describe what they will see or notice. -->
+
Users will create a shared pool of storage that will host all thin provisioned volumes and their associated snapshots. So in contrast to the old dm-snapshot implementation the user will not need to manage or monitor the free space of N snapshot volumes -- the storage for thin and snapshot volumes is allocated on-demand from the backing shared pool of storage.
  
 
== Dependencies ==
 
== Dependencies ==
<!-- What other packages (RPMs) depend on this package?  Are there changes outside the developers' control on which completion of this feature depends? In other words, completion of another feature owned by someone else and might cause you to not be able to finish on time or that you would need to coordinate?  Other upstream projects like the kernel (if this is not a kernel feature)? -->
+
No other packages depend on this feature (and vice-versa). If not ready the associated lvm2 thinp code, if included in lvm2, will error out accordingly.
  
 
== Contingency Plan ==
 
== Contingency Plan ==
<!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "None necessary, revert to previous release behaviour."  Or it might not.  If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy. -->
+
None necessary, no other packages or capabilities will depend on this feature.
  
 
== Documentation ==
 
== Documentation ==
<!-- Is there upstream documentation on this feature, or notes you have written yourself?  Link to that material here so other interested developers can get involved. -->
+
See documentation that is in the kernel tree:
*
+
* [https://github.com/jthornber/linux-2.6/blob/thin-stable/Documentation/device-mapper/thin-provisioning.txt thin-provisioning] -- overview and usage "how-to".
 +
 
 +
* [https://github.com/jthornber/linux-2.6/blob/thin-stable/Documentation/device-mapper/persistent-data.txt persistent-data] -- some details on the kernel library that enables storing metadata for DM targets (block and transaction manager, data structures, etc).
 +
 
 +
Man pages will cover the LVM2 extensions.
  
 
== Release Notes ==
 
== Release Notes ==
<!-- The Fedora Release Notes inform end-users about what is new in the release.  Examples of past release notes are here: http://docs.fedoraproject.org/release-notes/ -->
+
STORAGE:
<!-- The release notes also help users know how to deal with platform changes such as ABIs/APIs, configuration or data file formats, or upgrade concernsIf there are any such changes involved in this feature, indicate them hereYou can also link to upstream documentation if it satisfies this needThis information forms the basis of the release notes edited by the documentation team and shipped with the release. -->
+
Scalable snapshots of thinly provisioned volumes.
*
+
 
 +
The main highlight of this implementation, compared to the previous
 +
implementation of DM snapshots, is that it allows many virtual devices to
 +
be stored on the same data volume.  This simplifies administration and
 +
allows the sharing of data between volumes, thus reducing disk usage.
 +
 
 +
Another significant feature is support for an arbitrary depth of
 +
recursive snapshots (snapshots of snapshots of snapshots ...)The
 +
previous implementation of snapshots did this by chaining together
 +
lookup tables, and so performance was O(depth)This new
 +
implementation uses a single data structure to avoid this degradation
 +
with depthFragmentation may still be an issue, however, in some
 +
scenarios.
 +
 
 +
Metadata is stored on a separate device from data, giving the
 +
administrator some freedom, for example to:
 +
 
 +
* Improve metadata resilience by storing metadata on a mirrored volume but data on a non-mirrored one.
 +
 
 +
* Improve performance by storing the metadata on SSD.
 +
 
 +
The 'device-mapper-persistent-data' package provides tools to dump and restore the metadata for thinly provisioned volumes. LVM2 has been enhanced to allow the creation and management of thinly-provisioned volumes.
  
 
== Comments and Discussion ==
 
== Comments and Discussion ==
Line 81: Line 99:
  
  
[[Category:FeaturePageIncomplete]]
+
[[Category:FeatureAcceptedF17]]
 
<!-- When your feature page is completed and ready for review -->
 
<!-- When your feature page is completed and ready for review -->
 
<!-- remove Category:FeaturePageIncomplete and change it to Category:FeatureReadyForWrangler -->
 
<!-- remove Category:FeaturePageIncomplete and change it to Category:FeatureReadyForWrangler -->
 
<!-- After review, the feature wrangler will move your page to Category:FeatureReadyForFesco... if it still needs more work it will move back to Category:FeaturePageIncomplete-->
 
<!-- After review, the feature wrangler will move your page to Category:FeatureReadyForFesco... if it still needs more work it will move back to Category:FeaturePageIncomplete-->
 
<!-- A pretty picture of the page category usage is at: https://fedoraproject.org/wiki/Features/Policy/Process -->
 
<!-- A pretty picture of the page category usage is at: https://fedoraproject.org/wiki/Features/Policy/Process -->

Latest revision as of 11:58, 26 March 2012

Contents

[edit] ThinProvisioning

[edit] Summary

Provide the thin provisioning Device Mapper (DM) target and supporting userspace utilities. This DM target allows a single pool of storage to be the backing store of multiple thinly provisioned volumes. Numerous snapshots (and snapshots of snapshots) may be taken of the thinly provisioned volumes.

[edit] Owner

  • Name: Joe Thornber and Mike Snitzer
  • Email:
    • thornber AT redhat DOT com
    • snitzer AT redhat DOT com

[edit] Current status

  • Targeted release: Fedora 17
  • Last updated: 2012-03-23
  • Percentage of completion:
    • kernel: 100%
    • device-mapper-persistent-data tools: 100%
    • lvm2 thinp support: 100%

[edit] Detailed Description

The main highlight of this implementation, compared to the previous implementation of snapshots, is that it allows many virtual devices to be stored on the same data volume. This simplifies administration and allows the sharing of data between volumes, thus reducing disk usage.

Another significant feature is support for an arbitrary depth of recursive snapshots (snapshots of snapshots of snapshots ...). The previous implementation of snapshots did this by chaining together lookup tables, and so performance was O(depth). This new implementation uses a single data structure to avoid this degradation with depth. Fragmentation may still be an issue, however, in some scenarios.

Metadata is stored on a separate device from data, giving the administrator some freedom, for example to:

  • Improve metadata resilience by storing metadata on a mirrored volume but data on a non-mirrored one.
  • Improve performance by storing the metadata on SSD.

[edit] Benefit to Fedora

Scalable snapshots of thinly provisioned volumes may be used as the foundation of compelling virtualization and/or cloud services. Fedora would be positioned to be the first distribution to provide this unique advance in Linux block storage.

[edit] Scope

The bulk of the change is in the kernel (localized to the DM layer) but userspace tools for dumping, restoring and repairing the metadata are also under development. These tools will be provided in a new 'device-mapper-persistent-data' package. In addition the lvm2 package will be updated to ease configuration and management of thin provisioned volumes and their associated snapshots.

[edit] How To Test

A comprehensive test suite has been developed to verify the kernel code works as expected (depends on ruby, dt and dmsetup).

Any additional IO workloads (or benchmarks that model real workloads) that the community has an interest in would be welcomed tests. Data integrity is of utmost importance so all tests that increase confidence in the feature are encouraged.

See Documention for pointers to "how to" style usage/test guidance.

[edit] User Experience

Users will create a shared pool of storage that will host all thin provisioned volumes and their associated snapshots. So in contrast to the old dm-snapshot implementation the user will not need to manage or monitor the free space of N snapshot volumes -- the storage for thin and snapshot volumes is allocated on-demand from the backing shared pool of storage.

[edit] Dependencies

No other packages depend on this feature (and vice-versa). If not ready the associated lvm2 thinp code, if included in lvm2, will error out accordingly.

[edit] Contingency Plan

None necessary, no other packages or capabilities will depend on this feature.

[edit] Documentation

See documentation that is in the kernel tree:

  • persistent-data -- some details on the kernel library that enables storing metadata for DM targets (block and transaction manager, data structures, etc).

Man pages will cover the LVM2 extensions.

[edit] Release Notes

STORAGE: Scalable snapshots of thinly provisioned volumes.

The main highlight of this implementation, compared to the previous implementation of DM snapshots, is that it allows many virtual devices to be stored on the same data volume. This simplifies administration and allows the sharing of data between volumes, thus reducing disk usage.

Another significant feature is support for an arbitrary depth of recursive snapshots (snapshots of snapshots of snapshots ...). The previous implementation of snapshots did this by chaining together lookup tables, and so performance was O(depth). This new implementation uses a single data structure to avoid this degradation with depth. Fragmentation may still be an issue, however, in some scenarios.

Metadata is stored on a separate device from data, giving the administrator some freedom, for example to:

  • Improve metadata resilience by storing metadata on a mirrored volume but data on a non-mirrored one.
  • Improve performance by storing the metadata on SSD.

The 'device-mapper-persistent-data' package provides tools to dump and restore the metadata for thinly provisioned volumes. LVM2 has been enhanced to allow the creation and management of thinly-provisioned volumes.

[edit] Comments and Discussion