From Fedora Project Wiki
(Status update)
m (internal link cleaning)
 
(129 intermediate revisions by 3 users not shown)
Line 2: Line 2:


== Summary ==
== Summary ==
Better compression technology (LZMA) allows us to ship more software on our live images.
Enabling XZ (a variant of LZMA) for compression to allow us ship more software on our live images.


== Owner ==
== Owner ==
Line 10: Line 10:


== Current status ==
== Current status ==
* Targeted release: [[Releases/14 | Fedora 14 ]]
* Targeted release: [[Releases/15 | Fedora 15 ]]
* Last updated: 2010-02-14
* Last updated: 2011-03-06
* Percentage of completion: 40%
* Percentage of completion: 100%


== Detailed Description ==
== Detailed Description ==
[http://en.wikipedia.org/wiki/Lempel-Ziv-Markov_chain_algorithm LZMA compression] typically provides better compression than zlib while still providing relatively fast uncompression. Once squashfs file systems are supported by the kernel (which is anticipated in 2.6.34) we can use them for building live images. A test with the games spin resulted in a 10% reduction in size of the resulting live image.
[http://en.wikipedia.org/wiki/Lempel-Ziv-Markov_chain_algorithm LZMA compression] typically provides better compression than gzip while still providing relatively fast uncompression. Once squashfs file systems are supported by the kernel we can use them for building live images. A test with the games spin resulted in a 8.7% reduction (from 4242538496 bytes to 3873439744 bytes) in size of the resulting live image.


== Benefit to Fedora ==
== Benefit to Fedora ==
Line 21: Line 21:


== Scope ==
== Scope ==
* Lougher's patches need to be accepted upstream. Right now they are in linux-next and will probably be in 2.6.34.
* kernel
** Lougher made a pull request of Linus on 2010-03-01 for 2.6.34.
** F15 has had 2.6.38 kernels for a while and XZ is working fine.
*** Linus bounced it back asking for cleanup, not rejecting the idea itself.
** Fallback plan for the kernel is to do nothing.
* Lougher's 4.1 (currently prerelease) version of squashfs-tools needs to be in the release.
* livecd-creator
** [https://bugzilla.redhat.com/show_bug.cgi?id=504425 Bug 504425] documents the current status of that effort.
** Changes to livecd-creator:
*** It currently builds.
*** Change the default compression type to xz. (Done - in rawhide)
*** It uses multiple processors for both zlib and lzma compression.
*** Update the documentation to reflect the changed default. (Done - in rawhide)
*** A test games spin had a 10% reduction (from almost 4.4 GB to just under 4 GB) in size using lzma compression, saving about 400 MB. (This was a nonfunctional image because of lack of kernel support for lzma squashfs.)
*** These changes won't be made until after the kernel feature is available, so no fallback is needed.
*** A live image was built using this version of squashfs (using the default zlib compression) which functioned properly. No changes were made to livecd-creator to make the image.
* squashfs-tools
*** Currently there is just a scratch build of the dev squashfs-tools.
*** The 4.2 release is now in rawhide and F15 testing.
* livecd-creator needs to be modified to have mksquashfs use lzma compression. (Probably as the default, with other compression methods specifiable as command line options.)
** A [[QA:Testcase_squashfs-tools_compression|test case]] has been created to make it easier to get critical path testing.
* If possible this should be usable significantly ahead of feature freeze so that spin owners can be pretty safe relying on it for their spins.
* Out of scope is taking advantage of squashfs-tools new compression specific options.
** They will probably be exposed in some way in livecd-creator for F16 if it looks like they are useful to tweak for live images. But that won't be a feature.


== How To Test ==
== How To Test ==
* We need to make sure live images work.
* We need to make sure live images work.
* We need to make sure live images are smaller using LZMA compression.
** The nightly builds are now using xz compression.
* We need to make sure performance is acceptable. (I.e. that LZMA doesn't slow things down significantly on typical systems.)
** The Desktop spin from the nightly composes boots successfully from a live USB drive.
* We need to make sure live images are smaller using LZMA2 (XZ) compression.
** A test games spin from 2011-01-21 rawhide + a 2.6.38 scratch build went from 4499222528 bytes to 4111771648 bytes (8.6% decrease).
** A test desktop spin dropped from 753926144 bytes to 613416960 bytes (18.6% decrease).
* We need to make sure performance is acceptable.
** Using a live image needs to not be noticeably slower (hopefully it will be faster with typcial hardware) with LZMA than with GZIP compression.
*** The speed seemed comparable when run off a USB drive.
** It is OK if making a compressed file system using xz is somewhat slower than with gzip, because commonly time isn't as critical when making the image (compared to using it).
** squashfs-tools should be [[QA:Testcase_squashfs-tools_compression|tested]] to make sure all of the compression methods work.
* Specific test cases: (Note in early testing, the default will be gzip rather than xz.)
** Squashfs 4.2
*** No --compression-type
**** Should build using xz and image should work.
*** --compression-type=gzip
**** Should build using gzip and image should work.
*** --compression-type=xz
**** Should build using xz and image should work with a kernel that supports xz squashfs.
**** The image should be smaller than with gzip compression.
**** The image should run nearly as fast or faster than when using gzip compression on common hardware.
**** The time needed to build the image should not be inordinately longer than when using gzip on common hardware.
*** --compression-type=garbage
**** A reasonable error message should be displayed
** man livecd-creator
*** Should have the --compression-type option documented with the correct default. (gzip in eary testing, xz in later testing.)
** livecd-creator --help
*** Should have the --compression-type option documented with the correct default. (gzip in eary testing, xz in later testing.)


== User Experience ==
== User Experience ==
Line 43: Line 69:


== Dependencies ==
== Dependencies ==
* squashfs-tools
* kernel (may need to ask for backport)
* kernel (may need to ask for backport)
** This is unlikely. If the patches don't make it into 2.6.38, then due to kernel release timing it will be better to slip to F16.
* livecd-tools
* livecd-tools
* Spin creation process
** I have started a [https://fedorahosted.org/rel-eng/ticket/4343 dialogue] with releng about this.


== Contingency Plan ==
== Contingency Plan ==
* Since the default in the new squashfs is to use zlib compression, we shouldn't have to revert squashfs-tools.
* Since there isn't really anything further to do until the kernel feature is available, the fallback is pretty much not to do anything.
* livecd-creator would need to have at least some of the changes reverted.


== Documentation ==
== Documentation ==
Line 56: Line 83:
== Release Notes ==
== Release Notes ==


* Mention the squashfs version change, that it supports LZMA compression, point to its documentation and note that it is supported in the kernel as well.
* Mention the squashfs version change, that it supports XZ (LZMA2) compression, point to its documentation and note that it is supported in the kernel as well.
* Mention that livecd-creator uses LZMA by default, how to get zlib compression if desired for use with older kernels or other reasons.
**I have noted this in the file systems beat.
* Mention that livecd-creator uses XZ by default, how to get GZIP compression if desired for use with older kernels or other reasons.
** I have something in the doc beats about this.
 
== Thanks ==
 
I would like to thank Phillip Lougher for his work on squashfs and Lasse Collin for getting XZ squashfs support in the 2.6.38 kernel. Most of the work for this feature was done by them upstream.


== Comments and Discussion ==
== Comments and Discussion ==
* See [[Talk:Features/LZMA_for_Live_Images]]
* See [[Talk:Features/LZMA_for_Live_Images]]


[[Category:FeaturePageIncomplete]]
[[Category:FeatureAcceptedF15]]
<!-- When your feature page is completed and ready for review -->
<!-- remove Category:FeaturePageIncomplete and change it to Category:FeatureReadyForWrangler -->
<!-- After review, the feature wrangler will move your page to Category:FeatureReadyForFesco... if it still needs more work it will move back to Category:FeaturePageIncomplete-->
<!-- A pretty picture of the page category usage is at: https://fedoraproject.org/wiki/Features/Policy/Process -->

Latest revision as of 21:43, 17 September 2016

LZMA for Live Images

Summary

Enabling XZ (a variant of LZMA) for compression to allow us ship more software on our live images.

Owner

  • Email: <bruno@wolff.to>

Current status

  • Targeted release: Fedora 15
  • Last updated: 2011-03-06
  • Percentage of completion: 100%

Detailed Description

LZMA compression typically provides better compression than gzip while still providing relatively fast uncompression. Once squashfs file systems are supported by the kernel we can use them for building live images. A test with the games spin resulted in a 8.7% reduction (from 4242538496 bytes to 3873439744 bytes) in size of the resulting live image.

Benefit to Fedora

By allowing more packages to be put on a Live Image, we can make a few of our space constrained images better. For other images, having them be smaller will make them faster to download.

Scope

  • kernel
    • F15 has had 2.6.38 kernels for a while and XZ is working fine.
    • Fallback plan for the kernel is to do nothing.
  • livecd-creator
    • Changes to livecd-creator:
      • Change the default compression type to xz. (Done - in rawhide)
      • Update the documentation to reflect the changed default. (Done - in rawhide)
      • These changes won't be made until after the kernel feature is available, so no fallback is needed.
  • squashfs-tools
      • The 4.2 release is now in rawhide and F15 testing.
    • A test case has been created to make it easier to get critical path testing.
  • Out of scope is taking advantage of squashfs-tools new compression specific options.
    • They will probably be exposed in some way in livecd-creator for F16 if it looks like they are useful to tweak for live images. But that won't be a feature.

How To Test

  • We need to make sure live images work.
    • The nightly builds are now using xz compression.
    • The Desktop spin from the nightly composes boots successfully from a live USB drive.
  • We need to make sure live images are smaller using LZMA2 (XZ) compression.
    • A test games spin from 2011-01-21 rawhide + a 2.6.38 scratch build went from 4499222528 bytes to 4111771648 bytes (8.6% decrease).
    • A test desktop spin dropped from 753926144 bytes to 613416960 bytes (18.6% decrease).
  • We need to make sure performance is acceptable.
    • Using a live image needs to not be noticeably slower (hopefully it will be faster with typcial hardware) with LZMA than with GZIP compression.
      • The speed seemed comparable when run off a USB drive.
    • It is OK if making a compressed file system using xz is somewhat slower than with gzip, because commonly time isn't as critical when making the image (compared to using it).
    • squashfs-tools should be tested to make sure all of the compression methods work.
  • Specific test cases: (Note in early testing, the default will be gzip rather than xz.)
    • Squashfs 4.2
      • No --compression-type
        • Should build using xz and image should work.
      • --compression-type=gzip
        • Should build using gzip and image should work.
      • --compression-type=xz
        • Should build using xz and image should work with a kernel that supports xz squashfs.
        • The image should be smaller than with gzip compression.
        • The image should run nearly as fast or faster than when using gzip compression on common hardware.
        • The time needed to build the image should not be inordinately longer than when using gzip on common hardware.
      • --compression-type=garbage
        • A reasonable error message should be displayed
    • man livecd-creator
      • Should have the --compression-type option documented with the correct default. (gzip in eary testing, xz in later testing.)
    • livecd-creator --help
      • Should have the --compression-type option documented with the correct default. (gzip in eary testing, xz in later testing.)

User Experience

Live images will be smaller or have more included packages.

Dependencies

  • kernel (may need to ask for backport)
    • This is unlikely. If the patches don't make it into 2.6.38, then due to kernel release timing it will be better to slip to F16.
  • livecd-tools
  • Spin creation process
    • I have started a dialogue with releng about this.

Contingency Plan

  • Since there isn't really anything further to do until the kernel feature is available, the fallback is pretty much not to do anything.

Documentation

  • Most likely livecd-creator would be getting a new option and would need its help updated along with the code changes.

Release Notes

  • Mention the squashfs version change, that it supports XZ (LZMA2) compression, point to its documentation and note that it is supported in the kernel as well.
    • I have noted this in the file systems beat.
  • Mention that livecd-creator uses XZ by default, how to get GZIP compression if desired for use with older kernels or other reasons.
    • I have something in the doc beats about this.

Thanks

I would like to thank Phillip Lougher for his work on squashfs and Lasse Collin for getting XZ squashfs support in the 2.6.38 kernel. Most of the work for this feature was done by them upstream.

Comments and Discussion