From Fedora Project Wiki
No edit summary
m (internal link cleaning)
 
(99 intermediate revisions by 3 users not shown)
Line 2: Line 2:


== Summary ==
== Summary ==
Better compression technology (LZMA) allows us to ship more software on our live images.
Enabling XZ (a variant of LZMA) for compression to allow us ship more software on our live images.


== Owner ==
== Owner ==
Line 10: Line 10:


== Current status ==
== Current status ==
* Targeted release: [[Releases/14 | Fedora 14 ]]
* Targeted release: [[Releases/15 | Fedora 15 ]]
* Last updated: 2010-06-11
* Last updated: 2011-03-06
* Percentage of completion: 55%
* Percentage of completion: 100%


== Detailed Description ==
== Detailed Description ==
[http://en.wikipedia.org/wiki/Lempel-Ziv-Markov_chain_algorithm LZMA compression] typically provides better compression than zlib while still providing relatively fast uncompression. Once squashfs file systems are supported by the kernel we can use them for building live images. A test with the games spin resulted in a 10% reduction in size of the resulting live image.
[http://en.wikipedia.org/wiki/Lempel-Ziv-Markov_chain_algorithm LZMA compression] typically provides better compression than gzip while still providing relatively fast uncompression. Once squashfs file systems are supported by the kernel we can use them for building live images. A test with the games spin resulted in a 8.7% reduction (from 4242538496 bytes to 3873439744 bytes) in size of the resulting live image.


== Benefit to Fedora ==
== Benefit to Fedora ==
Line 22: Line 22:
== Scope ==
== Scope ==
* kernel
* kernel
** Lougher needs to do a new set of patches since the ones for 2.6.34 were only partially accepted.
** F15 has had 2.6.38 kernels for a while and XZ is working fine.
*** Patches for 2.6.35 were not supplied during the merge window. This makes it unlikely that they will be accepted for 2.6.35 and that 2.6.36 will be the next chance. If the patches do get accepted for 2.6.36, we can probably have the squashfs support in place for the release (maybe backporting patches, since F14 will probably release with 2.6.35). I'll need to have squashfs and livecd-creator updates ready to go. If I need to wait until the 2.6.36 merge window to know if the feature will be doable or not, there will be little time after that to get it tested. So we need to be able to start testing promptly.
*** Statements from Lougher:
**** https://bugzilla.redhat.com/show_bug.cgi?id=504425#c19
**** http://marc.info/?l=linux-kernel&m=127190194229951&w=2
*** It looks like [http://git.kernel.org/?p=linux/kernel/git/pkl/squashfs-lzma.git;a=summary some patches] were being worked on, but weren't ready for 2.6.35 as there was no pull request for them.
** Fallback plan for the kernel is to do nothing.
** Fallback plan for the kernel is to do nothing.
* livecd-creator
** Changes to livecd-creator:
*** Change the default compression type to xz. (Done - in rawhide)
*** Update the documentation to reflect the changed default. (Done - in rawhide)
*** These changes won't be made until after the kernel feature is available, so no fallback is needed.
* squashfs-tools
* squashfs-tools
** Lougher's 4.1 has been built for F14.
*** The 4.2 release is now in rawhide and F15 testing.
*** It uses multiple processors for both zlib and lzma compression.
** A [[QA:Testcase_squashfs-tools_compression|test case]] has been created to make it easier to get critical path testing.
*** A test games spin had a 10% reduction (from almost 4.4 GB to just under 4 GB) in size using lzma compression, saving about 400 MB. (This was a nonfunctional image because of lack of kernel support for lzma squashfs.)
* Out of scope is taking advantage of squashfs-tools new compression specific options.
*** A live image was built using this version of squashfs (using the default zlib compression) which functioned properly. No changes were made to livecd-creator to make the image.
** They will probably be exposed in some way in livecd-creator for F16 if it looks like they are useful to tweak for live images. But that won't be a feature.
** The fallback plan if squashfs-tools works (or at least doesn't have regressions for zlib compression) but some other part of feature can't be completed is to continue using the dev version.
** The fallback plan if the dev version of squashfs-tools is found to have regressions affecting zlib compression, is to revert by untagging the build if this apparent before the alpha release. After the alpha release we'd want to bump the epoch.
* livecd-creator
** livecd-creator needs to be modified to have mksquashfs use lzma compression. (Probably as the default, with other compression methods specifiable as command line options.)
** If possible this should be usable significantly ahead of feature freeze so that spin owners can be pretty safe relying on it for their spins.
** Proposed changes to livecd-creator:
*** Add a --compression_type= parameter that takes a string that will normally be passed to mksquashfs.
*** Add a default compression type config. This will initially be 'zlib', but will be changed to 'lzma', once we believe that is a better default for most people. (After there aren't regressions and performance is know to be acceptible in most cases.)
*** Add a default compression for mksquashfs type, so that if the compression type requested (or defaulted) matches this type, no compression option is specified for mksquashfs. This will allow the use of older versions of squashfs-tools as long as the mksquashfs default ('zlib') compression type is used.
*** Document the 'zlib' and 'lzma' types. Warn that 'lzma' needs kernel support for mounting and live images.
** The simple fallback plan for livecd-creator is to document that only 'zlib' compression is available currently. If the default was changed to 'lzma', change it back to 'zlib. The more intrusive fallback is to revert all of the compression type changes.
** A patch is undergoing testing now.


== How To Test ==
== How To Test ==
* We need to make sure live images work.
* We need to make sure live images work.
* We need to make sure live images are smaller using LZMA compression.
** The nightly builds are now using xz compression.
** The Desktop spin from the nightly composes boots successfully from a live USB drive.
* We need to make sure live images are smaller using LZMA2 (XZ) compression.
** A test games spin from 2011-01-21 rawhide + a 2.6.38 scratch build went from 4499222528 bytes to 4111771648 bytes (8.6% decrease).
** A test desktop spin dropped from 753926144 bytes to 613416960 bytes (18.6% decrease).
* We need to make sure performance is acceptable.
* We need to make sure performance is acceptable.
** Using a live image needs to not be noticeably slower (hopefully it will be faster with typcial hardware) with LZMA than with zlib compression.
** Using a live image needs to not be noticeably slower (hopefully it will be faster with typcial hardware) with LZMA than with GZIP compression.
** Making a compressed file system using LZMA can be somewhat slower than with zlib, because commonly time isn't as critical when making the image (compared to using it).
*** The speed seemed comparable when run off a USB drive.
** We need to test unsquashing squashed images work using unsquashfs (both LZMA and zlib).
** It is OK if making a compressed file system using xz is somewhat slower than with gzip, because commonly time isn't as critical when making the image (compared to using it).
* Specific test cases:
** squashfs-tools should be [[QA:Testcase_squashfs-tools_compression|tested]] to make sure all of the compression methods work.
** Squashfs 4.1
* Specific test cases: (Note in early testing, the default will be gzip rather than xz.)
** Squashfs 4.2
*** No --compression-type
*** No --compression-type
**** Should build using zlib and image should work.
**** Should build using xz and image should work.
*** --compression-type=zlib
*** --compression-type=gzip
**** Should build using zlib and image should work.
**** Should build using gzip and image should work.
*** --compression-type=lzma
*** --compression-type=xz
**** Should build using lzma and image should work with a kernel that supports lzma squashfs.
**** Should build using xz and image should work with a kernel that supports xz squashfs.
**** The image should be smaller than with zlib compression.
**** The image should be smaller than with gzip compression.
**** The image should run nearly as fast or faster than with zlib compression on common hardware.
**** The image should run nearly as fast or faster than when using gzip compression on common hardware.
**** The time needed to build the image should not be inordinately longer than with zlib on common hardware.
**** The time needed to build the image should not be inordinately longer than when using gzip on common hardware.
*** --compression-type=garbage
**** A reasonable error message should be displayed
** Squashfs 4.0
*** No --compression-type
**** Should build using zlib and image should work.
*** --compression-type=zlib
**** Should build using zlib and image should work.
*** --compression-type=lzma
**** A reasonable error message should be displayed
*** --compression-type=garbage
*** --compression-type=garbage
**** A reasonable error message should be displayed
**** A reasonable error message should be displayed
** man livecd-creator
*** Should have the --compression-type option documented with the correct default. (gzip in eary testing, xz in later testing.)
** livecd-creator --help
*** Should have the --compression-type option documented with the correct default. (gzip in eary testing, xz in later testing.)


== User Experience ==
== User Experience ==
Line 81: Line 69:


== Dependencies ==
== Dependencies ==
* squashfs-tools
* kernel (may need to ask for backport)
* kernel (may need to ask for backport)
** This is unlikely. If the patches don't make it into 2.6.38, then due to kernel release timing it will be better to slip to F16.
* livecd-tools
* livecd-tools
* Spin creation process
** I have started a [https://fedorahosted.org/rel-eng/ticket/4343 dialogue] with releng about this.


== Contingency Plan ==
== Contingency Plan ==
* Since the default in the new squashfs is to use zlib compression, we shouldn't have to revert squashfs-tools.
* Since there isn't really anything further to do until the kernel feature is available, the fallback is pretty much not to do anything.
* The livecd-creator changes are being implemented in a way that up until the default is changed, nothing should have to be reverted. If the default gets changed to lzma, then that would need to get reverted if a problem was found.


== Documentation ==
== Documentation ==
Line 94: Line 83:
== Release Notes ==
== Release Notes ==


* Mention the squashfs version change, that it supports LZMA compression, point to its documentation and note that it is supported in the kernel as well.
* Mention the squashfs version change, that it supports XZ (LZMA2) compression, point to its documentation and note that it is supported in the kernel as well.
* Mention that livecd-creator uses LZMA by default, how to get zlib compression if desired for use with older kernels or other reasons.
**I have noted this in the file systems beat.
* Mention that livecd-creator uses XZ by default, how to get GZIP compression if desired for use with older kernels or other reasons.
** I have something in the doc beats about this.
 
== Thanks ==
 
I would like to thank Phillip Lougher for his work on squashfs and Lasse Collin for getting XZ squashfs support in the 2.6.38 kernel. Most of the work for this feature was done by them upstream.


== Comments and Discussion ==
== Comments and Discussion ==
* See [[Talk:Features/LZMA_for_Live_Images]]
* See [[Talk:Features/LZMA_for_Live_Images]]


[[Category:FeatureReadyForFesco]]
[[Category:FeatureAcceptedF15]]
<!-- When your feature page is completed and ready for review -->
<!-- remove Category:FeaturePageIncomplete and change it to Category:FeatureReadyForWrangler -->
<!-- After review, the feature wrangler will move your page to Category:FeatureReadyForFesco... if it still needs more work it will move back to Category:FeaturePageIncomplete-->
<!-- A pretty picture of the page category usage is at: https://fedoraproject.org/wiki/Features/Policy/Process -->

Latest revision as of 21:43, 17 September 2016

LZMA for Live Images

Summary

Enabling XZ (a variant of LZMA) for compression to allow us ship more software on our live images.

Owner

  • Email: <bruno@wolff.to>

Current status

  • Targeted release: Fedora 15
  • Last updated: 2011-03-06
  • Percentage of completion: 100%

Detailed Description

LZMA compression typically provides better compression than gzip while still providing relatively fast uncompression. Once squashfs file systems are supported by the kernel we can use them for building live images. A test with the games spin resulted in a 8.7% reduction (from 4242538496 bytes to 3873439744 bytes) in size of the resulting live image.

Benefit to Fedora

By allowing more packages to be put on a Live Image, we can make a few of our space constrained images better. For other images, having them be smaller will make them faster to download.

Scope

  • kernel
    • F15 has had 2.6.38 kernels for a while and XZ is working fine.
    • Fallback plan for the kernel is to do nothing.
  • livecd-creator
    • Changes to livecd-creator:
      • Change the default compression type to xz. (Done - in rawhide)
      • Update the documentation to reflect the changed default. (Done - in rawhide)
      • These changes won't be made until after the kernel feature is available, so no fallback is needed.
  • squashfs-tools
      • The 4.2 release is now in rawhide and F15 testing.
    • A test case has been created to make it easier to get critical path testing.
  • Out of scope is taking advantage of squashfs-tools new compression specific options.
    • They will probably be exposed in some way in livecd-creator for F16 if it looks like they are useful to tweak for live images. But that won't be a feature.

How To Test

  • We need to make sure live images work.
    • The nightly builds are now using xz compression.
    • The Desktop spin from the nightly composes boots successfully from a live USB drive.
  • We need to make sure live images are smaller using LZMA2 (XZ) compression.
    • A test games spin from 2011-01-21 rawhide + a 2.6.38 scratch build went from 4499222528 bytes to 4111771648 bytes (8.6% decrease).
    • A test desktop spin dropped from 753926144 bytes to 613416960 bytes (18.6% decrease).
  • We need to make sure performance is acceptable.
    • Using a live image needs to not be noticeably slower (hopefully it will be faster with typcial hardware) with LZMA than with GZIP compression.
      • The speed seemed comparable when run off a USB drive.
    • It is OK if making a compressed file system using xz is somewhat slower than with gzip, because commonly time isn't as critical when making the image (compared to using it).
    • squashfs-tools should be tested to make sure all of the compression methods work.
  • Specific test cases: (Note in early testing, the default will be gzip rather than xz.)
    • Squashfs 4.2
      • No --compression-type
        • Should build using xz and image should work.
      • --compression-type=gzip
        • Should build using gzip and image should work.
      • --compression-type=xz
        • Should build using xz and image should work with a kernel that supports xz squashfs.
        • The image should be smaller than with gzip compression.
        • The image should run nearly as fast or faster than when using gzip compression on common hardware.
        • The time needed to build the image should not be inordinately longer than when using gzip on common hardware.
      • --compression-type=garbage
        • A reasonable error message should be displayed
    • man livecd-creator
      • Should have the --compression-type option documented with the correct default. (gzip in eary testing, xz in later testing.)
    • livecd-creator --help
      • Should have the --compression-type option documented with the correct default. (gzip in eary testing, xz in later testing.)

User Experience

Live images will be smaller or have more included packages.

Dependencies

  • kernel (may need to ask for backport)
    • This is unlikely. If the patches don't make it into 2.6.38, then due to kernel release timing it will be better to slip to F16.
  • livecd-tools
  • Spin creation process
    • I have started a dialogue with releng about this.

Contingency Plan

  • Since there isn't really anything further to do until the kernel feature is available, the fallback is pretty much not to do anything.

Documentation

  • Most likely livecd-creator would be getting a new option and would need its help updated along with the code changes.

Release Notes

  • Mention the squashfs version change, that it supports XZ (LZMA2) compression, point to its documentation and note that it is supported in the kernel as well.
    • I have noted this in the file systems beat.
  • Mention that livecd-creator uses XZ by default, how to get GZIP compression if desired for use with older kernels or other reasons.
    • I have something in the doc beats about this.

Thanks

I would like to thank Phillip Lougher for his work on squashfs and Lasse Collin for getting XZ squashfs support in the 2.6.38 kernel. Most of the work for this feature was done by them upstream.

Comments and Discussion