From Fedora Project Wiki
(Ad pointer to recent lzma xz discussion)
(It looks like xz (lzma2) may be replacing lzma)
Line 29: Line 29:
*** It looks like [http://git.kernel.org/?p=linux/kernel/git/pkl/squashfs-lzma.git;a=summary some patches] were being worked on, but weren't ready for 2.6.35 (or 2.6.36) as there was no pull request for them.
*** It looks like [http://git.kernel.org/?p=linux/kernel/git/pkl/squashfs-lzma.git;a=summary some patches] were being worked on, but weren't ready for 2.6.35 (or 2.6.36) as there was no pull request for them.
** [http://sourceforge.net/mailarchive/forum.php?forum_name=squashfs-devel&max_rows=25&style=nested&viewmonth=201011 Discussion] about lzma / xz support in the kernel on the squashfs-devel list.
** [http://sourceforge.net/mailarchive/forum.php?forum_name=squashfs-devel&max_rows=25&style=nested&viewmonth=201011 Discussion] about lzma / xz support in the kernel on the squashfs-devel list.
** New development related to xz (lzma2) and squashfs kernel support:
*** http://marc.info/?l=linux-kernel&m=129187477131797&w=2
*** http://marc.info/?l=linux-kernel&m=129131729423308&w=2
*** http://git.kernel.org/?p=linux/kernel/git/pkl/squashfs-xz.git;a=summary
*** http://sourceforge.net/projects/squashfs/develop
** Fallback plan for the kernel is to do nothing.
** Fallback plan for the kernel is to do nothing.
* livecd-creator
* livecd-creator
Line 35: Line 40:
*** Update the documentation to reflect the changed default.
*** Update the documentation to reflect the changed default.
*** These changes won't be made until after the kernel feature is available, so no fallback is needed.
*** These changes won't be made until after the kernel feature is available, so no fallback is needed.
* squashfs-tools
** squashfs-tools might be back on the table as it looks like xz will be separate from lzma


== How To Test ==
== How To Test ==

Revision as of 18:43, 10 December 2010

LZMA for Live Images

Summary

Enabling LZMA for compression to allow us ship more software on our live images.

Owner

  • Email: <bruno@wolff.to>

Current status

  • Targeted release: Fedora 15
  • Last updated: 2010-10-11
  • Percentage of completion: 0%

Detailed Description

LZMA compression typically provides better compression than gzip while still providing relatively fast uncompression. Once squashfs file systems are supported by the kernel we can use them for building live images. A test with the games spin resulted in a 8.7% reduction (from 4242538496 bytes to 3873439744 bytes) in size of the resulting live image.

Benefit to Fedora

By allowing more packages to be put on a Live Image, we can make a few of our space constrained images better. For other images, having them be smaller will make them faster to download.

Scope

How To Test

  • We need to make sure live images work.
  • We need to make sure live images are smaller using LZMA compression.
  • We need to make sure performance is acceptable.
    • Using a live image needs to not be noticeably slower (hopefully it will be faster with typcial hardware) with LZMA than with gzip compression.
    • Making a compressed file system using LZMA can be somewhat slower than with gzip, because commonly time isn't as critical when making the image (compared to using it).
    • We need to test unsquashing squashed images work using unsquashfs (both LZMA and gzip).
  • Specific test cases: (Note in early testing, the default will be gzip rather than lzma.)
    • Squashfs 4.1
      • No --compression-type
        • Should build using lzma and image should work.
      • --compression-type=gzip
        • Should build using gzip and image should work.
      • --compression-type=lzma
        • Should build using lzma and image should work with a kernel that supports lzma squashfs.
        • The image should be smaller than with gzip compression.
        • The image should run nearly as fast or faster than with gzip compression on common hardware.
        • The time needed to build the image should not be inordinately longer than with gzip on common hardware.
      • --compression-type=garbage
        • A reasonable error message should be displayed
      • Test using a 2.6.33 kernel (with gzip) to make sure the xattr feature doesn't cause a problem for kernels that don't support that in squashfs file systems.
      • Test livecd-iso-to-disk on an image using lzma with a system with a 4.0 version of squashfs-tools.
      • Test livecd-iso-to-disk on an image using gzip with a system with a 4.0 version of squashfs-tools.
    • Squashfs 4.0
      • No --compression-type
        • A reasonable error message should be displayed
      • --compression-type=gzip
        • Should build using gzip and image should work.
      • --compression-type=lzma
        • A reasonable error message should be displayed
      • --compression-type=garbage
        • A reasonable error message should be displayed
    • man livecd-creator
      • Should have the --compression-type option documented with the correct default. (gzip in eary testing, lzma in later testing.)
    • livecd-creator --help
      • Should have the --compression-type option documented with the correct default. (gzip in eary testing, lzma in later testing.)

User Experience

Live images will be smaller or have more included packages.

Dependencies

  • kernel (may need to ask for backport)
  • livecd-tools

Contingency Plan

  • Since there isn't really anything further to do until the kernel feature is available, the fallback is pretty much not to do anything.

Documentation

  • Most likely livecd-creator would be getting a new option and would need its help updated along with the code changes.

Release Notes

  • Mention the squashfs version change, that it supports LZMA compression, point to its documentation and note that it is supported in the kernel as well.
  • Mention that livecd-creator uses LZMA by default, how to get gzip compression if desired for use with older kernels or other reasons.

Thanks

I would like to thank Phillip Lougher for his work on squashfs. The key work needed for this feature was done by Phillip upstream.

Comments and Discussion