From Fedora Project Wiki

m (→‎Image recompression and overlay refreshment: tune merge progress display)
No edit summary
Line 66: Line 66:
** there may be other good candidates...
** there may be other good candidates...


===Image recompression and overlay refreshment===
===Merge overlay into new image===


The Device-mapper snapshot-merge target allows one to merge changes in a persistent snapshot into the original filesystem (See notes from [[{{TALKPAGENAMEE}}|Andrew Gilmore]]). In a LiveOS image the original filesystem is compressed within a read-only SquashFS image.  If one has sufficient free disk space available (for example, 4.3 GiB of free space for Fedora 17 Live Desktop—typically on an attached hard drive—plus about 660 MiB or more, depending on additions of software or root filesystem files, of free space on the LiveOS device filesystem), one can copy (and uncompress) the original filesystem to a working folder, invoke the snapshot-merge Device-mapper target, delete the old SquashFS image, replace it with a recompressed version of the merged LiveOS snapshot, and reset the original snapshot overlay before rebooting the LiveOS device.
A new root filesystem image can be constructed by using some Device-mapper tools.


The following Bash script demonstrates the process where {{Code|$1}} is the working folder and {{Code|$2}} is the LiveOS device (such as {{Code|/run/initramfs/livedev}}, for the currently booted LiveOS image):
If one has sufficient free disk space available, typically on an attached hard drive, one can copy (and uncompress) the current filesystem to a working folder. (This may require, for example, 4.3 GiB of free space for Fedora 17 Live Desktop.)
 
Device-mapper's mirror target allows one to create a 'mirror' image of the LiveOS root filesystem (the snapshot based on the SquashFS and overlay device).  The mirror image can be then be recompressed, and if one has an additional 660 MiB or more of free space (depending on how much software or other root filesystem files have been added) on your LiveOS device filesystem (the USB/SD card filesystem), one can delete the old SquashFS image and replace it with the recompressed version.
 
The snapshot overlay can then be reset before rebooting the LiveOS device.
 
The following Bash script demonstrates the process, where the {{Code|$1}} parameter is a working folder and {{Code|$2}} is the LiveOS device node name (such as, {{Code|/run/initramfs/livedev}}, for the currently booted LiveOS image, or {{Code|/dev/sdc1, for example, for an attacked LiveOS device):
{{admon/warning|Test and practice the following script!|This procedure will delete the LiveOS filesystem before replacing it with a recompressed merged version. Be sure to verify that you will have sufficient space for the replacement before running the full script.  (You could comment out the deletion, and squash into a directory on a large working drive to determine the new filesystem size.)}}
{{admon/warning|Test and practice the following script!|This procedure will delete the LiveOS filesystem before replacing it with a recompressed merged version. Be sure to verify that you will have sufficient space for the replacement before running the full script.  (You could comment out the deletion, and squash into a directory on a large working drive to determine the new filesystem size.)}}
  <nowiki>#!/bin/bash
<pre>
t0=$(date +%s)
 
# Prepare working directory.
TMPDIR=$1
mkdir $TMPDIR/LiveOS 2>/dev/null
 
# Mount the source device and SquashFS.
SRCMNT=$(mktemp -d /media/XXXXXX)
mount $2 $SRCMNT
 
if [[ -e $SRCMNT/LiveOS/squashfs.img ]]; then
    SQUASHMNT=$(mktemp -d /media/XXXXXX)
    mount $SRCMNT/LiveOS/squashfs.img $SQUASHMNT --read-only
    ORIGFS=$SQUASHMNT/LiveOS/ext3fs.img
else
    ORIGFS=$SRCMNT/LiveOS/ext3fs.img
fi
 
# Prepare temporary devices for Device-mapper mirror target.
FSTYPE=$(blkid -s TYPE -o value $ORIGFS || :)
FSLABEL=$(blkid -s LABEL -o value $ORIGFS || :)
stinfo=($(stat -c '%b %B %o' $ORIGFS || :))
BLOCKS=${stinfo[0]}
BLOCKSZ=${stinfo[1]}
IOBLKSZ=${stinfo[2]}
NEWFS=$TMPDIR/LiveOS/ext3fs.img
dd if=/dev/null of=$NEWFS count=1 bs=$BLOCKSZ seek=$BLOCKS
mkfs.$FSTYPE -F -L $FSLABEL -m 1 -b $IOBLKSZ $NEWFS
tune2fs -c0 -i0 -Odir_index -ouser_xattr,acl $NEWFS
 
ORIGFSDEV=$(losetup -f --show $ORIGFS --read-only)
OVDEV=$(losetup -f --show $SRCMNT/LiveOS/overlay-* --read-only)
NEWFSDEV=$(losetup -f --show $NEWFS)
 
dmsetup create orig --readonly <<< "0 $BLOCKS snapshot $ORIGFSDEV $OVDEV P 8"
 
# Invoke mirror target device.
mirror="0 $BLOCKS mirror core 2 32 sync 2 /dev/mapper/orig 0 $NEWFSDEV 0"
dmsetup create merge <<< "$mirror"
 
# Wait for mirror completion.
while state=$(dmsetup status merge)
state=${state#*mirror 2 * * }
alloc=${state%/*}
size=${state#*/}
size=${size%% *}
[[ $alloc != $size ]]; do
    percent=$(dc <<< "8k 10 0.05 100 $alloc $size /*+*p")
    percent=$(dc <<< "1k $percent 10 /p")
    printf '\r  Mirroring %5.1f %% complete.  ' $percent
    printf '\b|'
    sleep 0.5
    printf '\b/'
    sleep 0.5
    printf '\b-'
    sleep 0.5
    printf '\b\'
    sleep 0.5
done
printf '\r Mirroring 100 %% complete.    \n'
 
# Clean up from mirror.
dmsetup remove merge orig
sync
 
dt=$(($(date +%s)-t0))
h=$((dt/3600))
m=$((dt%3600/60))
s=$((dt%60))
printf '\nTime elaspsed: %02d:%02d:%02d hh:mm:ss\n' $h $m $s
 
# Replace SquashFS image
rm $SRCMNT/LiveOS/squashfs.img
mksquashfs $TMPDIR/LiveOS $SRCMNT/LiveOS/squashfs.img -comp xz -keep-as-directory
 
# Reset overlay.
losetup -d $OVDEV
OVDEV=$(losetup -f --show $SRCMNT/LiveOS/overlay-*)
dd if=/dev/zero of=$OVDEV bs=64k count=1 conv=notrunc,fsync
 
# Clean up.
sleep 2
losetup -d $OVDEV $ORIGFSDEV $NEWFSDEV
sleep 2
umount $SQUASHMNT
sleep 2
umount $SRCMNT
rmdir $SQUASHMNT $SRCMNT
</pre>
 
The Device-mapper snapshot-merge target also allows one to merge changes in a persistent snapshot into the original filesystem (See notes from [[{{TALKPAGENAMEE}}|Andrew Gilmore]]).
{| class="collapsible collapsed"
|-
! Merge overlay with snapshot-merge
|-
|<pre>#!/bin/bash
# 2012-08-20 00:27:28
# 2012-08-20 00:27:28


Line 79: Line 181:
mkdir $TMPDIR/LiveOS 2>/dev/null
mkdir $TMPDIR/LiveOS 2>/dev/null


# Mount SquashFS, copy out the original root filesystem, unmount SquashFS.
# Mount source device, SquashFS, copy out the original root filesystem.
SRCMNT=$(mktemp -d /media/XXXXXX)
SRCMNT=$(mktemp -d /media/XXXXXX)
mount $2 $SRCMNT
mount $2 $SRCMNT
SQUASHMNT=$(mktemp -d /media/XXXXXX)
if [[ -e $SRCMNT/LiveOS/squashfs.img ]]; then
mount $SRCMNT/LiveOS/squashfs.img $SQUASHMNT
    SQUASHMNT=$(mktemp -d /media/XXXXXX)
    mount $SRCMNT/LiveOS/squashfs.img $SQUASHMNT --read-only
    ORIGFS=$SQUASHMNT/LiveOS/ext3fs.img
else
    ORIGFS=$SRCMNT/LiveOS/ext3fs.img
fi
if [[ -x /usr/bin/rsync ]]; then
if [[ -x /usr/bin/rsync ]]; then
     rsync --inplace --8-bit-output --progress $SQUASHMNT/LiveOS/ext3fs.img \
     rsync --inplace --progress $ORIGFS $TMPDIR/LiveOS
                                              $TMPDIR/LiveOS
else
else
     cp $SQUASHMNT/LiveOS/ext3fs.img $TMPDIR/LiveOS
     cp $ORIGFS $TMPDIR/LiveOS
fi
fi
umount $SQUASHMNT
umount $SQUASHMNT
BLOCKS=$(stat -c '%b' $ORIGFS || :)
NEWFS=$TMPDIR/LiveOS/ext3fs.img


# Prepare temporary devices for Device-mapper snapshot-merge.
# Prepare temporary devices for Device-mapper snapshot-merge.
FSDEV=$(losetup -f --show $TMPDIR/LiveOS/ext3fs.img)
NEWFSDEV=$(losetup -f --show $NEWFS --read-only)
OVDEV=$(losetup -f --show $SRCMNT/LiveOS/overlay-*)
OVDEV=$(losetup -f --show $SRCMNT/LiveOS/overlay-*)
FS_SIZE=$(blockdev -q --getsz $FSDEV)


# Invoke snapshot-merge target device.
# Invoke snapshot-merge target device.
dmsetup create merged <<< "0 $FS_SIZE snapshot-merge $FSDEV $OVDEV P 8"
dmsetup create merge <<< "0 $BLOCKS snapshot-merge $NEWFSDEV $OVDEV P 8"


# Wait for merge completion.
# Wait for merge completion.
while state=$(dmsetup status merged)
while state=$(dmsetup status merge)
state=${state#*snapshot-merge }
state=${state#*snapshot-merge }
meta=${state#* }
meta=${state#* }
Line 118: Line 227:


# Clean up from merge.
# Clean up from merge.
dmsetup remove merged
dmsetup remove merge
losetup -d $OVDEV $FSDEV
losetup -d $OVDEV $FSDEV


Line 132: Line 241:
losetup -d $OVDEV $FSDEV
losetup -d $OVDEV $FSDEV
umount $SRCMNT
umount $SRCMNT
rmdir $SQUASHMNT $SRCMNT</nowiki>
rmdir $SQUASHMNT $SRCMNT</pre>
|}
The mirror method copies and merges in one pass, while the snapshot-merge method copies the original filesystem first and then merges in the changes. I've found the mirror method to be about 15% faster.


===Overlay recovery===
===Overlay recovery===

Revision as of 16:36, 26 August 2012



Introduction

Fedora has developed Live CD USB DVD images for their GNU/Linux operating system. Since the image file systems are stored in the /LiveOS folder of the image, this is the name we'll use to reference the product.

This page shares some information about the LiveOS design that helps users take better advantage of their disc resources.

References

Storage

When a Live CD or Live DVD (a LiveOS image on read-only disc media) is booted, temporary storage is prepared for the system in RAM on each boot by /sbin/dmsquash-live-root in initrd0, the initial ram disk filesystem. An in-memory, copy-on-write, system overlay is used (see File Systems below).

The Fedora LiveOS system allows for persistent storage in 3 locations:

  1. An all-purpose, persistent overlay - a write-once, fixed-size file space that will save updates and changes to the LiveOS image (Activities, operating system changes, anything written in the LiveOS file space).
  2. Persistent home folder - a re-writable, re-sizable (with difficulty), uncompressed, optionally-encryptable, file space for anything that goes in the user's /home/ folder. A persistent home folder is an option that may be selected at the time of installation of the LiveOS image (although with some effort, one could manually create a home.img filesystem in /LiveOS/ and move the /home/ folder to it). This installation option is only available through the script, 'livecd-iso-to-disk'. The Windows and Fedora 'Live USB Creator' installers do not provide this option at present.
  3. Host device file space - this is the USB stick or SD card file system that is outside of the LiveOS file tree, but which is accessible through the /mnt/live or /run/initramfs/live (since Fedora 17) mount point of a booted LiveOS installation. There, one finds the boot configuration files and anything else one had on the device before loading the LiveOS image. One may save files here without consuming the other, limited file spaces. (This file space is limited by the device capacity).

The all-purpose, persistent overlay is needed for operating system changes and updates.

The file systems are prepared on each boot by the /etc/rc.d/init.d/livesys script.

Home filesystem

One may find many advantages to installing the LiveOS, with a persistent, home folder (using the --home-size-mb NN --delete-home options), which will hold all the user files and documents one wishes and, perhaps later, throw away—all without consuming the write-once overlay, which can be consumed very quickly (and overlay file space is not normally reusable).

Device filesystem

Additionally, keeping some storage space on the device disc outside of the LiveOS system will let you copy, carry, and delete large resource files, such as image.iso files, or anything you might want to use or share.

Installation options

Fedora 17 Live Desktop may be installed on a 1 GB USB device using the following options with livecd-iso-to-disk (on a single, terminal or console command line, even though the wiki may wrap the following text to accommodate your browser window size):

./livecd-iso-to-disk --reset-mbr --overlay-size-mb 200 --home-size-mb 140 --delete-home --unencrypted-home /path/to/source/iso/or/device /dev/sd?1
where '?' in the final parameter represents the target bootable device node, such as sdb1 or sdc1, etc.

The above configuration would allow space for the home folder, the operating system, and a little on the device root.

But with a larger capacity device, one may allocate the resources to suit the anticipated use, as described above.

File Systems

The Fedora LiveOS uses the Device-mapper service of the Linux kernel to manage the file stores on the device. This is the same service that is used by Logical Volume Manager to provide disc partition services.

One limitation, mentioned above, is that the LiveOS persistent overlay is a write-once file space. This is related to its use of device mapper snapshots to combine a read-only file system image (copied from the compressed SquashFS.img on the read-only LiveCD or installation .iso file) with a Copy-on-write service that tracks only changed blocks of data in the snapshot (overlay) file and then re-referencing file pointers to the updated blocks.[1][2] Any changes to the operating system files are stored as differences from the base. As such, "deletions" of files are saved as additional difference references, and the originals are hidden. With this mechanism, physical storage space is consumed in the write-once file space rather than recovered.

Consumption of the space allocated for persistent storage in the snapshot overlay file may be tracked with the device mapper dmsetup status report.

Several developments in the LiveOS design may be considered to maximize the endurance of the LiveOS image.

Avoid persistent overlay consumption

  • Use a persistent /home folder (discussed above).
    This is a very effective method to avoid consumption and will also make the user files more available for sharing and backup purposes.
  • Move the root user's home to /home/root in a persistent home.img
    This would avoid some consumption, but would compromise the root user account if there was a boot problem and the /home folder was not mounted.
  • Mount more folders onto temporary, in-memory filesystems (like /var/cache/yum, /var/tmp, & /tmp are now).
    • /var/lib/NetworkManager (holds a timestamps file that is deleted and refreshed quite often, every 5 minutes)
    • /var/log/audit (holds the SELinux audit.log that records a great many file accesses.)
    • /var/spool/abrt (holds often large, error reports and core dumps). Users could be advised to act on any abrt reports in the current session or copy the reports to other permanent storage, such as in /home/ or external storage.
    • there may be other good candidates...

Merge overlay into new image

A new root filesystem image can be constructed by using some Device-mapper tools.

If one has sufficient free disk space available, typically on an attached hard drive, one can copy (and uncompress) the current filesystem to a working folder. (This may require, for example, 4.3 GiB of free space for Fedora 17 Live Desktop.)

Device-mapper's mirror target allows one to create a 'mirror' image of the LiveOS root filesystem (the snapshot based on the SquashFS and overlay device). The mirror image can be then be recompressed, and if one has an additional 660 MiB or more of free space (depending on how much software or other root filesystem files have been added) on your LiveOS device filesystem (the USB/SD card filesystem), one can delete the old SquashFS image and replace it with the recompressed version.

The snapshot overlay can then be reset before rebooting the LiveOS device.

The following Bash script demonstrates the process, where the $1 parameter is a working folder and $2 is the LiveOS device node name (such as, /run/initramfs/livedev, for the currently booted LiveOS image, or {{Code|/dev/sdc1, for example, for an attacked LiveOS device):

Warning.png
Test and practice the following script!
This procedure will delete the LiveOS filesystem before replacing it with a recompressed merged version. Be sure to verify that you will have sufficient space for the replacement before running the full script. (You could comment out the deletion, and squash into a directory on a large working drive to determine the new filesystem size.)
t0=$(date +%s)

# Prepare working directory.
TMPDIR=$1
mkdir $TMPDIR/LiveOS 2>/dev/null

# Mount the source device and SquashFS.
SRCMNT=$(mktemp -d /media/XXXXXX)
mount $2 $SRCMNT

if [[ -e $SRCMNT/LiveOS/squashfs.img ]]; then
    SQUASHMNT=$(mktemp -d /media/XXXXXX)
    mount $SRCMNT/LiveOS/squashfs.img $SQUASHMNT --read-only
    ORIGFS=$SQUASHMNT/LiveOS/ext3fs.img
else
    ORIGFS=$SRCMNT/LiveOS/ext3fs.img
fi

# Prepare temporary devices for Device-mapper mirror target.
FSTYPE=$(blkid -s TYPE -o value $ORIGFS || :)
FSLABEL=$(blkid -s LABEL -o value $ORIGFS || :)
stinfo=($(stat -c '%b %B %o' $ORIGFS || :))
BLOCKS=${stinfo[0]}
BLOCKSZ=${stinfo[1]}
IOBLKSZ=${stinfo[2]}
NEWFS=$TMPDIR/LiveOS/ext3fs.img
dd if=/dev/null of=$NEWFS count=1 bs=$BLOCKSZ seek=$BLOCKS
mkfs.$FSTYPE -F -L $FSLABEL -m 1 -b $IOBLKSZ $NEWFS
tune2fs -c0 -i0 -Odir_index -ouser_xattr,acl $NEWFS

ORIGFSDEV=$(losetup -f --show $ORIGFS --read-only)
OVDEV=$(losetup -f --show $SRCMNT/LiveOS/overlay-* --read-only)
NEWFSDEV=$(losetup -f --show $NEWFS)

dmsetup create orig --readonly <<< "0 $BLOCKS snapshot $ORIGFSDEV $OVDEV P 8"

# Invoke mirror target device.
mirror="0 $BLOCKS mirror core 2 32 sync 2 /dev/mapper/orig 0 $NEWFSDEV 0"
dmsetup create merge <<< "$mirror"

# Wait for mirror completion.
while state=$(dmsetup status merge)
state=${state#*mirror 2 * * }
alloc=${state%/*}
size=${state#*/}
size=${size%% *}
[[ $alloc != $size ]]; do
    percent=$(dc <<< "8k 10 0.05 100 $alloc $size /*+*p")
    percent=$(dc <<< "1k $percent 10 /p")
    printf '\r  Mirroring %5.1f %% complete.  ' $percent
    printf '\b|'
    sleep 0.5
    printf '\b/'
    sleep 0.5
    printf '\b-'
    sleep 0.5
    printf '\b\'
    sleep 0.5
done
printf '\r  Mirroring 100 %% complete.    \n'

# Clean up from mirror.
dmsetup remove merge orig
sync

dt=$(($(date +%s)-t0))
h=$((dt/3600))
m=$((dt%3600/60))
s=$((dt%60))
printf '\nTime elaspsed: %02d:%02d:%02d hh:mm:ss\n' $h $m $s

# Replace SquashFS image
rm $SRCMNT/LiveOS/squashfs.img
mksquashfs $TMPDIR/LiveOS $SRCMNT/LiveOS/squashfs.img -comp xz -keep-as-directory

# Reset overlay.
losetup -d $OVDEV
OVDEV=$(losetup -f --show $SRCMNT/LiveOS/overlay-*)
dd if=/dev/zero of=$OVDEV bs=64k count=1 conv=notrunc,fsync

# Clean up.
sleep 2
losetup -d $OVDEV $ORIGFSDEV $NEWFSDEV
sleep 2
umount $SQUASHMNT
sleep 2
umount $SRCMNT
rmdir $SQUASHMNT $SRCMNT

The Device-mapper snapshot-merge target also allows one to merge changes in a persistent snapshot into the original filesystem (See notes from Andrew Gilmore).

The mirror method copies and merges in one pass, while the snapshot-merge method copies the original filesystem first and then merges in the changes. I've found the mirror method to be about 15% faster.

Overlay recovery

Warning.png
Test and practice the following
This procedure is not thoroughly validated, and may destroy your data.

If one 'exhausts' the limited storage capacity of a LiveOS overlay, Device-mapper will mark the filesystem as 'Invalid', as shown by the dmsetup status command executed in Terminal or a console (if you haven't crashed):

# dmsetup status
live-osimg-min: 0 8388608 snapshot 2464/2464 24
live-rw: 0 8388608 snapshot Invalid

The invalid bit is 00 at byte 5 of the overlay. You might try to recover the overlay by switching it to 01 with the following command line:

# echo $'\x01' | dd of=/path/to/overlay-file bs=1 count=1 seek=4 conv=notrunc

where /path/to/overlay may be /mnt/live/LiveOS/overlay-devicename-discUUID or
/media/devicename/LiveOS/overlay-devicename-discUUID for an attached device.

Follow this by registering the LiveOS image with Device-mapper:

# mount /media/devicename/LiveOS/squashfs.img /mnt/some_mountpoint
# losetup /dev/loop1 /mnt/some_mountpoint/LiveOS/ext3fs.img -r
# losetup /dev/loop2 /media/devicename/LiveOS/overlay-devicename-discUUID
# dmsetup create devicename --table "0 8388608 snapshot 7:1 7:2 P 8"
  • devicename in the last line may be any string.
  • loop1 and loop2 (and the corresponding 7:1 7:2) may be any free loop devices; just substitute the appropriate numbers.
  • 8388608 is the size of the ext3fs.img file in 512 byte units. This can be read by the following command:
# blockdev -q --getsz /dev/loop1

Execute

# dmsetup status

to check the the virtual filesystem has been configured.

Then try to repair any damage with the following command:

# e2fsck -f -y /dev/mapper/devicename

where devicename is the string you used in the dmsetup create command.

Run a second check,

# e2fsck -p /dev/mapper/devicename

to verify if the filesystem could be repaired. At this point you will want to enlarge the overlay, or backup or rebuild the image.

Remove the virtual filesystem registration,

# dmsetup remove devicename

Determine the size of the overlay file:

# blockdev -q --getsz /dev/loop2
# losetup -d /dev/loop2
# losetup -d /dev/loop1
# dd if=/dev/zero of=/path/to/overlay-file seek=overlay_size count=size_increase

where overlay_size and size_increase are now 512 byte units.

Once you have a working filesystem, you should proceed to merge your overlay into the original filesystem as describe above.

References

See this dm-devel thread.