Jigdo Release

Contents

  1. Jigdo Release
    1. Summary
    2. Owner
      1. Interested People
    3. Current status
    4. Detailed Description
      1. Current Jigdo Restraints
      2. Issues That pyJigdo Solves
    5. Benefit to Fedora
    6. Scope
    7. Test Plan
    8. User Experience
    9. Dependencies
    10. Contingency Plan
    11. Documentation
    12. Release Notes
    13. Questions

Summary

Jigdo is a distribution method for ISO images. A server hosts a number of files contained in an ISO image along with a .jigdo file and a template for each ISO image. A client downloads the template files and using the .jigdo, downloads the many small parts (possibly from many servers) that make up the final ISO image.

As the footprint for a Fedora Project mirror increases, Jigdo might resolve some of these problems by eliminating the need to host both the ISO (in releases/$releasever/Fedora/$arch/iso/) and the expanded release tree (in releases/$releasever/Fedora/$arch/os/).

Owner

Interested People

Current status

Detailed Description

For more information on Jigdo, see it's home page at http://atterer.net/jigdo/.

The size of a DVD image is over 3.0 GB (Source DVD image), up to 3.9 GB of PPC, while an ISO DVD template for jigdo is approximately 23 MB. Decreasing the footprint of a release becomes more important as sizes of images increases (such as with supporting multiple architectures, or releasing multiple spins or multiple types of media -CD/DVD/Everything Spin?).

Jigdo scans the ISO image(s) for files (slices) that match certain files in a given tree, such as the expanded release tree. Each match becomes referenced in the .template for the ISO image and gets an entry in the .jigdo file users use to download the slices with. In the .jigdo file, you can specify a number of resources to get the slices from, such as the Fedora Project mirrorlist, a local (private) mirror or an already downloaded DVD image.

Right now the Fedora Project releases a DVD iso as well as a Rescue CD, 14 GB in total: 

$ cd /data/fedora/releases/8/Fedora/
$ du -sch `find -type f -name "*.iso" | grep -v boot.iso`
3.7G    ./x86_64/iso/Fedora-8-x86_64-DVD.iso
104M    ./x86_64/iso/Fedora-8-x86_64-rescuecd.iso
3.0G    ./source/iso/Fedora-8-source-DVD.iso
3.2G    ./i386/iso/Fedora-8-i386-DVD.iso
103M    ./i386/iso/Fedora-8-i386-rescuecd.iso
135M    ./ppc/iso/Fedora-8-ppc-rescuecd.iso
3.9G    ./ppc/iso/Fedora-8-ppc-DVD.iso
14G     total

In addition, the expanded installation tree is published and mirrored, 8.8 GB in total (keeping in mind linked files between i386 and x86_64, but not files linked from Everything): 

# du -sch `find -type d -name "os"`
3.7G    ./x86_64/os
2.2G    ./i386/os
3.1G    ./ppc/os
8.8G    total

This is 22.8GB of ISO images and expanded installation trees in total, per release. The expanded installation trees (8.8GB) with one overall .jigdo file, and .template files for each ISO (22MB each, approximately 154MB in total), would, using the expanded installation tree as a source, decrease the amount of data needing to be hosted and mirrored drastically.

Current Jigdo Restraints

Issues That pyJigdo Solves

Benefit to Fedora

Besides potentially decreasing the footprint of a Fedora release (the original ISO would not need to be mirrored), Jigdo facilitates having multiple architectures, multiple types of media, and can make optimal use of the Fedora Project mirrorlist (by redirecting the users to a local mirror or even a private mirror should they have one).

Using Jigdo, releasing an "Everything Spin" would consume approximately ~30 MB of .jigdo and .template files with the bit-extracted slices already being in the releases/$releasever/Everything/$arch/os/ tree and expanded installation tree at releases/$releasever/Fedora/$arch/os/).

Custom Installation Media Spins (such as with respins, rebranded or remixed content) would be able to use the same distribution method relying on the Fedora Project infrastructure without needing the Fedora Project to actually add the distribution to it's mirrors, BitTorrent seed/tracker or web pages.

Scope

  1. Integrate the Jigdo into the Release Process.
  2. Getting the distribution via Jigdo advertised on /get-fedora or spins.fedoraproject.org.

About 2), if 1) isn't met; not integrating the jigdo generation into the release process would mean the jigdo is available shortly after release (~4 hours).

Test Plan

  1. Compose CD images from the officially released Fedora 8 DVD and release them over Jigdo (DONE, http://fedoraunity.org/news-archives/fedora-8-cd-sets-released)

    • These CD images have been composed against the releases/8/Fedora/$arch/os/ expanded installation tree using the following command: 

      $ jigdo-file make-template \
--image=/data/fedora/spins/Fedora-8-i386-CD/Fedora-8-i386-CD1.iso \              # The input ISO image file
/data/fedora/releases/8/Fedora/i386/os/ \                                        # The tree to scan for files (slices)
--label Base-i386=/data/fedora/releases/8/Fedora/i386/os/ \                      # The label for the tree to scan
--jigdo=/var/www/jigdo/templates/Fedora-8-CD/Fedora-8-CD.jigdo \                 # The output jigdo file
--template=/var/www/jigdo/templates/Fedora-8-CD/Fedora-8-i386-CD1.iso.template \ # The output jigdo template for this ISO image
--no-servers-section \                                                           # No servers section as we specify this later on in the .jigdo file
--force \                                                                        # Force
--merge=/var/www/jigdo/templates/Fedora-8-CD/Fedora-8-CD.jigdo \                 # Merge with the existing .jigdo so one .jigdo file can
\                                                                                # re-compose multiple ISO images
--cache=/var/tmp/Fedora-8-CD.cache                                               # Cache what slices we've come across

  2. For further notes, see also http://kanarip.blogspot.com/2007/11/lessons-learned.html

    • Summarizing:

      • The '+' character in filenames (mostly RPM filenames) confuses wget/jigdo in combination with MirrorManager and need to be converted to '%2b' in the jigdo file.

      • It is best to generate the i386 ISO templates first
      • Templates should not be generated against a loop-mounted ISO.

A Fedora 9 Alpha release -as well as many Fedora 7 and Fedora 8 custom spins and Re-Spins, have been made available via Jigdo: http://spins.fedoraunity.org/spins

User Experience

Dependencies

Contingency Plan

Documentation

Release Notes

"""The Fedora Project is proud to announce it's releases are now also available via Jigdo. This particular distribution method has been used by the Debian distribution for a long, long time and could improve the speed at which you can obtain the installation ISO images. Instead of waiting for torrent downloads to complete, Jigdo seeks the fastest mirrors it can find via the Fedora Project Mirror Manager infrastructure, downloading the bits it needs from these mirrors. To optimize seeking these bits you can tell it to scan a DVD or CD you already have and prevent bits from being downloaded off the net, twice. This feature becomes particularly useful if you either

  1. Download all the test releases and then get the final release. You have 90% of the data already with each subsequent download.
  2. Download both the DVD and the CD set. The DVD holds 95% of the data on you need for the CD sets.
  3. A combination of the above.

Questions

JesseKeating:

CategoryAcceptedFedora9