From Fedora Project Wiki

Zchunk Metadata

Summary

All dnf repository metadata will be compressed with the zchunk format in addition to xz or gzip.

Owner

Current status

Detailed Description

Currently Fedora's repository metadata is compressed using the xz and gzip formats. Zchunk is a new compression format designed to allow for highly efficient deltas. When Fedora's metadata is compressed using zchunk, dnf will download only the differences between any earlier copies of the metadata and the current version.

Benefit to Fedora

Dnf and related tools will see significant reductions in the size of the metadata they download, especially if they are run on a regular basis.

Scope

  • Other developers: Fedora Infrastructure needs to start creating zchunked metadata
  • Policies and guidelines: Packaging guidelines are not affected by this change.
  • Trademark approval: N/A (not needed for this Change)

Upgrade/compatibility impact

Old-style xz and gzip metadata will continue to be generated by createrepo_c, so old versions of Fedora will use the xz/gzip metadata while upgrading to the new version, and then use the new zchunk metadata after the upgrade.

How To Test

Check that zchunk metadata is being downloaded:

  1. Remove /var/cache/dnf/*
  2. Run dnf update
  3. Check that the files in /var/cache/dnf/updates-*/repodata have .zck extensions

Check that zchunk metadata won't be redownloaded when it matches the current metadata:

  1. Remove /var/cache/dnf/updates-*/repodata/repomd.xml
  2. Run dnf update
  3. Check that the repodata download size matches the size of just repomd.xml

Check that only different chunks are being downloaded:
Todo: setup two snapshots of updates that are a few days apart

  1. Create /etc/yum.repos.d/test.repo that contains the following lines:
    [test]
    name=Fedora $releasever - $basearch - Test
    failovermethod=priority
    baseurl=first_snapshot_from_todo
    enabled=0
    gpgcheck=0
    skip_if_unavailable=True
  2. Run dnf --enablerepo=test update
  3. Change /etc/yum.repos.d/test.repo as follows:
    baseurl=second_snapshot_from_todo
  4. Run dnf --enablerepo=test update
  5. Check that the repodata download size matches delta size from todo bytes

User Experience

This change will allow Fedora users to see significant reductions in the size of the metadata they download. If they check for updates daily, they should see a savings of up to 95%.

Dependencies

Upstream needs to merge the zchunk pull requests, and Fedora 29's dnf, libdnf, librepo, libsolv and createrepo_c will need to pull the patches in.

Contingency Plan

  • Contingency mechanism:
    • If there are problems generating zchunked metadata, Fedora Infrastructure will need to disable zchunk metadata generation
    • If there are problems with the zchunk dnf/libdnf integration, we will:
      • First disable the downloading of zchunk metadata
      • If that doesn't fix it, we will revert the zchunk patches
  • Contingency deadline: 2018-08-28 (Beta Freeze)
  • Blocks release? No, as long as the contingency is enacted
  • Blocks product? No

Documentation

Introduction to zchunk
What is zchunk and how does it work?
How to use the zchunk utilities
The zchunk file format

Release Notes