All dnf repository metadata will be compressed with the zchunk format in addition to xz or gzip.
- Targeted release: Fedora 30
- Last updated: 2019-02-25
- Tracker bug: #1601595
- Release notes tracker: #285
Currently Fedora's repository metadata is compressed using the xz and gzip formats. Zchunk is a new compression format designed to allow for highly efficient deltas. When Fedora's metadata is compressed using zchunk, dnf will download only the differences between any earlier copies of the metadata and the current version.
Benefit to Fedora
Dnf and related tools will see significant reductions in the size of the metadata they download, especially if they are run on a regular basis.
- Proposal owners:
Package zchunk for Fedora- Done, see https://bugzilla.redhat.com/show_bug.cgi?id=1597287 Get the pull requests to enable zchunk in dnf, libdnf, librepo, libsolv and createrepo_c merged upstream https://github.com/rpm-software-management/librepo/pull/127 https://github.com/openSUSE/libsolv/pull/270 https://github.com/rpm-software-management/dnf/pull/1107No longer needed https://github.com/rpm-software-management/libdnf/pull/478 https://github.com/rpm-software-management/createrepo_c/pull/92
Create a new package for Fedora's zchunk dictionaries- See https://bugzilla.redhat.com/show_bug.cgi?id=1656627
- Other developers:
Fedora Infrastructure needs to start creating zchunked metadataGenerated since 2018-12-27 Build zchunk-enabled libdnf(see PR at https://github.com/rpm-software-management/libdnf/pull/663) Build librepo with patch to properly calculate download size for zchunk metadata(see PR at https://github.com/rpm-software-management/librepo/pull/138) Build createrepo_c with patch to generate zchunked metadata for all records(see PR at https://github.com/rpm-software-management/createrepo_c/pull/115)
- Policies and guidelines: Packaging guidelines are not affected by this change.
- Trademark approval: N/A (not needed for this Change)
Old-style xz and gzip metadata will continue to be generated by createrepo_c, so old versions of Fedora will use the xz/gzip metadata while upgrading to the new version, and then use the new zchunk metadata after the upgrade.
How To Test
Check that zchunk metadata is being downloaded:
- Remove /var/cache/dnf/*
- Run dnf update
- Check that the files in /var/cache/dnf/rawhide-*/repodata have .zck extensions
Check that zchunk metadata won't be redownloaded when it matches the current metadata:
- Remove /var/cache/dnf/rawhide-*/repodata/repomd.xml
- Run dnf update
- Check that the repodata download size matches the size of just repomd.xml
Check that only different chunks are being downloaded:
- Run dnf update
- Wait 24-48 hours
- Run dnf update again
- Validate that per-repository metadata download is less than 60MB (should be less than 5MB)
This change will allow Fedora users to see significant reductions in the size of the metadata they download. If they check for updates daily, they should see a savings of up to 95%.
Upstream needs to merge the zchunk pull requests, and Fedora 29's dnf, libdnf, librepo, libsolv and createrepo_c will need to pull the patches in.
- Contingency mechanism:
- If there are problems generating zchunked metadata, Fedora Infrastructure will need to disable zchunk metadata generation
- If there are problems with the zchunk dnf/libdnf integration, we will:
- First disable the downloading of zchunk metadata
- If that doesn't fix it, we will revert the zchunk patches
- Contingency deadline: 2019-03-05 (Beta Freeze)
- Blocks release? No, as long as the contingency is enacted
- Blocks product? No