Features/NoarchSubpackages

From FedoraProject

< Features(Difference between revisions)
Jump to: navigation, search
(Detailed Description: update progress)
(Documentation: first sketch)
Line 99: Line 99:
  
 
== Documentation ==
 
== Documentation ==
* '''Incomplete''': Documentation will be created as part of this Feature (see [[#Detailed Description]]).
+
* '''Incomplete Draft'''
 +
 
 +
=== What's this all about? ===
 +
 
 +
With version 4.6.0 RPM supports subpackages being noarch by just
 +
adding "BuildArch: noarch" to their subpackage section in the spec
 +
file.
 +
 
 +
The noarch subpackages build on the different arches are going to be
 +
compared by koji with rpmdiff ignoring time stamp, size and md5 sum of
 +
files. If any other differences are found the build will be rejected.
 +
Even with those automatic checks in place it is the
 +
responsibility of the packager to make sure that the package is really
 +
arch independent - as for regular noarch packages, too.
 +
 
 +
=== Candidates for being switched to noarch ===
 +
 
 +
 
 +
To get a list with good candidates all x86_64 packages that contain no binaries/libs (as known to rpm) and no files
 +
in /lib64 or /usr/lib64 were selected as a starting point. There are a few false positives - mostly
 +
development packages that put files in different locations. To further refine the selection and get an idea what can go wrong
 +
rpmdiff was run against the i386 sister packages - both with the relaxed and the -t setting. This showed a small number of false positives
 +
and several packages with inconsistent owner settings on some files (TODO: find out reason). The sub packages are marked by one
 +
surrounding '*' if they fail the rpmdiff check as used by koji and by two if they also fails the more strict rpmdiff -t check. It is assumed that packages without '*' can be directly switched to noarch (assuming they don't do weird stuff on other arches). One '*' will require a more detailed look but should be OK in most cases and two '*'s is either a false positive of a packaging bug (file ownage problem)
 +
 
 +
* [[media:NoarchCandidates.txt]]
 +
* [[media:NoarchRpmdiff.txt]]
 +
* [[media:NoarchStrictRpmdiff.txt]]
 +
 
 +
=== What about other packages? ===
 +
 
 +
A lot of other packages could also make use of this feature. When
 +
considering to split up your package please avoid too complicated spec
 +
files. We still have to develop packaging strategies to be applied
 +
throughout the distribution and it doesn't look like this is going to
 +
happen in the F11 time frame.
 +
 
 +
=== What can you do as a packager? ===
 +
 
 +
There are still fixes for koji that must hit the
 +
Fedora build system first. So noarch subpackages DO NOT WORK within Fedora
 +
yet. We hope that this can be solved soon.
 +
 
 +
If you are interested you can already play with noarch subpackages by
 +
building with mock and comparing the results on different arches with
 +
rpmdiff -t (Files differing in S and 5 are ok). There is going to be
 +
little time between support in koji and the
 +
feature freeze. So being prepared for this short time slot is a good
 +
thing.
 +
 
 +
=== What if you don't want to change your packages? ===
 +
 
 +
That's perfectly fine. There is no plan to force packager to use
 +
noarch subpackages. I hope we can develop a more detailed plan on how to
 +
make use of this feature in future Fedora releases. You might be
 +
interested in taking part in this discussion.
 +
 
 +
=== What does that mean for the Packaging Policy? ===
 +
 
 +
The packaging policy will require a few additions. See [[/PolicyChanges]].
 +
Any comment and help is welcome.
  
 
== Release Notes ==
 
== Release Notes ==

Revision as of 16:55, 11 February 2009

Contents

Support Noarch Sub Packages in Fedora

Summary

Since some months RPM supports sub packages being noarch. Right now the Fedora infrastructure does not support this feature. This feature will provide the technical abilities to use noarch sub packages and also provide help to use them within packages all over the distribution.

Owner

Current status

  • Targeted release: Fedora 11
  • Last updated: --Ffesti 19:17, 29 January 2009 (UTC)
  • Percentage of completion: 25%

Detailed Description

There are several steps needed:

  • Support in rpm (100%)
  • Support in koji (75%)
    • see Ticket
    • Fedora infrastructure still needs to be updated
  • Support in other parts of the infrastructure (unknown)
  • Support in test/verification tools (unknown)
    • rpmlint (?)
    • ... (?)
  • Get a list of possible candidates (sub packages) (90%)
  • Write a mail to f-d-l and package owners (0%)
  • Write best practise documentation (0%)
  • Get packaging policy adjusted (see /PolicyChanges) (10%)
  • Get the packages changed (0)

Benefit to Fedora

Noarch packages have several benefits over arch dependent packages:

  • They can be shared between different architectures and thus use up less disk space and bandwidth on both the Fedora infrastructure and our mirrors
  • They avoid double installation of data for multilib packages.
  • They tell the user that the content of the package is arch independent.

By increasing the use of noarch packages we also increase the effect of these benefits.

Additionally we can get rid of some hacks that are used to generate noarch sub packages for very few packages right now.

Scope

A small statistic on Fedora rawhide x86_64 (2009-01-22) to give an idea how many packages/files/bytes could be affected:

The files are split into 3 groups:

  1. binary: files rpm knows that they are arch dependent
  2. libdir: files that are not binaries but reside in (/usr)/lib(64)
  3. noarch: everything else

Libdir files should be noarch in most cases. Sizes are (uncompressed) bytes in files and though do not directly map to the size of packages nor to used disk space.

15204 packages (44 GB in 2.0 M files, 100%)
        41 k binary files (8.8 GB, ~20%)
       298 k libdir files (7.1 GB, ~16%)
       1.7 M noarch files (28 GB, ~64%)

8762 x86_64 packages (25 GB in 1.0 M files, 100%)
        31 k binary files (6.7 GB, ~27%)
       182 k libdir files (5.9 GB, ~24%)
       826 k noarch files (12 GB, ~48%)
3132 i386 packages (5.3 GB in 280 k files, 100%)
        10 k binary files (2.0 GB, ~38%)
        32 k libdir files (551 MB, ~11%)
       237 k noarch files (2.7 GB, ~51%)
3308 noarch packages (13 GB in 755 k files, 100%)
         88  binary files (2.3 MB, ~0.2%)
        84 k libdir files (635 MB, ~5%)
       671 k noarch files (12 GB, 95%)

903 (sub) packages in 571 source packages could be directly switched to noarch (filtering out 32 bit packages): media:NoarchCandidates.txt. These are all x86_64 packages that do neither contain binary files (as known to rpm) nor files in (/usr)/lib64/.

Test Plan

  1. Create one noarch subpackage by adding BuildArch: noarch to the subpackage section
  2. Scratch build the package to see whether there are any problems with koji
  3. Build package for rawhide - check that it correctly shows up in the repository and is shown as noarch package in the metadata
  4. See if the package installs correctly via yum
  5. Check if updating from a arch dependent previous version to the new noarch package works

User Experience

  • Slightly improved mirrors due to less transfer size
  • Only packages containing binaries will be arch dependent

Dependencies

Contingency Plan

  • Move target to Fedora 12
  • As soon as the technical problems have been fixed moving more sub packages to noarch can be a continuing process.

Documentation

  • Incomplete Draft

What's this all about?

With version 4.6.0 RPM supports subpackages being noarch by just adding "BuildArch: noarch" to their subpackage section in the spec file.

The noarch subpackages build on the different arches are going to be compared by koji with rpmdiff ignoring time stamp, size and md5 sum of files. If any other differences are found the build will be rejected. Even with those automatic checks in place it is the responsibility of the packager to make sure that the package is really arch independent - as for regular noarch packages, too.

Candidates for being switched to noarch

To get a list with good candidates all x86_64 packages that contain no binaries/libs (as known to rpm) and no files in /lib64 or /usr/lib64 were selected as a starting point. There are a few false positives - mostly development packages that put files in different locations. To further refine the selection and get an idea what can go wrong rpmdiff was run against the i386 sister packages - both with the relaxed and the -t setting. This showed a small number of false positives and several packages with inconsistent owner settings on some files (TODO: find out reason). The sub packages are marked by one surrounding '*' if they fail the rpmdiff check as used by koji and by two if they also fails the more strict rpmdiff -t check. It is assumed that packages without '*' can be directly switched to noarch (assuming they don't do weird stuff on other arches). One '*' will require a more detailed look but should be OK in most cases and two '*'s is either a false positive of a packaging bug (file ownage problem)

What about other packages?

A lot of other packages could also make use of this feature. When considering to split up your package please avoid too complicated spec files. We still have to develop packaging strategies to be applied throughout the distribution and it doesn't look like this is going to happen in the F11 time frame.

What can you do as a packager?

There are still fixes for koji that must hit the Fedora build system first. So noarch subpackages DO NOT WORK within Fedora yet. We hope that this can be solved soon.

If you are interested you can already play with noarch subpackages by building with mock and comparing the results on different arches with rpmdiff -t (Files differing in S and 5 are ok). There is going to be little time between support in koji and the feature freeze. So being prepared for this short time slot is a good thing.

What if you don't want to change your packages?

That's perfectly fine. There is no plan to force packager to use noarch subpackages. I hope we can develop a more detailed plan on how to make use of this feature in future Fedora releases. You might be interested in taking part in this discussion.

What does that mean for the Packaging Policy?

The packaging policy will require a few additions. See /PolicyChanges. Any comment and help is welcome.

Release Notes

Not applicable as visibility for the users is low and developers need to know before the release.

Comments and Discussion