Changes/MemoryConstraintsMacros

Memory Constraints macros for RPM

Summary

Introduce macros, similar to openSUSE's memory-constraints), for optionally limiting build parallelism for build-time memory-bound packages

Owner

• Name: Michel Alexandre Salim
• Email: michel AT michel-slm DOT name

Current status

• Targeted release: Fedora Linux 35
• Last updated: 2021-07-15
• FESCo issue: #2642
• Tracker bug: #1982748
• Release notes tracker: <will not be assigned by the Wrangler - no external impact>

Detailed Description

Some source packages have a memory usage per build thread higher than the RAM:CPU ratio available in some of our builders. Further, this ratio can be different for different build server on different architectures.

At the moment, such packages (ceph, chromium, mcrouter) have to implement their own logic for determining the safe amount of parallelism, and redefine _smp_build_ncpus.

When this proposal is implemented, they can instead declaratively specify the amount of RAM needed per build thread, e.g.

 %limit_build -m 8192

for declaring a build thread should be allocated 8GB of RAM.

Since Koji supports setting default values for macros, there will be a macro for the default memory limit (name TBD) that, if set, will be used to cap _smp_build_ncpus unless overridden by %limit_build -m. 0 I'm proposing to tentatively call the macro package build-constraints-rpm-macros to allow the possibility of adding macros for related needs e.g. build timeouts to the same package.

Benefit to Fedora

This change simplifies maintaining specs for software that are memory-bounded rather than CPU-bounded on our build servers

It could potentially improve build reliability for these packages, by reducing the number of jobs failing because of OOM errors, and reduce the need for package maintainers to debug these failures.

By keeping the user-facing API aligned with what openSUSE is using, we open up the possibility to collaborate with them and with the upstream RPM project to get such macros upstreamed into RPM itself (see previous attempt). **note** that is not in scope for this Change.

Scope

• Proposal owners:
  • Introduce new macros
  • Update known packages to use the new macros, replacing their custom _smp_build_ncpus overrides

• Other developers:
  • The proposal owners might not catch all references of such logic. Individual package maintainers can try refactoring their packages using these new macros

• Release engineering: #10188

No mass rebuild needed. Affected packages should be rebuilt using the new macro

• Policies and guidelines: Packaging guideline can be updated to recommend using these macros for build-time memory-bound packages

• Trademark approval: N/A (not needed for this Change)

• Alignment with Objectives: N/A

Upgrade/compatibility impact

No impact, affects package building only. Also, the use of the new macros are optional.

How To Test

1. Install build-constraints-rpm-macros 2. Modify spec to set %limit_build -n AMOUNT_IN_MB in %build 3. Rebuild in koji and make sure it passes on all supported architectures

Dependencies

This can optionally be added as dependencies of redhat-rpm-config and epel-rpm-macros, depending on how many packages need this

Contingency Plan

• Contingency mechanism: (What to do? Who will do it?)

Revert changed packages to their previous way of capping the number of build jobs

• Contingency deadline: beta
• Blocks release? No

Documentation

Previous discussion: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/GWSXTGOCMNXHPGMTW32LI5EKPGHDKCFU/#GWSXTGOCMNXHPGMTW32LI5EKPGHDKCFU

openSUSE implementation: https://build.opensuse.org/package/show/openSUSE:Factory/memory-constraints