< Changes
m (fix openSUSE link to point to the main implementation) |
(going live) |
||
| Line 13: | Line 13: | ||
== Current status == | == Current status == | ||
[[Category: | [[Category:ChangeReadyForWrangler]] | ||
<!-- When your change proposal page is completed and ready for review and announcement --> | <!-- When your change proposal page is completed and ready for review and announcement --> | ||
<!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler --> | <!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler --> | ||
| Line 70: | Line 70: | ||
** The proposal owners might not catch all references of such logic. Individual package maintainers can try refactoring their packages using these new macros | ** The proposal owners might not catch all references of such logic. Individual package maintainers can try refactoring their packages using these new macros | ||
* Release engineering: [https://pagure.io/releng/ | * Release engineering: [https://pagure.io/releng/issue/10188 #10188] <!-- REQUIRED FOR SYSTEM WIDE CHANGES --> | ||
<!-- Does this feature require coordination with release engineering (e.g. changes to installer image generation or update package delivery)? Is a mass rebuild required? include a link to the releng issue. | <!-- Does this feature require coordination with release engineering (e.g. changes to installer image generation or update package delivery)? Is a mass rebuild required? include a link to the releng issue. | ||
The issue is required to be filed prior to feature submission, to ensure that someone is on board to do any process development work and testing and that all changes make it into the pipeline; a bullet point in a change is not sufficient communication --> | The issue is required to be filed prior to feature submission, to ensure that someone is on board to do any process development work and testing and that all changes make it into the pipeline; a bullet point in a change is not sufficient communication --> | ||
Revision as of 19:55, 29 June 2021
Memory Constraints macros for RPM
Summary
Introduce macros, similar to openSUSE's memory-constraints), for optionally limiting build parallelism for build-time memory-bound packages
Owner
- Name: Michel Alexandre Salim
- Email: michel AT michel-slm DOT name
Current status
- Targeted release: Fedora Linux 35
- Last updated: 2021-06-29
- FESCo issue: <will be assigned by the Wrangler>
- Tracker bug: <will be assigned by the Wrangler>
- Release notes tracker: <will be assigned by the Wrangler>
Detailed Description
Some source packages have a memory usage per build thread higher than the RAM:CPU ratio available in some of our builders. Further, this ratio can be different for different build server on different architectures.
At the moment, such packages (ceph, chromium, mcrouter) have to implement their own logic for determining the safe amount of parallelism, and redefine _smp_build_ncpus.
When this proposal is implemented, they can instead declaratively specify the amount of RAM needed per build thread, e.g.
%limit_build -m 8192
for declaring a build thread should be allocated 8GB of RAM.
Since Koji supports setting default values for macros, there will be a macro for the default memory limit (name TBD) that, if set, will be used to cap _smp_build_ncpus unless overridden by %limit_build -m.
0
I'm proposing to tentatively call the macro package build-constraints-rpm-macros to allow the possibility of adding macros for related needs e.g. build timeouts to the same package.
Feedback
Benefit to Fedora
This change simplifies maintaining specs for software that are memory-bounded rather than CPU-bounded on our build servers
It could potentially improve build reliability for these packages, by reducing the number of jobs failing because of OOM errors, and reduce the need for package maintainers to debug these failures.
By keeping the user-facing API aligned with what openSUSE is using, we open up the possibility to collaborate with them and with the upstream RPM project to get such macros upstreamed into RPM itself (see previous attempt). **note** that is not in scope for this Change.
Scope
- Proposal owners:
- Introduce new macros
- Update known packages to use the new macros, replacing their custom
_smp_build_ncpusoverrides
- Other developers:
- The proposal owners might not catch all references of such logic. Individual package maintainers can try refactoring their packages using these new macros
- Release engineering: #10188
No mass rebuild needed. Affected packages should be rebuilt using the new macro
- Policies and guidelines: N/A (not needed for this Change)
Packaging guideline can be updated to recommend using these macros for build-time memory-bound packages
- Trademark approval: N/A (not needed for this Change)
- Alignment with Objectives: N/A
Upgrade/compatibility impact
No impact, affects package building only. Also, the use of the new macros are optional.
How To Test
1. Install build-constraints-rpm-macros
2. Modify spec to set %limit_build -n AMOUNT_IN_MB in %build
3. Rebuild in koji and make sure it passes on all supported architectures
User Experience
Dependencies
This can optionally be added as dependencies of redhat-rpm-config and epel-rpm-macros, depending on how many packages need this
Contingency Plan
- Contingency mechanism: (What to do? Who will do it?)
Revert changed packages to their previous way of capping the number of build jobs
- Contingency deadline: beta
- Blocks release? No
Documentation
Previous discussion: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/GWSXTGOCMNXHPGMTW32LI5EKPGHDKCFU/#GWSXTGOCMNXHPGMTW32LI5EKPGHDKCFU
openSUSE implementation: https://build.opensuse.org/package/show/openSUSE:Factory/memory-constraints