From Fedora Project Wiki
No edit summary
 
(43 intermediate revisions by 3 users not shown)
Line 1: Line 1:
= Changes/CompilerPolicy Change <!-- The name of your change proposal --> =
= CompilerPolicy Change =


== Summary ==
== Summary ==
Fedora has historically forced packages to build with GCC unless the upstream project for the package only supported Clang/LLVM.  This change proposal replaces that policy with one where compiler selection for Fedora follows the package's upstream preferences.
Fedora has historically forced packages to build with GCC unless the upstream project for the package only supported Clang/LLVM.  This change proposal replaces that policy with one where, given a good technical reason, a packager may:
* Choose to build with their package with clang even if the upstream project supports gcc.
* Choose to build with gcc even if upstream does not support it.
 
The goal is to give the packager the ability to use their own technical judgement to choose the best compiler.
 
Note this change is only for compiler selection.  It does not change existing policies WRT runtime library selection, linker selection, debuggers, etc.


== Owner ==
== Owner ==
* Name: Jeff Law
* Name: Jeff Law
* Email: law@redhat.com
* Name: Tom Stellard
* Email: tstellar@redhat.com


== Current status ==
== Current status ==
[[Category:ChangePageIncomplete]]
[[Category:ChangeAcceptedF35]]
<!-- When your change proposal page is completed and ready for review and announcement -->
<!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler -->
<!-- The Wrangler announces the Change to the devel-announce list and changes the category to Category:ChangeAnnounced (no action required) -->
<!-- After review, the Wrangler will move your page to Category:ChangeReadyForFesco... if it still needs more work it will move back to Category:ChangePageIncomplete-->


[[Category:SystemWideChange]]
[[Category:SystemWideChange]]


* Targeted release: Fedora 33
* Targeted release: Fedora Linux 35
* Last updated: {{REVISIONYEAR}}-{{REVISIONMONTH}}-{{REVISIONDAY2}}  
* Last updated: {{REVISIONYEAR}}-{{REVISIONMONTH}}-{{REVISIONDAY2}}  
<!-- After the change proposal is accepted by FESCo, tracking bug is created in Bugzilla and linked to this page  
<!-- After the change proposal is accepted by FESCo, tracking bug is created in Bugzilla and linked to this page  
Line 26: Line 29:
CLOSED as NEXTRELEASE -> change is completed and verified and will be delivered in next release under development
CLOSED as NEXTRELEASE -> change is completed and verified and will be delivered in next release under development
-->
-->
* FESCo issue: <will be assigned by the Wrangler>
* FESCo issue: [https://pagure.io/fesco/issue/2603 #2603]
* Tracker bug: <will be assigned by the Wrangler>
** Previously: [https://pagure.io/fesco/issue/2409 #2409]
* Release notes tracker: <will be assigned by the Wrangler>
* Tracker bug: [https://bugzilla.redhat.com/show_bug.cgi?id=1960295 #1960295]
* Release notes tracker: <will NOT be assigned by the Wrangler: not externally facing>


== Detailed Description ==
== Detailed Description ==


The main goal here is to make selection of the compiler to build a package flow from upstream in an effort to preserve our development resourcesIn cases where there is no upstream the Fedora package maintainer should be allowed to make the compiler choice for the package. For packages where upstream does not have a strong compiler preference, we should (for now) stick with the status quo to avoid unnecessary churn.
The main goal here is to give a packager some freedom to select the best compiler for their package.  This policy change will allow packagers to choose to use a non-default compiler for their package, if they have a valid technical reason to do soThe default compiler for Fedora is gcc or when upstream doesn't support gcc, then the default compiler is clang. Examples of valid technical reasons to not use the default compiler, include but are not limited to:
 
* The default compiler cannot build a package correctly.
* The packager needs to disable a compiler feature (e.g. LTO) in order for the default compiler to correctly compile their package.
* The default compiler takes significantly longer to build a package.
* The default compiler is missing a feature that would benefit the package.
 
There will also be no requirement that the compiler choice is consistent across all architectures, this will be up to the packager to decide.


== Benefit to Fedora ==
If this policy is implemented, it may be beneficial for packagers to use conditional macros to switch between different compilers.  This could be useful when trying to make a comparison between the two compilers or to make it easy to change back to the default compiler.  Therefore, in addition to updating the compiler policy, we are also proposing to add the following macros to redhat-rpm-config to help facilitate easily switching between different compilers:
The new macros are:


This change allows packages to be built with whatever compiler the upstream project recommends/supports (so long as that compiler is in Fedora). Thus, Fedora package owners no longer need to spend time making a package work with GCC if the upstream project is using Clang/LLVM.
* %cc  - equivalent to %__cc (C Compiler)
* %cxx - equivalent to %__cxx (C++ Compiler)
* %cpp  - equivalent to %_cpp (C preprocessor)


An obvious example is Firefox.  Upstream, the Firefox project builds primarily with Clang/LLVM.  Yet we force the Fedora package owner to find and fix issues building with GCC then either carry those custom fixes forward in Fedora or negotiate with upstream to get those changes upstreamed.  While this process can be helpful in finding non-portable code, this is ultimately a poor use of the packager's time.
The policy update will also recommend that packagers use standardized macro names when using conditional options to control the compiler choice:


Additionally Fedora loses the benefit of the testing provided by other distributions where Firefox is compiled in the same way as the upstream project -- when issues arise the Fedora team must consider the possibility that the problem is due to using GCC instead of Clang/LLVM or the patches to make that possible.  Again, this is a poor use of Fedora developer's time.
%bcond_with toolchain_clang
%bcond_with toolchain_gcc


In the immediate term this change in policy only affects a few packages (Firefox, Chromium and perhaps a few others).  The benefit will likely expand over time.
[https://pagure.io/packaging-committee/pull-request/1066 Pull request] with proposed policy changes.


== Scope ==
Note this change is only for compiler selection.  It does not change existing policies WRT runtime library selection, linker selection, debuggers, etc.
* Proposal owners:
Update the Fedora Packaging Guidelines to reflect the policy change.


It is worth noting that Clang/LLVM's implementation of -fstack-clash-protection, which is one of Fedora's default compiler flags, is under development and does not yet have an implementation for AArch64.


* Other developers:
== Feedback ==
Developers working with packages where upstream builds with Clang/LLVM, but Fedora policy has forced the package to build with GCC should update their packages to build with Clang/LLVM, dropping all Fedora specific changes that were necessary to make the package build with GCC.


Firefox and Chromium are known to be impacted, there may be other packages. While there is no specific timeframe where this would need to be accomplished as the existing packages are already building with GCC, getting the conversion done earlier rather than later seems beneficial.  Failure to identify all the impacted packages is not a catastrophic failure, we just fail to reap the benefits in the immediate term.
<!-- Summarize the feedback from the community and address why you chose not to accept proposed alternatives. This section is optional for all change proposals but is strongly suggested. Incorporating feedback here as it is raised gives FESCo a clearer view of your proposal and leaves a good record for the future. If you get no feedback, that is useful to note in this section as well. For innovative or possibly controversial ideas, consider collecting feedback before you file the change proposal. -->


* Release engineering: [https://pagure.io/releng/issues #Releng issue number] (a check of an impact with Release Engineering is needed) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* The original proposal stated that Fedora packagers should follow upstream's compiler preferences, but based on feedback, the proposal was updated to allow a packager to choose the compiler they feel is best as long as there is a valid technical reason to do so.
I do not believe this change requires any coordination with release engineering.  No mass rebuild is required.


<!-- Does this feature require coordination with release engineering (e.g. changes to installer image generation or update package delivery)? Is a mass rebuild required?  include a link to the releng issue.
* Remove examples of specific packages that might use clangThere was some feedback that the justifications for firefox using clang were not valid, but this proposal is not about firefox or any other specific package, it's about changing the policy to
The issue is required to be filed prior to feature submission, to ensure that someone is on board to do any process development work and testing, and that all changes make it into the pipeline; a bullet point in a change is not sufficient communication -->
let packagers make their own technical decisions.  Having the examples seems to distract from the goal of the proposals, so they were removed.


* Policies and guidelines:
== Benefit to Fedora ==
Yes, the packaging guidelines certainly need to be updated for this feature.  That can happen as soon as the exact text is agreed upon.  Development work on packages such as Firefox or Chrome can happen as soon FESCO agrees to the change while the final packaging guideline text is hammered out.


This change allows packagers more freedom to use the compiler that produces the best version of their package, which helps to give our users a better overall experience.  Also, giving package maintainers more freedom to select the best compiler, let's them spend more time focusing on package improvements and less time on debugging compiler issues.


== Scope ==
* Proposal owners:
** Update the Fedora Packaging Guidelines to reflect the policy change and submit patch with new macros to redhat-rpm-config.
* Other developers:
** Developers that want to use a new compiler in accordance with the policy may update their packages if they want, but there is no required action for packagers with this change.
* Release engineering: [https://pagure.io/releng/issue/9503] (a check of an impact with Release Engineering is needed)
** I do not believe this change requires any coordination with release engineering.  No mass rebuild is required.
* Policies and guidelines:
** Yes, the packaging guidelines certainly need to be updated for this feature.  That can happen as soon as the exact text is agreed upon.
* Trademark approval: N/A (not needed for this Change)
* Trademark approval: N/A (not needed for this Change)


Line 72: Line 93:
For packages where the compiler should change, the package owner will need to update the spec file and build the package with the new compiler.  Once done, the package's testsuite should be run (if it's not part of the standard build process).
For packages where the compiler should change, the package owner will need to update the spec file and build the package with the new compiler.  Once done, the package's testsuite should be run (if it's not part of the standard build process).


In general, I would think the standard Fedora QE work should be sufficient here, perhaps with a bit of additional attention to the affected packages.  The graphical nature of some of the affected packages like Firefox and Chrome will make testing difficult on some of Fedora's architectures.
In general, I would think the standard Fedora QE work should be sufficient here, perhaps with a bit of additional attention to the affected packages.  The graphical nature of some of the potentially affected packages like Firefox and Chrome will make testing difficult on some of Fedora's architectures.


== User Experience ==
== User Experience ==
Line 78: Line 99:


== Dependencies ==
== Dependencies ==
One the policy change is made a set of packages will need to be updated.  Firefox and Chrome are known to be affected.  There may be others.  It seems like the Fedora package owners are in the best position to know if their package is affected by the policy change.
There are no dependencies, once the policy change is made, if packagers choose to update their packages, they can do it at any time.
 
It is useful to remember that the packages as they exist today still work.  Therefore if a package which should change is not identified now, it will continue to work.  Similarly if the package owner does not have the time to implement the change right now, the existing package will continue to work.


== Contingency Plan ==
== Contingency Plan ==
The backup plan is trivial.  We can keep the current policy in place.
The backup plan is trivial.  We can keep the current policy in place.
If the policy change is approved, but a particular package has not switched to the upstream preferred compiler, the package can continue to build with GCC until the package maintainer has the time to make the change for their particular package.
Failure to convert any particular package should not create any downstream or distribution wide delays.


* Contingency mechanism:  
* Contingency mechanism:  
It seems like we could institute the policy change anytime we choose.  But it also seems like once the policy change is in place, packages that are going to convert should do so before beta freeze.
** It seems like we could institute the policy change anytime we choose.  But it also seems like once the policy change is in place, packages that are going to convert should do so before beta freeze.
 
* Contingency deadline: Fedora can ship with this feature in an incomplete state.
* Contingency deadline: Fedora can ship with this feature in an incomplete state.
* Blocks release? No
* Blocks release? No
Line 113: Line 127:


The Red Hat tools team believes that LLVM/Clang and GCC should be considered equals from
The Red Hat tools team believes that LLVM/Clang and GCC should be considered equals from
a Fedora policy standpoint.  Selection of one toolchain over the should should be
a Fedora policy standpoint.  Selection of one toolchain over the other should be
driven by the upstream project's preferences not by Fedora specific policy.
driven by the packager's best technical judgement not by Fedora specific policy. The only policy restriction should be that the compiler must exist in Fedora.
 
What that means in practice is that if the project upstream prefers Clang/LLVM,
then Fedora should in turn be using Clang/LLVM to build those packages.  As a
concrete example, let's consider Chromium.
 
Chromium upstream has been building with Clang/LLVM for several years.  Yet policy
has forced Fedora package owners to shoulder a significant burden to make it
build with GCC.  Furthermore, Fedora does not get as much benefit at it could by
forcing Chromium to be built with GCC since most other instances are built with
Clang/LLVM.
 
By changing policy Fedora package maintainers no longer have to waste time trying
to make Chromium build/work with GCC and Fedora gains additional "many eyes" and
"many users" benefits by relying on the same tools to build Chrome as the
upstream developers and other distributions.
 
Additionally, if an upstream project currently uses GCC, but switches to
Clang/LLVM (or vice-versa), then the package in Fedora should switch in a similar
manner.  The only policy restriction should be that the compiler must exist in Fedora.
 
In some ways this means there is no "default" compiler for Fedora.  The default
is whatever the upstream project supports/recommends.  However, there are
probably many packages with upstreams that are ambivalent about their compiler
choice.  For those packages I would recommend we keep the status quo at the
current time.  For a package with a dead upstream, the Fedora packager should be
able to select the compiler they want to use for the package.


== Release Notes ==
== Release Notes ==
This change should not have any end user impacts nor does it strictly require a release note.  However, a short release note could be written if FESCO or the development community thinks it would be useful.
This change should not have any end user impacts nor does it strictly require a release note.  However, a short release note could be written if FESCO or the development community thinks it would be useful.

Latest revision as of 14:49, 13 May 2021

CompilerPolicy Change

Summary

Fedora has historically forced packages to build with GCC unless the upstream project for the package only supported Clang/LLVM. This change proposal replaces that policy with one where, given a good technical reason, a packager may:

  • Choose to build with their package with clang even if the upstream project supports gcc.
  • Choose to build with gcc even if upstream does not support it.

The goal is to give the packager the ability to use their own technical judgement to choose the best compiler.

Note this change is only for compiler selection. It does not change existing policies WRT runtime library selection, linker selection, debuggers, etc.

Owner

  • Name: Jeff Law
  • Name: Tom Stellard
  • Email: tstellar@redhat.com

Current status

  • Targeted release: Fedora Linux 35
  • Last updated: 2021-05-13
  • FESCo issue: #2603
  • Tracker bug: #1960295
  • Release notes tracker: <will NOT be assigned by the Wrangler: not externally facing>

Detailed Description

The main goal here is to give a packager some freedom to select the best compiler for their package. This policy change will allow packagers to choose to use a non-default compiler for their package, if they have a valid technical reason to do so. The default compiler for Fedora is gcc or when upstream doesn't support gcc, then the default compiler is clang. Examples of valid technical reasons to not use the default compiler, include but are not limited to:

  • The default compiler cannot build a package correctly.
  • The packager needs to disable a compiler feature (e.g. LTO) in order for the default compiler to correctly compile their package.
  • The default compiler takes significantly longer to build a package.
  • The default compiler is missing a feature that would benefit the package.

There will also be no requirement that the compiler choice is consistent across all architectures, this will be up to the packager to decide.

If this policy is implemented, it may be beneficial for packagers to use conditional macros to switch between different compilers. This could be useful when trying to make a comparison between the two compilers or to make it easy to change back to the default compiler. Therefore, in addition to updating the compiler policy, we are also proposing to add the following macros to redhat-rpm-config to help facilitate easily switching between different compilers: The new macros are:

  • %cc - equivalent to %__cc (C Compiler)
  • %cxx - equivalent to %__cxx (C++ Compiler)
  • %cpp - equivalent to %_cpp (C preprocessor)

The policy update will also recommend that packagers use standardized macro names when using conditional options to control the compiler choice:

%bcond_with toolchain_clang %bcond_with toolchain_gcc

Pull request with proposed policy changes.

Note this change is only for compiler selection. It does not change existing policies WRT runtime library selection, linker selection, debuggers, etc.

It is worth noting that Clang/LLVM's implementation of -fstack-clash-protection, which is one of Fedora's default compiler flags, is under development and does not yet have an implementation for AArch64.

Feedback

  • The original proposal stated that Fedora packagers should follow upstream's compiler preferences, but based on feedback, the proposal was updated to allow a packager to choose the compiler they feel is best as long as there is a valid technical reason to do so.
  • Remove examples of specific packages that might use clang. There was some feedback that the justifications for firefox using clang were not valid, but this proposal is not about firefox or any other specific package, it's about changing the policy to

let packagers make their own technical decisions. Having the examples seems to distract from the goal of the proposals, so they were removed.

Benefit to Fedora

This change allows packagers more freedom to use the compiler that produces the best version of their package, which helps to give our users a better overall experience. Also, giving package maintainers more freedom to select the best compiler, let's them spend more time focusing on package improvements and less time on debugging compiler issues.

Scope

  • Proposal owners:
    • Update the Fedora Packaging Guidelines to reflect the policy change and submit patch with new macros to redhat-rpm-config.
  • Other developers:
    • Developers that want to use a new compiler in accordance with the policy may update their packages if they want, but there is no required action for packagers with this change.
  • Release engineering: [1] (a check of an impact with Release Engineering is needed)
    • I do not believe this change requires any coordination with release engineering. No mass rebuild is required.
  • Policies and guidelines:
    • Yes, the packaging guidelines certainly need to be updated for this feature. That can happen as soon as the exact text is agreed upon.
  • Trademark approval: N/A (not needed for this Change)

Upgrade/compatibility impact

This should not require any configuration changes or data migration, nor should it change existing functionality.

How To Test

For packages where the compiler should change, the package owner will need to update the spec file and build the package with the new compiler. Once done, the package's testsuite should be run (if it's not part of the standard build process).

In general, I would think the standard Fedora QE work should be sufficient here, perhaps with a bit of additional attention to the affected packages. The graphical nature of some of the potentially affected packages like Firefox and Chrome will make testing difficult on some of Fedora's architectures.

User Experience

Users should not notice any change.

Dependencies

There are no dependencies, once the policy change is made, if packagers choose to update their packages, they can do it at any time.

Contingency Plan

The backup plan is trivial. We can keep the current policy in place.

  • Contingency mechanism:
    • It seems like we could institute the policy change anytime we choose. But it also seems like once the policy change is in place, packages that are going to convert should do so before beta freeze.
  • Contingency deadline: Fedora can ship with this feature in an incomplete state.
  • Blocks release? No
  • Blocks product? N/A

Documentation

Several years ago Red Hat's tools team championed for Fedora policy to strongly discourage the use of LLVM/Clang for package building. Exceptions were made for packages that could only be built with Clang/LLVM:

https://docs.fedoraproject.org/en-US/packaging-guidelines/#compiler


At that point in history Red Hat had no Clang/LLVM engineers or expertise. In fact, the LLVM packages were actually maintained by an engineer on the desktop team (they had a hard requirement for llvm-pipe, so they got to own the Clang/LLVM bits). The policy essentially was a risk management strategy for Fedora.

Times have changed and as a result we should revisit that Fedora policy.

The Red Hat tools team believes that LLVM/Clang and GCC should be considered equals from a Fedora policy standpoint. Selection of one toolchain over the other should be driven by the packager's best technical judgement not by Fedora specific policy. The only policy restriction should be that the compiler must exist in Fedora.

Release Notes

This change should not have any end user impacts nor does it strictly require a release note. However, a short release note could be written if FESCO or the development community thinks it would be useful.