From Fedora Project Wiki

No edit summary
No edit summary
 
(40 intermediate revisions by 3 users not shown)
Line 1: Line 1:
{{old}}
{{Admon/note | Accepted proposal |
This proposal has been accepted and is now part of Packaging Guidelines, section [https://docs.fedoraproject.org/en-US/packaging-guidelines/SourceURL/#_using_forges_hosted_revision_control Referencing Source]}}
Projects published on a “forge” can be packaged using the '''forgemeta''' macro.
Projects published on a “forge” can be packaged using the '''forgemeta''' macro.


{{Admon/note | What is a “forge” for the forgemeta macro? |
This proposal is a requisite for the [[More_Go_packaging|More Go packaging draft]].
Any software publishing website, permitting the download of source code archives on normalized URLs, that can be deduced from a project root URL on the website plus ''version''/''commit''/''tag''/''scm''/''extension'' values, is a “forge”.
 
{{Admon/note | What is a “forge”? |
Any software publishing website, permitting the download of source code archives on normalized URLs, that can be deduced from ''version'', ''commit'', ''tag'', ''scm'', ''extension'' values and a project root URL on the website, is a “forge” that can be supported by ''forgemeta''.
}}
}}


== Links : ==
== Links ==
* [https://docs.fedoraproject.org/en-US/packaging-guidelines/SourceURL/#_using_forges_hosted_revision_control Accepted guidelines]
* [https://pagure.io/packaging-committee/issue/719 FPC ticket]
* [https://pagure.io/packaging-committee/issue/719 FPC ticket]
* [https://bugzilla.redhat.com/show_bug.cgi?id=1523779 fedora-rpm-macros RFE with the macro file]
* [https://bugzilla.redhat.com/show_bug.cgi?id=1523779 redhat-rpm-macros RFE] with the macro file → '''DONE'''
* [https://bugzilla.redhat.com/show_bug.cgi?id=1524192 fedora-release RFE] to make dist munging safe
* [https://lists.fedoraproject.org/archives/list/packaging@lists.fedoraproject.org/thread/5K4CED5IBVTS567QJEYGUWLPBFYONDLN/ packaging@lists.fedoraproject.org discussion]
* [https://lists.fedoraproject.org/archives/list/packaging@lists.fedoraproject.org/thread/5K4CED5IBVTS567QJEYGUWLPBFYONDLN/ packaging@lists.fedoraproject.org discussion]
* [https://pagure.io/pagure/issue/861 pagure.io RFE to make pagure.io compatible with forgemeta]
* [https://pagure.io/pagure/issue/861 pagure.io RFEto make pagure.io compatible with forgemeta
* [https://gitlab.com/gitlab-org/gitlab-ce/issues/38830 GitLab CE RFE to remove GitLab quirks]
* [https://gitlab.com/gitlab-org/gitlab-ce/issues/38830 GitLab CE RFE] to remove GitLab quirks
* [https://github.com/rpm-software-management/rpm/issues/368 rpm RFE to get rid of the remaining %setup special cases]
* [[More_Go_packaging|More Go packaging draft]] that depends on this proposal.
 
==Benefits and limitations==
* Benefits:
# Fedora packagers no longer need to know about the URL structure of well-known forges.
# Spec files are simpler, less error-prone, easier to maintain and audit.
# Forge URLs can be defined and fixed in a single place, without waiting for guidelines to percolate.
# The macros are mostly written in Lua, making them more verbose but easier to adjust and extend.
# The macros are written in Lua and can perform error handling.
# It is very easy to switch from commit to tag to version (or any combination of those).
# ''spectool'' just works©.
# scm snapshot date is exact and does not rely on a variable which may or may not have been updated.
# Almost all the computations are done in optional rpm variables that can be ignored by the packager if he does not need or does not like the result.
# Fedora forge know-how can be capitalized over time.
# The macro normalizes common spec constructs such as ''archivename''.
 
* Limitations:
# The macro is not intended to be used in spec files that package multiple forge archives (it could probably be extended, is the added flexibility worth the complexity?).
# The macro needs to be updated when a forge changes its structure (but better changing one place than lots of spec files).
# New forges need to be added to the macro before it can be used with them.
# ''dist'' munging can not be easily reverted once ''forgemeta'' has been called. The solution would be to change fedora-release to define  ''dist'' as ''%{?distprefix}.fcxx''
# ''dist'' commit date computation relies on source files with the correct modification time (incorrect source file modification time → incorrect ''dist'' date).
# The macro highlights some historical rpm design mistakes: bad separation of upstream and package metadata, special magic variables. This is why ''version'' declaration occurs in an unusual (for rpm) order and requires a specific syntax.


== Usage ==
== Usage ==
Line 20: Line 50:
* '''commit:''' the commit hash to package, if any
* '''commit:''' the commit hash to package, if any
*  '''tag:''' the project tag to package, if any
*  '''tag:''' the project tag to package, if any
* '''scm''': the source code management engine used by the software hosting service, if packaging a tag or a commit, and if the service allows different engines.
B. The packager calls the '''%forgemeta''' macro to compute rpm-oriented metadata:
* '''forgesource:''' the corresponding source file URL, that can be used as ''Source:''
* '''shortcommit:''' a commit hash reduction
* '''setupargs:'''  the arguments that ''%setup'' will need
* '''archivename:''' the corresponding archive name (often used by ''setupargs'')
* '''archiveext:''' the corresponding archive extension
* '''archiveurl:'''  the archive URL, without renaming
* '''dist:''' auto-adjusted when packaging a tag or commit
C. (Optionnal) The packager calls the '''%forgemetacheck''' macro to display the values of computed variables
D. The packager uses the computed metadata as needed.
== Packaging examples ==
=== Packaging a release (except for GitLab) ===
%global forgeurl    https://github.com/foo/bar
Version:            8.18.1
 
%forgemeta
Name:    foobar
Release: 1%{?dist}
URL:    %{forgeurl}
Source: %{forgesource}
%prep
%setup  %{?setupargs}
=== Packaging a release (GitLab) ===
{{Admon/note | GitLab is inconvenient |
* Any download from GitLab requires knowledge of the corresponding ''commit''.
* Release download depends on a Git tag, which is free-form. If your release tag value does not match ''Version'' (for example, if it is ''v%{version}''), you also need to declare ''tag'' before calling '''forgemeta'''.
}}


%global forgeurl    https://gitlab.example.com/foo/bar
B. The packager calls '''%forgemeta'''This macro will attempt to compute and set the following variables if they are not already set by the packager:
%global commit      2a810629566a2d0f0d4107df244e8828b9f7bd5c
Version:            8.18.1
 
%forgemeta
Name:    foobar
Release: 1%{?dist}
URL:    %{forgeurl}
  Source: %{forgesource}
%prep
%setup  %{?setupargs}


=== Packaging a specific commit ===
* '''forgesource''': an URL that can be used as ''SourceX:'' value
%global forgeurl    https://code.googlesource.com/foobar
* '''forgesetupargs''': the correct arguments to pass to ''%setup'' for this source; used by ''%forgesetup'' and ''%forgeautosetup''
%global commit      2a810629566a2d0f0d4107df244e8828b9f7bd5c
* '''archivename''': the source archive filename, without extentions
* '''archiveext''': the source archive filename extensions, without leading dot
%forgemeta
* '''archiveurl''': the url that can be used to download the source archive, without renaming
* '''scm''': the scm type, when packaging code snapshots (commits or tags)
Name:    foobar
Version: 0
Release: 0.1%{?dist}
URL:   %{forgeurl}
Source: %{forgesource}
%prep
%setup  %{?setupargs}


=== Packaging a release accessed through a tag===
If the macro is unable to parse the ''%{forgeurl}'' value the packager should set at least ''%{archivename}'' and ''%{archiveurl}'' before calling it.
%global forgeurl    https://github.com/foo/bar
Version:            0.10.0
%global tag        %{version}
 
%forgemeta
Name:    foobar
Release: 1%{?dist}
URL:    %{forgeurl}
Source: %{forgesource}
%prep
%setup  %{?setupargs}


=== Packaging a pre-release commit ===
The macro also accepts the following optional parameters:
{{Admon/note | On Version declaration order |
*  '''-u''' ''<url>'':  ignore ''%{forgeurl}'' even if it exists and use <url> instead; note that the macro will still end up setting <url> as ''%{forgeurl}'' if it manages to parse it
It is generally safe to declare ''Version'' before calling ''%forgemeta'' even for pre or post-releases. '''Except for GitLab'''. Due to GitLab's use of ''commit'' for release downloads, you MUST declare ''Version'' after calling ''%forgemeta'' when the packaged ''commit'' does not match the release commit.
* '''-s''':  silently ignore problems in ''%{forgeurl}'', use it if it can be parsed, ignore it otherwise
}}
* '''-p ''': restore problem handling, override ''-s''
  %global forgeurl    https://gitlab.example.com/foo/bar/
* '''-v''' : be verbose and print every variable the macro sets
  %global commit      51637bc0960002b811e1c0c7be8671cf9a1cc5be
* '''-i'''Print some info about the state of variables the macro may use or set at the end of the processing
    
    
%forgemeta
C. The packager uses the computed metadata as needed.
Name:    foobar
Version: 3.2.6
Release: 0.1%{?dist}
URL:    %{forgeurl}
Source: %{forgesource}
%prep
%setup %{?setupargs}
 
Post-release packaging is similar. See also [[Packaging:Versioning#Release_and_post-release_versions]].
 
== %forgemetacheck output examples ==
 
{{Admon/note | On variable resolution |
'''forgemetacheck''' resolves the values of rpm variables at one point of the spec file. Since most of those variables are themselves composed of other variables, it is possible for those values to change if one of the variables used in the composition is overridden by the packager after the '''forgemetacheck''' call.
}}


'''forgemetacheck''' emits rpm warnings. It should be safe to call from any point of the spec file.
== ''%{dist}'' modification for code snapshots ==


Forge-specific packaging variables
Most of the macro behavior is optional and safe. You can override the computed rpm variables values before or after the ''%forgemeta'' call, or ignore those variables altogether and don't use them in the rest of the spec file. There is one exception:
  forgeurl:        “https://github.com/satori/uuid”
  forgesource:    “https://github.com/satori/uuid/archive/v1.0.0.tar.gz#/go-uuid-1.0.0.tar.gz”
  shortcommit:    “”
Generic variables
  setupargs:      “-n go-uuid-1.0.0”
  archivename:    “go-uuid-1.0.0”
  archiveext:      “tar.gz”
  archiveurl:      “https://github.com/satori/uuid/archive/v1.0.0.tar.gz”
  scm:            “”
  tag:            “”
  commit:          “”
  dist:            “.fc28” (snapshot date is computed once the corresponding source file is available)


Forge-specific packaging variables
The macro will change the value of ''%{dist}'' in a non-reversible way, when packaging a code snapshot (commit or tag). Redefine ''%{dist}'' manually after the ''%forgemeta'' call if you don't like the result. Alternatively, don't use ''%forgemeta''.
  forgeurl:        “https://github.com/apache/thrift”
  forgesource:    “https://github.com/apache/thrift/archive/0.10.0.tar.gz#/thrift-0.10.0.tar.gz”
  shortcommit:    “”
Generic variables
  setupargs:      “-n thrift-0.10.0”
  archivename:    “thrift-0.10.0”
  archiveext:      “tar.gz”
  archiveurl:      “https://github.com/apache/thrift/archive/0.10.0.tar.gz”
  scm:            “git”
  tag:            “0.10.0”
  commit:          “”
  dist:            “.fc28” (snapshot date is computed once the corresponding source file is available)


Forge-specific packaging variables
The root cause is that Fedora [https://bugzilla.redhat.com/show_bug.cgi?id=1524192 does not provide currently provide] a placeholder prefix macro inside ''%{dist}'' that could be adjusted without redefining ''%{dist}'' as a whole.
  forgeurl:        “https://code.googlesource.com/google-api-go-client”
  forgesource:    “https://code.googlesource.com/google-api-go-client/+archive/3a1d936b7575b82197a1fea0632218dd07b1e65c.tar.gz#/google-api-go-client-3a1d936b7575b82197a1fea0632218dd07b1e65c.tar.gz”
  shortcommit:    “”
Generic variables
  setupargs:      “-c”
  archivename:    “google-api-go-client-3a1d936b7575b82197a1fea0632218dd07b1e65c”
  archiveext:      “tar.gz”
  archiveurl:      “https://code.googlesource.com/google-api-go-client/+archive/3a1d936b7575b82197a1fea0632218dd07b1e65c.tar.gz”
  scm:            “git”
  tag:            “”
  commit:          “3a1d936b7575b82197a1fea0632218dd07b1e65c”
  dist:            “.git3a1d936.fc28” (snapshot date is computed once the corresponding source file is available)


==Extending the macro==
==Extending the macro==
Line 185: Line 87:
* copy the definition block closest to your needs after the other definition blocks. It should look like :
* copy the definition block closest to your needs after the other definition blocks. It should look like :
   if (forge == "'''vvv'''") then
   if (forge == "'''vvv'''") then
   …
   []
   end
   end
or
or
   if (string.match(forge, "'''vvv'''")) then
   if (string.match(forge, "'''vvv'''")) then
   …
   []
   end
   end
* change the '''"vvv"''' value
* change the '''"vvv"''' value
Line 198: Line 100:
     forgeurl = string.match(forgeurl, "'''www'''")
     forgeurl = string.match(forgeurl, "'''www'''")
* adapt the initial error message
* adapt the initial error message
      error("'''xxx''' URLs must match '''yyy'''!")
      rpm.expand("%{error:"'''xxx''' URLs must match '''yyy'''!\\n}")
* set the hosting service defaults
* set the hosting service defaults
     if (archiveext == "") then
     safeset("archiveext",    "'''tar.bz2'''")
      rpm.define("archiveext '''tar.xz'''")
     safeset("forgesetupargs", "'''-n %%{archivename}'''")
     end
    if (setupargs == "") then
      rpm.define("setupargs '''-n %{archivename}'''")
    end
     if (commit ~= "") or (tag ~= "") then
     if (commit ~= "") or (tag ~= "") then
       rpm.define("scm '''git'''")
       safeset("scm", "'''git'''")
     end
     end
* scrap the variables you need from '''forgeurl''' using:
* scrap the variables you need from '''forgeurl''' using:
Line 213: Line 111:
* use your variables to define '''archivename''' and '''archiveurl''' in the following if ''tag''/''commit''/''version'' block
* use your variables to define '''archivename''' and '''archiveurl''' in the following if ''tag''/''commit''/''version'' block
* install the result in ''/usr/lib/rpm/macros.d/macros.forge-srpm''
* install the result in ''/usr/lib/rpm/macros.d/macros.forge-srpm''
* test with '''forgemetacheck''' and '''rpmbuild -bs''' ''myspecfile.spec''
* test with '''forgemeta -i''' or  '''forgemeta -v''' and '''rpmbuild -bs''' ''myspecfile.spec''
* submit the enhancement for inclusion in [https://bugzilla.redhat.com/enter_bug.cgi?product=Fedora&component=fedora-rpm-macros fedora-rpm-macros] once it works satisfactorily.
* submit the enhancement for inclusion in [https://bugzilla.redhat.com/enter_bug.cgi?product=Fedora&component=redhat-rpm-macros redhat-rpm-macros] once it works satisfactorily.
 
==Benefits and limitations==
* Benefits:
# Fedora packagers no longer need to know about the URL structure of well-known forges.
# Spec files are simpler, less error-prone, easier to maintain and audit.
# Forge URLs can be defined and fixed in a single place, without waiting for guidelines to percolate.
# The macros are mostly written in Lua, making them more verbose but easier to adjust and extend.
# The macros are written in Lua and can perform error handling.
# It is very easy to switch from commit to tag to version (or any combination of those).
# ''spectool'' just works© (in Fedora).
# scm snapshot date is exact and does not rely on a variable which may or may not have been updated.
# Almost all the computations are done in optional rpm variables that can be ignored by the packager if he does not need or does not like the result.
# Fedora forge know-how can be capitalized over time.
# The macro normalizes common spec constructs such as ''archivename''.
 
* Limitations:
# The macro is not intended to be used in spec files that package multiple forge archives (it could probably be extended, is the added flexibility worth the complexity?).
# The macro needs to be updated when a forge changes its structure (but better changing one place than lots of spec files).
# New forges need to be added to the macro before it can be used with them.
# ''dist'' munging can not be easily reverted once ''forgemeta'' has been called. The alternative would be to 1. change fedora-release to define  ''dist'' as ''%{?snapid}%{?distroid}'' and set ''distroid'' to the value ''dist'' is set today, and 2. let other macros such as ''forgemeta'' define ''snapid'' later. Cleaner but a lot more invasive and more difficult to propagate downstream.
# ''dist'' commit date computation relies on source files with the correct modification time (incorrect source file modification time → incorrect ''dist'' date).
# ''spectool'' may not work in older Fedora derivatives (but just use ''forgemetacheck'' and copy the source URL form its output).
# The macro highlights some historical rpm design mistakes: bad separation of upstream and package metadata, special magic variables. This is why ''version'' declaration occurs in an unusual (for rpm) order and requires a specific syntax.
 
==Testing==
 
Just drop in ''/usr/lib/rpm/macros.d/''  the file proposed [https://bugzilla.redhat.com/show_bug.cgi?id=1523779 here] for inclusion in '''fedora-rpm-macros''', and play with the result. The file name must be prefixed with “macros.”.


[[Category:Packaging guidelines drafts]]
[[Category:Packaging guidelines drafts]]

Latest revision as of 16:43, 24 February 2021

Old page
This page has been marked as "old", and likely contains content that is irrelevant or incorrect. If you can, please update this page. This page will be deleted if action is not taken.
Accepted proposal
This proposal has been accepted and is now part of Packaging Guidelines, section Referencing Source

Projects published on a “forge” can be packaged using the forgemeta macro.

This proposal is a requisite for the More Go packaging draft.

What is a “forge”?
Any software publishing website, permitting the download of source code archives on normalized URLs, that can be deduced from version, commit, tag, scm, extension values and a project root URL on the website, is a “forge” that can be supported by forgemeta.

Links

Benefits and limitations

  • Benefits:
  1. Fedora packagers no longer need to know about the URL structure of well-known forges.
  2. Spec files are simpler, less error-prone, easier to maintain and audit.
  3. Forge URLs can be defined and fixed in a single place, without waiting for guidelines to percolate.
  4. The macros are mostly written in Lua, making them more verbose but easier to adjust and extend.
  5. The macros are written in Lua and can perform error handling.
  6. It is very easy to switch from commit to tag to version (or any combination of those).
  7. spectool just works©.
  8. scm snapshot date is exact and does not rely on a variable which may or may not have been updated.
  9. Almost all the computations are done in optional rpm variables that can be ignored by the packager if he does not need or does not like the result.
  10. Fedora forge know-how can be capitalized over time.
  11. The macro normalizes common spec constructs such as archivename.
  • Limitations:
  1. The macro is not intended to be used in spec files that package multiple forge archives (it could probably be extended, is the added flexibility worth the complexity?).
  2. The macro needs to be updated when a forge changes its structure (but better changing one place than lots of spec files).
  3. New forges need to be added to the macro before it can be used with them.
  4. dist munging can not be easily reverted once forgemeta has been called. The solution would be to change fedora-release to define dist as %{?distprefix}.fcxx
  5. dist commit date computation relies on source files with the correct modification time (incorrect source file modification time → incorrect dist date).
  6. The macro highlights some historical rpm design mistakes: bad separation of upstream and package metadata, special magic variables. This is why version declaration occurs in an unusual (for rpm) order and requires a specific syntax.

Usage

A. The packager declares upstream-dependent metadata :

  • forgeurl: the project URL on the target software hosting service
  • version: the project release to package, if non nil (as Version: xxx)
  • commit: the commit hash to package, if any
  • tag: the project tag to package, if any

B. The packager calls %forgemeta. This macro will attempt to compute and set the following variables if they are not already set by the packager:

  • forgesource: an URL that can be used as SourceX: value
  • forgesetupargs: the correct arguments to pass to %setup for this source; used by %forgesetup and %forgeautosetup
  • archivename: the source archive filename, without extentions
  • archiveext: the source archive filename extensions, without leading dot
  • archiveurl: the url that can be used to download the source archive, without renaming
  • scm: the scm type, when packaging code snapshots (commits or tags)

If the macro is unable to parse the %{forgeurl} value the packager should set at least %{archivename} and %{archiveurl} before calling it.

The macro also accepts the following optional parameters:

  • -u <url>: ignore %{forgeurl} even if it exists and use <url> instead; note that the macro will still end up setting <url> as %{forgeurl} if it manages to parse it
  • -s: silently ignore problems in %{forgeurl}, use it if it can be parsed, ignore it otherwise
  • -p : restore problem handling, override -s
  • -v : be verbose and print every variable the macro sets
  • -i: Print some info about the state of variables the macro may use or set at the end of the processing

C. The packager uses the computed metadata as needed.

%{dist} modification for code snapshots

Most of the macro behavior is optional and safe. You can override the computed rpm variables values before or after the %forgemeta call, or ignore those variables altogether and don't use them in the rest of the spec file. There is one exception:

The macro will change the value of %{dist} in a non-reversible way, when packaging a code snapshot (commit or tag). Redefine %{dist} manually after the %forgemeta call if you don't like the result. Alternatively, don't use %forgemeta.

The root cause is that Fedora does not provide currently provide a placeholder prefix macro inside %{dist} that could be adjusted without redefining %{dist} as a whole.

Extending the macro

If the project you're packaging is published on a software publishing service forgemeta has no knowledge of, don't be sad, that's pretty easy to fix.

  • note down the archive URLs you want to generate for versions, tags and commits
  • locate the latest version of the forgemeta macro (it should be installed in /usr/lib/rpm/macros.d/macros.forge-srpm by fedora-rpm-macros)
  • copy the definition block closest to your needs after the other definition blocks. It should look like :
 if (forge == "vvv") then
 […]
 end

or

 if (string.match(forge, "vvv")) then
 […]
 end
  • change the "vvv" value
Don't be afraid of Lua patterns!
While the Lua pattern syntax is a bit unusual, it is quite simple. You can launch an interactive interpreter to experiment in with the lua command. Most of the times the existing forgemeta macro will already contain something similar to what you need to accomplish.
  • adapt the initial normalization rule
    forgeurl = string.match(forgeurl, "www")
  • adapt the initial error message
     rpm.expand("%{error:"xxx URLs must match yyy!\\n}")
  • set the hosting service defaults
   safeset("archiveext",     "tar.bz2")
   safeset("forgesetupargs", "-n %%{archivename}")
   if (commit ~= "") or (tag ~= "") then
     safeset("scm", "git")
   end
  • scrap the variables you need from forgeurl using:
      local myvariable = string.match(forgeurl, "zzz")
  • use your variables to define archivename and archiveurl in the following if tag/commit/version block
  • install the result in /usr/lib/rpm/macros.d/macros.forge-srpm
  • test with forgemeta -i or forgemeta -v and rpmbuild -bs myspecfile.spec
  • submit the enhancement for inclusion in redhat-rpm-macros once it works satisfactorily.