From Fedora Project Wiki

No edit summary
No edit summary
Line 25: Line 25:
B. The packager calls the '''%forgemeta''' macro to compute rpm-oriented metadata:
B. The packager calls the '''%forgemeta''' macro to compute rpm-oriented metadata:
* '''forgesource:''' the corresponding source file URL, that can be used as ''Source:''
* '''forgesource:''' the corresponding source file URL, that can be used as ''Source:''
* '''forgesetupargs:''' arguments that will be passed to ''%setup'' by ''%forgesetup''
* '''shortcommit:''' a commit hash reduction
* '''shortcommit:''' a commit hash reduction
* '''setupargs:''' arguments that will be passed to ''%setup''
* '''archivename:''' the corresponding archive name (often used by ''%{forgesetupargs}'')
* '''archivename:''' the corresponding archive name (often used by ''setupargs'')
* '''archiveext:''' the corresponding archive extension
* '''archiveext:''' the corresponding archive extension
* '''archiveurl:'''  the archive URL, without renaming
* '''archiveurl:'''  the archive URL, without renaming
Line 40: Line 40:
D. The packager uses the computed metadata as needed.
D. The packager uses the computed metadata as needed.


== Magic ==
== ''%{dist}'' modification for code snapshots ==


Most of the macro behavior is optional and safe. You can override the computed rpm variables values before or after the ''%forgemeta'' call, or ignore those variables altogether and don't use them in the rest of the spec file. There are two exceptions however:
Most of the macro behavior is optional and safe. You can override the computed rpm variables values before or after the ''%forgemeta'' call, or ignore those variables altogether and don't use them in the rest of the spec file. There is one exception:
 
=== ''%{dist}'' modification for code snapshots ===


The macro will change the value of ''%{dist}'' in a non-reversible way, when packaging a code snapshot (commit or tag). Redefine ''%{dist}'' manually after the ''%forgemeta'' call if you don't like the result. Alternatively, don't use ''%forgemeta''.
The macro will change the value of ''%{dist}'' in a non-reversible way, when packaging a code snapshot (commit or tag). Redefine ''%{dist}'' manually after the ''%forgemeta'' call if you don't like the result. Alternatively, don't use ''%forgemeta''.


The root cause is that Fedora [https://bugzilla.redhat.com/show_bug.cgi?id=1524192 does not provide currently provide] a placeholder prefix macro inside ''%{dist}'' that could be adjusted without redefining ''%{dist}'' as a whole.
The root cause is that Fedora [https://bugzilla.redhat.com/show_bug.cgi?id=1524192 does not provide currently provide] a placeholder prefix macro inside ''%{dist}'' that could be adjusted without redefining ''%{dist}'' as a whole.
=== ''%setup'' overload to ''%setup %{?setupargs}'' ===
The macro will overload ''%setup'' and change it to ''%setup %{?setupargs}'' (and compute the correct ''%{setupargs}'' value).
This is generally what you want but may be annoying if you need to pass your own arguments to ''%setup''. Possible solutions:
* redefine ''%{setupargs}'' yourself
%global setupargs '''xxx'''
* set it to "" after the ''%forgemeta'' call
%forgemeta
%global setupargs ""


== Packaging examples ==
== Packaging examples ==
Line 81: Line 68:
  …
  …
  %prep
  %prep
  %setup
  '''%forgesetup'''


''%forgemetacheck'' output:
''%forgemetacheck'' output:


  Forge-specific packaging variables
  Forge-specific packaging variables
   forgeurl:       “https://github.com/alecthomas/kingpin”
   forgeurl:       “https://github.com/alecthomas/kingpin”
   forgesource:     “https://github.com/alecthomas/kingpin/archive/v2.2.5.tar.gz#/kingpin-2.2.5.tar.gz”
   forgesource:   “https://github.com/alecthomas/kingpin/archive/v2.2.5.tar.gz#/kingpin-2.2.5.tar.gz”
   shortcommit:     “”
  forgesetupargs: “-n kingpin-2.2.5”
   shortcommit:   “”
  Generic variables
  Generic variables
  setupargs:      “-n kingpin-2.2.5”
   archivename:    “kingpin-2.2.5”
   archivename:    “kingpin-2.2.5”
   archiveext:      “tar.gz”
   archiveext:      “tar.gz”
Line 120: Line 107:
  …
  …
  %prep
  %prep
  %setup
  '''%forgesetup'''


=== Packaging a release tag===
=== Packaging a release tag===
Line 137: Line 124:
  …
  …
  %prep
  %prep
  %setup
  '''%forgesetup'''


''%forgemetacheck'' output:
''%forgemetacheck'' output:
Line 144: Line 131:
   forgeurl:        “https://github.com/apache/thrift”
   forgeurl:        “https://github.com/apache/thrift”
   forgesource:    “https://github.com/apache/thrift/archive/0.10.0.tar.gz#/thrift-0.10.0.tar.gz”
   forgesource:    “https://github.com/apache/thrift/archive/0.10.0.tar.gz#/thrift-0.10.0.tar.gz”
  forgesetupargs:  “-n thrift-0.10.0”
   shortcommit:    “”
   shortcommit:    “”
  Generic variables
  Generic variables
  setupargs:      “-n thrift-0.10.0”
   archivename:    “thrift-0.10.0”
   archivename:    “thrift-0.10.0”
   archiveext:      “tar.gz”
   archiveext:      “tar.gz”
Line 170: Line 157:
  …
  …
  %prep
  %prep
  %setup
  '''%forgesetup'''
'''
'''
''%forgemetacheck'' output:
''%forgemetacheck'' output:
Line 177: Line 164:
   forgeurl:        “https://code.googlesource.com/google-api-go-client”
   forgeurl:        “https://code.googlesource.com/google-api-go-client”
   forgesource:    “https://code.googlesource.com/google-api-go-client/+archive/3a1d936b7575b82197a1fea0632218dd07b1e65c.tar.gz#/google-api-go-client-3a1d936b7575b82197a1fea0632218dd07b1e65c.tar.gz”
   forgesource:    “https://code.googlesource.com/google-api-go-client/+archive/3a1d936b7575b82197a1fea0632218dd07b1e65c.tar.gz#/google-api-go-client-3a1d936b7575b82197a1fea0632218dd07b1e65c.tar.gz”
  forgesetupargs:  “-c”
   shortcommit:    “”
   shortcommit:    “”
  Generic variables
  Generic variables
  setupargs:      “-c”
   archivename:    “google-api-go-client-3a1d936b7575b82197a1fea0632218dd07b1e65c”
   archivename:    “google-api-go-client-3a1d936b7575b82197a1fea0632218dd07b1e65c”
   archiveext:      “tar.gz”
   archiveext:      “tar.gz”
Line 206: Line 193:
  …
  …
  %prep
  %prep
  %setup
  '''%forgesetup'''


Post-release packaging is similar. See also [[Packaging:Versioning#Release_and_post-release_versions]].
Post-release packaging is similar. See also [[Packaging:Versioning#Release_and_post-release_versions]].
Line 231: Line 218:
     forgeurl = string.match(forgeurl, "'''www'''")
     forgeurl = string.match(forgeurl, "'''www'''")
* adapt the initial error message
* adapt the initial error message
      error("'''xxx''' URLs must match “'''yyy'''”!")
      print("%{error:"'''xxx''' URLs must match “'''yyy'''”!\\n}")
* set the hosting service defaults
* set the hosting service defaults
     if (archiveext == "") then
     if (archiveext == "") then
Line 267: Line 254:
# The macro needs to be updated when a forge changes its structure (but better changing one place than lots of spec files).
# The macro needs to be updated when a forge changes its structure (but better changing one place than lots of spec files).
# New forges need to be added to the macro before it can be used with them.
# New forges need to be added to the macro before it can be used with them.
# ''dist'' munging can not be easily reverted once ''forgemeta'' has been called. The alternative would be to 1. change fedora-release to define  ''dist'' as ''%{?snapid}%{?distroid}'' and set ''distroid'' to the value ''dist'' is set today, and 2. let other macros such as ''forgemeta'' define ''snapid'' later. Cleaner but a lot more invasive and more difficult to propagate downstream.
# ''dist'' munging can not be easily reverted once ''forgemeta'' has been called. The solution would be to change fedora-release to define  ''dist'' as ''%{?distprefix}.fcxx''
# ''%setup'' overloading may be annoying. The alternative would be to define another macro (for example ''%forgesetup''). This would be safer but would also require more packaging work in the common case. Right now the macro chose to make the common case easy and advanced cases with multiple ''%setup'' calls possible.
# ''dist'' commit date computation relies on source files with the correct modification time (incorrect source file modification time → incorrect ''dist'' date).
# ''dist'' commit date computation relies on source files with the correct modification time (incorrect source file modification time → incorrect ''dist'' date).
# ''spectool'' may not work in older Fedora derivatives (but just use ''forgemetacheck'' and copy the source URL form its output).
# ''spectool'' may not work in older Fedora derivatives (but just use ''forgemetacheck'' and copy the source URL form its output).

Revision as of 16:03, 11 December 2017

Projects published on a “forge” can be packaged using the forgemeta macro.

Note.png
What is a “forge”?
Any software publishing website, permitting the download of source code archives on normalized URLs, that can be deduced from version, commit, tag, scm, extension values and a project root URL on the website, is a “forge” that can be supported by forgemeta.

Links

Usage

A. The packager declares upstream-dependent metadata :

  • forgeurl: the project URL on the target software hosting service
  • version: the project release to package, if non nil (as Version: xxx)
  • commit: the commit hash to package, if any
  • tag: the project tag to package, if any
  • scm: the source code management engine used by the software hosting service, if packaging a tag or a commit, and if the service allows different engines.

B. The packager calls the %forgemeta macro to compute rpm-oriented metadata:

  • forgesource: the corresponding source file URL, that can be used as Source:
  • forgesetupargs: arguments that will be passed to %setup by %forgesetup
  • shortcommit: a commit hash reduction
  • archivename: the corresponding archive name (often used by %{forgesetupargs})
  • archiveext: the corresponding archive extension
  • archiveurl: the archive URL, without renaming
  • dist: auto-adjusted when packaging a tag or commit

C. (Optional) The packager calls the %forgemetacheck macro to display the values of computed variables. %forgemetacheck emits rpm warnings. It should be safe to call from any point of the spec file.

Note.png
On variable resolution
forgemetacheck resolves the values of rpm variables at one point of the spec file. Since most of those variables are themselves composed of other variables, it is possible for those values to change if one of the variables used in the composition is overridden by the packager after the forgemetacheck call.

D. The packager uses the computed metadata as needed.

%{dist} modification for code snapshots

Most of the macro behavior is optional and safe. You can override the computed rpm variables values before or after the %forgemeta call, or ignore those variables altogether and don't use them in the rest of the spec file. There is one exception:

The macro will change the value of %{dist} in a non-reversible way, when packaging a code snapshot (commit or tag). Redefine %{dist} manually after the %forgemeta call if you don't like the result. Alternatively, don't use %forgemeta.

The root cause is that Fedora does not provide currently provide a placeholder prefix macro inside %{dist} that could be adjusted without redefining %{dist} as a whole.

Packaging examples

Note.png
“%{forgeurl}“ use as “URL:”
You do not have to use %{forgeurl} as URL: value if the packaged project has a better customized home page. It is a convenience, nothing more.

Packaging a release (except for GitLab)

%global forgeurl    https://github.com/alecthomas/kingpin/
Version:            2.2.5

%forgemeta
%forgemetacheck (optionnal)

…
Release: 1%{?dist}
…
URL:    %{forgeurl}
Source: %{forgesource}
…
%prep
%forgesetup

%forgemetacheck output:

Forge-specific packaging variables
  forgeurl:       “https://github.com/alecthomas/kingpin”
  forgesource:    “https://github.com/alecthomas/kingpin/archive/v2.2.5.tar.gz#/kingpin-2.2.5.tar.gz”
  forgesetupargs: “-n kingpin-2.2.5”
  shortcommit:    “”
Generic variables
  archivename:     “kingpin-2.2.5”
  archiveext:      “tar.gz”
  archiveurl:      “https://github.com/alecthomas/kingpin/archive/v2.2.5.tar.gz”
  scm:             “”
  tag:             “”
  commit:          “”
  dist:            “.fc28” (snapshot date is computed once the corresponding source file is available)

Packaging a release (GitLab)

Note.png
GitLab is inconvenient
  • Any download from GitLab requires knowledge of the corresponding commit.
  • Release download depends on a Git tag, which is free-form. If your release tag value does not match Version (for example, if it is v%{version}), you also need to declare tag before calling forgemeta.
  • Please vote on the GitLab CE RFE to remove commit quirks from releases.
%global forgeurl    https://gitlab.example.com/foo/bar
%global commit      2a810629566a2d0f0d4107df244e8828b9f7bd5c
Version:            8.18.1
 
%forgemeta

…
Release: 1%{?dist}
…
URL:    %{forgeurl}
Source: %{forgesource}
…
%prep
%forgesetup

Packaging a release tag

%global forgeurl    https://github.com/apache/thrift/
Version:            0.10.0
%global tag         %{version}
 
%forgemeta
%forgemetacheck (optional)

…
Release: 1%{?dist}
…
URL:    %{forgeurl}
Source: %{forgesource}
…
%prep
%forgesetup

%forgemetacheck output:

Forge-specific packaging variables
  forgeurl:        “https://github.com/apache/thrift”
  forgesource:     “https://github.com/apache/thrift/archive/0.10.0.tar.gz#/thrift-0.10.0.tar.gz”
  forgesetupargs:  “-n thrift-0.10.0”
  shortcommit:     “”
Generic variables
  archivename:     “thrift-0.10.0”
  archiveext:      “tar.gz”
  archiveurl:      “https://github.com/apache/thrift/archive/0.10.0.tar.gz”
  scm:             “git”
  tag:             “0.10.0”
  commit:          “”
  dist:            “.fc28” (snapshot date is computed once the corresponding source file is available)

Packaging a commit

%global forgeurl    https://code.googlesource.com/google-api-go-client/
%global commit      3a1d936b7575b82197a1fea0632218dd07b1e65c

%forgemeta
%forgemetacheck (optional)

…
Version: 0
Release: 0.1%{?dist}
…
URL:    %{forgeurl}
Source: %{forgesource}
…
%prep
%forgesetup

%forgemetacheck output:

Forge-specific packaging variables
  forgeurl:        “https://code.googlesource.com/google-api-go-client”
  forgesource:     “https://code.googlesource.com/google-api-go-client/+archive/3a1d936b7575b82197a1fea0632218dd07b1e65c.tar.gz#/google-api-go-client-3a1d936b7575b82197a1fea0632218dd07b1e65c.tar.gz”
  forgesetupargs:  “-c”
  shortcommit:     “”
Generic variables
  archivename:     “google-api-go-client-3a1d936b7575b82197a1fea0632218dd07b1e65c”
  archiveext:      “tar.gz”
  archiveurl:      “https://code.googlesource.com/google-api-go-client/+archive/3a1d936b7575b82197a1fea0632218dd07b1e65c.tar.gz”
  scm:             “git”
  tag:             “”
  commit:          “3a1d936b7575b82197a1fea0632218dd07b1e65c”
  dist:            “.git3a1d936.fc28” (snapshot date is computed once the corresponding source file is available)

Packaging a pre-release commit

Note.png
On Version declaration order
  • It is generally safe to declare Version before calling %forgemeta even for pre or post-releases. Except for GitLab. Due to GitLab's use of commit for release downloads, you MUST declare Version after calling %forgemeta when the packaged commit does not match the release commit.
  • Please vote on the GitLab CE RFE to remove commit quirks from releases.
%global forgeurl     https://gitlab.example.com/foo/bar/
%global commit       51637bc0960002b811e1c0c7be8671cf9a1cc5be
 
%forgemetaVersion: 3.2.6
Release: 0.1%{?dist}
…
URL:    %{forgeurl}
Source: %{forgesource}
…
%prep
%forgesetup

Post-release packaging is similar. See also Packaging:Versioning#Release_and_post-release_versions.

Extending the macro

If the project you're packaging is published on a software publishing service forgemeta has no knowledge of, don't be sad, that's pretty easy to fix.

  • note down the archive URLs you want to generate for versions, tags and commits
  • locate the latest version of the forgemeta macro (it should be installed in /usr/lib/rpm/macros.d/macros.forge-srpm by fedora-rpm-macros)
  • copy the definition block closest to your needs after the other definition blocks. It should look like :
 if (forge == "vvv") then
 …
 end

or

 if (string.match(forge, "vvv")) then
 …
 end
  • change the "vvv" value
Note.png
Don't be afraid of Lua patterns!
While the Lua pattern syntax is a bit unusual, it is quite simple. You can launch an interactive interpreter to experiment in with the lua command. Most of the times the existing forgemeta macro will already contain something similar to what you need to accomplish.
  • adapt the initial normalization rule
    forgeurl = string.match(forgeurl, "www")
  • adapt the initial error message
     print("%{error:"xxx URLs must match “yyy”!\\n}")
  • set the hosting service defaults
   if (archiveext == "") then
     rpm.define("archiveext tar.xz")
   end
   if (setupargs == "") then
     rpm.define("setupargs -n %{archivename}")
   end
   if (commit ~= "") or (tag ~= "") then
     rpm.define("scm git")
   end
  • scrap the variables you need from forgeurl using:
      local myvariable = string.match(forgeurl, "zzz")
  • use your variables to define archivename and archiveurl in the following if tag/commit/version block
  • install the result in /usr/lib/rpm/macros.d/macros.forge-srpm
  • test with forgemetacheck and rpmbuild -bs myspecfile.spec
  • submit the enhancement for inclusion in fedora-rpm-macros once it works satisfactorily.

Benefits and limitations

  • Benefits:
  1. Fedora packagers no longer need to know about the URL structure of well-known forges.
  2. Spec files are simpler, less error-prone, easier to maintain and audit.
  3. Forge URLs can be defined and fixed in a single place, without waiting for guidelines to percolate.
  4. The macros are mostly written in Lua, making them more verbose but easier to adjust and extend.
  5. The macros are written in Lua and can perform error handling.
  6. It is very easy to switch from commit to tag to version (or any combination of those).
  7. spectool just works© (in Fedora).
  8. scm snapshot date is exact and does not rely on a variable which may or may not have been updated.
  9. Almost all the computations are done in optional rpm variables that can be ignored by the packager if he does not need or does not like the result.
  10. Fedora forge know-how can be capitalized over time.
  11. The macro normalizes common spec constructs such as archivename.
  • Limitations:
  1. The macro is not intended to be used in spec files that package multiple forge archives (it could probably be extended, is the added flexibility worth the complexity?).
  2. The macro needs to be updated when a forge changes its structure (but better changing one place than lots of spec files).
  3. New forges need to be added to the macro before it can be used with them.
  4. dist munging can not be easily reverted once forgemeta has been called. The solution would be to change fedora-release to define dist as %{?distprefix}.fcxx
  5. dist commit date computation relies on source files with the correct modification time (incorrect source file modification time → incorrect dist date).
  6. spectool may not work in older Fedora derivatives (but just use forgemetacheck and copy the source URL form its output).
  7. The macro highlights some historical rpm design mistakes: bad separation of upstream and package metadata, special magic variables. This is why version declaration occurs in an unusual (for rpm) order and requires a specific syntax.

Testing

Just drop in /usr/lib/rpm/macros.d/ the file proposed here for inclusion in fedora-rpm-macros, and play with the result. The file name must be prefixed with “macros.”.