Packaging:Minutes20070529

Present

 * DavidLutterkort
 * JasonTibbitts
 * JesseKeating
 * RexDieter
 * TomCallaway
 * ToshioKuratomi
 * VilleSkyttä

Writeups
The following drafts have been accepted by FESCO and are to be written into the guidelines:


 * Tweaks to static library packaging guidelines: http://fedoraproject.org/wiki/PackagingDrafts/StaticLibraryChanges

Votes
No proposals were voted upon this week.

Other Discussions
The following additional items were discussed; see the logs for full details.

of users and groups inside rpm scriptlets. For instance, %pre cannot be allowed to fail, because it seems that it is not possible to abort an RPM transaction.
 * Guidelines for adding Users and Groups
 * http://fedoraproject.org/wiki/PackagingDrafts/UsersAndGroups
 * Serious fundamental issues exist in rpm that prevent any reliable handling
 * Clever ideas working around the tough problems would be warmly welcomed.


 * Fix incorrect information in http://fedoraproject.org/wiki/Packaging/ScriptletSnippets
 * http://www.redhat.com/archives/fedora-packaging/2007-May/msg00116.html
 * scop will draft up various changes.


 * Including in binary packages test suite code that upstream does not install.
 * There are multiple issues: including things as %doc and actually packaging test binaries in /usr/bin.
 * There is little committee consensus on this; some members don't care what gets packaged as %doc, some don't want any test suite code packaged at all.
 * Community opinions are solicited here.
 * If someone feels strongly enough about the issue, please write up a draft.

IRC Logs
[12:01] * f13 stumbles in [12:02]  So, FPC meeting today? [12:02] I recall that we had plenty to talk about. [12:03] spot: ? I know your music is playing... [12:03] yeah, i'm here [12:03] tibbs: blah blah static-libs blah blah... :) [12:03] i'm knee deep in licensing stuff. [12:04]  ok, i count f13, rdieter, tibbs, and me [12:05]  add one to that [12:05]  abadger1999, lutter? [12:05]  I'm here. [12:05]  ok, thats 6. [12:06]  abadger1999: is the OCaml draft ready for review? [12:06]  Not yet. [12:06]  ok. [12:06]  Richard has refined it quite a bit but I'd like to go through one more week on the lists. [12:06]  scop: emacsen bits? [12:07]  some progress with emacs stuff, haven't found time to look at the last round of changes [12:07]  ok, we'll set that aside for next week too [12:08]  scop: how about User/Groups? any changes there from last week? [12:08]  no changes - I was supposed to add something about a registry, but the more I approached doing something the more I started to feel that it could come afterwards [12:09]  ie. a somewhat separate issue [12:09]  spot: here now, sorry [12:09]  So, there are a few changes that I'd like to see [12:09] 1. Use Requires: shadow-utils [12:10] or Requires(pre), rather [12:10] then, we can safely lose the pathing for the useradd/groupadd calls [12:10] I have no problem with that - however IIRC quite a few people prefer being explicit about stuff and requiring the exact executables we'd use [12:11] useradd/groupadd have been in shadow-utils for... years. [12:11] yeah, unless there are objections, I'll make that change [12:12] did anyone test to see if yum gets mad if the %pre fails? [12:12] I guess shadow-utils could grow a couple of provides, but I think it's pointless to worry about it. [12:12] rdieter? [12:12] I had someone wanting to require /bin/ping because it might move out of iputils one day. [12:12] tibbs: *shudder* [12:13] scop: sorry, -ENOTIME lately. [12:13] i'd feel a lot warmer about this draft if i knew yum would handle the pre failure case properly [12:13] skvidal: ping? [12:13] If we don't fail the pre, what would the consequences be? [12:14] %pre failing means the package won't be installed [12:14] Package gets installed with everything owned by root? [12:14] jeremy: what about other packages in the transaction set? [12:14] does yum stop entirely? [12:14] or packages before it in the transaction [12:15] spot: things will continue onward and you get absolutely no indication that something failed other than a line of output if you're running yum interactively [12:15] (on installs) [12:15] oh, that seems bad to me. [12:15] on upgrade, I think the old package remains installed [12:15] if foo deps on bar, and foo fails in pre, bar gets installed? [12:15] or rather, if bar deps on foo. [12:15] spot: pong [12:15] and yes, it is bad. if you are intentionally making %pre fail and expecting it to mean something, you're going to lose :-) [12:16] jeremy: is this a %pre with an exit 1? [12:16]  exit $nonzero [12:16]  exit != 0 [12:16]  it seems to me like we do not want %pre to fail [12:16]  so you're aborting the transaction [12:16]  no, don't do that [12:16]  not if you want anything to behave properly [12:16]  and there is a pretty obvious failure case for %pre in this draft [12:17]  instead, let the transaction go through, knowing that things won't work because it failed? [12:17]  scop: broken system vs broken package? [12:17]  which is the worse outcome? :) [12:17] "broken system"? [12:17] * jeremy goes to read the actual draft... [12:17] ok. lets say [12:17] what would be the desired failure mode? [12:17] libfoo and bar are being installed [12:17] ie: what is it you're hoping would happen? [12:18] skvidal: http://fedoraproject.org/wiki/PackagingDrafts/UsersAndGroups [12:18] we'd like the txn to abort cleanly [12:18] there is no way at all from a package to abort a transaction "cleanly" [12:18] at least, don't do anything for the package whose %pre failed and any of its deps [12:18] the concept isn't even defined [12:18] if %pre fails on adding a user, then that package doesn't get installed. [12:18] but any other packages in the same transaction _do_ get installed [12:19] what? [12:19] you want it to swallow the failure? [12:19] uh uh. [12:19] i'm not advocating that at all. :) [12:19] okay, then I'm confused how is: [12:19]  " but any other packages in the same transaction _do_ get installed" [12:19]  not "swallow the failure" [12:19]  ideally, i'd like to see yum detect the failure, scream, and revert the transaction entirely [12:20]  revert the transaction [12:20]  oh that's amusing [12:20]  i tell yum [12:20]  install foo bar baz [12:20]  foo goes in fine [12:20]  or abort gracefully early on... [12:20]  bar fails in pre [12:20]  yum does what now? [12:20]  skips installing bar, nothing else changes [12:20]  no need to revert everything, only packages that have dependencies to the one(s) that got dropped [12:20]  but what if baz depended on bar? [12:20]  b/c rpm doesn't error out [12:20]  spot: baz still tries to get installed [12:20]  will it fail because bar didn't go in? [12:21]  no [12:21]  jeremy: and the ts would believe it had it satisfied it [12:21]  skvidal: correct [12:21] ok, so now we've got bar missing and baz broken [12:21] spot: no. it will happily go along [12:21] ouch .. so failing %pre is really bad [12:21] on a larger transaction, this is DOOM [12:21] * rdieter thinks we're screwed. :) [12:21] thus, two issues. [12:21]  there's no definition of failure modes for %scriptlets [12:21]  1. utopia: yum should be able to handle that case. [12:21]  2. NEVER EVER let %pre fail [12:21]  if it makes you feel any better apt-deb doesn't have anything it can do then, either. [12:22]  spot: it's really rpm that needs to handle the failure, not yum [12:22]  i am totally not pointing fingers there. [12:22]  spot: I'm not worried, fixing this in yum would be, umm, amusing :) [12:22] if we're going to use %pre to add users groups, we need to have a fallout case [12:22] spot: correct [12:23] before that we need to define the modes of failure and all the cases [12:23] I know .. just trying to keep tyhe moving parts styraight [12:23] (and if you're not going to use %pre to add users and groups, I'm not really sure what you're planning on doing, but hey, what do I know? ;-) [12:23] <-- cweyl|work has left this server (Read error: 110 (Connection timed out)). [12:23] skvidal: i did point out utopia, didn't i? [12:23]  spot: I mean the broader definitions [12:23] Well, rpm could take care of adding users and groups internally, I guess. [12:23] skvidal: ok, i see what you mean. [12:24] so, the failure cases i see here: [12:24] spot: I mean we don't just care about adding users - all the other hurky shit out there in %scriptlets [12:24] ie: if a trigger fails? [12:24] skvidal: doesn't rpm throw a exit code on triggers? [12:24] I don't think it aborts the transaction anymore than a %pre error [12:24] it doesn't [12:24]  i didn't ask that. [12:24] and I'm pretty sure it doesn't get passed up to the library layer [12:25] it's probably just emitted [12:25] well, eww. [12:25] no kiddin [12:25] so, to bring this back on topic [12:25] if we want to use %pre for user/group adds [12:25] we need to note all the possible cases [12:25] and plan for them. [12:26] <-- nim-ni1 has left this server ("Leaving."). [12:26] case 1: user/group doesn't exist. [12:26] or option 48: stick head in sand, pray to *diety*. [12:26] case 2: user/group already exists (previous package put them there) [12:26] case 3: user/group already exists (amanda is my username, yay!) [12:26] best we can do is have the user/group additions need to always succeed; if there were problems, they need to be recorded somewhere and give ppl another tool to check for them [12:27] well, we can trap the exit code from useradd/groupadd [12:27] and say "if this exit code is non-zero, fallback to using $USER or $GROUP" [12:27] you can't fall back %files sections [12:28] I can't see that working, these packages (often) have hard-coded user/group references [12:28] you either have the user/group or not - if not, you'll get them owned by root [12:28] and if they're setuid/setgid? (to root?) [12:29] * scop was thinking exactly the same [12:29] iirc, they don't get the suid/sgid bit if the right user/group doesn't exist [12:29] ok, so I'm slowly coming to the conclusion that there is no safe way to add users/groups in the package [12:29] jeremy: good, thanks. [12:29] that we need to be thinking about this differently. [12:29] (I'd have to go reread the code again to be 100% certain about it) [12:29] spot: back to 'setup'? [12:30] rdieter: yes, i'm thinking setup. [12:30] I could not disagree more [12:30] scop: ok, well, here's my logic path [12:30] pre cannot fail -> fallback will not work because %files isn't flexible [12:31] or hybrid solution, setup for well-defined users/groups ahead-of-time, and then put some (1,2, many?) place-holder user/groups there too, for use until the next iteration of setup. ? [12:31] the only workaround I can think of is HORRIFYING [12:31] and thus, i will not bring it up. [12:31] spot: do anyway [12:31] lutter: ok. detect that we've failed in %pre with a macro define, fallback to root ownership [12:31] %files will do the right thing [12:32] check for the macro define in %post [12:32] and manually fix everything. [12:32] Can we do the user addition in a separate package? [12:32] the horror! [12:32] Or is that the horrifying workaround? [12:32] spot: what would 'fix everything' do ? [12:32] chown root.root for everything in %files that had ownership perms [12:32] tibbs: I thought about that, but it doesn't buy us anything (it has most of the same problem(s)). [12:32] tibbs: I think we'd run into the same problem, since rpm doesn't abort the txn [12:33] We're sure it won't abort even though the package will have unsatisfied dependencies in that case? [12:33] tibbs: i tend to trust skvidal when he says it won't. [12:33]  tibbs: the package that adds the user would abort -> same problem. [12:33] yep .. that's how I understood jeremy and skvidal said [12:33] <-- mdomsch has left this server ("Leaving"). [12:34] I had the impression it just wouldn't stop the installation of the package with the failing pre. [12:34] skvidal: ok, come back, we need you to be explicit [12:35] yum install foo bar baz [12:35] we have the option of simply assuming user/group adds always succeed. :) [12:35] scrollback seems to indicate that I'm wrong. [12:35]  baz depends on bar [12:35]  bar fails on %pre [12:35]  does baz get installed? [12:35]  yes [12:35]  Note also that we don't have to use %pre in this case. [12:35]  ? [12:35]  tibbs: %pre is still the way to go (I think) [12:36]  If there were some other hook we could use that has better behavior.... [12:36]  tibbs: we don't want a pkg that Provides: user(foo) to get installed, then fail in %post, do we? [12:36]  %pretrans? [12:36]  %pretrans is way, way early alright. [12:37]  maybe failure there is less horrifying. :) [12:37] i'm not sure it is. [12:37] any pre-install scriptlet failure will cause that package to not be installed [12:37] but it will not affect the transaction in-flight [12:38] ugh (we want the whole transation stopped). [12:38] if failing %pretrans doesn't cancel the whole transaction, that must be a bug [12:38] but then again I'm not sure we can always expect shadow-utils to be available for %pretrans [12:38] scop: ouch, you're right. [12:39] --> nim-nim has joined this channel (n=nim-nim@m37.net81-64-156.noos.fr). [12:39] hmm, ok. thinking out loud. [12:39] can we just check for the existence of the desired user/group in %pre? [12:39] we already do for the username [12:39] we don't fail if we find it, we just binary macro flip yes/no [12:40] eh [12:40]  conditionalize the Provides: user(foo) around that binary macro [12:40] in %post, check the binary macro, add user if its not there [12:40] I don't think you can affect such things in scriptlets [12:41] are you sure? [12:41] 99% [12:41]  * rdieter though macros were defined at *build* time, not runtime. [12:41] because that would be slippery, but it might get us over the top [12:41] spot: why not wrap the user/group add in a script that (a) is somewhat tolerant to the various failure modes and (b) keeps track of what user/groups it would have liked to create; that at least gives admins a fighting chance to figure out if there were problems after the fact [12:41] I fail to see how that would be better than just letting %pre fail if the user/group creation fails [12:42] scop: ok, let me repeat. we don't ever want %pre to fail. [12:42] i will vote against any draft that lets %pre fail. [12:42] lutter: yeah, I think we need a helper app/scriptlet that can do more for us here. [12:42] ok, more thinking out loud [12:43] can we do this with a yum plugin? [12:43] no [12:43]  why not? [12:43] is there ever a situation where we are not able to run a script and have user X exist afterwards ? The user might have the wrong uid/home dir etc., but that's a sperate issue ;) [12:43] doesn't help apt, smart, rpm [12:43]  i'm all for choice, but couldn't they also get plugins? [12:43]  well, maybe not rpm. ok. [12:43]  Well, rpm doesn't do plugins. [12:43]  I think such a script should go into shadow-utils [12:43]  I will vote against any draft that requires writing plugins for this [12:44]  and there's no guarantee that ppl have that plugin enabled in yum [12:44]  so, shadow-utils should provide a utility that adds a user/group safely, handles rpm provides, and file ownership? [12:45]  i think we're violently slamming against the wall of what is possible with rpm [12:45]  how would it handle rpm provides? [12:45]  spot: everything but rpm provides,  yes. [12:45]  not rpm provides; not sure what to do about file ownership [12:45]  Provides: user(foo) [12:45]  but a script in shadow-utils? [12:46] * scop notes I have 2 smaller other things I'd like to discuss for a bit today [12:46] ok, lets set user/groups aside [12:46] some sort of useradd/groupadd queue (and sets perms), where stuff gets removed only on success. [12:46] i have some other ideas i want to try before i discuss [12:46] if the user at least exists by the time %files does its thing, file ownership isn't as big an issue. To guard against the amanda situation (ordinary user suddenly can run programs they weren't supposed to) we'd need a separate way to disable such programs .. it's ugly [12:46] (this is starting to sound like some SuSE madness I've seen before) [12:47] scop: go ahead, whatcha got? [12:47] for the record, I'm now convinced that non-failing %pre is better than a failing one even if the user/group creation fails [12:47] there's another, even ickier option: fix rpm to do the right thing when %pre fails (where we define 'the right thing') [12:47] %post could do some cleanup/disabling/chown to nobody if the user didn't get created, I guess. [12:47] 1st little thing: http://www.redhat.com/archives/fedora-packaging/2007-May/msg00116.html [12:48] lutter: rule number 1 of packaging club: Fixing rpm is not an option, outside of our scope. [12:48] I suppose fixing the scriptletsnippets page according to mlichvar's points doesn't require much discussion? [12:48] scop: ok, so someone should probably fix scriptletsnippers [12:48] scop: Well, what's wrong with a little defensive programming? [12:49] I think we just came to the conclusion that aborting scriplets leads to madness, and is not an option. [12:49] tibbs, you're referring to mlichvar's post? [12:49] scop: Yes. [12:49] We can save three characters per line if we make an assumption about the internals of rpm. [12:49] he points out that some of the info in ScriptletSnippets is plain wrong, and he's right [12:49] I think the part that talks about omitting ||: is sane and scriptletsnipers should be updated accordingly [12:50] Doesn't seem like a terribly good tradeoff to me, personally. [12:50] I don't really see the point either. [12:50] "This is important because the scriptlet as a whole will error the moment it tries to execute a command that has a non-zero exit status." [12:50] that's incorrect [12:51] ah, that part should be fixed, the rest, not so sure. [12:51] any obvious errors in scriptletsnippets should be fixed. [12:51] that page got grandfathered in... [12:52] scop: I agree. That needs to be changed otherwise people will have false expectations of [12:52] %post [12:52] /bin/false [12:52] /bin/true || : [12:52] ok, so do people want this drafted or me to just go ahead and fix any obvious errors? [12:53] +1 Just fix it. [12:53] I guess that whole thing needs to be changed to note that we now know that you can never allow a scriptlet to fail. [12:53] +1 just fix it [12:53]  +1 [12:53] on the other hand, if we can't allow a scriptlet to fail, that's something that our rpm which drives us into that kind of situation should have built in [12:53]  I guess it depends on what you intend to just fix. [12:53] +1 [12:54]  * scop sighs [12:54] Because if you want to add mlichvar's info about appending "|| exit 1" then we'd have a problem. [12:54] But if you just want to remove the incorrect info then fine. [12:54] that's not in the "obvious errors" category I volunteered to fix [12:54] scop: why don't you draft it [12:54]  just to make sure everyone is happy [12:54] --> JSchmitt has joined this channel (n=s4504kr@p54B10442.dip0.t-ipconnect.de). [12:55] ok [12:55]  I don't want to force an extra runthrough. [12:55] scop: I think you could just write a one sentence description of the change here and we vote on it right now [12:55] lutter: i think its more than one sentence [12:55] scop: you can just copy the page into PackagingDrafts [12:55] edit it there [12:55] I'll draft something for next week [12:56] next, including test suite code that upstream doesn't install in binary packages [12:56] I'm not sure if this is on our plate though [12:56] Ah, what cweyl's doing for all of his perl packages now. [12:56] yeah. i think i'm not really in support of this. [12:56] i think test suite code not installed by make install goes in -devel [12:56] or not at all [12:56] I have to admit it bothers me, but in many cases the example code is useful as documentation. [12:57] I'm not at all either [12:57] tibbs: not in /usr/bin [12:57] i don't care what goes in %doc that isn't executable. :) [12:57] OK, so this isn't what Cweyl's doing. [12:57]  you can dump the kitchen sink in there. :) [12:57] spot: Even tarballs? [12:57] * spot squirms a little [12:57] i like that better than test binaries in /usr/bin [12:58] I do care about %docs as well, some rationale here: https://bugzilla.redhat.com/239193#c4 [12:58] I just reviewed a package that included several empty directories and two tarballs as %doc. [12:58] empty dirs shouldn't be in %doc without good reason [12:58] They're needed by the "example code", aka the test suite that gets packaged. [12:59] * scop thinks empty dirs are less harmful than the average test suite code [12:59] installing an rpm shouldn't set you up with a dev environment for that package .. at some point, you have to check out the code from upstream [12:59] So we have multiple issues. [12:59] What packages are putting test stuff in /usr/bin? [13:00] i think packaging up test code as %doc is alright. [13:00] as long as it is non-executable [13:00] Perl tests are generally non-executable. [13:00] hm, so the difference is "./t/foo.t" vs "perl ./t/foo.t"? [13:02] the former is a possible accident, the later is very unlikely to be accidental. [13:02] i admit it is a thin line. [13:02] seems like it boils down to whether the tests are written in a compiled or in a scripting language [13:02] but really, test suite code in non-*-devel, non-*-test packages? [13:03] ld.so ./t/blah ? [13:03] scop: other than in %doc, i say no. [13:03]  Or -doc packages [13:03] abadger1999++ [13:03] IMO not in main package, %doc or not [13:03] I have to agree. Tests as documentation are suboptimal, but occasionally that's all you get. [13:04] but if you are interested at that level, you might as well check out from upstream [13:04] Sorry, I was agreeing with spot, not scop. [13:04] *** knurd is now known as knurd_afk. [13:04] we shouldn't require it, but its packager's discretion to have tests in %doc, -devel, or -test [13:04] if its in %doc, it can't be executable (chmod -x) [13:04] or -doc (as %doc) if there are a lot of docs [13:05] but test code shouldn't be in %bindir or %sbindir [13:05] Theoretically, we could require all perl modules to split their documentation to -devel, since it's only needed by developers and not by end-users. [13:05] But that would be madness, I think. [13:05] * spot shudders [13:05] I'd much prefer if test code be left out of the main package entirely; it's just bloat for most people [13:05] i can hear the knashing of teeth now [13:06] if -devel already exists, fine, but not just for docs. [13:06] i think if the argument is that test code is useful for -devel, there is a -devel. [13:07] how about (if a -devel, put it there. if not, -doc/%doc is ok for small quantities of test code) [13:07] spot: knashing if you use kde, gnashing otherwise. [13:07] if not small quanties, use -test. [13:07] -1 to blanket approval for test code that isn't installed by upstream [13:07] quantities [13:08] anything else juicy, me gotta run soon. [13:08] * spot sighs [13:08] if someone cares about this enough to draft it formally, then i'll vote on it. [13:08] scop: Do you have a different opinion of example code? [13:08] tibbs, different than what? [13:08] i'm starving. [13:09] scop: You don't like test code that isn't installed by upstream, but what about an "examples" directory included as %doc? [13:09] And if you view that differently, why is test suite code not acceptable as an example? [13:09] well, that's pretty clearly designated as an example how to use stuff [13:09] tibbs: Does libfl.a need anything from us? [13:10] +1 to what spot said (about test code, and starving) [13:10] abadger1999: It's FESCo that approves those, isn't it? [13:10] yeah, static libs ain't us. [13:10] we just tell you how to package them. :) [13:10] I think that FESCo kind of has to approve flex or else we break lots of things. [13:10]  I think the static library changes we voted on last week makes it so FESCo doesn't have to approve inclusion anymore. [13:11]  Just approval of linking. [13:11]  abadger1999: i agree [13:11]  static libraries can be packaged. static linked binaries need permission. [13:11]  abadger1999: Which would mean any package that does BR: flex needs an ack? [13:11]  (from FESCo) [13:11]  Fun. [13:12]  tibbs: FESCo could blanket approve flex linked binaries. [13:12]  I think they'd have to in this case. [13:12]  tibbs: That's why I asked :-) [13:12] but i don't speak for FESCo as a whole. :) [13:12] * spot puts on his tin-foil hat [13:12]  Maybe libfl.a should be in a -devel package? [13:12]  imo, yes. [13:12]  the guidelines would say that, yes. [13:13]  ok, one last thing before i stop the madness. ;) [13:13] http://fedoraproject.org/wiki/Packaging/GuidelinesTodo [13:13] lutter, rdieter [13:13] * lutter hides (again) [13:13] you know what i'm going to say. get it done, pls. [13:14] abadger1999: flex itself is a "-devel" package, like gcc. [13:14] yeah, yeah. F7-tasks are mostly done, I can catch-up on things now. [13:14] And we could add something to the guidelines that say BR: *-static needs approval,  BR: *-devel is okay as we can get the same information out of it. [13:14] tibbs: good point, ok. :) [13:14] ok, folks. lunch time for me. thanks for all the headaches. :) [13:15] *** rdieter is now known as rdieter_away. [13:15] Thanks spot. [13:15] Lunch! [13:16] tibbs: Hmm... Let me think of a way to reword so we don't have to worry about flex.