Packaging talk:Ruby

The following metadata was found in MoinMoin that could not be converted to a useful value in MediaWiki:

• : These two files are symlinks into %gemdir created in %install

• acl: Known:read,write All:read

The short tutorial how to put an rpm file created with gem2rpm in compliance is here: http://lukas.zapletalovi.com/2011/01/how-to-prepare-gemfile-package-for.html

Testsuite execution

The most, if not all gems contains their test suite, which is commonly executed during build. This might be in most cases achieved by running rake task such as:

%check
rake spec

However this has some cavities. The problem is mainly that Rakefile usually contains not just the "spec" or "test" task for test suite execution, but also packaging tasks etc. These tasks usually depends on other gems, such as "Hoe". This brings in unnecessary dependencies and polution. Therefore, please prefer to execute the test suite as follows:

Test::Unit

RUBYOPT="I%{buildroot}%{geminstdir}/lib Itest" testrb test/test_*

MiniTest

RUBYOPT="I%{buildroot}%{geminstdir}/lib Itest" ruby -e "Dir.glob('test/**/test_*').each {|t| require t}"

RSpec

RUBYOPT="I%{buildroot}%{geminstdir}/lib Ispec" spec spec/

Additional notes

If the test suite has more dependencies, RUBYOPT can additionally enforce usage of RubyGems. This apply for all test frameworks mentioned above.

RUBYOPT="rubygems I%{buildroot}%{geminstdir}/lib Ispec" spec spec/

Why? Just adding "BR: rubygem(hoe)" and executing $ rake test" is much simpler and easier to maintain. Also using $rake check is useful because it can check if the Rakefile itself is valid or not. Please avoid such comprecated and unneeded craft as much as possible which makes spec file less readable and less maintainable. Note that BuildRequires != Requires. Adding BuildRequires does not add "runtime" dependency. If running $ rake check requires additional packages, please try to add them to Fedora project when possible. It is more useful than resorting to unusual way. - 2011-02-01 UTC 12:45 Mamoru

By the way for rspec it seems that some testsuite need rspec 1, and others need rspec 2. So should we rename rspec 2 to rubygem-rspec2 and try new review request? (And again, if some testsuite needs rspec 2 for %check, we should package it, not try to avoid using it) - Mamoru

2011-02-01 UTC 13:15 vondruch: I have nothing against adding required packages into Fedora, but I am against package pollution. Packages such as Hoe, Bundler, Rubyforge, they are useful for developer. Useful if you want to have every development dependency or create gem. They should be of course available in Fedora. But requiring them because you would like to be sure that test suite passes? It is pain.

If you want to test something in mock and the package your test "depends" on is not available in repository yet, then you are doing nothing else than installing manually packages for mock. If you require rake, it brings in immediately also dependency on RubyGems which means you have immediatelly polluted your load path with libraries you don't know. This makes execution of test suite as fragile as my magic commands.

I even don't want to mention test suites I met in recent time, which tried to install their dependencies during test suite execution. This is plain wrong.

And regarding the RSpec ... The gems I have packaged recently depends on RSpec2, but with some effort, the conversion for RSpec1 was possible with some minor cavities, like conditional execution of some example groups which is not supported by RSpec1.

If we are going ahead with direct upgrade of Rails 2.3 to Rails 3, without any alternatives, I would like to choose the same approach for RSpec (although I am aware that it might be more painful).

The only problem could be rspec-rails gem IMO. But I don't believe there are many gems which depends on it. How many Rails applications we have packaged? But they will be enforced to be migrate somehow to Rails 3 anyway.

2011-02-01 17:00UTC - Mamoru Testing is for development purpose (that is why we usually split test/ spec/ directories or so into -doc subpackage), so it is fairly natural to add (BuildRequires) dependency for testing suite. If you say installing them "pollutes" your environ, feel free to say so, however again it is fairly natural. It is more and more fragile to use such complicated craft than to just installing needed dependency, again your way is much harder to read and maintain. Please just package and install the needed dependency. Again %check suite is for developing purpose, i.e. - It is useful for people developing or maintaining the package, not needed on runtime (i.e. normal user). It is why we distinguish "BuildRequires" and "Requires". Normal user wouldn't need "BR" pkgs or won't have to do test suite.

This is similar to that normal people wouldn't install gcc, foo-devel or other developent-purpose pkgs. You seem to just be confused about "BuildRequires" and "Requires", or "maintainer" and "normal user". Please don't feel it "pain" to install development pkgs such as hoe, rubyforge and so on, if you are maintainer.

Also it is preferable that we as "maintainer" check if $ rake test as provided by the upstream actually works or not after installing the needed pkgs for testsuite. If it does not work, we just report it upstream as maintainer. However again it is not what "normal user" is expected to do and they won't have to install such development-purpose pkgs.

2011-02-10 15:32 UTC vondruch: I am not confused about BuildRequires. I understand it perfectly as far as I can say. And I am basing my statement on experiences from recent time. And I understand perfectly that it is very beneficial to execute test suite (for example Sinatra was broken in Rawhide until today, because its test suite was not executed during build time). The only purpose of the test suite from packager point of view is to ensure that whoever will build the package later will get correct result (current mass rebuild is good example).

The developer have different needs and it is completely different topic (we would call them DevelRequires ;) ). And I am speaking about developer in sense of the upstream package developer. He needs to provide gem for rubygems.org or rubyforge. Therefore it is natural that he needs hoe for example.

Nevertheless, I have to oppose your statement about "complicated craft". Actually my proposal is using the most straight forward way how to execute test suite. And that is exactly what I want during build time. Using "rake test" or "rake spec" is nowhere standardized. What is happening later is just magic. Have you ever noticed that RubyGems are required by rake? Is that correct in the days of Bundler, which doesn't rely on RubyGems in runtime? Some projects doesn't provide Rakefile at all.

From what I have seen in recent times, even developers of upstream gems doesn't understand their test suites. For example every test/spec file should be possible to execute separately. Which is not case for example for sunspot gem. And you want to rely on such developers or test suites? You believe that the rakefile is correct? Will you just blindly setup dependencies, because they are require by Rakefile itself? Honestly, will you check them? If there is some obsolete dependency, which is not in use anymore, how will you discover it?

2011-02-15 11:00 UTC - Mamoru

> using "rake test" or "rake spec" is nowhere standardized. And the way you provided is really far away from what you call "standard". Do you think that "for f in tests/* ; do <run> $f ; done" is the standard way for general "Makefile"-based pkgs? We should base on what we do on standard autotool-based pkgs, that is why "Rakefile" is provided. > Have you ever noticed that RubyGems are required by rake? Why not? rake is installed as "rubygem-rake" on Fedora, so why wouldn't I notice it? (I am also listed in %changelog). And why it is annoyance for you?

> Some projects doesn't provide Rakefile at all. Where did I say that every projects must provide Rakefile?

> From what I have seen in recent times, even developers of upstream > gems doesn't understand their test suites. If so we package maintainers should teach the upstream to fix it.

> You believe that the rakefile is correct? Again it Rakefile is not correct, we should fix it or ask the upstream to fix it. It is what maintainers are expected to do.

> Will you just blindly setup dependencies, because they are require by > Rakefile itself? Honestly, will you check them? If there is some obsolete > dependency, which is not in use anymore, how will you discover it? Won't you check it?? Reviewers would check such dependency of course, I guess?

Please know that I have commented in review requests many times that "this dependency is written in Rakefile, however it is actually not needed. Kill such dependency". Again like that you should fix that if the scripts in gem file are broken you must fix it, if Rakefile is broken you should fix it.

Think in this way. Wouldn't you check the dependencies written in gemspec file or installed ruby scripts? (for example, wouldn't you check "require " statement in all the installed ruby scripts? I always check them) This is not specific to ruby based packages. For python-based scripts we should check every "import " statement during review request. So why don't you check Rakefile's dependency? Well, you are saying that _you_ don't check the "rightness" of Rakefile?

2011-02-16 15:00 UTC - vondruch autotools and rake are in different position IMO. You need autotools to build the package while you don't need Rake to build rubygems (with some rare execptions). However, I am far from expert on autotoos, so I won't continue discussion about autotools.

Btw I am trying to establish standard or better "best practices guide". So my guidelines apply fine for scenarios where Rakefile is provided or not. Everybody can benefit from this. Since he/she follows the guidelines anyway, it will be obvious to them. I am trimming the preconditions down. If you want to execute test using rake, you have to be familiar with rake, if Rakefile is used for gem build, then you will need to be familiar with hoe, etc. At the end you will need again to know what is done behind the scene to execute the tests to be sure that you are testing what is needed. So I am trying to trim all these unnecessary knowledge just to the basic one, how to execute test suite.

To have rake on my system, or required for build is annoyance because it needs to be installed for build, therefore my build takes longer. Don't take me wrong, I would fight against every unnecessary dependency, it is not about Rake. Rakefiles just have the tendency to bundle every dependency inside, therefore you have to later bundle a lot of unnecessary bloat. So I would like to cut off the dependency chain right from the start. Rake is just development tool, nothing else. And we are not developing, we are packaging the development output.

If Rakefile is not included in default and if Rakefile is not used for test execution, I really do not check the correctness of Rakefile. There is no universal way how to ensure Rakefile correctness. You might check some parts of Rakefile but not all. Checking Rakefile if it can produce gem is pointless IMO. To be honest, IMO Rakefile should not ever be part of gem neither part of our RPM. As I stated, it is just development tool. I cannot remember I would ever need Rakefile of some installed gem, except I really twiddled around the gem and that is better to do in clone of Git repo anyway. Take Rails for example. You will not find Rakefile in any of the gems. And that is correct. I, as a Rails user, don't care about gem Rakefiles.

2011-02-18 20:20UTC mtasaka Well, after updating rspec-{core,expectations,mocks} to 2.5.x for review requests (uploaded, ready for review), I changed my mind because dependency for testsuite executed with $ rake foo is actually too complicated... Actually in rubygem-rspec-foo spec file, I am using $ ruby -rubygems -Ilib/ -S rspec spec/rspec/*/*_spec.rb or so.

2011-02-24 15:30 UTC - vondruch I noticed that. Nice. Just few notes. I have proposed usage of RUBYOPT which IMO fits better for this style. First reason was that I was not aware of "-S" flag ;) Anyway, I still find better to use RUBYOPT, because you avoid influence of potential flags which might already be set in RUBYOPT, i.e. you have better control (although in mock, there is not RUBYOPT set at all). And another reason is readibility. IMO it is better to see some environment variable setting followed by the command, then search through the command line and to understand what parameter are passed where etc. And you completely exclude the call to ruby, which is unimportant.

And one more nit is the "spec/rspec/*/*_spec.rb" pattern, which is IMO too specific. If there is no reason, like some failing tests you want to exclude, I believe that it is better to let the RSpec do their work and specify just the top folder. RSpec will try to recursively find every *_spec.rb file and execute it.

Applications

By applications are typically understood packages, which:

• provides user-level tools or
• web application typically build using Rails, Sinatra or similar frameworks.

Their RPM packages should obey FHS rules, it means that they should be installed into %{_datadir}. Following macro can help you:

%global app_root %{_datadir}/%{name}

These packages typically has no "Provides" section, since no other library or application depends on them.

RubyGems

While upstream might provide the package using RubyGems for convenience and easy support of all platforms supported by Ruby, please think about these packages more in way of ordinary .tar.gz packages than RubyGems packages. The packages should follow the FHS rules on Fedora. Although "gem install" places the files into %{gemdir}, they should be relocated into %{_datadir}. It also means that ruby- or rubygem- package name prefixes should not be used for these packages.

Note that the binary wrapper generated by RubyGems have to be replaced by new version without the RubyGems dependency:

#!/usr/bin/ruby

root_path = File.join('/', 'usr', 'share', 'foo-app')

$:.unshift File.join(root_path, 'lib')

load File.join(root_path, 'bin', 'foo-app')

Global macros

Global macros can be put in one file: /etc/rpm/macros.ruby. This file should be owned by ruby or rubygems. Good candidates for these macros are:

%global geminstdir %{gemdir}/gems/%{gemname}-%{version}

%global gemdir %(ruby -rubygems -e 'puts Gem::dir' 2>/dev/null)

Cached .gem file

The package should exclude the cached .gem file in files section:

%exclude %{gemdir}/cache/%{gemname}-%{version}.gem

Since the gem is installed using RPM, it makes no sense to include the cached .gem file. This file is used typically with 'gem pristine' command to restore gem into its original state, but this could be achieved by equivalent RPM command.