From Fedora Project Wiki

< SIGs‎ | bigdata‎ | packaging

Line 109: Line 109:
  |
  |
  | job server for Spark-as-a-service
  | job server for Spark-as-a-service
|-
| [https://github.com/spray/spray spray]
|
|
|
| REST toolkit for Akka
|-
| [https://github.com/spray/spray-json spray-json]
|
|
|
| Spray's JSON library (used by Ooyala job server)
  |-
  |-
  | [https://github.com/sbt/sbt-ghpages <code>sbt-ghpages</code>]
  | [https://github.com/sbt/sbt-ghpages <code>sbt-ghpages</code>]

Revision as of 14:06, 19 March 2014

Scala and sbt

Overview

Scala combines a lot of excellent features (functional-style pattern matching, an expressive type system, closures, etc.) with JVM compatibility and a very interesting developer ecosystem. Fedora has included a package for Scala itself for some time (although the Fedora Scala package is currently broken). However, Fedora doesn't include any of the Scala ecosystem projects. The main obstacle to having Scala ecosystem projects in Fedora is that many projects use sbt, the Simple Build Tool, but there is no native Fedora sbt package. If sbt were available on Fedora, it would greatly facilitate packaging other Scala packages for Fedora. We are taking steps to make Scala and sbt available in Fedora; see here for more background.

If you just want to get started helping out, check out these tips for packaging with sbt and our current packaging wishlist.

Building Scala

The latest stable release of the Scala language is 2.10. It isn't possible to build 2.9 on Fedora 19 or later. Some interesting ecosystem projects don't officially support 2.10, although 2.10 is mostly source-compatible with version 2.9. (Since version 0.9.0, Spark supports Scala 2.10.)

A revised Scala package for F19 and up is available from Fedora testing repositories.

Challenges

We had to overcome several challenges in order to build Scala for Fedora:

  • The upstream build uses ant and relies on maven-ant-tasks in order to pull down dependencies. For a variety of reasons, maven-ant-tasks is a poor fit for Fedora, but patching ant build.xml files is tedious and brittle. In the future, it would be best to have a more automated approach to ant build patching (a la the functionality available to packagers via maven pom macros). Another possibility is to maintain a separate, leaner build file for Fedora. Gil Cattaneo developed such a script (currently missing documentation generation and some POM files); Will made a first pass at using it for packaging.
  • Scala depends on a forked version of JLine that is roughly API-compatible with JLine 2.5. In prior Fedora releases, it was possible to simply replace references to the Scala JLine with references to the system JLine. However, the Scala JLine API is incompatible with stock JLine 2.7 and up, so Scala requires a more involved patch in order to work with Fedora 19 and later.
  • A bug in the Scala optimizer interacts poorly with JLine (either version), so it is necessary to build Scala with optimizations turned off.
  • The test suite and scala-partest artifact have missing dependencies in Fedora; getting them to run locally (viz., outside of a build) required some pruning to remove spurious failures (e.g. here, here, and here). (In particular, the binary-compatibility test checked files compiled by an optimizing scalac against files compiled by our scalac; since we need the optimizer turned off, this exposed a problem like this one.)

Limitations

  • We'd like to have the test suite and scala-partest artifact in a build and run the former as part of %check.

Missing dependencies

These are immediate dependencies for the Scala test suite (there may be other transitive dependencies). We anticipate that getting the Scala test suite running under mock will be a nontrivial effort.

Missing/Questionable Dependencies
Project State Review BZ Packager Notes
com.googlecode.java-diff-utils:diffutils Review 1013039 willb check out the packaging repo; the open issue is removing some cut-and-paste code reuse to comply with bundling policy
org.ops4j.pax.exam:pax-exam-container-native Available version 2.5.0
org.ops4j.pax.exam:pax-exam-junit4 Available version 2.5.0
org.ops4j.pax.exam:pax-exam-link-assembly Available version 2.5.0
org.ops4j.pax.url:pax-url-aether Available version 1.4.0; Gil has a pax-url spec
com.typesafe:mima-reporter_2.9.2 Available version 0.1.5 (needs Scala to build; used for binary-compatibility testing)

Other Scala projects to package

These are immediate build and test dependencies for other projects that use Scala and sbt. In some cases, we've been able to work around their omission, but having these available would make it possible to package more Scala projects and would bring the packages we have available more in line with the upstream builds.

Please take some of these and package them, if you're so inclined!

Project State Review BZ Packager Notes
scalacheck in testing 1056335 willb property-based testing (à la QuickCheck)
chill Scala extensions for Kryo
specs2 willb has a preliminary package
scalatest Note that the current version (2.0) is more recent than that required by most ecosystem projects.
Ooyala job server job server for Spark-as-a-service
spray REST toolkit for Akka
spray-json Spray's JSON library (used by Ooyala job server)
sbt-ghpages sbt plugin to generate GitHub pages-style documentation
sbt-site sbt plugin to generate project websites
sbt-git sbt plugin to access git functionality
sbt-release sbt plugin to automate the release process (including generating dependency metadata, etc.)
sbt-unidoc sbt plugin to generate unified API documentation across projects
sbt-osgi sbt plugin to generate unified OSGi bundles

Building sbt

We currently have sbt 0.13.1 under review in Fedora. It includes a patch to work with Ivy 2.3.0. Having sbt available in Fedora is currently the main obstacle to getting interesting parts of the Scala ecosystem into Fedora. The rest of this section will discuss some of the pitfalls involved in packaging sbt.

One of the big obstacles to building sbt is its reliance on Ivy repositories with metadata. We have both short-term and longer-term solutions to this problem: a patch to make sbt work with bare Ivy repositories, a tool to make a temporary Ivy repository from installed Fedora packages, and improved Fedora support for Ivy. In the future (F21+), it will be possible to use the latter as part of a much simpler build process.

We need to build sbt 0.13.x — and not sbt 0.12.x — because we use Scala 2.10, and we need to build sbt 0.13.1 in particular because sbt's bootstrap process depends on a particular version of the Scala compiler and 0.13.1 is the first version that works with (and requires) Scala 2.10.3. As a consequence, Scala and sbt in Fedora should necessarily move more slowly than other packages; we'll need to ensure that everything we want to build will work with a new compiler or sbt before upgrading any pieces.