Build systems, versioning and dependency

(written by Lawrence Krubner, however indented passages are often quotes)

In my mind, the most impressive thing about the Ruby On Rails eco-system is Bundle, which manages dependencies among gems (in other words, it is a great package manager). It locks down a given build/version when you run “bundle install” but it doesn’t allow this when you deploy, as they didn’t want versions changing at deploy time.

Maven, in the Java ecosystem, has problems because it will include changing versions:

This is the largest practical issue I’ve had with developing a Scala project.

One major cause of the problem seems to be that Scala is stuck with the “precompiled bytecode in JARs” model of software distribution; while this may be appropriate for Java (where core developers go through great pains to maintain compatibility between versions) it seems like a poor fit for Scala. Not only do the core library APIs change between versions, but the conventions used by the compiler to “link” Scala code together are also altered, for good reason.

An alternative distribution mechanism that’s more resistant to these kinds of issues might make sense. Even just distributing source code and build files (like, say, Ruby) could be a better solution. Building dependencies might be slow at first, but centralized caching could address this. Another solution might be an intermediate representation (after type-checking and other compiler phases, but before “linking”) that gives the Scala compiler the knowledge that’s missing from compiled class files.
Even namespace-mangling or class-loader magic could be helpful here. Loading multiple versions of a single dependency should in theory be possible, since OSGi manages it.

Sadly, there seems to be a pretty large investment by the Scala community in the whole Java/Maven/JAR ecosystem. It’d take a concerted effort by many people to create a more robust solution.