Is there a single answer to the problem of package management?

Interesting:

Why are there so many goddamn package managers? They sprawl across both operating systems (apt, yum, pacman, Homebrew) as well as for programming languages (Bundler, Cabal, Composer, CPAN, CRAN, CTAN, EasyInstall, Go Get, Maven, npm, NuGet, OPAM, PEAR, pip, RubyGems, etc etc etc). “It is a truth universally acknowledged that a programming language must be in want of a package manager.” What is the fatal attraction of package management that makes programming language after programming language jump off this cliff? Why can’t we just, you know, reuse an existing package manager?

You can probably think of a few reasons why trying to use apt to manage your Ruby gems would end in tears. “System and language package managers are completely different! Distributions are vetted, but that’s completely unreasonable for most libraries tossed up on GitHub. Distributions move too slowly. Every programming language is different. The different communities don’t talk to each other. Distributions install packages globally. I want control over what libraries are used.” These reasons are all right, but they are missing the essence of the problem.

I can identify at least three distinct approaches to the problem among the emerging generation of package managers, each of which has their benefits and downsides.

Pinned versions. Perhaps the most popular school of thought is that developers should aggressively pin package versions; this approach advocated by Ruby’s Bundler, PHP’s Composer, Python’s virtualenv and pip, and generally any package manager which describes itself as inspired by the Ruby/node.js communities (e.g. Java’s Gradle, Rust’s Cargo). Reproduceability of builds is king: these package managers solve the decentralization problem by simply pretending the ecosystem doesn’t exist once you have pinned the versions. The primary benefit of this approach is that you are always in control of the code you are running. Of course, the downside of this approach is that you are always in control of the code you are running. An all-to-common occurrence is for dependencies to be pinned, and then forgotten about, even if there are important security updates to the libraries involved. Keeping bundled dependencies up-to-date requires developer cycles–cycles that more often than not are spent on other things (like new features).

A stable distribution. If bundling requires every individual application developer to spend effort keeping dependencies up-to-date and testing if they keep working with their application, we might wonder if there is a way to centralize this effort. This leads to the second school of thought: to centralize the package repository, creating a blessed distribution of packages which are known to play well together, and which will receive bug fixes and security fixes while maintaining backwards compatibility. In programming languages, this is much less common: the two I am aware of are Anaconda for Python and Stackage for Haskell. But if we look closely, this model is exactly the same as the model of most operating system distributions. As a system administrator, I often recommend my users use libraries that are provided by the operating system as much as possible. They won’t take backwards incompatible changes until we do a release upgrade, and at the same time you’ll still get bugfixes and security updates for your code. (You won’t get the new hotness, but that’s essentially contradictory with stability!)

Embracing decentralization. Up until now, both of these approaches have thrown out decentralization, requiring a central authority, either the application developer or the distribution manager, for updates. Is this throwing out the baby with the bathwater? The primary downside of centralization is the huge amount of work it takes to maintain a stable distribution or keep an individual application up-to-date. Furthermore, one might not expect the entirety of the universe to be compatible with one another, but this doesn’t stop subsets of packages from being useful together. An ideal decentralized ecosystem distributes the problem of identifying what subsets of packages work across everyone participating in the system. Which brings us to the fundamental, unanswered question of programming languages package management:

How can we create a decentralized package ecosystem that works?

This seems like an interesting possible solution:

Advancing semantic versioning. In a decentralized system, it’s especially important that library writers give accurate information, so that tools and users can make informed decisions. Wishful, invented version ranges and artistic version number bumps simply exacerbate an already hard problem (as I mentioned in my previous post). If you can enforce semantic versioning, or better yet, ditch semantic versions and record the true, type-level dependency on interfaces, our tools can make better choices. The gold standard of information in a decentralized system is, “Is package A compatible with package B”, and this information is often difficult (or impossible, for dynamically typed systems) to calculate.

But for now, I like how things work with Clojure, which relies on Leinengen, which relies on Maven: allow duplicates.

Of course, if a library uses some library internally, but this choice is entirely an implementation detail, this shouldn’t result in any sort of global constraint. Node.js’s NPM takes this choice to its logical extreme: by default, it doesn’t deduplicate dependencies at all, giving each library its own copy of each of its dependencies. While I’m a little dubious about duplicating everything (it certainly occurs in the Java/Maven ecosystem), I certainly agree that keeping dependency constraints local improves composability.