May 3rd, 2015
(written by lawrence krubner, however indented passages are often quotes). You can contact lawrence at: email@example.com
I find it interesting to consider the question that maybe Clojure has so many great systems for distributed processing that it does not need the classic job queue.
It’s worth considering that if you don’t need something very robust (and depending on your situation), you could just use Redis + Carmine directly to pass messages around (possibly representing jobs), and have workers pull from the message queue. There is nothing else you really need for this; it’s quite straight forward. The rub is that you don’t get any of the higher level features; if a job dies for instance, there’s no way to know about that without building it yourself.
It’s also worth considering whether a job queue is really the right mode of distributed computing. This area is a real strength of Clojure’s, and there are a lot of offerings here. I say this because the offerings are a lot slimmer in Ruby land (at least when I was working with it), and so if you’re thinking queues based on prior experience with Ruby, I’d definitely dig a little more to make sure it’s the right model for you. It might seem like “already having Redis in your system” makes it a good goto, but if you’re bending the model it’ll end up being more work. Some other things to consider:
* quasar (https://github.com/puniverse/quasar)
* pulsar (http://docs.paralleluniverse.co/pulsar)
Also, I should mention that Ruby doesn’t have very good built in parallelism support (no true threads when I was using, though this might have changed). As such, I’ve seen a fair bit of usage of Resque running on a single machine. This would be an insane overcomplication in Clojure given all it’s concurrency/parallelism support. So unless you’re actually planning to distribute tasks across a cluster, keep it simple and stick to things like the built in Clojure reference types and core.async.