The simplest step toward circuit breakers in Clojure

(written by lawrence krubner, however indented passages are often quotes). You can contact lawrence at: lawrence@krubner.com

I think I missed the importance of this the first time I read it:

Consider you have this line within a service response:

{:body @(future (client/get “http://soundcloud.com/blah/wah”)) :status 200}

Now http://soundcloud.com/blah/wah goes down and those client requests start getting blocked on the request. In Clojure all future calls acquire a thread from the same thread pool. In our example the service is blocked up, is pilling new requests onto the blocked pool and we are in trouble.

My first solution to this problem was to introduce circuit breakers (https://github.com/josephwilk/circuit-breaker). I also stop using @ to dereference futures and used deref http://clojuredocs.org/clojure_core/clojure.core/deref which supports defaults and timeouts.

(defncircuitbreaker :blah-http {:timeout 30 :threshold 2})

(def future-timeout 1000)
(def timeout-value nil)

(defn http-get [url]
  (with-circuit-breaker :blah-http {
    :connected (fn [] (client/get "http://soundcloud.com/blah/wah"))
    :tripped (fn [] nil)}))

{:body (http-get http://www.soundcloud.com/blah/wah) :status 200}

Problem solved, now even though the thread pool may become blocked we back off the following requests and avoid pilling more work onto the blocked thread pool.

So this is the simplest and most obvious thing you can do to add circuit breakers to Clojure. And I suspect, you can go a long way with just this. I suspect, for the additional complexity of better approaches, the Law Of Diminish Returns cuts in with a vengeance.

Indeed, consider how difficult it is to get cache invalidation right (There are only two hard things in Computer Science: cache invalidation and naming things — Phil Karlton) and then consider that the next line in the article is:

This worked pretty well, but then we decided we would to try and go even further in gracefully degrading. Why don’t we serve from a cache on failure, slightly stale data is better than none.

Source