What I learned from Clojure

Someone on Hacker News asked “Why use Functional Programming”. I wrote this in response:

I don’t have an answer for you, but for the last year I have been learning Clojure, and I have found it fascinating. I do not think this will satisfy you, but I would like to share some of the things that I find interesting about Clojure.

Lately I’ve been studying the source code of Aleph, a web server written by Zach Tellman. Zach is a very smart guy, and reading his source code is an education (I’ve actually found this true of the majority of Clojure projects: the quality of the code is very good, and Clojure is the only language I’ve worked with so far where I have been able to learn so much reading the source code of the 3rd party libraries that I rely on).
Let’s start with an interesting comment about Java, on StackOverflow:

Why doesn’t more Java code use PipedInputStream / PipedOutputStream?

http://stackoverflow.com/questions/484119/why-doesnt-more-java-code-use-pipedinputstream-pipedoutputstream

In response to that question, matt_b suggests this is part of the reason why people don’t use pipedinputstream or pipedoutputstream:

“Typically, data is read from a PipedInputStream object by one thread and data is written to the corresponding PipedOutputStream by some other thread. Attempting to use both objects from a single thread is not recommended, as it may deadlock the thread.”

So now, look at how PipedInputStream and PipedOutputStream are used in Aleph:

https://github.com/ztellman/aleph/blob/perf/src/aleph/formats.clj

In particular, look at this function:

(defn channel->input-stream “Consumes messages from a channel that emits bytes, and feeds them into an InputStream.”
([ch]

(channel->input-stream ch “utf-8”))

([ch charset]

(let [out (PipedOutputStream.)
in (PipedInputStream. out 16384)
bytes (map* #(bytes->byte-array % charset) ch)]

(future
(try
(loop []
(when-let [msg @(read-channel* bytes :on-drained nil)]
(.write out ^bytes msg)
(recur)))
(finally
(.close out))))

in)))

I think this is very clean. He uses (future) to put the PipedOutputStream on its own thread, meanwhile he returns a PipedInputStream to whoever is calling this function. This is a good use of PipedInputStream/PipedOutputStream, and he’s ensured they run in separate threads, which answers the concern that matt_b raised on StackOverflow.
On a different subject, I have learned a new style of programming by using closures in a functional language. Maybe I am a bit stupid, but it took me a long time before I began to realize how useful closures are. My only other exposure to closures was with Ruby and Javascript, but I never did concurrent programming with Ruby or Javascript, and it seems to me the greatest use of closures is when you are dealing with a lot of threads. When I first started writing Clojure code, I did not use closures at all. I wrote some clumsy code. For instance, I wrote a function that got some data from the database, and I wrote another function that stored that data in a hashmap in memory (using the memory as a cache) and I wrote another function that would check the hashmap for the data, before making the call to the database, and then I wrote another function that ran every 15 minutes and erased items in the hashmap, thus invalidating the cache. But then I realized I could combine several of these operations by creating a closure at the moment that I store data in the hashmap: the closure could call Thread.sleep(900000) and then delete the data from the hashmap. I found it convenient to create the closure while I was storing the data in the hashmap as, at that moment, I have all the information I need, including the key with which that data is being stored. Then I push the closure onto a queue and it gets processed in 15 minutes. This simplifies things quite a bit.

Again, Zach Tellman’s code has been educating me about how powerful closures can be in the context of concurrent programming. Look here:

https://github.com/ztellman/aleph/blob/perf/src/aleph/http.clj

And look especially at this function:

(defn wrap-ring-handler
“Takes a normal Ring handler, and turns it into a handler that can be consumed by Aleph’s start-http-server. If the Ring handler returns an async-promise, this will be handled properly and sent along as a long-poll response whenever it is realized. This should be the outermost middleware around your function. To use an Aleph handler at a particular endpoint within the scope of this middleware, use wrap-aleph-handler.”
[f]

(fn [ch request]

(run-pipeline request
{:error-handler (fn [ex] (error ch ex))}

;; call into handler
(fn [{:keys [body content-type character-encoding] :as request}]
(if (channel? body)

(if (options/channel-ring-requests? request)

;; leave channels as is
(f (assoc request ::channel ch))

;; move onto another thread, since there will be blocking reads
(task “input-stream-reader”
(f (assoc request
::channel ch
:body (formats/channel->input-stream body character-encoding)))))

(f (assoc request
::channel ch
:body (formats/bytes->input-stream body character-encoding)))))

;; send response
(fn [response]
(when-not (::ignore response)
(enqueue ch response))))))

Here you have a function that returns a function that returns another function. The first returned function calls (run-pipeline) and gives (run-pipeline) a closure that will be called later, but with data that we have right now, in this context.

You are asking why functional programming is a big deal. I’m sure different people will have different answers. I think the difficulty in answering is that there is no one thing that by itself makes an answer, rather, it is a whole series of things that combine in powerful ways. For me, the immutable data structures of Clojure are an education. I have learned a lot. I think using closures in the context of concurrent programming could be an absolute nightmare, if it was not for the safety offered by immutable data structures. And yet, I don’t think immutable data structures are a big deal by themselves, nor do I think that closures are a big deal by themselves, nor do I think concurrency is a big deal by itself. Only when I look at the combination of these 3 do I find myself enlightened, like I’ve just discovered a whole new level of programming, a level of power I have never felt in any other language.

A final note: Adam Bard has a blog post “Effortless async by design” in which he says “I dare you to find a thread-safer language. Well, maybe Erlang.” And then he also uses some code from Zach Tellman to demonstrate what he means. Bard’s post is worth a read:

http://adambard.com/blog/why-clojure-part-2-async-magic/

I could list a lot of things that seem to me very powerful about the coding style I’ve learned this last year: composability, currying, closures, immutable data structures, thread-safe, etc. But I think descriptions like that tend to be a bunch of meaningless buzz words until you’ve dived in deep enough to see how they all interact with each other. Because its the interaction of these features that is really powerful, not any of them alone.

To which someone responded:

Uh, hey, I’m Zach. Thanks for the shout out. I’m mildly uncomfortable with you using Aleph as an example of good, idiomatic Clojure, but I suppose I can live with it.
If you’re looking through my code in the future and have questions, please feel free to get in touch.