Why Appsflyer gave up on Python and switched to Clojure

(written by lawrence krubner, however indented passages are often quotes). You can contact lawrence at: lawrence@krubner.com


At AppsFlyer we actually started our code base in Python. Two years later this wasn’t enough to handle the growing number of users and requests. We started to encounter issues like one of the critical Python processes taking too long to digest the incoming messages, caused mainly by string manipulations and Python’s own memory management system. Even partitioning the messages amongst several processes and servers could not overcome this. This eventually killed the process and caused data loss – the first ‘Python victim’ was the reporting service.

Taking the functional approach
As these kinds of difficulties accumulated, we had to choose between 2 options:

Rewrite some of our services in C (great performance, but less fun to code) and wrap it with Python interop code (easy to do)

Rewrite some of our services in a programming language more suitable for data processing

It is important to mention at this point, that we took the asynchronous event-driven approach to handle incoming messages, which allows the system to easily scale as traffic grows.

We’ve been toying around with the idea of introducing Functional Programming into the company for some time before the rogue reporting service started failing. It’s a good fit with our way of thinking and architecture, so it was logical to make the change – especially since the reporting service failures encouraged us to make the call already. After deciding to go with it, came the second hurdle, which language should we choose?

Scala Vs. OCaml Vs. Haskell Vs. Clojure
Scala was out of the picture because it’s a hybrid of Object Oriented Programming & Functional Programming and leans more towards OOP. OCaml was discarded because of the relatively small community and the Global Interpreter Lock (GIL) that allows only one thread to execute at a time – even on multicore machines (which was also a problem for us in Python). Monads in Haskell made us cringe in fear, so we were left with Clojure.

But that’s not the only reason we chose this path, Clojure won because of 2 major issues. First, it runs on the JVM and second, it’s a functional language with easy access to a mutable state if you need it, even in a heavily concurrent environment.