July 21st, 2016
(written by lawrence krubner, however indented passages are often quotes). You can contact lawrence at: firstname.lastname@example.org
For my purposes, for the next 6 months, it seems that Kestrel will be all I need. I’d have to be very successful before I would need anything as complex as Kafka.
The biggest conceptual difference is that Kestrel is a simple stand alone queue where as Kafka is a full fledged queuing system.
Kestrel runs on a single machine and has no concept of clustering or failover or any other features you might expect in a queuing system. Instead the clients need to be aware of the kestrel servers and balance or partition the messages across the servers. The simplicity of this system has some great benefits and is not unlike how you would partition and balance against memcached or redis.
Kafka on the other hand adds a lot of smarts to the clients which are all coordinated using Zookeeper. Zookeeper keeps track of which partitions of data goes to which Kafak node and can also coordinate multiple consumers collaborating to process a collection of messages. Additionally, as of recent releases, Kafka can replicate data and provide a better fail over story. One other very interesting feature of Kafka is that it stores the data for a configurablely long time so that you can replay messages easily if a process crashes or had a bug and needs to reprocess messages. This is an interesting difference from Kestrel and most other messaging systems which delete messages once they are properly acknowledged.