Instant dashboard with the ELK stack?

(written by lawrence krubner, however indented passages are often quotes). You can contact lawrence at: lawrence@krubner.com, or follow me on Twitter.

I need to build some dashboards for a business. I am impressed with Riak but I would have to find some way to build the dashboard. In theory, the ELK stack gives an almost instant dashboard. I am not at all interested in Logstash, but the rest of this sounds interesting:

We’ve built a data analysis and dashboarding infrastructure for one of our clients over the past few weeks. They collect about 10 million data points a day. Yes, that’s big data.

My highest priority was to allow them to browse the data they collect so that they can ensure that the data points are consistent and contain all the attributes required to generate the reports and dashboards they need.

I chose to give the ELK stack a try: ElasticSearch, logstash and Kibana.

ElasticSearch is a schema-less database that has powerful search capabilities and is easy to scale horizontally. Schema-less means that you just throw JSON at it and it updates the schema as you go. It indexes every single field, so you can search anything (with full-text search) and it will aggregate and group the data. Registering a new node to a cluster is a matter of installing ElasticSearch on a machine and editing a config file. ElasticSearch takes care of spreading data around and splitting out requests over multiple servers.

logstash allows you to pipeline data to and from anywhere. This is called an ETL (for Extract, Transform, Load) pipeline in the Business Intelligence and Data warehousing world, and it is what allows us to fetch, transform, and store events into ElasticSearch.

Kibana is a web-based data analysis and dashboarding tool for ElasticSearch. It leverages ElasticSearch’s search capabilities to visualise your (big) data in seconds.

Post external references

  1. 1
    http://brewhouse.io/blog/2014/11/04/big-data-with-elk-stack.html
Source