Wednesday, October 2, 2013

Logstash 1.2 and Kibana2 auto-install for Ubuntu 12.04

Logstash and Kibana auto-install

Last time I brought you the auto-install script for logstash and kibana 2. I'm updating the install now to work with logstash 1.2 and Kibana 3.

You can read through the change logs if you're really curious about what's different between the two. The biggest change is the facelift in the Kibana project, which looks quite nice.

This auto-install will automatically install everything you need to start using logstash and kibana on ubuntu 12.04. Simply clone the project and run:

$ ./bootstrap

If you notice any issues please feel free to comment here or to head over to the github page and open up an issue.

About the project

My goal was to make the install as easy and automated as possible. You may want to adjust the settings (like the memory allowed by logstash), but overall the settings are general enough to be good for most folks.

I've opted to use nginx to host Kibana, but it's not hard to swap that out with apache if that's your thing.

A note about Kibana

You may notice that you're unable to connect to your Kibana instance to elasticsearch. They've changed the frontend to use client-side javascript, which needs the fully qualified domain name of you elasticsearch instance. This means your browser will basically make the request to ES and not the server running Kibana. Don't get confused. If you can't connect then take a look at the following:
  1. Verify the FQDN of your elasticsearch host is in config.js. You cannot use localhost here or your browser will make a request to localhost.
  2. Your browser has permissions to request information from ES.


  1. Stephen, this script is awesome. It's saved me a load of time spinning up a basic logstash server.

    I have one question. in your default logstash indexer.conf, There's a comment that says:
    # This takes information straight from redis and loads it into elasticsearch.
    # It assumes you've already done any groking or mutating.

    Does that mean I can't add logstash filters into this file? And if not where else should they go?

    1. PS I'm using beaver as my log shipper:

    2. Jay I'm glad you found it useful. You can definitely add mutations and filters to this indexer. Personally, I prefer to do all my mutating on the _shipping_ machines, which saves a little bit on the CPU of the final indexer.