Auto Install Logstash and Kibana on Ubuntu 12.04

What is Logstash?

Logstash is a log indexer built on top of elasticsearch. It aggregates logs from multiple sources and allows you to query them using the Apache Lucene query parser syntax.

Logstash is built on elasticsearch, which allows your data to scale easily. This is an important factor when dealing with bigdata because you never really know how big your logs are going to get. 

Now it's totally automated! Run this script and you'll be shipping and indexing logs in no time.

How it works

Logstash has two parts, the indexer and the server. The indexer works on a specific datasource to collect logs and ship them to the server. The indexer can also be something totally unrelated to Logstash (for example, rsyslogd). 

If you do use logstash to ship logs you can do interesting things, such as mutate them, add tags, or disregard them altogether.

Adding tags to certain types of logs allows you to quickly retrieve them and keep track of trending information.

The server keeps logs in a redis queue until the logs can be drained into elasticsearch. Neither redis nor elasticsearch are required to be on the server, but they are nevertheless required and installed here.

The frontend

While not a direct part of the logstash project, Kibana works on top of logstash to give you visualization and monitoring tools. Kibana also gives you the flexibility to define patterns and filters and then watch the stream for these matches as they happen in realtime.

Setup

The entire setup has been automated. Simply run:

$ sudo ./logstash_server

All of the logstash services (elasticsearch, logstash, and Kibana) will be listening on their default port except Kibana which is running on port 80.

You may want to change the default data directory for Elasticsearch.

For more information or if you found a bug, please visit my github repo for this project here. I've tested the install on fresh installs of Ubuntu 12.04. In production I am indexing about 800 logs per second and it is handling it quite nicely.

Converting EC2 pem file to ssh public key

EC2 supplies an OpenSSL encrypted "pem" private key. Sometimes you just need the ssh public key version of this file.

There are a few ways to do this using openssl and other tools, but I prefer using ssh-keygen to create a public key from a private key.

 ssh-keygen -y -f yourprivatekey.pem  

This will print out the public key from any given private key. Works with EC2 pem files.

Just drop the output of this command into ~/.ssh/authorized_keys and you should be good to go!

Using sources.list.d instead of endlessly appending to sources.list

There's a "new way" to using all of your custom repo lists, and that's to append it to a special fine in the  /etc/apt/sources.list.d/ directory. This helps keep things a tidy and organized.

Simply add any sources you would like to use to a separate file within this directory.

Don't be afraid if you get this message:

N: Ignoring file 'percona' in directory '/etc/apt/sources.list.d/' as it has no filename extension

It just means you need to rename the file and append ".list" to the end of it.

Zabbix mysql blocked by bin_log errors

Zabbix never fails to find new and unusual ways to break. The most recent breakage occured because the housekeeper is running "unsafe" sql. You'll find this repeated a thousand times in the mysql-error.logs:

130617 17:30:35 [Note] The following warning was suppressed 50 times during the last 38 seconds in the error log
130617 17:30:35 [Warning] Unsafe statement written to the binary log using statement format since BINLOG_FORMAT = STATEMENT. The statement is unsafe because it uses a LIMIT clause. This is unsafe because the set of rows included cannot be predicted. Statement: delete from history_uint where itemid=2506343 limit 500

I just decided to disable the housekeeper. It slows performance of the database down anyway. In /etc/zabbix/zabbix_server.conf uncomment the following line:

DisableHousekeeping=1

And then restart Zabbix. You should be good to go. You can also vote on my bug report here.

Setting up a firewall on Your Raspberry Pi

Raspberry Pi Firewall

You have two good options for protecting your raspberry pi with a software firewall. The first is the tried and true iptables. The second is much more easy to use and configure, and that's debian's "ufw" service. I'll show you how to firewall your Raspberry Pi with ufw.

Before we start messing around with firewall rules, I always like to leave myself a backdoor. We're going to continually open up port 22 to our local network. We'll open up a screen session and start a loop. When we're sure everything is good, we'll close our screen session.

You can learn more about the awesome program screen here.

$ apt-get install -y screen
$ screen -S firewall
$ while true; do ufw allow from 192.168.1.0/24; sleep 60;done
 (disconnect from the screen session by type in "ctrl+a d")

Great, now we have a backdoor in case we lock ourselves out. Every 60 seconds our session will try allow every address from 192.168.1.1-255 to access every port on the host. You'll only be locked out for up to a minute. Trust me you do not want to skip this step.

We can use ufw to add different ports. Here's my basic setup.

# Allow port 22 to everyone in the world
sudo ufw allow 22

# Allow all ports on my local network
sudo ufw allow from 192.168.1.0/24

# Allow web ports to everyone
sudo ufw allow 80

sudo ufw --force enable

You can check the status:

$ ufw status
Status: active

To                         Action      From
--                         ------      ----
Anywhere                   ALLOW       192.168.0.0/24
Anywhere                   ALLOW       192.168.1.0/24
80                         ALLOW       Anywhere
22                         ALLOW       Anywhere
80                         ALLOW       Anywhere (v6)
22                         ALLOW       Anywhere (v6)

Now all of the Raspberry Pi's ports are exposed to our local network, but everything else can communicate with port 22 and 80. If you're done making changes to the firewall and are positive you're not locked out, then go ahead and kill the screen loop:

$ screen -r
(ctrl + d once inside the session)

Now you've got every port locked down from the outside but 22 and 80. But your raspberry pi probably isn't yet expose to the public internet. For this to happen we're going to add our Raspberry Pi to the DMZ on our wireless router's firewall.

A firewall DMZ means that every port will be forwarded to this specific host by default. This will make our raspberry pi the first port of entry into our home network. You can connect to it anywhere, and even use your raspberry pi as an ssh tunnel.

You can usually find the dmz settings by logging into your router, which is typically found at 192.168.1.1 or 192.168.0.1.

DMZ Settings for Tomato wireless firmware

Now you can run some external port scans and make sure the ports are actually open. You can use inCloak's tool. Since we opened up every port to our local network, we'll need to use an external port scanner.

Here's the scan on my network, which has my Raspberry Pi in the DMZ.

Success. It looks like port 22 and 80 are open. Everything else is closed off. Now you can "safely" expose your raspberry pi to the public internet.

Up next: Protecting Your Raspberry Pi With fail2ban and SSH Private Keys

Failed to generate PXELinux template: undefined method `template' for nil:NilClass

Having trouble with your foreman installation where machines aren't marked as "built"? You probably deleted the PXE Local Default file and now things silently crash. There's a bug report about it here.

When it happens you'll get a message like this in /var/log/foreman/production.log:

Started GET "/unattended/built" for 10.8.0.205 at Wed Jun 12 19:00:16 -0400 2013
  Processing by UnattendedController#built as
Found swift-store-05.va.moz.local
unattended: swift-store-05.va.moz.local is Built!
Add the TFTP configuration for swift-store-05.va.moz.local
Failed to generate PXELinux template: undefined method `template' for nil:NilClass
Delete the autosign entry for swift-store-05.va.moz.local
Rolling back due to a problem:
Adding autosign entry for swift-store-05.va.moz.local
Delete the TFTP configuration for swift-store-05.va.moz.local
Completed 409 Conflict in 301ms

The solution is to create a template named "PXE Localboot Default". This is case sensitive so don't fat finger it! This template is applied to each host after the build process is complete.

By the way, the build process is completely when you're host sends this:

/usr/bin/wget --quiet --output-document=/dev/null --no-check-certificate <%= foreman_url %>

You can create the file from scratch. The default one usually works fine

DEFAULT menu
PROMPT 0
MENU TITLE PXE Menu
TIMEOUT 200
TOTALTIMEOUT 6000
ONTIMEOUT local

LABEL local
     MENU LABEL (local)
     MENU DEFAULT
     LOCALBOOT 0

Fix for "gzip: stdout: No space left on device"

Just installed a new OS and you're getting that message when you run apt-get update?

Here's one thing that might be causing you grief. Make sure that you don't have your /boot partition set to read only. Installing any package that modifies this directory will fail if you can't write to it. Duh. At least that was my reaction after I realized what I had done.

Simply edit your /etc/fstab entry and remove the "ro" portion of the /boot record. Make a backup first.

Don't be tempted to restart your server to see if the fix works. You're messing with the dang /boot directory after all! Just umount and mount it.

 cp /etc/fstab /etc/fstab.backup  
 vim /etc/fstab (make your changes)  
 umount /boot  
 mount /boot  
 apt-get update