Installing Raspberry Pi on SD card on mac or linux

If you're on mac or linux, installing the raspberry pi operating system on a sd card is quick and easy. Here's the instructions for doing it using the tool dd.

Warning: with one wrong typo you can wipe out your entire hard drive with the command dd. Be very careful with the dd step that you're pointing it at your sd card.

We'll be doing this all in the terminal. Open up your terminal by typing "command + space" then typing in terminal if you're on a mac. If you're using ubuntu it's "ctrl + t".

Download the newest raspberry pi image and unzip it

 $ cd ~/Downloads  
 $ wget http://files.velocix.com/c1410/images/raspbian/2013-02-09-wheezy-raspbian/2013-02-09-wheezy-raspbian.zip .  
 $ unzip 2013-02-09-wheezy-raspbian.zip 

Find out what the device name is, and then unmount it. Ignore the "s1" part after the device name if you're on a mac. In this case it's "/dev/disk1".

$ df -lh  
 Filesystem   Size  Used Avail Capacity iused  ifree %iused Mounted on  
 /dev/disk0s2 233Gi  97Gi 135Gi  42% 25586022 35483418  42%  /  
 /dev/disk1s1  56Mi  19Mi  37Mi  34%   512    0 100%  /Volumes/Untitled  
   
 $ diskutil umount /Volumes/Untitled  

Send the data using the DD command.

WARNING: Be very careful with this step. The "of=" MUST be the device name for the sd card. Measure twice and cut once. There's a reason sysadmins sometimes refer to dd as "disk destroyer."

$ sudo dd if=2013-02-09-wheezy-raspbian.img of=/dev/disk1 bs=1  

You can check the progressing by sending a "SIGINFO" (ctrl + t):

load: 0.85 cmd: dd 24671 uninterruptible 31.47u 174.72s  
 112168961+0 records in  
 112168960+0 records out  
 112168960 bytes transferred in 410.701456 secs (273116 bytes/sec)  

That's it! You're done. 

Raspberry Pi as an SSH Tunnel Gateway

The Raspberry Pi is a neat little Linux box that costs $25 and is the size of a credit card. I've been tinkering with mine for about a month now. I know a few people who bought them and are wondering what they can do with it, so I'm going to do a series called "10 slices of Pi"

In this first post, I'm going to show you how to set up your Raspberry Pi to tunnel SSH traffic through your home network. This will help you browse privately and securely from behind firewalls and public wireless hotspot.

Why tunnel SSH traffic?

There are a few good reasons you may want to tunnel your web traffic through your home.

By tunneling your traffic through SSH, it is encrypted to any prying eyes that may be hoping to sniff passwords and sensitive information on a public wireless hotspot. 

Encrypting your traffic defeats website-blocking firewalls at your school, work, or oppressive country.

Requirements

1. An "always-on" SSH server on a trusted network (like your home). This is what the raspberry pi will become.
2. An SSH client on the computer or laptop you wish to do the tunneling (mac has a native ssh application, and for windows you can use PuTTY.
3. Firewall rules that allow you to access your open-ssh server from the outside.
4. A browser that supports SOCKS proxies.

Setting up the Raspberry Pi

Fortunately, there's very little setup we'll actually do on the RPI. Most of what will take place will happen on your router or laptop. The first thing we'll need to do is make sure that your openssh-server is running and accepting connections.

If you're logged into your RPI, simply check if openssh is listening on port 22

$ nc localhost 22

SSH-2.0-OpenSSH_6.0p1 Debian-3

Looks like it is! If it wasn't listening, netcat would simply hang up on you. If openssh isn't listening, you can run the config again to enable the server.

$  sudo dpkg-reconfigure

Give the Raspberry Pi a static IP

Before you go on to the port forwarding step, you'll want to take a moment to give your RPI a static IP on the network. Since it's a regular linux computer the steps are the same. 

$ sudo vi /etc/network/interfaces

    auto eth0
    iface eth0 inet static
        address 192.168.1.3
        netmask 255.255.255.0
        gateway 192.168.1.1

$ ifdown eth0$ ifup eth0

Granting Access To Port 22

The next step is to make sure you can access your raspberry pi from the outside world. This is where it gets tricky. Most everyone has their computers protected from the outside world via the firewall on their router, but we need a way to snake our way through the firewall and onto our RPI. For purposes like this, routers come with a feature called "port forwarding" which allows you to route specific traffic to a particular host.

Settings on your router

To log into your router, typically you type in 192.168.1.1 or 192.168.0.1 into your browser window. You'll find port forwarding usually around access control. 

Since port 22 is a privileged port and is frequently scanned. I'm going to pick a random and unprivileged port (those above port number 1024).

In this example, we're forward external port 7000 from the outside to internal port 22 on the RPI server. 

This means when you connect to your router on port 7000, it will send that traffic to your rpi on port 22. You'll need to forward it to your raspberry pi internal network address. 

Here's what my home network settings look like:

Put in your appropriate settings and restart your router. Now it's time to test if it's listening properly to the outside world. You can use the port forwarding tester over at yougetsignal.com

Setting up your SOCKS proxy

Now you need to connect to your RPI using your SSH client to establish a SOCKS proxy. 

Directions for doing that in Windows using PuTTY can be found here.

If you're on a mac you can look at my previous post here: How to SSH Tunnel on a Mac.

Setting up your browser

Our last step is to set up our browser to use the new SOCKS proxy for all web traffic. I usually keep a copy of portable firefox just for this. You can even stash firefox and portable PuTTY on a flash drive and carry it around with you.

In Firefox, you can adjust the settings by doing the following:

1. Go to preferences, and select the "Advanced" tab.
2. Select "Network" button and choose "Settings"
3. Select "Manual proxy configuration" and under "SOCKS Host", you're going to put your local loopback address (127.0.0.1) and the port you selected when you opened up the proxy 

You are now relaying your internet traffic through your raspberry pi as a proxy. 

Caveats

This will forward your web traffic (port 80, 443), but will not forward DNS queries (port 53). Your DNS server could be keeping a log, so your browsing isn't completely private. 

Macbook Pro Displayport Audio Out

I was unable to get digital audio out working with my macbook pro. The fix was simple but took some digging. It turns out out that my receiver didn't support the format that the mac osx outputted by default. It's pretty easy to test and change this if you're having trouble.

If you're having trouble getting audio to work with your displayport adapter, try this:

Open up the audio settings first and make sure that displayport is set as the audio out source.

Next open up Audio MIDI Setup and make sure that the format is something that your receiver can handle. I had to change the bitrate from 48 to 44 Hz. Play around with it.

Bootstrap your Raspberry Pi with a Bash Script

I love my raspberry pi. I use it for fun projects. It's inexpensive, easy to use, and fun! Coincidentally, it also makes an easy dev environment for small projects.

I like to treat mine as a disposable test bed for different things I'm working on. I find myself doing repetitive configuration changes to the pi, so I've instead decided to script all the initial installation stuff.

Here's my setup script for all to use and enjoy. After my raspberry pi finishes it's general config screen, bootstrapping it is very easy (and interactive). If you have any comments, feel free to post them on Git or right here on the blog.

You can see the raw or full mode on github here.

If you'd like to run it and test it yourself, just run from root:

$ curl -q "https://raw.github.com/stephendotexe/raspberrypi/master/configuration/setup_pi_interactive" > ~/setup_pi_interactive 

(open up the file -- never run anything from the web as root without first looking it over)

$ ~/setup_pi_interactive


Let's break it down:

#!/bin/bash

# Interactive Raspberry Pi setup script
# Author: Stephen Wood (www.heystephenwood.com)

# Die on any errors
set -e 

if [[ `whoami` != "root" ]]
then
  echo "Script must be run as root."
  exit 1
fi

# Variables for the rest of the script
echo -n "Choose a hostname: "
read NEW_HOSTNAME
echo -n "User: "
read NEW_USER
echo -n "Password for user (leave blank for disabled): "
read PASS_PROMPT
echo -n "Paste public key (leave blank for disabled): "
read PUBLIC_KEY

I left this interactive for the sake of anyone else running the script. On my own personal version, I set this variables by hand and run it straight off, which is nice because you can do it in one line (curl "http://script" | bash), just make sure it's securely hosted locally if you are putting passwords in it.

apt-get -y update
apt-get -y upgrade

# Install some base packages
apt-get install -y --force-yes dnsutils g++ gcc ipython \
make ntp python python-pip vim vlc

Install whatever packages you want. I find this to be ones I like on every system.

# Update hostname
echo "$NEW_HOSTNAME" > /etc/hostname
sed -i 's/pi/$NEW_HOSTNAME/' /etc/hosts

# Set VIM as the default editor
update-alternatives --set editor /usr/bin/vim.basic

# Add user and authorized_keys
if [[ "$PASS_PROMPT" = "" ]]
then
  useradd -b /home --create-home -s /bin/bash -G sudo $NEW_USER
else
  useradd -b /home --create-home -s /bin/bash -G sudo $NEW_USER -p `echo "$PASS_PROMPT" | openssl passwd -1 -stdin` 
fi

Personally, I recommend you do SSH key-pairs for everything, but sometimes a password is just easier.

# Remove Pi user's password
passwd -d pi

The whole world knows the default password to this user, so disable the password or the user, your choice. I remove the password here. If you want to delete the user, you won't be able to do it with sudo on the first run.

if [[ "$PUBLIC_KEY" != "" ]]
then
  mkdir -p /home/$NEW_USER/.ssh/
  echo "$PUBLIC_KEY" > /home/$NEW_USER/.ssh/authorized_keys
fi
chown -R $NEW_USER:$NEW_USER /home/$NEW_USER

# Allow users in the sudo group to sudo without password
sed -i 's/%sudo.*/%sudo   ALL=NOPASSWD: ALL/g' /etc/sudoers

# Turn off password authentication 
sed -i 's/#   PasswordAuthentication yes/    PasswordAuthentication no/g' /etc/ssh/ssh_config

# Now for some memory tweaks!
# Remove unnecessary consoles
sed -ie 's|l4:4:wait:/etc/init.d/rc 4|#l4:4:wait:/etc/init.d/rc 4|g' /etc/inittab
sed -ie 's|l5:5:wait:/etc/init.d/rc 5|#l5:5:wait:/etc/init.d/rc 5|g' /etc/inittab
sed -ie 's|l6:6:wait:/etc/init.d/rc 6|#l6:6:wait:/etc/init.d/rc 6|g' /etc/inittab
# Also disable serial console
sed -ie 's|T0:23:respawn:/sbin/getty -L ttyAMA0 115200 vt100|#T0:23:respawn:/sbin/getty -L ttyAMA0 115200 vt100|g' /etc/inittab 

echo "Installation Complete. Some changes might require a reboot."

I got these tweaks from the blog Gordon @ Drogon.

Zabbix Server - query failed duplicate entry solution

After backing up and restoring one of my Zabbix servers, I was getting the following error that was preventing all of my auto-registration from working:

18317:20121211:224348.314 [Z3005] query failed: [1062] Duplicate entry '296101' for key 'PRIMARY' [insert into events (eventid,source,object,objectid,clock,ns,value,value_changed) values (296101,2,3,606,1355265828,0,1,0)]

The Zabbix server worked perfectly fine otherwise, but auto-registration was completely broken. It turns out that the eventid table needs to be cleared if it's full of old hosts, which is very possible if you did a dump and restore. It's pretty simple to clear.

Warning, this should only be done on new servers which you are restoring. You would not want to run this command on an established Zabbix server:

mysql -e zabbix 'delete from events'

Let it purge the events table. This allowed auto-registration to work again properly. Hope that helps somebody else out there.

Easy Command Line Math for Mac and Linux

I found myself going to google frequently for basic math that my feeble brain couldn't handle.

Here's two ways to do it easily on the command line on mac osx and linux variants.  Caveat, floats are  not supported.

Using echo:
$ echo $(( 6 / 2 ))
3
$ echo $(( 6 * 2 ))
12
$ echo $(( 6 / 7 ))
0

exp -- evaluate expression
expr 300 + 9230
9530
$ expr 6 \* 32
192
$ expr 32 \/ 4
8
$ expr 5 + 5.5
expr: non-numeric argument
$ expr 6 \/ 7
0

I found myself calculating epoch timestamps for 5 or 10 minutes in the future. I typically used this as an expiration in some api method. We can use our new tool to calculate this easily:

$ echo $(( `date +"%s"` + 300 )) # Five minutes in the future

You can wrap this up in a bash function for easy use later. In my bashrc I have the following function:

timestamp(){
echo $(( `date +"%s"` + $1 ))
}

Which allows me to call it directly from the command line when I need a new expiration:

$ timestamp 300
1351897187

Now we can put off learning basic math solving skills for a little while longer.

Logrotate fills up root partition

Production machines were dropping dead. A few minutes (or sometimes seconds) before death , I received alerts the root partition was full. Then poof, access to the machine is gone forever.

Logrotate ended up being the culprit. Even though our big log producer (Varnish) stored its access logs on a separate, large partition apart from root, logrotate still uses /tmp to compress logs. Rotation would attempt to rotate a log several times larger than its root partition and then croak.

The solution for us was to double the amount of logs stored but cut the time in half. Rather than try and speed up the amount of time logrotate runs, we just removed the varnish logrotate configuration file from /etc/logrotate.d and ran it in a cron every 30 minutes:

*/30 * * * * /usr/sbin/logrotate -f $LOGROTATECONF > /dev/null

This forces a rotation every 30 minutes using our custom varnish logrotate conf. Within the configuration file we set:

rotate 48

To effectively keep 24 hours of logs available for debugging.