Tuesday, June 4, 2013

Use your raspberry Pi as a DNS cache to speed up your internet

I want my internet to go faster

Here's something you can do with your raspberry pi that will make your internet experience faster.

Set up your raspberry pi to cache DNS queries so that they can be answered locally in a fraction of time and ditch your slow ISP domain name server.

If you'd rather not read the windy explanation of what this is or how this works, you can skip all the way down to the bottom for the script example, or simply run:

$ curl "https://raw.github.com/stephen-mw/raspberrypi/master/roles/dnsmasq_server" | sudo sh

But never run a command like that without first checking the source file.

A little background on DNS

When you type in "www.heystephenwood.com" into your browser and hit enter, there's a little transaction that goes on under the hood. In order to visit the website, your machine must translate the alphanumeric website name into a series of numbers called an IP address. It accomplishes this by sending the query to a domain name server.

The whole trip can take anywhere between 1 ms to 250 ms or more depending on your connection. This happens every time you visit a url (depending on your browser).

You can test it yourself using the tool dig.

  $ dig heystephenwood.com   
  ; <<>> DiG 9.7.6-P1 <<>> heystephenwood.com   
  ;; global options: +cmd   
  ;; Got answer:   
  ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42972   
  ;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 0   
  ;; QUESTION SECTION:   
  ;heystephenwood.com. IN A   
  ;; ANSWER SECTION:   
  heystephenwood.com. 14359 IN A 64.90.54.29   
  heystephenwood.com. 14359 IN A 216.239.38.21   
  heystephenwood.com. 14359 IN A 216.239.36.21   
  heystephenwood.com. 14359 IN A 216.239.34.21   
  heystephenwood.com. 14359 IN A 216.239.32.21   
  ;; Query time: 130 msec   
  ;; SERVER: 192.168.0.1#53(192.168.0.1)   
  ;; WHEN: Sun Feb 10 22:09:26 2013   
  ;; MSG SIZE rcvd: 116  

Dig is a very powerful and fun tool. Here you'll see the question and answer. The important part is right in the middle:

  ;; Query time: 130 msec   

That query took 130 ms. That means from the time I hit "enter" on my browser to the point where I was actually able to fetch and display a webpage there was a 120 ms pause while I waited to get the server's address. Those little pauses add up!

We're going to make our internet experience faster by storing these records in memory on our own DNS cache so that they can be fetched nearly instantaneously. The best part of this is that the cache is shared for all users, so the more people you have browser the web at your house the bigger and better the cache.

Introducing dnsmasq

Dnsasq is a very lightweight dns and dhcpd server. The benefit of dnsmasq is that it's fast, very easy to use, and offers the ability to use both dns and dhcpd services under one roof. The raspberry pi makes for a perfect vessel for a DNS server because of its low power consumption and easy setup.

Testing DNS query times

These next steps are optional but make the transition to your own raspberry pi dns server more more fun. We'll run the loop below to get a good working baseline for our DNS speed. You can run this from your raspberry pi or from a linux or mac osx box.

In this example I'm using the default Comcast public DNS server which they supply. Here's a bash loop without any averages:

 $ for i in {1..30}; do dig heystephenwood.com | grep time; done   
  ;; Query time: 159 msec   
  ;; Query time: 17 msec   
  ;; Query time: 23 msec   
  ;; Query time: 18 msec   
  ;; Query time: 156 msec   
  ;; Query time: 93 msec   
  ...  
  ... 

Doing it serially can take awhile, so let's pipe the whole thing into xargs and run it in parallel of sets of 10:

$ for i in {1..30}; do echo heystephenwood.com;done | xargs -I^ -P10 dig ^ | grep time 

Let's average them and get a baseline, because what's the point in trying to optimize a system if we can't actually see if it's improved? We're going to run the same command but we'll pipe it to awk to sum the columns for us to grab our mean.

  $ for i in {1..30}; do echo heystephenwood.com; done | xargs -I^ -P10 dig ^ | grep time | awk /time/'{sum+=$4} END { print "Average query = ",sum/NR,"ms"}'   
  Average query = 41.3 ms  

Let's take a look at just how far our packets need to travel to our DNS server:

$ traceroute 75.75.75.75   
  traceroute to 75.75.75.75 (75.75.75.75), 64 hops max, 52 byte packets   
  1 192.168.0.1 (192.168.0.1) 2.989 ms 0.927 ms 0.982 ms   
  2 73.97.112.1 (73.97.112.1) 10.238 ms 8.582 ms 10.133 ms   
  3 te-0-0-0-7-ur08.seattle.wa.seattle.comcast.net (68.87.207.89) 14.287 ms 19.168 ms 10.108 ms   
  4 ae-20-0-ar03.seattle.wa.seattle.comcast.net (69.139.164.129) 24.351 ms 12.557 ms 14.335 ms   
  5 he-1-6-0-0-10-cr01.seattle.wa.ibone.comcast.net (68.86.94.69) 15.803 ms   
   he-1-8-0-0-11-cr01.seattle.wa.ibone.comcast.net (68.86.95.249) 15.441 ms 23.240 ms   
  6 so-6-1-0-0-ar03.troutdale.or.bverton.comcast.net (68.86.90.214) 14.503 ms 16.077 ms 14.468 ms   
  7 te-7-4-ur01-d.beaverton.or.bverton.comcast.net (68.87.216.41) 15.748 ms 17.018 ms 18.251 ms   
  8 cdns01.comcast.net (75.75.75.75) 15.407 ms 23.341 ms 14.946 ms   

In this case it's 8 hops. Generally speaking, the less hops a packet needs to travel to a destination the better. By using a cache, we'll only need to make this trip once, then we'll store the record for quick retrieval.

Let's talk cache

Installing dnsmasq is easy on your raspberry pi:

 $ sudo apt-get install -y dnsmasq  

Stop the service so that we can tinker with its configurations

 $ sudo service dnsmasq stop   

Let's open the conf file and get the settings we want. For reference I've included my own configuration.

You'll find the configuration at /etc/dnsmasq.conf if you installed from the repo.


# Dnsmasq.conf for raspberry pi
# /etc/dnsmasq.conf
# http://www.thekelleys.org.uk/dnsmasq/docs/dnsmasq.conf.example

# Set up your local domain here
domain=raspberry.local
resolv-file=/etc/resolv.dnsmasq
min-port=4096
server=8.8.8.8
server=8.8.4.4

# Max cache size dnsmasq can give us, and we want all of it!
cache-size=10000

# Below are settings for dhcp. Comment them out if you dont want
# dnsmasq to serve up dhcpd requests.
# dhcp-range=192.168.0.100,192.168.0.149,255.255.255.0,1440m
# dhcp-option=3,192.168.0.1
# dhcp-authoritative

The "server=" setting is declaring the upstream domain name servers. dnsmasq never actually looks at root hints to resolve an ip address. If it doesn't know the answer, it just asks a different dns (in this case it's google).

I've included the settings for having dnsmasq be the dhcp server as well. This is optional, but it gives you a few extra perks: the dnsmasq dns will be the default, and you can control the lease space.

Note: these settings assume that 192.160.0.1 is your default gateway. If it's not, you'll need to adjust the settings to reflect it.

Don't forget to start the service again

 $ service dnsmasq start  

Test your new dns cache

Let's run that same command earlier. Once the record is retrieved, dnsmasq will store the record locally.

 $ for i in {1..30}; do dig heystephenwood.com | grep time; sleep 1; done | awk /time/'{sum+=$4} END { print "Average query = ",sum/NR,"ms"}'  
 Average query = 2.33333 ms  

From 41 ms down to 2.3? I'll take that!

Set up your computer to use your new DNS server

This part can be accomplished a few different ways. For mac, you can see this guide, or for windows you can see here. I recommend you use your raspberry pi as a dhcp server as well, that way every device on your network will use your fancy new dns server by default.

Script the bootstrap

Anytime you need to do something on your linux server, you should get into two habits: 1) distilling your creation into scripts that can be called over and over again, and 2) storing those configurations in source control.

Let's bootstrap the whole gosh darn thing with a single bash script.

You can call the script simply enough right from your raspberry pi

$ curl "https://raw.github.com/stephen-w/raspberrypi/master/roles/dnsmasq_server" 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
#!/bin/bash
set -e

# Raspberry Pi dnsmasq script
# Stephen Wood
# www.heystephenwood.com
#
# Usage: $ sudo ./raspberrypi_dnsmasq
#
# Net install:
#   $ curl https://raw.github.com/stephen-mw/raspberrypi/master/roles/dnsmasq_server | sudo sh

# Must be run as root
if [[ `whoami` != "root" ]]
then
  echo "This install must be run as root or with sudo."
  exit
fi

apt-get install -y dnsmasq
cat - > /etc/dnsmasq.conf <<DNSMASQCONF
# Dnsmasq.conf for raspberry pi
# By Stephen Wood heystephenwood.com
# Full examples found here:
# http://www.thekelleys.org.uk/dnsmasq/docs/dnsmasq.conf.example

# Set up your local domain here
domain=raspberry.local
resolv-file=/etc/resolv.dnsmasq
min-port=4096
server=8.8.8.8
server=8.8.4.4

# Max cache size dnsmasq can give us, and we want all of it!
cache-size=10000

# Below are settings for dhcp. Comment them out if you dont want
# dnsmasq to serve up dhcpd requests.
# dhcp-range=192.168.0.100,192.168.0.149,255.255.255.0,1440m
# dhcp-option=3,192.168.0.1
# dhcp-authoritative

DNSMASQCONF

service dnsmasq restart

echo "Testing dns performance with random urls"

# We'll generate a list of urls that we're moderately certain doesn't exist in our cache to get a good base line for speed increases.
URLS=`for i in {1..50}; do echo www.$RANDOM.com;done`

# Make the requests in parallel
echo $URLS | xargs -I^ -P50 dig @127.0.0.1 grep time | awk /time/'{sum+=$4} END { print "average response = ", sum/NR,"ms"}'
echo $URLS | xargs -I^ -P50 dig @127.0.0.1 grep time | awk /time/'{sum+=$4} END { print "average response = ", sum/NR,"ms"}'
echo $URLS | xargs -I^ -P50 dig @127.0.0.1 grep time | awk /time/'{sum+=$4} END { print "average response = ", sum/NR,"ms"}'

echo 'Installation complete. Enjoy!'







20 comments:

  1. Replies
    1. I'm not sure if windows has a DNS cache or how it functions, but there would still be upsides to using a central dns on your home network. Besides configurability, you can leverage the opportunity to use a cache that someone else on your network has warmed up :)

      Delete
  2. I keep getting this in my syslog...what can i do?
    23:17:06 raspberrypi dnsmasq[902]: failed to access /etc/resolv.dnsmasq: No such file or directory

    ReplyDelete
    Replies
    1. You may be running a new version of dnsmasq that requires this file. I would just created it

      $ touch /etc/resolv.dnsmasq

      Delete
  3. Justed wanted to add for those toying around with performance the command to specify a destionation DNS server becomes:
    for i in {1..30}; do echo google.fr; done | xargs -I^ -P10 dig @8.8.8.8 ^ | grep time | awk /time/'{sum+=$4} END { print "Average query = ",sum/NR,"ms"}'

    ReplyDelete
    Replies
    1. Thanks, it worked to check the differences b/w 8.8.8.8 and the pi

      Delete
  4. Nice tutorial, but it doesn't seem to be working. When I do testing (removed the averaging), seems I should see the first look-up be longer, but the 2nd and later should all be short, right?

    # for i in {1..30}; do echo msnbc.com; done | xargs -I^ -P10 dig ^ | grep time
    ;; Query time: 80 msec
    ;; Query time: 42 msec
    ;; Query time: 76 msec
    ;; Query time: 38 msec
    ;; Query time: 5 msec
    ;; Query time: 2 msec
    ;; Query time: 47 msec
    ;; Query time: 55 msec
    ;; Query time: 39 msec
    ;; Query time: 67 msec
    ;; Query time: 1 msec
    ;; Query time: 1 msec
    ;; Query time: 1 msec
    ;; Query time: 8 msec
    ;; Query time: 1 msec
    ;; Query time: 3 msec
    ;; Query time: 2 msec
    ;; Query time: 1 msec
    ;; Query time: 1 msec
    ;; Query time: 1 msec
    ;; Query time: 1 msec
    ;; Query time: 1 msec
    ;; Query time: 1 msec
    ;; Query time: 8 msec
    ;; Query time: 7 msec
    ;; Query time: 1 msec
    ;; Query time: 1 msec
    ;; Query time: 1 msec
    ;; Query time: 1 msec
    ;; Query time: 1 msec

    So not sure where to look now. Any ideas?

    ReplyDelete
    Replies
    1. Not unless the first 10 reads were concurrent (see the -P10 flag).

      Delete
    2. I recognise this is an old thread, but the reason you are seeing these results is xargs runs the queries in parallel batches of 10. As a result, the first 10 queries are all run before the first one returns a result that dnsmasq can cache, so they all have to go out to your offsite DNS server. You'll see that the subsequent queries (11-30) all have much shorter (<10ms) response times. The cache is working as intended.

      Delete
    3. It's working fine - the "-P10" switch in xargs causes it to run 10 queries in parallel, which means the first 10 queries are run against a cold cache. After that, the cache is warmed up and the subsequent queries run in <10ms.

      Delete
  5. Thanks for the taking the time to share. I used your guide to successfully setup a DNS cache. My only concern is that it seems the cache is cleared after dnsmasq is restarted. What could be the reason for this?

    ReplyDelete
    Replies
    1. Cache is stored in memory and memory is not in a committed state. When you save a file, it saves that information to a hard drive, it persists that information while the computer is not online. When in cache, the information lives in the ram, if the power goes out, all of the things that lived in ram are cleared out. Likewise, when you have too many things in cache, you'll see things leave it and if it is used again, it is not cached any longer and must re-cache.

      Delete
    2. I have the same issue. The cache resets when the service is restarted. I looked in all the different files dnsmasq.leases, resolv.dnsmasq, and resolv.conf and I do not see anything in the files, they are all empty.

      Delete
  6. This looks great. Will tryit at home! Just one question, can I add my own entries, so I can give local devices an address? Also can I override a dns entry? Specifically the internet connectivity check that Android does?

    ReplyDelete
  7. Thanks, It seems it worked for me..

    ReplyDelete
  8. Hey Stephen, just tried running your curl script but it seems to be out of date. Does it still exist somewhere? I'd love to get dnsmasq working but I'm new to Linux and doing all the config with one click is super useful for me.

    Thanks!

    ReplyDelete
  9. Hey Stephen, just tried running your curl script but it seems to be out of date. Does it still exist somewhere? I'd love to get dnsmasq working but I'm new to Linux and doing all the config with one click is super useful for me.

    Thanks!

    ReplyDelete
    Replies
    1. The script location has been fixed. Sorry it took me almost 5 years :-)

      Delete
  10. Is there an updated curl script to get this up and running? New to Linux and got stuck editing the config file, it would be awesome to have a one click install for this :)

    ReplyDelete