Saturday, November 22, 2014

Encrypting and decrypting files with Gnu Privacy Guard (GPG)

Asymmetrical encryption with GnuPG


You should start using asymmetrical encryption for your backups. Here's a little guide.
With asymmetrical key encryption, other hosts and people can encrypt data that only you can decrypt, meaning that you don't have to share a secret key with anyone and you can distribute the public key freely.
At Elastica this is how we do backups. Every host gets preseeded with one of our strong backup keys. This allows them to gobble up sensitive data and store it in places like S3. If our S3 account was ever compromised then the intruder would have a bunch of encrypted files that would take them 20 years of computing power to decrypt.
Since decryption speed is not an issue with cold storage backups, I recommend the maximum 4096-bit RSA key.
Here's a quick guide to creating a strong crypto key and encrypting some files.
Also remember that gpg can be installed on your mac via brew.
Generate the key:
$ gpg --gen-key
(follow the prompts)
If you'd rather do this automatically or in a script, you can feed the options in:
$ gpg --batch --gen-key <<EOF
    Key-Type: rsa
    Key-Length: 4096
    Name-Real: Stephen Wood
    Name-Comment: For Testing
    Name-Email: smwood4@gmail.com
    Expire-Date: 0
    %pubring my_key.pub
    %secring my_key.sec
    %commit
    %echo done
EOF
You'll want to import the private key on any hosts that you want to be able to decrypt data.
$ gpg --allow-secret-key-import --import my_key.key
Remember to securely delete your secret key if you're not going to put the file elsewhere.
$ srm my_key.key

Encrypting and decrypting

Encrypting can be done on any host that has your public key. First, import the public key to the local hosts's gpg key ring
$ gpg --import my_key.pub
To encrypt a single file:
$ gpg --recipient smwood4@gmail.com --encrypt ping.heystephenwood.com.cert
(creates ping.heystephenwood.com.cert.gpg)
Note that the key is referred to as an email address. This is how gpg typically works.
Decrypting the file is easy if you have the private key:
$ gpg --decrypt ping.heystephenwood.com.cert.gpg > ping.heystephenwood.com.cert
By default gpg will decrypt a file to stdout, so we redirect it to a file. You can tell gpg to decrypt it to the original filename by passing in the --use-embedded-filename flag:
$ gpg --use-embedded-filename --decrypt ping.heystephenwood.com.cert.gpg
(creates ping.heystephenwood.com.cert.gpg)
Notice that we didn't even have to specify which key to use. That's because gpg will use imbedded metadata in the file to decrypt the key. You can see this information by passing in the --list-packets flag:
$ gpg --list-packets ping.heystephenwood.com.cert.gpg
:pubkey enc packet: version 3, algo 1, keyid 6FC17A3C5E45F65C
    data: [4096 bits]
:encrypted data packet:
    length: unknown
    mdc_method: 2
gpg: encrypted with 4096-bit RSA key, ID 5E45F65C, created 2014-06-13
      "Stephen Wood <smwood4@gmail.com>"
:compressed packet: algo=2
:literal data packet:
    mode b (62), created 1416686288, name="ping.heystephenwood.com.cert",
    raw data: 2171 bytes
Pretty nifty, huh?

Multiple files

Encrypting multiple files is much easier to do via the tar command. Since gpg files are compressed automatically, it's not even necessary to gzip them!
Here's how to encrypt an entire folder:
$ tar cf - some_folder | gpg --recipient smwood4@gmail.com
(creates some_folder.gpg)
Since gpg defaults to stdout, we simply need to pass that back to tar
$ gpg --decrypt some_folder.gpg | tar xf -
(creates some_folder)

Other fun things

You can also use your private key to sign a message that others can verify for authenticity.
Let's create a document:
cat > my_message.txt <<MESSAGE
Help! The NSA is after me! They've tapped my phone, TV, microwave, and
Nike Air Jordans! Also the fillings in my teeth won't stop playing episodes
of "This American Life" and Ira Glass is getting on my nerves.
MESSAGE
Now let's sign the message:
$ gpg -u smwood4@gmail.com --clearsign my_message.txt 
(produces my_message.txt.asc)
You've got yourself a signed document.
$ cat my_message.txt.asc
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Help! The NSA is after me! They've tapped my phone, TV, microwave, and
Nike Air Jordans! Also the fillings in my teeth won't stop playing episodes
of "This American Life" and Ira Glass is getting on my nerves.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBAgAGBQJUcPACAAoJEKSJEkYj0afvor0QALvrYflk52NkNmwpq09FM+2x
06RKsmzQgXa8W+3yHpTUlAMhVPoToeETp9wrM4zod+6nmiyan/aqP1QzXq0QC+yc
SEOGIn7r69+d1ybCXHnC2ebbB2pVj8NwH30a6hLNUHlmZ9xGjX4bnqUREIo1booV
MnWqUab7PFjEJ97fVW9b47kpZ+Mi0P8MhphGA+QHzGl56UtDo5wPci8A3TKqfCsq
didoYGAJurVnVIwvjXPkwKnKwL4E0AvTRN5iiiN6Rib+be8Pmy8wUNA7LdwjERO1
H8zVCgyFvOzA16PbdmKJF5nnUAlJASO4ZStUD5z/AIYir9cXuNSlx45FFVse3lFu
yyftQkvJPJbzVdg/JaYswj7vMFPPmRBweFzFZjBqrDu+o1aPWougbOFnU3ZDuf3s
4SmbI7gJDDet7cvYHWn+8doGGI5BpPb0gXjfhW+WBuvMZ01GJbXwychi5d4xC1Qi
XYkd4RgdxxIhyLrO+whWYHFZlI3UY6cPPyyiDjTLR/UCP2IrtdL7gcPJicXGSGG3
EJLjG1mK+nk44QT4oW7Ghha6G1oI2YoqjAWrlWhaSyxlNSv4qw75x5w6WtzoL2yR
mayROMUmmHrBYfgFtgk64X+qcd3Li8/InZUw95A5SnV69r2h5p05fPiE70sgu73R
ozOu9eBEjYDTEb4PYS+5
=JiEL
-----END PGP SIGNATURE-----
Anyone can verify the message. They don't even need your public key:

gpg --verify my_message.txt.asc
gpg: Signature made Sat Nov 22 12:29:17 2014 PST using RSA key ID 23D1A7EF
gpg: Good signature from "Stephen Wood <smwood4@gmail.com>"
And what happens when somebody intercepts and alters your original message?

$ cat my_message.txt.asc
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dear friend. All is well here.
-----BEGIN PGP SIGNATURE-----
...
gpg --verify my_message.txt.asc
gpg: Signature made Sat Nov 22 12:20:18 2014 PST using RSA key ID 23D1A7EF
gpg: BAD signature from "Stephen Wood <smwood4@gmail.com>"
Looks like your friend is dead. I'm so sorry.
This can be taken a step further if you've already securely exchanged your public key with your "friend". They can both verify your message hasn't been altered, and also verify that it was sent from you. They just need to verify that the Key ID is the same as the one you've imported.
That's it for now. Encrypt your backups! It's easy to do.

Monday, September 29, 2014

OpenVPN & Raspberry Pi auto-installer

Openvpn server on your Raspberry PI

From wikipedia:
OpenVPN is an open source software application that implements virtual private network (VPN) techniques for creating secure point-to-point or site-to-site connections in routed or bridged configurations and remote access facilities. It uses a custom security protocol that utilizes SSL/TLS for key exchange.
What that means is that openvpn will allow you to safely and securely route your internet traffic through an untrusted network to a trusted one. It does this by encrypting your traffic so nobody can read it until it goes out from your openvpn server.
There are some benefits to this:
  1. Prevent others from snooping on your traffic.
  2. Access websites that are blocked by your work, school, or oppressive governments.
  3. Access assets on your home network from anywhere.
This auto-install script will turn your raspberry pi into an openvpn server so you can browse the internet safely and securely.
Then why use it? Because sometimes you end up on insecure networks (think starbucks, stadiums, etc). This will protect your privacy in those situations. It will not prevent people from finding you if you are stupid and do something illegal.

Running the installer

The auto-installer is completely automated and can be run directly from the web.
From your raspberry pi:
# Set the user
$ OPENVPN_USER='stephen.openvpn.local'

# Run the installer
$ curl "https://raw.githubusercontent.com/stephen-mw/raspberrypi-openvpn-autoinstall/master/bootstrap" | sudo bash
Remember that you should always inspect these types of files before ever running them. You can download it locally and run it like so:
# Download the file
$ wget "https://raw.githubusercontent.com/stephen-mw/raspberrypi-openvpn-autoinstall/master/bootstrap"

# Make sure it's legit
$ less bootstrap

# Execute it
$ chmod +x bootstrap
$ sudo ./bootstrap stephen.openvpn.local

What's the installer do

The installer script will download openvpn and generate all of the necessary root certificates for you. Then it will generate and sign a new certificate for a user. Lastly it will create the ovpn file and place it in /root/client_<some_user>.
All you need to do is download that file into your openvpn client software and you'll be able to safely and securely connect to the host.

Testing

I've included a Vagrantfile for you to run tests. Simply clone the repo and then run:
$ vagrant up
==> default: Forcing shutdown of VM...
==> default: Destroying VM and associated drives...
==> default: Running cleanup tasks for 'shell' provisioner...
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'wheezy64'...
...
==> default: Make sure that your firewall allows incoming UDP connections to port 1194.
==> default:
==> default: The last thing necessary is to securely copy the configuration file over to your
==> default: computer and then load it. The configuration file is located at:
==> default:
==> default:   /root/client_test.ovpn
The vagrant guest will be running the openvpn server. You can pull down the client file and connect to it locally.

Clone the repo

Tuesday, September 23, 2014

Mac OS X time machine backups to a windows or linux samba share

Time machine is a great product from Apple. I used it just recently when I replaced my laptop. All I had to do was select the time machine disk when booting up for the first time and everything was just how I left it. Fantastic!

Creating backups is now very important to me. With a USB hard drive this is very simple, but I wanted to take backups to my home server, where there is plenty of secure space.

Making a backup to network drives (like windows and linux samba shares) is not supported out of the box. I'll show you how to do it.

Setting up the network drive

This guide assumes you have a network-attached drive that's available and mounted. You can use samba to create this drive on either windows or linux.

Creating the sparse disk

A sparse image is an image that grows in size. It's a mac thing. This image will become our dmg file that gets mounted.

Open up up the disk utility. You can do this by searching for it in finder:


  • Give it a name with something you'll remember
  • Before you do anything else, change the image format to "sparse disk image". This will allow us to create a 300GB disk image without allocating all of that space at once.
  • For format select "Mac OS extended"
  • For size select "custom" and put in your desired size. The image will grow to this size.
  • For partition select "single partition -- apple partition map"
  • Save it to your network network mount


You'll see that the disk is 300GB (or whatever size) large but you're only using around 300MB of it.

Next we need to run a command that will force time machine to use our new disk. Find the mount point for the disk. If you select the disk in disk utilities, you can find it at the bottom. In my case it's "/Volumes/stephen_time"


Open up your terminal and paste in the following command:

sudo tmutil setdestination <mount point>

Now open up your time machine preferences and check out the new disk:



Hurray! You've got your disk loaded and ready to use. Now just do the backups as normal. You'll need to remount the disk (find it and double click it) when you want to make backups.

Monday, September 22, 2014

Dyn DNS cli tool

Get the file: https://github.com/stephen-mw/dyncli

Usage

usage: dyncli [-h] [-e ENDPOINT] [-c CREATE] [--cname] [-v VALUE] [-t TTL]
              [-d DELETE] [-u UPDATE_RECORD] [-U USER] [-P PASSWORD]
              [-A ACCOUNT] [-Z ZONE] [-l] [--verbose]

optional arguments:
  -h, --help            show this help message and exit
  -e ENDPOINT, --endpoint ENDPOINT
                        The endpoint to make API requests.
  -c CREATE, --create CREATE
                        Create a new record. Defaults to A record unless
                        --cname is set.
  --cname               Create a cname record instead of an A record.
  -v VALUE, --value VALUE
                        The value to set the DNS record to. Eg 10.0.0.101.
  -t TTL, --ttl TTL     TTL of record in seconds.
  -d DELETE, --delete DELETE
                        Delete an existing record
  -u UPDATE_RECORD, --update UPDATE_RECORD
                        Perform a DNS update on an existing record.
  -U USER, --user USER  The username to connect to the dyn api. Can also be
                        set as DYN_USER in environment.
  -P PASSWORD, --password PASSWORD
                        The password to use with the dyn api. Can also be set
                        as DYN_PW in environment.
  -A ACCOUNT, --account ACCOUNT
                        The account name used to make API requests. Can be set
                        as DYN_ACCOUNT in environment.
  -Z ZONE, --zone ZONE  The zone to take action on. Eg example.com. Can be set
                        as DYN_ZONE in environment.
  -l, --list            List all dns records as a csv file.
  --verbose             Print out api responses.
Dyn is a nice DNS service. It's an alternative to Amazon's Route 53. Though I do not believe that dyn's API is as robust as amazone, with the right set of tools is just dandy.
This tool allows for the manipulation of DNS records via the restful API.
It currently supports the following:
  • Create A and CNAME records
  • Update A and CNAME records
  • Delete A and CNAME records
  • List all of the records as a CSV

Requirements

The requests & argparse packages are required but not currently part of the standard library. Get them with pip:
pip install -r requirements.txt
You can pass your username and password directly into the script, but it's better if you setup a few environment variables. dyncli will take advantage of the following environment variables if they are present:
DYN_USER
DYN_PW
DYN_ZONE
DYN_ACCOUNT
You can activate these by appending them to your ~/.bash_profile:
export DYN_USER='foo'
export DYN_PW='bar'
export DYN_ZONE='example.com'
export DYN_ACCOUNT='example'

Creating a record

Right now both A and CNAME record creation is supported. Create a record uses the -c flag along with the optional --cname flag if that's your desired record type.
The default record type is an address (A) record.
# Create an A record with default TTL
$ ./dyncli -c myfqdn.example.net -v 10.0.0.100

# Create an A record with custom TTL
$ ./dyncli -c myfqdn.example.net -v 10.0.0.100 --ttl 3600

# Create a CNAME record
$ ./dyncli -c myfqdn.example.net -v otherfqdn.example.net --cname

Updating a record

Once records are created, they can be easily updated with the -u or --update flag.
# Update the TTL of an existing record
$ ./dyncli -u myfqdn.example.net --ttl 3600

# Update the value of an existing cname record
$ ./dyncli -u myfqdn.example.net -v someotherfqdn.example.net

Deleting a record

Records are deleted by passing the -d or --delete flag.
# Delete an address (A) record
$ ./dyncli -d myfqdn.example.net

# Delete a cname record (requires the --cname flag)
$ ./dyncli -d myfqdn.example.net --cname

List all records (csv format)

Sometimes it's useful to list all of the records. We use this to do a simple nightly backup of our DNS zone.
The output format is a simple CSV file. Beware that if you have commas in your TXT record, they'll be translated to a pipe (|).
$ ./dyncli --list
zone,ttl,fqdn,record_type,data
example.net,300,test.example.net,CNAME,test2.example.net
example.net,300,foo.example.net,A,172,16.0.100
example.net,3600,example.net,TXT,public_key=sdjflksdj
..
Sometimes it's also useful to check if a zone exists before you go creating one willy nilly
$ ./dyncli --list | grep database
example.net,300,database.example.net,A,172.16.0.100
example.net,300,database.internal.example.net,A,172.16.0.100
example.net,300,database.something.example.net,A,172.16.0.100

(See what happens when you go on creating records without checking first?)
If you notice any bugs please open an issue.

Monday, September 15, 2014

Disposable minecraft servers in a flash with Vagrant

Minecraft Vagrant


Disposable Minecraft Servers

Minecraft is a game where you run around punching blocks and building stuff with your imagination, basically like legos for boys and girls born in the 21st century.
Vagrant is a virtualization service for creating disposable dev environments. Now the two of them are together for running easy, shareable minecraft servers.

What does this do

This is a hands-off, automated minecraft server builder. You can clone this repo on any machine that has vagrant installed to have a minecraft server up and running in minutes.
You can run vagrant up on your windows, linux, or mac machine and have a minecraft server running in a compartmentalized virtual server. When you run vagrant destroy the virtual machine is destroyed but your persistent data is kept in persistent_data.

Why

Initially I created this vagrant config so I could quickly and easily test mods. Just start the vagrant instance and do whatever you want. If you break something or decide you don't want to use that mod any longer, simply destroy the image and the persistent data and start it again. It will be just like new.
Backing up and restoring your minecraft server is very easy, since all of your data -- including the information necessary for running the server -- is contained in just a few files and folders. You can even copy the entire directory over to a new server and run it there.
Lastly, I bet there are parents out there who want to run minecraft servers for their children. This let's you do this easily from just about any platform without having to configure anything.

Running the server

First install Vagrant if you haven't already done that.
Next clone the repo:
git clone git@github.com:stephen-mw/vagrant_minecraft_auto_launch.git
Pop into the vagrant server and run vagrant up
cd vagrant_minecraft_auto_launch
vagrant up
The server will bootstrap and install everything it needs to run minecraft. It will also start the service automatically for you.
After the initial bootstrap you can run vagrant ssh to go into the server. You'll be greated with a MOTD with some helpful information:
WELCOME TO MINECRAFT!

To stop, start, restart or check the status of the the minecraft server
process, simply run:

  $ service minecraft <stop|start|restart|status>

The runtime logs are available at /opt/minecraft/logs

To change the min or max server memory values, make the change in:

  /etc/init/minecraft.conf

And then restart the server proces.

Status of minecraft server:

minecraft start/running, process 2344

Have fun!

Under the hood

The server will launch an Ubuntu 12.04 image that will download and install everything you need to run minecraft automatically. It will also start the service and take care of logs. Everything related to your server will be in persistent_data and that folder will survive the destruction and creation of these vagrant instances.
After the installation completes, you and others will be able to connect to your server at your host's ip address. That means if you launch the vagrant instance on a laptop that has the ip address of 192.168.0.101, then you can connect to the server at the same address (regardless of what your guest ip address is).
The instance itself doesn't need to be destroyed between runs. You can also halt it. Either way, when you bring it back online it should be running the server.

How to get it

Sunday, September 14, 2014

Minecraft server upstart script

I see a lot of people running minecraft server in strange ways: nohup, bg & disown, tmux, screen, etc. The problem with those options is that they are all ephemeral. If you restart your server you'll need to do it all over again.

Here's a simple upstart script that starts and watches your minecraft server. It runs as a service and works for centos 6.5+ and ubuntu 12.04+

Once you create the file at /etc/init/minecraft.conf, simply run:

initctl reload-configuration
service minecraft <start|stop|status>

upstart script:
# Minecraft upstart script

start on (local-filesystems and net-device-up IFACE!=lo)
stop on [!12345]

respawn

chdir SOME_MINECRAFT_DIRECTORY

exec java -Xmx1024M -Xms1024M -jar minecraft_server.1.8.jar nogui

Logs for the server will be rotated by the server and are available at /var/log/upstart/minecraft.log

Monday, July 28, 2014

ec2list -- quickly and easily list ec2 instances across all accounts and regions

Here's a quick tool I cooked up after working in with multiple AWS accounts in many regions. Sometimes I just need a particular piece of information about a host and I only had its instance id. Before this script I would have to run multiple aws calls in different regions until I found the instance.

The script itself is a wrapper around the aws cli tools. It will gather a list of instances in all accounts you specify and in all of the regions. It's multithreaded and I find it finishes in about 4 to 5 seconds. You'll get better performance if you comment out regions that you're not using.

Just clone the repo and execute.

Usage

usage: ec2list [-h] [-i INSTANCES] [-v] [-a] [-f FILTER]

optional arguments:
  -h, --help            show this help message and exit
  -i INSTANCES, --instances INSTANCES
                        Comma-separated list of instances to filter results.
  -v, --verbose         Print all host metadata.
  -a, --all             List all hosts, even those that aren't running.
  -f FILTER, --filter FILTER
                        key value filters to pass directly to command. Eg:
                        Name=instance-type,Value=m1.small

Prerequisites

You must have a $HOME/.aws/config file setup with the proper credentials.
An example profile:
[profile production]
aws_access_key_id=AKIAFOO
aws_secret_access_key=grzOgbar
region=us-west-1

[profile staging]
aws_access_key_id=AKIAIFOO
aws_secret_access_key=BAYS232bar
region=us-west-1
You can test that the different profiles are active by running the aws cli tool
aws ec2 describe-instances --profile=production
aws ec2 describe-instances --profile=staging

Listing all instances in a succinct manner.

The columns when run this way will always be predictable (unlike -v). Useful if you're using awk or other things.
$ ./list
i-7e15d971 stephen_test_001 10.0.0.100 54.183.84.15 m3.large production
i-7e15d971 stephen_test_002 10.0.0.100 54.183.84.16 m3.large staging
...

List instances that are also stopped

By default only running instances will be shown. Use the -a or --all flag to show all instances, even those that are stopped.
$ ./list -a

Be verbose 

Verbose prints everything returned by ec2. The return values are unordered.
./list -v
i-7e15d971:
  Monitoring: {"State": "disabled"}
  PublicDnsName: ec2-54-131-88-95.us-west-2.compute.amazonaws.com
  RootDeviceType: ebs
  State: {"Code": 16, "Name": "running"}
  EbsOptimized: False
  ...
i-bba5d6e6:
  Monitoring: {"State": "disabled"}
  PublicDnsName: ec2-54-210-162-78.us-west-1.compute.amazonaws.com
  RootDeviceType: ebs
  State: {"Code": 16, "Name": "running"}
  EbsOptimized: False
  ...

Filter by a list of instance IDs

To easily filter instances by instance id, they can be passed as a comma-separated list with the -i or --instances flag.
./list -i i-bba5d6e6,i-cad1b0c2,i-afa406f2

Print only instances that are an m1.small, or other things

Note that the filters arg allows you to pass filters down directory to the underlying aws ec2 command. See the ec2 api for information on the types of filtering you can do.
$ ./list -f Name=instance-type,Values=m1.small