Wednesday, November 18, 2015

List all keys in an S3 bucket using golang

Here's a golang snippet that will list all of the keys under your S3 bucket using the official aws-sdk-go:
package main

import (

func main() {
    svc := s3.New(session.New(), &aws.Config{Region: aws.String("us-east-1")})

    params := &s3.ListObjectsInput{
        Bucket: aws.String("bucket"),

    resp, _ := svc.ListObjects(params)
    for _, key := range resp.Contents {

Sunday, November 15, 2015

What's the difference between a PRNG and a CSPRNG?


What's the different between a pseudo-random number generator and a cryptographically-secure pseudo-random number generator? A recommendation from the NIST.
The NIST ran various tests (defined in a publication known as SP800-22) to determine which PRNG algorithms produced the best output given a certain criteria.
The result and recommendation for these tests can be read in another publication called SP800-90. Skip down to Appendix C (Informative) DRBG Mechanism Selection to see the good stuff.
Three algorithms are determined to be cryptographically-secure pseudo-random number generators:
* Hash_DRBG
So there you go. Those three algorithms are CSPRNG because they've been tested and recommended by the NIST*.
*There are other organizations and tests that determine a PRNG to be a CSPRNG, but the NIST is the 800-pound Gorilla.


It's worth noting that the original publication actually had 4 CSPRNG recommendations. The now infamous Dual_EC_DRBG was quietly removed in the revised publication SP800-9Ar1.
There has been a lot of controversy[1][2] surrounding Dual_EC_DRBG. Whether or not any of the controversy is true, it's clear that trust in the NIST's authority has been undermined by its recommendation of Dual_EC_DRBG. I'm grateful that the NIST has removed the algorithm from its newest revision.
This also serves as a warning to anyone trying to influence these types of publications. The people that design, implement and critique this type of cryptography are smart. Very smart. Smarter than you and me. A flawed implementation will eventually be exposed.

Monday, July 20, 2015

Try patch instead of sed in shell scripts

A script is better seen than sed

I'd like to offer all of the shell scripters out there an alternative to sed replacements: patch.
Patch has several important benefits:
  • Context for all changes
  • Easy backup via the -b flag
  • Easy rollback
  • Easy multi-line replacements
Sed is great but can often times lead to enigmatic behavior. This is especially true when using wildcard expressions or other shell magic. And as the saying goes, code is read more often than it is written, so you should write for readability.
Here's a common type of sed replacement you'll find in any old off-the-shelf shell script:
# Replace some mystery var with this one
sed -i 's/setting_foo: */setting_foo: 5'
This kind of thing makes sense to the author, but the reader is at a disadvantage. What is the original value of setting_foo? What type of setting is it and why change it? These are things that are lost in translation with these commands.
And it can easily and often times be worse:
sed -ie 's/[]\/$*.^|[]/\\&/g' file
Here's a real example of updating a mysql.conf. Since multiple lines potentially start with port, you need to use a regular expression to match the word and whitespace to prevent accidents. Also, it's difficult to maintain the original justification of the file. And who knows what the original values were
# Change the port. Since this argument appears on several lines, we only want to change the first one
sed -ie '0,/^port\s*=\s*/port        = 8000/' /etc/mysql

# Change port using whitespace
sed -i 's/max_allowed_packet             = 16M/max_allowed_packet             = 32M/' /etc/mysql

# Allow more packets
sed -ie 's/^max_allowed_packet\s*=*/max_allowed_packet = 32M/' /etc/mysql.conf
Pretty ugly.
Now compare these same changes with patch
patch -b /etc/mysql.conf <<PATCH
--- mysql.conf_old    2015-07-20 19:53:25.000000000 -0700
+++ mysql.conf        2015-07-20 19:58:28.000000000 -0700
@@ -8,14 +8,14 @@


-port                           = 3306
+port                           = 8000
 socket                         = /var/run/mysqld/mysql.sock


-max_allowed_packet             = 16M
+max_allowed_packet             = 32M
 prompt                         = '\u@\h [\d]> '
 default_character_set          = utf8                                # Possibly this setting is correct for most recent Linux systems

Clean and simple! You can see we've obviously added more lines to the script, but we have context to our changes. We've also used the -b flag to automatically backup the file.

Closing thoughts

Of course this isn't always ideal. A good example is when you don't actually know the value that you're replacing. But nevertheless I think it's a good alternative that should be used when readability is important.

Saturday, February 28, 2015

List running EC2 instances with golang and the aws-sdk-go

Managing multiple AWS accounts can sometimes be tough, even when doing something as simple as matching a private IP address with a hostname.

In the past I used the aws cli tools, but I had to constantly switch both the accounts and the regions when making requests:

# Make it's in the prod-5 account in us-west-1
aws ec2 --profile=prod-5 --region=us-west-1 | grep my_instance

# Okay not that account or region, let's try eu-west-1
aws ec2 --profile=prod-5 --region=eu-west-2 | grep my_instance

Repeat 1x for each account and region

As you can imagine this is extremely time consuming, even when using the CLI tools. I wrote a small tool that will find every single instance you can view using every account available to you (according to your ~/.aws/config). The aggregate results can then be searched.

I choose to use the existing ~/.aws/config file so that it works along side your normal aws cli tools. 

This is a very good use-case for Golang, which has nice concurrency primitives. With five accounts this script (once compiled) will display all of the results in under 1.5 seconds. Not bad.

The result is a space-separated list with some additional values added. It should be easy to find what you're looking for:

$ ./aws_list
i-71930187 prod-manage005 t2.micro prod-account
i-71930187 stage-controller7 t2.medium staging-account
i-71930187 prod-vpn01 m4.large None prod-account
i-71930187 stephen-test-host m3.large test-account

The output is a space-separated file and only "running" and "pending" instances are displayed. For a list of filters or instance attributes consult the official documentation.

package main

import (

func check(e error) {
    if e != nil {

printIds accepts an aws credentials file and a region, and prints out all
instances within the region in a format that's acceptable to us. Currently that
format is like this:

  instance_id name private_ip instance_type public_ip account

Any values that aren't available (such as public ip) will be printed out as

Because the "name" parameter is user-defined, we'll run QueryEscape on it so that
our output stays as a space-separated line.
func printIds(creds aws.CredentialsProvider, account string, region string, wg *sync.WaitGroup) {
    defer wg.Done()

    svc := ec2.New(&aws.Config{
        Credentials: creds,
        Region:      region,

    // Here we create an input that will filter any instances that aren't either
    // of these two states. This is generally what we want
    params := &ec2.DescribeInstancesInput{
        Filters: []*ec2.Filter{
                Name: aws.String("instance-state-name"),
                Values: []*string{

    // TODO: Actually care if we can't connect to a host
    resp, _ := svc.DescribeInstances(params)
    // if err != nil {
    //      panic(err)
    // }

    // Loop through the instances. They don't always have a name-tag so set it
    // to None if we can't find anything.
    for idx, _ := range resp.Reservations {
        for _, inst := range resp.Reservations[idx].Instances {

            // We need to see if the Name is one of the tags. It's not always
            // present and not required in Ec2.
            name := "None"
            for _, keys := range inst.Tags {
                if *keys.Key == "Name" {
                    name = url.QueryEscape(*keys.Value)

            important_vals := []*string{

            // Convert any nil value to a printable string in case it doesn't
            // doesn't exist, which is the case with certain values
            output_vals := []string{}
            for _, val := range important_vals {
                if val != nil {
                    output_vals = append(output_vals, *val)
                } else {
                    output_vals = append(output_vals, "None")
            // The values that we care about, in the order we want to print them
            fmt.Println(strings.Join(output_vals, " "))

func main() {
    // Go for it!

    // Make sure the config file exists
    config := os.Getenv("HOME") + "/.aws/config"
    if _, err := os.Stat(config); os.IsNotExist(err) {
        fmt.Println("No config file found at: %s", config)

    var wg sync.WaitGroup

    file, err := ini.LoadFile(config)

    for key, values := range file {
        profile := strings.Fields(key)

        // Don't print the default or non-standard profiles
        if len(profile) != 2 {

        // Where to find this host. The account isn't necessary for the creds
        // but it's something we expose to users when we print
        account := profile[1]
        key := values["aws_access_key_id"]
        pass := values["aws_secret_access_key"]
        creds := aws.Creds(key, pass, "")

        // Gather a list of all available AWS regions. Even though we're gathering
        // all regions, we still must use a region here for api calls.
        svc := ec2.New(&aws.Config{
            Credentials: creds,
            Region:      "us-west-1",

        // Iterate over every single stinking region to get a list of available
        // ec2 instances
        regions, err := svc.DescribeRegions(&ec2.DescribeRegionsInput{})
        for _, region := range regions.Regions {
            go printIds(creds, account, *region.RegionName, &wg)

    // Allow the goroutines to finish printing

Wednesday, February 25, 2015

Generate a unique, strong password on the command line (linux, mac osx)

Find yourself generating a lot of random passwords? Here's a way to generate quick, random, and secure passwords on the command line:
echo $(head -c 64 /dev/urandom | base64) $(date +%s) | shasum | awk '{print $1}'
This command will read 64 bytes of random data from /dev/urandom, base64 encode it, add a small salt (the current data in epoch time), and then create a sha1 hash of the data.
I like this because it's cryptographically secure and the chance of a collision (provided your PNRG isn't totally borked), is infinitely small. It's also a hexadecimal string, so I don't have to worry about quoting it in weird ways or escaping special characters. I can just double-click it in iterm and it's automatically added to my clipboard!

Go ahead and double click the shas below and then click the password from 1password. You'll know what I'm talking about.

The drawback being you can't possibly remember these passwords unless you're US memory champion Nelson Dellis, but you use a password manager anyway, right? Right?!
I do this so frequently that I created an alias, so I just have to type "pw" on the command line to get a random password.
alias pw="echo \$(head -c 64 /dev/urandom | base64) \$(date +%s) | shasum | awk '{print \$1}'"
Now you can create random passwords all day long.
[stephen ~]$ pw
[stephen ~]$ pw
[stephen ~]$ while true; do pw; done
# Here's a password from 1password. Go ahead and double click it to add it to your clipboard

Saturday, November 22, 2014

Encrypting and decrypting files with Gnu Privacy Guard (GPG)

Asymmetrical encryption with GnuPG

You should start using asymmetrical encryption for your backups. Here's a little guide.
With asymmetrical key encryption, other hosts and people can encrypt data that only you can decrypt, meaning that you don't have to share a secret key with anyone and you can distribute the public key freely.
At Elastica this is how we do backups. Every host gets preseeded with one of our strong backup keys. This allows them to gobble up sensitive data and store it in places like S3. If our S3 account was ever compromised then the intruder would have a bunch of encrypted files that would take them 20 years of computing power to decrypt.
Since decryption speed is not an issue with cold storage backups, I recommend the maximum 4096-bit RSA key.
Here's a quick guide to creating a strong crypto key and encrypting some files.
Also remember that gpg can be installed on your mac via brew.
Generate the key:
$ gpg --gen-key
(follow the prompts)
If you'd rather do this automatically or in a script, you can feed the options in:
$ gpg --batch --gen-key <<EOF
    Key-Type: rsa
    Key-Length: 4096
    Name-Real: Stephen Wood
    Name-Comment: For Testing
    Name-Email: [email protected]
    Expire-Date: 0
    %secring my_key.sec
    %echo done
You'll want to import the private key on any hosts that you want to be able to decrypt data.
$ gpg --allow-secret-key-import --import my_key.key
Remember to securely delete your secret key if you're not going to put the file elsewhere.
$ srm my_key.key

Encrypting and decrypting

Encrypting can be done on any host that has your public key. First, import the public key to the local hosts's gpg key ring
$ gpg --import
To encrypt a single file:
$ gpg --recipient [email protected] --encrypt
Note that the key is referred to as an email address. This is how gpg typically works.
Decrypting the file is easy if you have the private key:
$ gpg --decrypt >
By default gpg will decrypt a file to stdout, so we redirect it to a file. You can tell gpg to decrypt it to the original filename by passing in the --use-embedded-filename flag:
$ gpg --use-embedded-filename --decrypt
Notice that we didn't even have to specify which key to use. That's because gpg will use imbedded metadata in the file to decrypt the key. You can see this information by passing in the --list-packets flag:
$ gpg --list-packets
:pubkey enc packet: version 3, algo 1, keyid 6FC17A3C5E45F65C
    data: [4096 bits]
:encrypted data packet:
    length: unknown
    mdc_method: 2
gpg: encrypted with 4096-bit RSA key, ID 5E45F65C, created 2014-06-13
      "Stephen Wood <[email protected]>"
:compressed packet: algo=2
:literal data packet:
    mode b (62), created 1416686288, name="",
    raw data: 2171 bytes
Pretty nifty, huh?

Multiple files

Encrypting multiple files is much easier to do via the tar command. Since gpg files are compressed automatically, it's not even necessary to gzip them!
Here's how to encrypt an entire folder:
$ tar cf - some_folder | gpg --recipient [email protected]
(creates some_folder.gpg)
Since gpg defaults to stdout, we simply need to pass that back to tar
$ gpg --decrypt some_folder.gpg | tar xf -
(creates some_folder)

Other fun things

You can also use your private key to sign a message that others can verify for authenticity.
Let's create a document:
cat > my_message.txt <<MESSAGE
Help! The NSA is after me! They've tapped my phone, TV, microwave, and
Nike Air Jordans! Also the fillings in my teeth won't stop playing episodes
of "This American Life" and Ira Glass is getting on my nerves.
Now let's sign the message:
$ gpg -u [email protected] --clearsign my_message.txt 
(produces my_message.txt.asc)
You've got yourself a signed document.
$ cat my_message.txt.asc
Hash: SHA1

Help! The NSA is after me! They've tapped my phone, TV, microwave, and
Nike Air Jordans! Also the fillings in my teeth won't stop playing episodes
of "This American Life" and Ira Glass is getting on my nerves.
Version: GnuPG v1

Anyone can verify the message. They don't even need your public key:

gpg --verify my_message.txt.asc
gpg: Signature made Sat Nov 22 12:29:17 2014 PST using RSA key ID 23D1A7EF
gpg: Good signature from "Stephen Wood <[email protected]>"
And what happens when somebody intercepts and alters your original message?

$ cat my_message.txt.asc
Hash: SHA1

Dear friend. All is well here.
gpg --verify my_message.txt.asc
gpg: Signature made Sat Nov 22 12:20:18 2014 PST using RSA key ID 23D1A7EF
gpg: BAD signature from "Stephen Wood <[email protected]>"
Looks like your friend is dead. I'm so sorry.
This can be taken a step further if you've already securely exchanged your public key with your "friend". They can both verify your message hasn't been altered, and also verify that it was sent from you. They just need to verify that the Key ID is the same as the one you've imported.
That's it for now. Encrypt your backups! It's easy to do.

Monday, September 29, 2014

OpenVPN & Raspberry Pi auto-installer

Openvpn server on your Raspberry PI

From wikipedia:
OpenVPN is an open source software application that implements virtual private network (VPN) techniques for creating secure point-to-point or site-to-site connections in routed or bridged configurations and remote access facilities. It uses a custom security protocol that utilizes SSL/TLS for key exchange.
What that means is that openvpn will allow you to safely and securely route your internet traffic through an untrusted network to a trusted one. It does this by encrypting your traffic so nobody can read it until it goes out from your openvpn server.
There are some benefits to this:
  1. Prevent others from snooping on your traffic.
  2. Access websites that are blocked by your work, school, or oppressive governments.
  3. Access assets on your home network from anywhere.
This auto-install script will turn your raspberry pi into an openvpn server so you can browse the internet safely and securely.
Then why use it? Because sometimes you end up on insecure networks (think starbucks, stadiums, etc). This will protect your privacy in those situations. It will not prevent people from finding you if you are stupid and do something illegal.

Running the installer

The auto-installer is completely automated and can be run directly from the web.
From your raspberry pi:
# Set the user
$ OPENVPN_USER='stephen.openvpn.local'

# Run the installer
$ curl "" | sudo bash

The default key size is 4096 bits long. You can change the key size by exporting the KEY_SIZE variable before running the installer:

# Change the key size to something other than 4096
$ KEY_SIZE=2048

Remember that you should always inspect these types of files before ever running them. You can download it locally and run it like so:
# Download the file
$ wget ""

# Make sure it's legit
$ less bootstrap

# Execute it
$ chmod +x bootstrap
$ sudo ./bootstrap stephen.openvpn.local

What's the installer do

The installer script will download openvpn and generate all of the necessary root certificates for you. Then it will generate and sign a new certificate for a user. Lastly it will create the ovpn file and place it in /root/client_<some_user>.
All you need to do is download that file into your openvpn client software and you'll be able to safely and securely connect to the host.


I've included a Vagrantfile for you to run tests. Simply clone the repo and then run:
$ vagrant up
==> default: Forcing shutdown of VM...
==> default: Destroying VM and associated drives...
==> default: Running cleanup tasks for 'shell' provisioner...
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'wheezy64'...
==> default: Make sure that your firewall allows incoming UDP connections to port 1194.
==> default:
==> default: The last thing necessary is to securely copy the configuration file over to your
==> default: computer and then load it. The configuration file is located at:
==> default:
==> default:   /root/client_test.ovpn
The vagrant guest will be running the openvpn server. You can pull down the client file and connect to it locally.

Clone the repo

Edit: 2014-12-23

The original bootstrap was missing some iptables rules that allowed the raspberry pi to masquarade and forward IP. I've updated the script and everything should be working out of the box now.