List running EC2 instances with golang and the aws-sdk-go

Managing multiple AWS accounts can sometimes be tough, even when doing something as simple as matching a private IP address with a hostname.

In the past I used the aws cli tools, but I had to constantly switch both the accounts and the regions when making requests:

# Make it's in the prod-5 account in us-west-1
aws ec2 --profile=prod-5 --region=us-west-1 | grep my_instance

# Okay not that account or region, let's try eu-west-1
aws ec2 --profile=prod-5 --region=eu-west-2 | grep my_instance

Repeat 1x for each account and region

As you can imagine this is extremely time consuming, even when using the CLI tools. I wrote a small tool that will find every single instance you can view using every account available to you (according to your ~/.aws/config). The aggregate results can then be searched.

I choose to use the existing ~/.aws/config file so that it works along side your normal aws cli tools. 

This is a very good use-case for Golang, which has nice concurrency primitives. With five accounts this script (once compiled) will display all of the results in under 1.5 seconds. Not bad.

The result is a space-separated list with some additional values added. It should be easy to find what you're looking for:

$ ./aws_list
i-71930187 prod-manage005 10.0.0.180 t2.micro 207.171.166.22 prod-account
i-71930187 stage-controller7 10.0.0.164 t2.medium 72.21.206.80 staging-account
i-71930187 prod-vpn01 10.0.0.239 m4.large None prod-account
i-71930187 stephen-test-host 10.0.0.216 m3.large 216.58.216.174 test-account
...

The output is a space-separated file and only "running" and "pending" instances are displayed. For a list of filters or instance attributes consult the official documentation.

package main

import (
    "fmt"
    "github.com/awslabs/aws-sdk-go/aws"
    "github.com/awslabs/aws-sdk-go/service/ec2"
    "github.com/vaughan0/go-ini"
    "net/url"
    "os"
    "runtime"
    "strings"
    "sync"
)

func check(e error) {
    if e != nil {
        panic(e)
    }
}

/*
printIds accepts an aws credentials file and a region, and prints out all
instances within the region in a format that's acceptable to us. Currently that
format is like this:

  instance_id name private_ip instance_type public_ip account

Any values that aren't available (such as public ip) will be printed out as
"None"

Because the "name" parameter is user-defined, we'll run QueryEscape on it so that
our output stays as a space-separated line.
*/
func printIds(creds aws.CredentialsProvider, account string, region string, wg *sync.WaitGroup) {
    defer wg.Done()

    svc := ec2.New(&aws.Config{
        Credentials: creds,
        Region:      region,
    })

    // Here we create an input that will filter any instances that aren't either
    // of these two states. This is generally what we want
    params := &ec2.DescribeInstancesInput{
        Filters: []*ec2.Filter{
            &ec2.Filter{
                Name: aws.String("instance-state-name"),
                Values: []*string{
                    aws.String("running"),
                    aws.String("pending"),
                },
            },
        },
    }

    // TODO: Actually care if we can't connect to a host
    resp, _ := svc.DescribeInstances(params)
    // if err != nil {
    //      panic(err)
    // }

    // Loop through the instances. They don't always have a name-tag so set it
    // to None if we can't find anything.
    for idx, _ := range resp.Reservations {
        for _, inst := range resp.Reservations[idx].Instances {

            // We need to see if the Name is one of the tags. It's not always
            // present and not required in Ec2.
            name := "None"
            for _, keys := range inst.Tags {
                if *keys.Key == "Name" {
                    name = url.QueryEscape(*keys.Value)
                }
            }

            important_vals := []*string{
                inst.InstanceID,
                &name,
                inst.PrivateIPAddress,
                inst.InstanceType,
                inst.PublicIPAddress,
                &account,
            }

            // Convert any nil value to a printable string in case it doesn't
            // doesn't exist, which is the case with certain values
            output_vals := []string{}
            for _, val := range important_vals {
                if val != nil {
                    output_vals = append(output_vals, *val)
                } else {
                    output_vals = append(output_vals, "None")
                }
            }
            // The values that we care about, in the order we want to print them
            fmt.Println(strings.Join(output_vals, " "))
        }
    }
}

func main() {
    // Go for it!
    runtime.GOMAXPROCS(runtime.NumCPU())

    // Make sure the config file exists
    config := os.Getenv("HOME") + "/.aws/config"
    if _, err := os.Stat(config); os.IsNotExist(err) {
        fmt.Println("No config file found at: %s", config)
        os.Exit(1)
    }

    var wg sync.WaitGroup

    file, err := ini.LoadFile(config)
    check(err)

    for key, values := range file {
        profile := strings.Fields(key)

        // Don't print the default or non-standard profiles
        if len(profile) != 2 {
            continue
        }

        // Where to find this host. The account isn't necessary for the creds
        // but it's something we expose to users when we print
        account := profile[1]
        key := values["aws_access_key_id"]
        pass := values["aws_secret_access_key"]
        creds := aws.Creds(key, pass, "")

        // Gather a list of all available AWS regions. Even though we're gathering
        // all regions, we still must use a region here for api calls.
        svc := ec2.New(&aws.Config{
            Credentials: creds,
            Region:      "us-west-1",
        })

        // Iterate over every single stinking region to get a list of available
        // ec2 instances
        regions, err := svc.DescribeRegions(&ec2.DescribeRegionsInput{})
        check(err)
        for _, region := range regions.Regions {
            wg.Add(1)
            go printIds(creds, account, *region.RegionName, &wg)
        }
    }

    // Allow the goroutines to finish printing
    wg.Wait()
}

Dyn DNS cli tool

Get the file: https://github.com/stephen-mw/dyncli

Usage

usage: dyncli [-h] [-e ENDPOINT] [-c CREATE] [--cname] [-v VALUE] [-t TTL]
              [-d DELETE] [-u UPDATE_RECORD] [-U USER] [-P PASSWORD]
              [-A ACCOUNT] [-Z ZONE] [-l] [--verbose]

optional arguments:
  -h, --help            show this help message and exit
  -e ENDPOINT, --endpoint ENDPOINT
                        The endpoint to make API requests.
  -c CREATE, --create CREATE
                        Create a new record. Defaults to A record unless
                        --cname is set.
  --cname               Create a cname record instead of an A record.
  -v VALUE, --value VALUE
                        The value to set the DNS record to. Eg 10.0.0.101.
  -t TTL, --ttl TTL     TTL of record in seconds.
  -d DELETE, --delete DELETE
                        Delete an existing record
  -u UPDATE_RECORD, --update UPDATE_RECORD
                        Perform a DNS update on an existing record.
  -U USER, --user USER  The username to connect to the dyn api. Can also be
                        set as DYN_USER in environment.
  -P PASSWORD, --password PASSWORD
                        The password to use with the dyn api. Can also be set
                        as DYN_PW in environment.
  -A ACCOUNT, --account ACCOUNT
                        The account name used to make API requests. Can be set
                        as DYN_ACCOUNT in environment.
  -Z ZONE, --zone ZONE  The zone to take action on. Eg example.com. Can be set
                        as DYN_ZONE in environment.
  -l, --list            List all dns records as a csv file.
  --verbose             Print out api responses.

Dyn is a nice DNS service. It's an alternative to Amazon's Route 53. Though I do not believe that dyn's API is as robust as amazon, with the right set of tools is just dandy.

This tool allows for the manipulation of DNS records via the restful API.

It currently supports the following:

• Create A and CNAME records
• Update A and CNAME records
• Delete A and CNAME records
• List all of the records as a CSV

Requirements

The requests & argparse packages are required but not currently part of the standard library. Get them with pip:

pip install -r requirements.txt

You can pass your username and password directly into the script, but it's better if you setup a few environment variables. dyncli will take advantage of the following environment variables if they are present:

DYN_USER
DYN_PW
DYN_ZONE
DYN_ACCOUNT

You can activate these by appending them to your ~/.bash_profile:

export DYN_USER='foo'
export DYN_PW='bar'
export DYN_ZONE='example.com'
export DYN_ACCOUNT='example'

Creating a record

Right now both A and CNAME record creation is supported. Create a record uses the -c flag along with the optional --cname flag if that's your desired record type.

The default record type is an address (A) record.

# Create an A record with default TTL
$ ./dyncli -c myfqdn.example.net -v 10.0.0.100

# Create an A record with custom TTL
$ ./dyncli -c myfqdn.example.net -v 10.0.0.100 --ttl 3600

# Create a CNAME record
$ ./dyncli -c myfqdn.example.net -v otherfqdn.example.net --cname

Updating a record

Once records are created, they can be easily updated with the -u or --update flag.

# Update the TTL of an existing record
$ ./dyncli -u myfqdn.example.net --ttl 3600

# Update the value of an existing cname record
$ ./dyncli -u myfqdn.example.net -v someotherfqdn.example.net

Deleting a record

Records are deleted by passing the -d or --delete flag.

# Delete an address (A) record
$ ./dyncli -d myfqdn.example.net

# Delete a cname record (requires the --cname flag)
$ ./dyncli -d myfqdn.example.net --cname

List all records (csv format)

Sometimes it's useful to list all of the records. We use this to do a simple nightly backup of our DNS zone.

The output format is a simple CSV file. Beware that if you have commas in your TXT record, they'll be translated to a pipe (|).

$ ./dyncli --list
zone,ttl,fqdn,record_type,data
example.net,300,test.example.net,CNAME,test2.example.net
example.net,300,foo.example.net,A,172,16.0.100
example.net,3600,example.net,TXT,public_key=sdjflksdj
..

Sometimes it's also useful to check if a zone exists before you go creating one willy nilly

$ ./dyncli --list | grep database
example.net,300,database.example.net,A,172.16.0.100
example.net,300,database.internal.example.net,A,172.16.0.100
example.net,300,database.something.example.net,A,172.16.0.100

(See what happens when you go on creating records without checking first?)

If you notice any bugs please open an issue.