Monday, June 11, 2018

Working with nulls with sql databases in Golang

Go is typed with no generics, so it's a little tricky working with databases where null values abound.

the sql package includes structs such as sql.NullString, but it's confusing for users to get a json response that looks like this, which is how the value is represented internally:

 "foo": {
  "Valid": true,
  "Value": 5

Below is a working golang HTTP server that handles null strings. It reads and writes to a sqlite3 database (in memory). When rows contain null values, they'll be properly displayed as null.

The trick to making things work is the null.v3 library, which implements the json marshal and unmarshal methods necessary to return a null value when querying the database.

Gist here.

package main

import (

 _ ""

// DB is the database connector
var DB *sql.DB

// Person represents a single row in a database. Using the type null.
type Person struct {
 Name     string      `json:"id"`
 Age      int         `json:"age"`
 NickName null.String `json:"nickname"` // Optional

// InsertPerson adds a person to the database
func InsertPerson(p Person) {
 cnx, _ := DB.Prepare(`
          INSERT INTO people (name, age, nickname) VALUES (?, ?, ?)`)
 defer cnx.Close()
 log.Printf("Adding person: %v\n", p)
 cnx.Exec(p.Name, p.Age, p.NickName)

// GetPeople will retur N number of people from database
func GetPeople(n int) []Person {
 people := make([]Person, 0)
 rows, _ := DB.Query(`SELECT name, age, nickname from people LIMIT ?`, n)
 for rows.Next() {
  p := new(Person)
  rows.Scan(&p.Name, &p.Age, &p.NickName)
  people = append(people, *p)
 return people

func addPersonRouter(w http.ResponseWriter, r *http.Request) {

 age, _ := strconv.Atoi(r.FormValue("age"))

 // Get nickname from the form and create a new null.String. If the string
 // is empty, it will be considered invalid (null) in the database and not
 // empty
 nick := r.FormValue("nickname")
 nickName := null.NewString(
  nick, nick != "")

 p := Person{
  Name:     r.FormValue("name"),
  Age:      age,
  NickName: nickName,



func getPeopleRouter(w http.ResponseWriter, r *http.Request) {
 limit, _ := strconv.Atoi(r.FormValue("limit"))
 people := GetPeople(limit)

 peopleJSON, _ := json.Marshal(people)
 w.Header().Set("Content-Type", "application/json")

// CreateTable is a helper function to create the table for the first run
func CreateTable() error {

 createSQL := `
    CREATE TABLE people (
  name TXT NOT NULL,
  nickname TXT
 statement, err := DB.Prepare(createSQL)
 if err != nil {
  return err


 return nil

func main() {
 var err error
 DB, err = sql.Open("sqlite3", ":memory:")
 if err != nil {

 err = CreateTable()
 if err != nil {

 http.HandleFunc("/add", addPersonRouter)
 http.HandleFunc("/list", getPeopleRouter)
 http.ListenAndServe(":8080", nil)

Here's some examples of the server when it's running

$ curl -XPOST "localhost:8080/add?name=Joseph&age=25&nickname=Joe"
$ curl -XPOST "localhost:8080/add?name=Stephen&age=33"
$ curl -s 'localhost:8080/list?limit=2' | jq
    "id": "Joseph",
    "age": 25,
    "nickname": "Joe"
    "id": "Stephen",
    "age": 33,
    "nickname": null

There you go. Now you can read and write valid or invalid values to your database.

Friday, December 18, 2015

Git cheat sheet

Creating a feature branch

Creating a feature branch is a good way to keep your testing out of the main branch. You can hack away at your code without disturbing the general workflow.
The following command will create a feature branch that tracks master. Pushing it with the -u origin flag will autmatically make your local version of the branch track the remote version:
     $ git checkout master && git pull
     $ git checkout -t -b "some-feature-branch-name"
     $ git push -u origin "some-feature-branch-name"
Also, since this is a feature branch you're free to do force commits and cleanup your git history.
You can see what your branches are tracking and what their origin is with the following command:
git branch -vv
  1432-rb-force-safety-switch 62d5af7 [origin/1432-rb-force-safety-switch] Refuse rb if path contains keys.
* develop                     5f56434 [origin/develop: ahead 233, behind 2] Merge branch 'release-1.8.12'
And to see where "origin" points to:
$ git remote -v
origin (fetch)
origin (push)
upstream (fetch)
upstream (push)
In the above example, I can push to either "upstream" (which in this case is the official aws repo), or "origin", which is my forked repo.

Creating a Git Tag

A tag is a point-in-time reference to a git branch. Tags are immutable snapshots of the repo that can't be altered once they are created (though they can be deleted and recreated). They are useful for tasks such as deployments or ensuring your work won't be altered by others.
    # This will create a tag called "mysql-1380235429"
    $ git tag -a -m "Deploying mysql update" mysql-`date +%s`
    $ git push --tags
Now the tag can be checked out in the usual method.
    $ git checkout mysql-1380235429

Preparing Your Branch for a Pull Request

A pull request should be done before anything is merged into the master branch.
Before asking someone to review your work, you should take a few minutes and make sure it's really finished. You'd be surprised what things can slip through.

Diff Against Master

The first thing you want to do is diff your branch against the master branch. This allows you to catch simple bugs before pushing anything upstream.
git diff master
This will tell you exactly what's different between your branch and the master branch. It will also highlight things such as trailing whitespace.

Rebasing your branch against master (or a different branch)

You can think of rebasing as doing the following actions: 1. Take all recent changes and stash them away 2. Pull down the most recent changes from a different branch 3. Attempt to apply your stashed changes on top of the new changes one at a time.
Here's how to rebase your branch against master:
git checkout master
git pull
git checkout my_feature_branch
git rebase master
git push origin +my_feature_branch
Many times there's conflicts in this process if there were multiple changes to the same file. The way to fix a conflict is to change it during the rebase and add it again:
(fix the conflicts within the file)
git add some_conflicted_file
git rebase --continue
If you think there's a problem then you can abort the rebase with --abort.

Squashing Commits

It's important that you branch is tidy because it makes rolling back bad changes a lot easier. You can squash all of your commits into one or two commits using the rebase command.
First, run git log and find the commit that you want to squash into. Usually this is the first commit on your feature branch.
Next, interactively rebase against that commit:
git rebase --interactive shdf8032hfohsdofhsdohf80h^
This will rebase every commit after the sha.
Follow the instructions for rebasing. Usually you just want to change the "pick" to a "squash" or "s". In the below example, all commits will be squashed into the top commit (which is the oldest):
pick e775ebe Refactor regex to avoid duplication
s 3bb23fd Add support for spaces within unquoted values
s e36d71a Support '-' char in a key name
s 16eda81 Remove unused import in test module
s 403c7ec Add bugfixes to changelog
You will have a chance to rewrite the commit message to better include all of the changes.
After you're finished, do a diff against master again just to make sure things don't go wrong. Then force push your branch to origin.
# Make sure it still looks good
git diff master
git push origin +my_feature_branch

Send Out the PR

Go to and find your branch. Compare it with master and send out the PR. You should ask someone specifically to review your changes and include the following information:
  • Why did you change this?
  • What have you changed?
  • How have you tested these changes?
  • What are the risks involved with these changes?

Oh no! I rebase my branch and lost a lot of history! I'm doomed!

Fear not. With git nothing is ever truly lost. If you made some mistake and you're currently in the rebase process then you can abort it with git rebase --abort. If you've already commited, pushed, etc, then you can use the fantastic git reflog tool to go back to a different commit.

Using git reflog

In the following example, I'm going to force my branch back to the moment before I rebased and broke everything:
First, find the commit that came right before your rebase:
git reflog
5f56434 HEAD@{0}: pull upstream master: Fast-forward
ed610bc HEAD@{1}: checkout: moving from 1432-rb-force-safety-switch to develop
62d5af7 HEAD@{2}: rebase -i (finish): returning to refs/heads/1432-rb-force-safety-switch
62d5af7 HEAD@{3}: rebase -i (squash): Refuse rb if path contains keys.
781b4da HEAD@{4}: rebase -i (start): checkout 781b4da0dcc65736297464dd73da442daad4cf2c^
4350e25 HEAD@{5}: commit: Use two vars for readability. <---- ding ding ding
781b4da HEAD@{6}: checkout: moving from ed610bc9d38244feeaf0b640781da8ab01808f4e to 1432-rb-force-safety-switch
Next, we'll checkout that sha and then force push it
git checkout 4350e25
git log # make sure it's what you want
git push +some_branch

Saturday, December 12, 2015

Using ssh-import-id to manage authorized keys


While poking around in my ~/.ssh directory (in order to inspect and harden some of my SSH configurations -- more on that later), I noticed a file that I have never seen before:
I was surprised to this this file, especially in a directory related to openssh. Opening the file I saw this:
 "_comment_": "This file is JSON syntax and will be loaded by ssh-import-id to obtain the URL string, which defaults to  The following URL *must* be an https address with a valid, signed certificate!!!  %s is the variable that will be filled by the ssh-import-id utility.",
 "URL": ""
ssh-import-id is a utility included with Ubuntu 14.04+ that, according to the man page "will securely contact a public key server and retrieve one or more user's public keys". In other words it's a way to manage your authorized_keys file via an external API.
You have two options:'s user directory or github. Running the utility will fetch and update the authorized_keys file based on the remote API.
For example, the following command will pull down my authorized_keys on github and update the file/home/stephen/.ssh/authorized_keys (since that's the user running the command)
stephen@cato:/etc/ssh$ ssh-import-id gh:stephen-mw
2015-11-30 21:44:04,813 INFO Already authorized ['4096', 'SHA256:3bLv3IXbSzhQpCnchqQprIRHXWPoI+PPW4xwguR6ktE', 'stephen-mw@github/10248951', '(RSA)']
2015-11-30 21:44:04,817 INFO Already authorized ['4096', 'SHA256:5ZtG8hD7l9+yU7I1S17FunmrPR5u6tEcRi0xa6wQGD4', 'stephen-mw@github/12837805', '(RSA)']
2015-11-30 21:44:04,817 INFO [2] SSH keys [Authorized]
The way it works is pretty simple. Github exposes an API for authorized keys. The utility simply makes a request to this endpoint and loads the output into the file. The utility is smart enough to know when keys change (that is, if you added all of your keys with ssh-import-id) and will keep things up-to-date.
By the way, did you know that github has an API for retrieving any public key? If that weirds you out, remember that they're called public keys for a reason! Here's Linus Torvalds public key. It's a 2048 RSA key.
You can add something like this to your crontab to update your key once a day at 4 am, and then once again if ever there's a restart. The second option is to ensure that servers/hosts that have been turned off for a long time can be accessed immediately.
# Pull down my github keys and add them to my user
0 4 * * * ssh-import-id gh:stephen-mw
@reboot ssh-import-id gh:stephen-mw
I find this to be especially useful on small embedded computers, such as a raspberry pi. When the raspberry pi is started after a long period it will automatically pick up my newest keys.


My first problem was a file appearing magically in my ~/.ssh/ directory. I consider this directory a sacred place and don't like uninvited files here. Apart from that, the application bills itself as "secure" so I took a look at the source. Mostly it looks fine, but there are some things I would like to see different:
  • Github usernames can change and that string is the only thing used to pull down the key. If you change your name you'll need to hunt down any instance of this program and update it. That's annoying with embedded systems, which is exactly the problem I'm hoping to solve with this application.
  • For SSL, the application uses Python's urlib and attempts to fallback on shelling to wget. However, there's no guarantee that wget will honor https requests only. In fact this can be disabled via ~/.wgetrc. They're relying on wget's default behavior without being explicit.
  • It checks only if the SSL cert is valid, but doesn't try very hard to see how valid it is. I would have preferred to see it reject any TLS versions lower than 1.2 and only accept EV certificates, since both domains use EV and TLS 1.2.
The last issue worries me the most. SuperfishCNNIC, and eDellRoot all show that rogue certificate authorities are a real and not theoretical problem.
But like most things in the world, it's a trade-off. If you find the convenience outweighs the security risk -- and I do -- then give ssh-import-id a try.

Wednesday, November 18, 2015

List all keys in an S3 bucket using golang

Here's a golang snippet that will list all of the keys under your S3 bucket using the official aws-sdk-go:
package main

import (

func main() {
    svc := s3.New(session.New(), &aws.Config{Region: aws.String("us-east-1")})

    params := &s3.ListObjectsInput{
        Bucket: aws.String("bucket"),

    resp, _ := svc.ListObjects(params)
    for _, key := range resp.Contents {

Sunday, November 15, 2015

What's the difference between a PRNG and a CSPRNG?


What's the different between a pseudo-random number generator and a cryptographically-secure pseudo-random number generator? A recommendation from the NIST.
The NIST ran various tests (defined in a publication known as SP800-22) to determine which PRNG algorithms produced the best output given a certain criteria.
The result and recommendation for these tests can be read in another publication called SP800-90. Skip down to Appendix C (Informative) DRBG Mechanism Selection to see the good stuff.
Three algorithms are determined to be cryptographically-secure pseudo-random number generators:
* Hash_DRBG
So there you go. Those three algorithms are CSPRNG because they've been tested and recommended by the NIST*.
*There are other organizations and tests that determine a PRNG to be a CSPRNG, but the NIST is the 800-pound Gorilla.


It's worth noting that the original publication actually had 4 CSPRNG recommendations. The now infamous Dual_EC_DRBG was quietly removed in the revised publication SP800-9Ar1.
There has been a lot of controversy[1][2] surrounding Dual_EC_DRBG. Whether or not any of the controversy is true, it's clear that trust in the NIST's authority has been undermined by its recommendation of Dual_EC_DRBG. I'm grateful that the NIST has removed the algorithm from its newest revision.
This also serves as a warning to anyone trying to influence these types of publications. The people that design, implement and critique this type of cryptography are smart. Very smart. Smarter than you and me -- and they don't like shenanigans!

Monday, July 20, 2015

Try patch instead of sed in shell scripts

A script is better seen than sed

I'd like to offer all of the shell scripters out there an alternative to sed replacements: patch.
Patch has several important benefits:
  • Context for all changes
  • Easy backup via the -b flag
  • Easy rollback
  • Easy multi-line replacements
Sed is great but can often times lead to enigmatic behavior. This is especially true when using wildcard expressions or other shell magic. And as the saying goes, code is read more often than it is written, so you should write for readability.
Here's a common type of sed replacement you'll find in any old off-the-shelf shell script:
# Replace some mystery var with this one
sed -i 's/setting_foo: */setting_foo: 5'
This kind of thing makes sense to the author, but the reader is at a disadvantage. What is the original value of setting_foo? What type of setting is it and why change it? These are things that are lost in translation with these commands.
And it can easily and often times be worse:
sed -ie 's/[]\/$*.^|[]/\\&/g' file
Here's a real example of updating a mysql.conf. Since multiple lines potentially start with port, you need to use a regular expression to match the word and whitespace to prevent accidents. Also, it's difficult to maintain the original justification of the file. And who knows what the original values were
# Change the port. Since this argument appears on several lines, we only want to change the first one
sed -ie '0,/^port\s*=\s*/port        = 8000/' /etc/mysql

# Change port using whitespace
sed -i 's/max_allowed_packet             = 16M/max_allowed_packet             = 32M/' /etc/mysql

# Allow more packets
sed -ie 's/^max_allowed_packet\s*=*/max_allowed_packet = 32M/' /etc/mysql.conf
Pretty ugly.
Now compare these same changes with patch
patch -b /etc/mysql.conf <<PATCH
--- mysql.conf_old    2015-07-20 19:53:25.000000000 -0700
+++ mysql.conf        2015-07-20 19:58:28.000000000 -0700
@@ -8,14 +8,14 @@


-port                           = 3306
+port                           = 8000
 socket                         = /var/run/mysqld/mysql.sock


-max_allowed_packet             = 16M
+max_allowed_packet             = 32M
 prompt                         = '\u@\h [\d]> '
 default_character_set          = utf8                                # Possibly this setting is correct for most recent Linux systems

Clean and simple! You can see we've obviously added more lines to the script, but we have context to our changes. We've also used the -b flag to automatically backup the file.

Closing thoughts

Of course this isn't always ideal. A good example is when you don't actually know the value that you're replacing. But nevertheless I think it's a good alternative that should be used when readability is important.

Saturday, February 28, 2015

List running EC2 instances with golang and the aws-sdk-go

Managing multiple AWS accounts can sometimes be tough, even when doing something as simple as matching a private IP address with a hostname.

In the past I used the aws cli tools, but I had to constantly switch both the accounts and the regions when making requests:

# Make it's in the prod-5 account in us-west-1
aws ec2 --profile=prod-5 --region=us-west-1 | grep my_instance

# Okay not that account or region, let's try eu-west-1
aws ec2 --profile=prod-5 --region=eu-west-2 | grep my_instance

Repeat 1x for each account and region

As you can imagine this is extremely time consuming, even when using the CLI tools. I wrote a small tool that will find every single instance you can view using every account available to you (according to your ~/.aws/config). The aggregate results can then be searched.

I choose to use the existing ~/.aws/config file so that it works along side your normal aws cli tools. 

This is a very good use-case for Golang, which has nice concurrency primitives. With five accounts this script (once compiled) will display all of the results in under 1.5 seconds. Not bad.

The result is a space-separated list with some additional values added. It should be easy to find what you're looking for:

$ ./aws_list
i-71930187 prod-manage005 t2.micro prod-account
i-71930187 stage-controller7 t2.medium staging-account
i-71930187 prod-vpn01 m4.large None prod-account
i-71930187 stephen-test-host m3.large test-account

The output is a space-separated file and only "running" and "pending" instances are displayed. For a list of filters or instance attributes consult the official documentation.

package main

import (

func check(e error) {
    if e != nil {

printIds accepts an aws credentials file and a region, and prints out all
instances within the region in a format that's acceptable to us. Currently that
format is like this:

  instance_id name private_ip instance_type public_ip account

Any values that aren't available (such as public ip) will be printed out as

Because the "name" parameter is user-defined, we'll run QueryEscape on it so that
our output stays as a space-separated line.
func printIds(creds aws.CredentialsProvider, account string, region string, wg *sync.WaitGroup) {
    defer wg.Done()

    svc := ec2.New(&aws.Config{
        Credentials: creds,
        Region:      region,

    // Here we create an input that will filter any instances that aren't either
    // of these two states. This is generally what we want
    params := &ec2.DescribeInstancesInput{
        Filters: []*ec2.Filter{
                Name: aws.String("instance-state-name"),
                Values: []*string{

    // TODO: Actually care if we can't connect to a host
    resp, _ := svc.DescribeInstances(params)
    // if err != nil {
    //      panic(err)
    // }

    // Loop through the instances. They don't always have a name-tag so set it
    // to None if we can't find anything.
    for idx, _ := range resp.Reservations {
        for _, inst := range resp.Reservations[idx].Instances {

            // We need to see if the Name is one of the tags. It's not always
            // present and not required in Ec2.
            name := "None"
            for _, keys := range inst.Tags {
                if *keys.Key == "Name" {
                    name = url.QueryEscape(*keys.Value)

            important_vals := []*string{

            // Convert any nil value to a printable string in case it doesn't
            // doesn't exist, which is the case with certain values
            output_vals := []string{}
            for _, val := range important_vals {
                if val != nil {
                    output_vals = append(output_vals, *val)
                } else {
                    output_vals = append(output_vals, "None")
            // The values that we care about, in the order we want to print them
            fmt.Println(strings.Join(output_vals, " "))

func main() {
    // Go for it!

    // Make sure the config file exists
    config := os.Getenv("HOME") + "/.aws/config"
    if _, err := os.Stat(config); os.IsNotExist(err) {
        fmt.Println("No config file found at: %s", config)

    var wg sync.WaitGroup

    file, err := ini.LoadFile(config)

    for key, values := range file {
        profile := strings.Fields(key)

        // Don't print the default or non-standard profiles
        if len(profile) != 2 {

        // Where to find this host. The account isn't necessary for the creds
        // but it's something we expose to users when we print
        account := profile[1]
        key := values["aws_access_key_id"]
        pass := values["aws_secret_access_key"]
        creds := aws.Creds(key, pass, "")

        // Gather a list of all available AWS regions. Even though we're gathering
        // all regions, we still must use a region here for api calls.
        svc := ec2.New(&aws.Config{
            Credentials: creds,
            Region:      "us-west-1",

        // Iterate over every single stinking region to get a list of available
        // ec2 instances
        regions, err := svc.DescribeRegions(&ec2.DescribeRegionsInput{})
        for _, region := range regions.Regions {
            go printIds(creds, account, *region.RegionName, &wg)

    // Allow the goroutines to finish printing