as-create-launch-config: Service error: null AWSRequestId:null

Seemingly out of nowhere, amazon web services was giving me the following error when creating auto-scaling groups:

as-create-launch-config:  Service error: null  AWSRequestId:null

I've been using auto-scaling groups for months without any problems. It turns out this error is a result of a strange bug in the URL length when making a REST request to the amazon web server's api. Basically, the URL length becomes too long.

After some digging I found the solution on Amazon's forums.

The easiest way to solve this is to unset your "AWS_CREDENTIAL_FILE" variable and add the variables "EC2_CERT" and "EC2_PRIVATE_KEY" to point at your cert and pk file. This can be done on the command line as the -K and -C options, or you can add it to your bash_profile.

In your ~/.bash_profile, add the following variables:

export EC2_CERT="your_cert.pem"
export EC2_PRIVATE_KEY="your_pk.pem"

Then simply source your ~/.bash_profile to pick up the new changes.

$ source ~/.bash_profile

This will force the auto-scaling tools to make a SOAP request instead of a RESTful one to the aws api, which seems to get around the issue at hand.

SSH tunneling on a mac in 5 minutes

 

This blog post is brought to you by the SEA-TAC Airport's free public wifi -- and all of the potential snoops who could be listening over the wire. This technique also defeats internet censorship at your school or oppressive country.

Here's a quick guide for tunneling all of your traffic over SSH when you're on public wifi or behind a firewall that has port 22 open. All of your traffic will be encrypted with SSH between you and your web server, which will make the requests on your behalf.

What you'll need:

1. SSH access to a machine connected to the outside internet. If you don't have an SSH web server, now is a good time to recommend the free shell access provider Silence is Defeat.
2. Some sort of SSH client. In this case we'll use the default provided with mac. If you're using windows then stop reading and go here.
Open up your terminal and pick a non-privileged port, in this case we'll use 8888.

Here's the command that allows the tunnel. It's very simple:

$ ssh -D 8888 username@yourwebserver.com -vv

Make the obvious repalcements. The "-D" says to SSH to use this port for application level port forwarding. Here's a snippet from the SSH man page explaining the relevant magic:

Whenever a connection is made to this port, the connection is forwarded over the secure channel, and the application protocol is then used to determine where to connect to from the remote machine.  Currently the SOCKS4 and SOCKS5 protocols are supported, and ssh will act as a SOCKS server.  Only root can forward privileged ports.  Dynamic port forwardings can also be specified in the configuration file.

And the "-vv" argument tells SSH to use extra verbose output. This basically just affirms that things are working, but it is fun to watch.

Once it's connected, you'll need to set up a web browser to use the socks5 proxy. I keep Firefox on my mac just for this.

1. Go to preferences, and select the "Advanced" tab.
2. Select "Network" button and choose "Settings"
3. Select "Manual proxy configuration" and under "SOCKS Host", you're going to put your local loopback address (127.0.0.1) and the port you selected when you opened up the proxy

Settings for mac SSH tunneling.

Then click OK. The rest of the settings you can leave at default. To verify the tunnel is working, I like to use IP Geolocation and see where it says my request server is located before and after the tunnel.

Before using the SOCKS5 tunnel

After. Please note: that's the other Washington.

Look at that, the internet thinks I'm located in a datacenter in Washington, DC! Any traffic firefox uses will go back to your local tunnel to your webserver and out into the world, coming back in the same way it went out. Go ahead and browse facebook at work now, or become a Chinese dissident blogger.

I like to leave these settings as the default settings in Firefox. Then, when I want to start a tunnel, I just run that command in terminal and switch to using Firefox for awhile.  The instructions should work for Chrome or Internet Explorer as well.

How to Delete Large s3 Buckets Easily

You've got a problem: your s3 bucket is so massive it can't be deleted. It's big enough that s3cmd simply breaks when you try and run it. It's so big, in fact, that even if you deleted 10,000 keys a minute using s3nukem, you'd have to run it all day long for weeks.

Here's the easy way to delete a massive s3 bucket with large amounts of files: simply set a lifecycle policy of 1 day and wait.

Make sure you really want what's inside here gone forever.

1. Log into to the aws dashboard, go to s3, and then to the properties of the s3 bucket you'd like to purge.
2. Under the lifecycle tab, give it an expiration policy without a prefix of 1 day.

Let Amazon do the dirty work.

S3 will do its own housekeeping, and after awhile everything inside your bucket will be gone. Poof.

Before finding this solution, we would actually take a cluster of 200 machines and pound Amazon with s3nukem for several days. At one point Amazon actually deactivated our s3 credentials and called us on the phone asking us what the heck we were doing. Our buckets contained a web index with many, many millions of files. It still took several days.

Good luck!

How To Disable the Annoying Mac osx Tab Complete Alert Sound

We are a mac office here at work, and I love my mac, but they do something very annoying: beep every time you tab complete the terminal.

`ls` a directory that doesn't exist? Beep.
mv a file? Beep.
Any sort of tab complete? Beep, beep, beep.

When you fill an office with people who live inside their shell, that's a lot of beeping. It's almost as prolific as iPhone sms sounds coming from the marketing side of the room. Don't get me started on that.

So here's how to disable the annoying mac osx tab complete beep alert that you hear when auto-completing on your terminal.

Open up system preferences and click on "sound", then turn the "alert volume" all the way down. Die, alert sound, DIE!!!

Turn the "Alert volume" slider all the way down to disable the mac osx alert sound.

Be nice and mute your system or disable this sound.