Easily convert hex to ints and vice versa with python and the command line

I've started a lot of work lately with the ssd1306 oled display (which you can pick up for a few bucks on ebay) and my raspbery pi. While sending commands I've found it useful to be able to easily convert between ints and hexadecimal strings and back again.

To convert a hexadecimal value to an int on the command line:

python -c 'print int(0x7f)'
127

And back again:

python -c 'print hex(127)
0x7f

For floats you can use the float.hex method:

python -c 'print float.hex(120.5)'
0x1.e200000000000p+6

Maybe you want that trimmed:
python -c 'print float.hex(120.5)[:7]'

0x1.e20

It's worth noting that the '-c' flag being passed to python essentially tells python that the first argument is a script it should execute normally.

Verifying JSON easily on the command line

You can pipe stdout into python -mjson.tool to validate it. It makes for quick and easy json validation on the command line.

We'll create a simple json file:

> somefile.txt
{"someval": "something", "anotherval": 3}

Now pipe this into json.tools and check the output.

$ cat somefile.txt | python -mjson.tool
{
    "anotherval": 3,
    "someval": "something"
}

Cool. It even formatted it nicely for us. Let's break it and see what happens.

Single quotes are not valid according rfc4627.

> somefile.txt
{'someval': 'something', 'anotherval': 3}

$ cat somefile.txt | python -mjson.tool
Expecting property name: line 1 column 2 (char 1)

Not the most useful traceback, but at least you know it's not valid.

Using Saltstack to manage linux users

Saltstack makes it very easy to manage users. You can use pillars to predefine all of the users and add them later, or you can define them within a single sls state file.

By example here is an sls file that will add or remove any number of users to a system. If you want the user dropped from the system, change "present" to "absent" and run it again.

Thanks Will for fixes and pointing out that if you're not on a debian system you'll want to use something else for the "adm" group.

{% set users = {
  'someuser': {
    'state': 'present',
    'fullname': 'User One',
    'pub_key': 'ssh-rsa .... '
  },
  'someuser2': {
    'state': 'present',
    'fullname': 'User two',
    'pub_key': 'ssh-rsa .... '
  }
} %}
 
{% for name, user in users.items() %}
{{ name }}:
  {% set shell = user.shell | default('/bin/bash') %}
  {% set groups = user.groups | default(['sudo', 'adm']) %}
  user.{{ user.state }}:
    - fullname: {{ user.fullname }}
    - home: /home/{{ name }}
    - shell: {{ shell }}
    - groups:
    {% for group in groups %}
      - {{ group }}
    {% endfor %}
  {% if user.state == 'present' %}
ssh_key_{{ name }}:
  ssh_auth:
    - present
    - user: {{ name }}
    - names:
      - {{ user.pub_key }}
    - require:
      - {{ name }}
  {% endif %}
{% endfor %}

Connecting your raspberry pi to your wireless network

Update: 2015-05-01

This post is now out of date! Please see the official documentation for setting up wireless on your raspberry pi.

Why Wi-fi?

What good is a little tiny computer when it always needs to be attached to your router via and ethernet cable? Not very! I like that fact that my raspberry Pi is so small that I sometimes forget where I put them.

Literally. I have three and I can't remember where the third one is. It is online, though!

The best way to unwire is to go wireless. Disregard that dumb sentence and read this guide for connecting your raspberry pi to your wireless network using WPA encryption.

You need to get your hands on a linux-compatible wireless adapter. I recommend the Edimax Nano. It's cheap and works well with linux out of the box. No driver installation necessary.

It does have at least one major drawback, and that is that it supports wireless N but not at 5Ghz, making it effectively useless for wireless N networks. If you're fine with wireless G, then it's a decent and cheap solution.

Getting Started 

If it's not already installed, install wpa_supplicant.

$ sudo apt-get -y install wpasupplicant

We're going to change the adapter mode to "managed" and assign it an essid.

Managed is probably what you want, which says that the adapter is going to be roaming and connecting to different access points. You can find other options on the iwconfig man page.

$ iwconfig wlan0 mode Managed
$ iwconfig wlan0 essid <your clever router name>

Don't remember what your clever router is called? You can easily run a scan using iwlist scan

$ iwlist scan
wlan0     Scan completed :
          Cell 01 - Address: C0:C1:C0:7B:5A:03
                    ESSID:"Abraham Linksys"
                    Protocol:IEEE 802.11bgn
                    Mode:Master
                    Frequency:2.412 GHz (Channel 1)
                    Encryption key:on
                    Bit Rates:300 Mb/s
                    Extra:rsn_ie=30180100000fac020200000fac04000fac020100000fac020c00
                    IE: IEEE 802.11i/WPA2 Version 1
                        Group Cipher : TKIP
                        Pairwise Ciphers (2) : CCMP TKIP
                        Authentication Suites (1) : PSK
                    Quality=100/100  Signal level=100/100

Ah yes, that's what I called it.

Next up we need to provide wpa_supplicant with a password so it can negotiate the encryption. We'll use the command wpa_passphrase for that.

The utility takes two arguments. The first is your essid, and the second is your password. You can omit your password from the command-line and it will prompt you for it.

$ wpa_passphrase "Abraham Linksys" "password"

network={
 ssid="Abraham Linksys"
 #psk="password"
 psk=ca863518c2996944d5357729cdca5e6e459c46b05cd285e8600babaf3e76cb09
}

Looks good, let's create a conf file with it. Remember to put your actual password in the "password" field.

$ wpa_passphrase "Abraham Linksys" "password" > /etc/wpa.conf

We're almost done. Now we have a way to negotiate our encryption. The only thing left is to edit our /etc/network/intefaces file with the new stuff. Open it up and make the following changes:

$ sudo vim /etc/network/interfaces 

auto lo
iface lo inet loopback

iface eth0 inet dhcp 

auto wlan0
iface wlan0 inet dhcp

pre-up wpa_supplicant -B -Dwext -iwlan0 -c /etc/wpa.conf

You'll notice the pre-up command there. That says to load wpa_supplicant as a daemon when you bring up this interface. Here's what the various switches for wpa_supplicant do:

-B 

Daemonize, meaning run in the background

-Dwext 

Use "wext" as the driver. This is the most common one and works with edimax. 

-iwlan0

Our interface.

-c /etc/wpa.conf 

The config file to use, which we just created.

Save it and then it's time to fire it up and see if we get an IP.

$ sudo ifup wlan0
$ ifconfig wlan0

wlan0     Link encap:Ethernet  HWaddr 80:1f:02:7c:84:08
          inet addr:10.0.0.7  Bcast:10.0.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:426 errors:0 dropped:506 overruns:0 frame:0
          TX packets:189 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:67453 (65.8 KiB)  TX bytes:35122 (34.2 KiB)

It sometimes takes a few seconds for your adapter to connect to your access point and get an IP address.

And we're good to go. You've broken the cat5 yoke on your raspberry pi. Now you can put it anywhere in your house. Hook it up to monitor the temperature of your fridge, or put it behind your TV and stream wireless HD video from your PC. 

Nothing we did here is exclusive to the raspberry pi. In fact, this is exactly how I set up wireless on my ubuntu server laptops. That's the beauty of the RPI: the skills you learn are valuable linux skills.

Good luck and feel free to comment if you have any questions or problems.

The Deployment Manifesto

This has nothing to do with the word devops. Instead, this is a discussion about certain responsibilities in which both the ops teams and the developer teams overlap. This is about cooperation between teams to create a better product.

The 10 Requirements

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL
     NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED",  "MAY", and
     "OPTIONAL" in this document are to be interpreted as described in
     RFC 2119.

1. Configuration management MUST NOT be coupled to an external service, such as EC2, Openstack, Foreman, or anything else.

2. Devops SHOULD provide a self-service framework for the automatic creation and destruction of hosts from the ground up.

3. Devops SHOULD work with engineering teams to come up with continuous deployment strategy that doesn’t involve the destruction and creation of fresh operating systems.

4. All code required for deployments MUST be maintained in a centralized source repository. 

5. Deployments MUST use immutable snapshots -- such as a git tag -- from source code.

6. Hosts being provisioned MUST get their configurations from source control and MUST NOT rely on resources from an individual user or an engineer’s local computer.

7. Developers MUST provide a way to test code before it is deployed.

8. Devops MUST have an automated and tested rollback plan with every deployment.

9. Devops SHOULD provide feedback and planning support for hardware, infrastructure, and software dependencies necessary to run applications.

10. Devops MUST monitor all deployments and have clear, identified benchmarks for success or failure.

Logstash 1.2 and Kibana2 auto-install for Ubuntu 12.04

Logstash and Kibana auto-install

Last time I brought you the auto-install script for logstash and kibana 2. I'm updating the install now to work with logstash 1.2 and Kibana 3.

You can read through the change logs if you're really curious about what's different between the two. The biggest change is the facelift in the Kibana project, which looks quite nice.

This auto-install will automatically install everything you need to start using logstash and kibana on ubuntu 12.04. Simply clone the project and run:

$ ./bootstrap

If you notice any issues please feel free to comment here or to head over to the github page and open up an issue.

Clone the project

About the project

My goal was to make the install as easy and automated as possible. You may want to adjust the settings (like the memory allowed by logstash), but overall the settings are general enough to be good for most folks.

I've opted to use nginx to host Kibana, but it's not hard to swap that out with apache if that's your thing.

A note about Kibana

You may notice that you're unable to connect to your Kibana instance to elasticsearch. They've changed the frontend to use client-side javascript, which needs the fully qualified domain name of you elasticsearch instance. This means your browser will basically make the request to ES and not the server running Kibana. Don't get confused. If you can't connect then take a look at the following:

1. Verify the FQDN of your elasticsearch host is in config.js. You cannot use localhost here or your browser will make a request to localhost.
2. Your browser has permissions to request information from ES.

Install gnu grep on mac osx

One thing that bugs me about the bsd grep that comes with mac osx is that it doesn't offer you the -P flag. The -P flag will let you use the pcre engine in your regex matches, which is 100x more awesome than using the posix regex of egrep.

Not only that, but did you know gnu grep is significantly faster than the default bsd version? Go ahead and read Why is Gnu Grep So fast? where the author of gnu grep himself chimes in to explain why. Lots of nerdy tidbits in there.

You can install gnu grep on your mac very easily using brew. If you don't have brew you can install it following these instructions.

Installing gnu grep:

# Enable dupe and install
brew tap homebrew/dupes
brew install homebrew/dupes/grep

# Install the perl compatible regular expression library

brew install pcre

# Add the symlink to a place in $PATH
ln -s /usr/local/Cellar/grep/2.14/bin/ggrep /usr/bin/ggrep

# Add an alias

alias grep="ggrep"

# Verify you got it!
$ grep --version

grep (GNU grep) 2.14
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Mike Haertel and others, see <http://git.sv.gnu.org/cgit/grep.git/tree/AUTHORS>.

# If you want it to be permanent, you can add the alias line to your ~/.bash_profile

# You probably want the alias to stick after reboots

echo 'alias grep="ggrep"' >> ~/.bash_profile

All done!