The curious case of Python's datetime -0753 UTC offset

 "Time, the devourer of all things." - Ovid's Metamorphoses

Confused why your python datetime object's UTC offset is showing up as -0753 instead of -0700 (PDT) or -0800 (PST)? I certainly was.

The tl;dr is because Python's pytz does not do the right thing with daylight saving's time when simply adding a timezone to a naive datetime object, and you must localize it.

Here's an example of what I was encountering. I was parsing a timestamp which was in the America/Los_Angeles timezone. I needed to change the printed format, so I parsed the date and did a quick strftime and... -0753??

from datetime import datetime
import pytz

tz = pytz.timezone("America/Los_Angeles")

# Super duper sure this is America/Los_Angeles tz
niave_string = "2020-10-05 14:03:40"

dateobj = datetime.strptime(niave_string, "%Y-%m-%d %H:%M:%S")

# Make it a timezone-aware object (Or so you think)
dateobj = dateobj.replace(tzinfo=tz)

# Notice the -0753 UTC offset...
print(dateobj.strftime("%Y-%m-%d %H:%M:%S%z"))

'2020-10-05 14:03:40-0753'

The mismatch happens because you can't just slap a tz onto a naive datetime object and hope it will do the right thing. More info on what's happening here can be found on this great stack overflow answer.

The fix for this is to localize your date objects, not just replace the TZ:

from datetime import datetime
import pytz

tz = pytz.timezone("America/Los_Angeles")

# Super duper sure this is America/Los_Angeles tz
niave_string = "2020-10-05 14:03:40"

dateobj = datetime.strptime(niave_string, "%Y-%m-%d %H:%M:%S")

# Localize your date object to your timezone
dateobj = tz.localize(dateobj)

# 🤞🏻
print(dateobj.strftime("%Y-%m-%d %H:%M:%S%z"))

'2020-10-05 14:03:40-0700'

Much better.

Install Python 3 on raspberry pi and make it the system default

As of this blog post, the raspberry pi still ships with python2.7 as the default. There's also not really an easy way to install python3.8 (again the latest as of this post) as the system default.

I used a global pyenv so that the global version can be upgraded and changed easily.

The below script will automatically download, compile, and set python3.8.5 as the new system default for your raspberry pi. No prompts necessary.

You can run the gist to automatically do this:

curl 'https://gist.githubusercontent.com/stephen-mw/341c8194aefb694939b366204156037c/raw/fe2fc6060792dbe3e98bc3fc7830229e0d657bdf/install_python38_on_py.sh' | sudo bash

Or just copy below into your script and execute it:

#!/usr/bin/env bash
set -euo pipefail

# This script downloads, compiles, and installs python3.8 as the system default

export VERSION=3.8.5

apt install -y       \
    build-essential  \
    libbz2-dev       \
    libffi-dev       \
    liblzma-dev      \
    libncurses5-dev  \
    libncursesw5-dev \
    libreadline-dev  \
    libsqlite3-dev   \
    libssl-dev       \
    llvm             \
    python-openssl   \
    python-pip       \
    tk-dev           \
    xz-utils         \
    zlib1g-dev

export OLD_PIP=$(which pip)
export NEW_PIP=${OLD_PIP}2.7

mv ${OLD_PIP} ${NEW_PIP}
ln -s ${NEW_PIP} ${OLD_PIP}

export PYENV_ROOT=/etc/pyenv
export PATH="$PYENV_ROOT/bin:$PATH"

if [[ -d "${PYENV_ROOT}" ]]; then
    rm -rfv -- "${PYENV_ROOT}"
fi

git clone https://github.com/pyenv/pyenv.git ${PYENV_ROOT}

# Can't fit in /tmp because it's a ramfs
local TMPDIR=${PYENV_ROOT}/tmp
mkdir ${TMPDIR}

# The latest version as of now
echo "Setting system python to ${VERSION}. This may take several minutes..."
CFLAGS="-O2" TMPDIR=${TMPDIR} pyenv install ${VERSION}

rm -rf -- "${TMPDIR}"

# Set the global version
pyenv global ${VERSION}

# Latest
update-alternatives --install $(which python) python /etc/pyenv/versions/${VERSION}/bin/python 1
update-alternatives --install $(which pip) pip /etc/pyenv/versions/${VERSION}/bin/pip 1

# 2.7
update-alternatives --install $(which python) python /usr/bin/python2.7 2
update-alternatives --install $(which pip) pip /usr/bin/pip2.7 2

# Lastly, set the system python to our new version
update-alternatives --set python /etc/pyenv/versions/${VERSION}/bin/python
update-alternatives --set pip /etc/pyenv/versions/${VERSION}/bin/pip

pip install -U pip

cat <<"HELP"

Python${VERSION} is now your system default.

If you want to roll your system back to 2.7, simply run:

    sudo update-alternatives --set python /usr/bin/python2.7
    sudo update-alternatives --set pip /usr/bin/pip2.7

HELP

Using ssh-import-id to manage authorized keys

ssh-import-id

While poking around in my ~/.ssh directory (in order to inspect and harden some of my SSH configurations -- more on that later), I noticed a file that I have never seen before:

ssh_import_id

I was surprised to this this file, especially in a directory related to openssh. Opening the file I saw this:

{
 "_comment_": "This file is JSON syntax and will be loaded by ssh-import-id to obtain the URL string, which defaults to launchpad.net.  The following URL *must* be an https address with a valid, signed certificate!!!  %s is the variable that will be filled by the ssh-import-id utility.",
 "URL": "https://launchpad.net/~%s/+sshkeys"
}

ssh-import-id is a utility included with Ubuntu 14.04+ that, according to the man page "will securely contact a public key server and retrieve one or more user's public keys". In other words it's a way to manage your authorized_keys file via an external API.

You have two options: launchpad.net's user directory or github. Running the utility will fetch and update the authorized_keys file based on the remote API.

For example, the following command will pull down my authorized_keys on github and update the file/home/stephen/.ssh/authorized_keys (since that's the user running the command)

stephen@cato:/etc/ssh$ ssh-import-id gh:stephen-mw
2015-11-30 21:44:04,813 INFO Already authorized ['4096', 'SHA256:3bLv3IXbSzhQpCnchqQprIRHXWPoI+PPW4xwguR6ktE', 'stephen-mw@github/10248951', '(RSA)']
2015-11-30 21:44:04,817 INFO Already authorized ['4096', 'SHA256:5ZtG8hD7l9+yU7I1S17FunmrPR5u6tEcRi0xa6wQGD4', 'stephen-mw@github/12837805', '(RSA)']
2015-11-30 21:44:04,817 INFO [2] SSH keys [Authorized]

The way it works is pretty simple. Github exposes an API for authorized keys. The utility simply makes a request to this endpoint and loads the output into the file. The utility is smart enough to know when keys change (that is, if you added all of your keys with ssh-import-id) and will keep things up-to-date.

By the way, did you know that github has an API for retrieving any public key? If that weirds you out, remember that they're called public keys for a reason! Here's Linus Torvalds public key. It's a 2048 RSA key.

You can add something like this to your crontab to update your key once a day at 4 am, and then once again if ever there's a restart. The second option is to ensure that servers/hosts that have been turned off for a long time can be accessed immediately.

# Pull down my github keys and add them to my user
0 4 * * * ssh-import-id gh:stephen-mw
@reboot ssh-import-id gh:stephen-mw

I find this to be especially useful on small embedded computers, such as a raspberry pi. When the raspberry pi is started after a long period it will automatically pick up my newest keys.

Security

My first problem was a file appearing magically in my ~/.ssh/ directory. I consider this directory a sacred place and don't like uninvited files here. Apart from that, the application bills itself as "secure" so I took a look at the source. Mostly it looks fine, but there are some things I would like to see different:

• Github usernames can change and that string is the only thing used to pull down the key. If you change your name you'll need to hunt down any instance of this program and update it. That's annoying with embedded systems, which is exactly the problem I'm hoping to solve with this application.
• For SSL, the application uses Python's urlib and attempts to fallback on shelling to wget. However, there's no guarantee that wget will honor https requests only. In fact this can be disabled via ~/.wgetrc. They're relying on wget's default behavior without being explicit.
• It checks only if the SSL cert is valid, but doesn't try very hard to see how valid it is. I would have preferred to see it reject any TLS versions lower than 1.2 and only accept EV certificates, since both domains use EV and TLS 1.2.

The last issue worries me the most. Superfish, CNNIC, and eDellRoot all show that rogue certificate authorities are a real and not theoretical problem.

But like most things in the world, it's a trade-off. If you find the convenience outweighs the security risk -- and I do -- then give ssh-import-id a try.

Verifying JSON easily on the command line

You can pipe stdout into python -mjson.tool to validate it. It makes for quick and easy json validation on the command line.

We'll create a simple json file:

> somefile.txt
{"someval": "something", "anotherval": 3}

Now pipe this into json.tools and check the output.

$ cat somefile.txt | python -mjson.tool
{
    "anotherval": 3,
    "someval": "something"
}

Cool. It even formatted it nicely for us. Let's break it and see what happens.

Single quotes are not valid according rfc4627.

> somefile.txt
{'someval': 'something', 'anotherval': 3}

$ cat somefile.txt | python -mjson.tool
Expecting property name: line 1 column 2 (char 1)

Not the most useful traceback, but at least you know it's not valid.