Matthew Ernisse

February 20, 2019 @10:21


Ubiquiti's UniFi platform has the ability to run scheduled speed tests to keep an eye on your ISP's throughput from their USG router at a site. I discovered this back when I finished converting the network at the office over to UniFi and have been wanting to replicate this functionality at my other locations where I use OpenBSD routers. Currently I aggregate the data from those devices into my new Grafana-based monitoring platform which I wanted to continue to use so I could have a consolidated view into the infrastructure.

UniFi Speedtest


The Ubiquiti speed test functionality connects to an echo server running in Amazon AWS and reports back to the controller so the first thing I needed to do was either find an existing way to replicate that functionality or build something similar. Thankfully Debian ships a cli application that functions similarly to the Ubiquiti tester but leverages the already existing infrastructure. I ended up leveraging this and whipping up a really quick and dirty speedtester container that I could run periodically from cron(8).


Every hour cron fires the following script on one of my Docker engine hosts.

Container launch job


set -e

docker run --rm \
    -e INFLUX_HOST=[redacted] \
    -e INFLUX_USER=[redacted] \
    -e INFLUX_PASS=[redacted] \
    -e INFLUX_DB=[redacted] \
    --network grafana_backend \

Inside the container the entry point runs speedtest-cli with a few arguments to select the closest server and provide the output as a JSON formatted string which then gets piped into a small Python script to ship that data off to InfluxDB.

JSON to InfluxDB logger

#!/usr/bin/env python3
# -*- coding: UTF-8 -*-
''' (c) 2019 Matthew J Ernisse <>
All Rights Reserved.

Log the results of a run of speedtest-cli(1) to an InfluxDB database.

Redistribution and use in source and binary forms,
with or without modification, are permitted provided
that the following conditions are met:

    * Redistributions of source code must retain the
      above copyright notice, this list of conditions
      and the following disclaimer.
    * Redistributions in binary form must reproduce
      the above copyright notice, this list of conditions
      and the following disclaimer in the documentation
      and/or other materials provided with the distribution.

import json
import os
import sys
from influxdb import InfluxDBClient

class SpeedtestLogger(object):
    ''' Parse the JSON output of speedtest-cli and post the statistics
    up to an InfluxDB.  DB configuration is stored in the environment.
    def __init__(self):
        ''' Read the configuration environment variables and setup
        the InfluxDB client.

            INFLUX_HOST - Hostname of InfluxDB server
            INFLUX_PORT - Port (8086 by default)
            INFLUX_USER - Username to authenticate with
            INFLUX_PASS - Password to authenticate with
            INFLUX_DB   - Database to log to

        The Measurement will be called: speedtest

        The fields will be:
            download, upload, and ping

        The measurements will be tagged with:
            server_id, country_code, city, sponsor

        self.influx_config = {
            'host': os.environ.get('INFLUX_HOST'),
            'port': int(os.environ.get(
            'user': os.environ.get('INFLUX_USER'),
            'pass': os.environ.get('INFLUX_PASS'),
            'db': os.environ.get('INFLUX_DB')

        self.fields = {
            'download': 0.0,
            'upload': 0.0

        self.tags = {
            'city': '',
            'country_code': '',
            'server_id': '',
            'sponsor': ''

        self.timestamp = 0.0

        self.ifclient = InfluxDBClient(

    def parseJson(self, s):
        obj = json.loads(s)
        self.fields['download'] = obj['download']
        self.fields['upload'] = obj['upload']
        self.fields['ping'] = obj['ping']

        self.tags['city'] = obj['server']['name']
        self.tags['country_code'] = obj['server']['cc']
        self.tags['server_id'] = obj['server']['id']
        self.tags['sponsor'] = obj['server']['sponsor']

        self.timestamp = obj['timestamp']

    def postToInflux(self):
        if not self.timestamp:
            raise ValueError('No timestamp, did you call parseJson first?')

        measurements = [{
            'measurement': 'speedtest',
            'tags': self.tags,
            'time': self.timestamp,
            'fields': self.fields


if __name__ == '__main__':
    if sys.stdin.isatty():
        print('stdin is a tty, aborting!')

    logger = SpeedtestLogger()

    with sys.stdin as fd:
        input =


Now that the data is available to Grafana I was able to easily add some panels to my existing router dashboard to include the test measurements.

Grafana Router Dashboard


Start to finish it only took a few hours to get all of this put together. I didn't need to put the speed tester into a container but it seems like a reasonable idea to ensure I have a wide array of future deployment options open to myself. I already have some cloud hosted assets, so it makes sense to be able to extend the monitoring into those environments if the need arises. Even though I have less than a week of data I find it a bit surprising that my ISP has been fairly reliable. I'm currently on an up to 100Mbps / 10Mbps plan and the 95th percentile results over the last few days have been within 15% or so of meeting that claim. I remain impressed with the ease of use and flexibility of the tools, back when I worked for a national ISP we collected some similar information for billing dedicated Internet customers and it was all done with a large web of custom code.

Grafna panel configuration

Grafana made visualizing the information almost shockingly easy. Especially nice is the built in transforms that allow you to calculate the 95th percentiles over arbitrary time windows.

Honestly I'd love to see features like this built into consumer endpoint gear, I think it would help keep ISPs honest.

January 28, 2019 @21:01

Why are you a green bubble?

People often ask me why I have so much of the features of my phones turned off. My iPhone has iCloud, Siri, FaceTime and iMessage all firmly disabled and have since I originally setup the phone, my Mac has never signed into iCloud, and my Android phone has just about everything including Google Play Services disabled. My personal philosophy is that if it doesn't provide me with value, I disable it.


I will just leave this here

Add in all the lock screen bypass bugs and the fact that Apple still won't keep HomeKit turned off properly and I have to wonder why anyone involved in technology trusts any of this crap.

This is how you get ants

So I'm just going to leave this here so I can link it the next time someone asks me why I'm a green bubble.

January 05, 2019 @17:10

I own my own cable modem and have for the past 10 or so years. At first it was to save on the rental fee for the garbage equipment the local cable company supplied, but since they have stopped charging that it became more of a control thing. I have no need for the garbage router or wifi access point that they supply. I used to work for an ISP and so I'm well aware of the quality and support these devices receive. (Fun fact, the average cost per unit target when I was in the business for a residential CPE device (customer premise equipment) was between US $3 and US $8. For business devices it went up to a whopping US $25 or so...) I also prefer greatly the features and power that my OpenBSD routers give me and I've written more than a few posts about the various UniFi devices I've scattered around to provide WiFi. A few months ago the old Motorola/Arris SurfBoard 6141 I've had started playing up. It needed rebooting periodically to maintain the speeds provisioned. It was probably close to 7 years old and even though it's still a supported DOCSIS 3.0 modem the specs are starting to leave a bit to be desired...

I've used the SurfBoard products since I first got cable internet in the late 1990s and have always had good luck with them so I figured why change now and went and bought a new SB8200. The specs seem to be enough of an upgrade that I'll likely get another 6 or 7 years out of it barring fiber to the home finally coming to my area.

While playing around with the new modem I decided that I wanted to monitor the statistics it provided. Sadly it seems that the cable company, probably in response to the various information disclosure vulnerabilities decided to block the SNMP agent on the device. I'm all for good security practice but it would be nice to provide SNMP on the LAN side at least. Thankfully it still lets you access the web server so once again Python and BeautifulSoup to the rescue.

Arris SB8200 Web Interface

I pulled all the data from the web interface into the same InfluxDB and Grafana platform that I have been talking about lately and pulled together the following dashboard.

Arris Modem Statistics Dashboard

This is strictly a layer 2 look at the health of my circuit. There are separate dashboards monitoring my OpenBSD router for layer 3 information. It does give me a good look at what is going on and just adds to the toolbox of data that I have available to troubleshoot issues. Now that the better three-quarters is working from home full time this is even more important since while I always thought of the network here at home as being all in production, her income is now dependent upon it.

I've got to clean up the script a bit but once I do I'll post it in my miscellaneous script git repository if you want to look at it. It probably won't work with other versions of the Arris SurfBoard modems so be warned it won't be a copy and paste job.


December 28, 2018 @10:37

I like metrics. I've been working lately to convert some aging parts of my monitoring and alerting infrastructure over to something slightly more modern (in this case Grafana, InfluxDB, Telegraf and collectd). As part of this project I'm looking broadly at my environment and trying to decide what else to monitor (and what I don't need to monitor anymore). One of the things that came up was website performance.


Now if you go all the way back to my first post on this iteration of the blog you'll see that this website is static. In fact if you dig around a bit you'll see that I render all the pages from Jinja templates via a git post-receive hook so all my webserver is doing is sending you data off disk. Since server side metrics are practically useless in this case I naturally gravitated towards collecting RUM or Real User Metrics.

Privacy Goals

So now we enter what is (for me at least) the slippery slope. Once you decide to ship information back from the user agent to the server you open yourself up to a huge amount of options. Do I try to do cute stuff with cookies and LocalStorage to assign a user a unique ID (or maybe do some browser fingerprinting) to watch 'visits', do I try to gather detailed device information (ostensibly so I can test the site based on what people use), do I try to determine how people got to my site by history snooping? The possibilities for data collection in this fashion are pretty intense. For me though the answer is almost entirely a resounding no. As someone with a three tiered malware / tracking blocking infrastructure I don't want to be part of the problem. I firmly believe that privacy and security go hand in hand on the web so I refuse to collect any more information than what I need. I also think that this is just plain good security practice. Sadly this is in direct conflict with the urge and ability to just slurp everything into storage somewhere which seems to be a pretty successful business model (and generally horrific practice) these days. Realistically I don't think that someone's going to hack my site to get browsing details on the 20 or 30 visitors per day that I get but I see it as an excuse to lead by example here, to be able to say that you can collect useful real user metrics easily, fast and with an eye towards privacy.

Browser Side

If you view the source of this page, just before the footer you'll see a script tag loading 'timing.js'. It is pretty self explanatory, all this does is fire the sendLoadTiming() function once the DOM readyState hits the complete phase. This ensures that all the performance metrics are available in your browser. sendLoadTiming() then simply takes that PerformanceTiming data in your browser adds the current page location and your user agent string and sends it along to a Flask application.

Server Side

The Flask application receives the data (a JSON object) from timing.js and is responsible for sanitizing it and sending it off to InfluxDB so I can graph it. The first thing that happens to a request is that I compute 6 timing measurements from the data.

  1. How long DNS resolution took
  2. How long connection setup took
  3. How long the SSL handshake took
  4. How long the HTTP request took
  5. How long the HTTP response took
  6. How long the DOM took to load

I then sanitize the user agent string into browser and platform using the Request object provided by Flask which exposes the underlying Werkzeug useragents helper. Based on the platform name I (crudely) determine if the device is a mobile device or not. Finally I collect the URL, and HTTP version used in the request. All of this information becomes tags on the 6 measurements that get stored in InfluxDB.



Once the data is in InfluxDB it is a pretty simple task to create a Grafana dashboard. The JSON for the dashboard I use is available here, though it will likely need some customization for your environment.


So far I think it's been a success. Leveraging JavaScript for this has kept all of the automated traffic out of the data set meaning that I'm really only looking at data with people on the other end (which is the point). It has enough information to be useful but not so much as to be a potential invasion of privacy. All of the parts are open source if you want to look at them or use them yourself.

I may add a measurement for DOM interactive time as well as the DOM complete time. It looks like a LOT of people have some strange onLoad crap happening in their browser.

Some strange timing from Android

Also, once the new PerformanceNavigationTiming API is broadly supported (once Safari supports it, at least) I'll need to migrate to that but I don't see that as a major change. I'd love to hear from you if you found this useful or if you have any suggestions on expanding on it.


December 26, 2018 @14:52


Last year I wrote about my favorite podcasts so I figured I'd do the same this year. In no particular order, though I will call out the ones that I'm still listening to first.

Favorite Podcasts 2018

Still Listening

I have about 20 podcasts on my iPhone right now, but more than half are sitting there waiting to be listened to. Digging out of the backlog of some of these podcasts is quite the epic task.

King Falls AM

Though they have been on hiatus between seasons for a good part of the last few months this is still one of the few podcasts that I stay up to date with. It is epic. If you enjoy audio dramas, listen to this. I can't state enough how awesome this podcast is. 📻

Troy Hunt's Weekly Update Podcast

I wrote a good synopsis of this last year, and it all still holds. This is one of my touchstones for the InfoSec community. If you are interested in technology and security at all, it is worth checking Troy out.

Downloadable Content

Penny Arcade has been doing a lot in the last few years, largely on but the behind the scenes podcast is still going strong.

Above & Beyond: Group Therapy

Above and Beyond continue to provide the background music for any yard or house work that I end up having to do. Sit back, relax, and enjoy the chill vibes from the electronic music world... or get some work done. 🎧

New this year

Elite: Escape Velocity

From the world of Elite: Dangerous, set in the 33rd century Escape Velocity is a full cast audio scifi serial by Christopher Jarvis and The Radio Theatre Workshop. I think I blew through this in a weekend. I am anxiously waiting for Series 4.
Even if you aren't an Elite: Dangerous player check this out, I think the story still holds up. :🍸:

Jason Scott Talks His Way Out of It

If you don't know Jason Scott I don't think I can do him justice in a short intro. A Free Range Archivist at the venerable Internet Archive, curator of software oddities, documentarian, textfiles enthusiast, DEFCON speaker, and now podcaster. Jason tells lots of stories on a weekly basis in an effort to educate, entertain and, get himself out of debt.


I can't believe I only started listening to this podcast this year. I was a massive The History of Rome fan and Mike Duncan is back at it with his dry wit, and silky voice dropping wonderful narrative history in a weekly(ish) podcast. Focused on the great revolutions of history this is worth a listen even if you aren't a history buff. I'll also toss a quick plug in for his book The Storm Before the Storm which I read, listened to (he narrates the Audible version so it is like getting a huge bonus episode of The History of Rome), and loved every minute of.

Risky Business

I stumbled across this at some point this year while hunting for more InfoSec content. It is a really informative and well produced weekly look into what is going on in the information security world. If you are in IT at all, this is worth a listen.

The Adventure Zone

Another one that I can't believe I started listening to this year. I blew through the back catalog in like 3 months and listen to each episode as soon as it comes out. I wrote a dedicated article earlier this year about it so go check that out if you want to know more.

The History of Byzantium

Robin Pierson decided to do something amazing. He took it upon himself to take up the mantle of Mike Duncan and pick up the story of the Romans where The History of Rome left off. Styled as the spiritual successor Robin has done a flat out amazing job. I'm currently only in the 10th century AD, furiously trying to catch up but this podcast truly lives up to the creator's intent. If you like history, listen to this.

The Pirate History Podcast

I think pirates are cool. I like to know things about the things that I think are cool. This was a recommendation from a friend and it has been so worth it. Give it a listen and get Matt Albers take you through the golden age of piracy. 🏴‍☠️

The Invisible Network

Last but not least, a podcast I found quite literally two days prior to writing this. I listened to all 6 episodes currently released in one sitting. A really lovely look behind the scenes on how NASA has and continues to work to bring humanity to the stars. Produced by the Space Communications and Navigation program office at NASA this is a must listen for all space geeks out there. 🚀


If a podcast wasn't listed here that was on the 2017 list it means I've stopped keeping up to date on it this year. Most likely because of the vast back catalogs of the new podcasts this year that I have been chewing through so don't take their absence as a negative review. I have something like 16GB of unlistened to episodes on my phone right now that I'm working my way through. The ones that are back this year are something special though.

Hopefully 2019 will be as filled with great podcast content as 2018. 🍻

December 22, 2018 @16:10

Merry Christmas, Happy Hanukkah, Happy Saturnalia, Happy Festivus, Joyous Yule, and congratulations on surviving the solstice to everyone. Be safe and enjoy some time with the people that are important to you this holiday season as the Earth hurtles towards perihelion.

🎄 🎁

Merry Christmas from Bennie and I

December 21, 2018 @09:28

If you have read my previous post about monitoring my ADS-B receiver it probably won't come as a surprise that the impetus for this whole project has been to deprecate MRTG from my environment. MRTG was a fine enough tool when it was basically all we had (though I had rolled a few iterations of a replacement for personal projects over the years) but these days it is woefully dated. The biggest issues lie in the data gathering engine. Even a moderately sized environment is asking for trouble, dropped polls, and stuck perl processes. MRTG also fails to provide any information beyond the aggregated traffic statistics.


Years ago I wrote a small script that renders some web pages to display the switchports on the network linked to their MRTG graphs. Each port is enumerated by operational status and description to make it easy to find what you are looking for. It turns out it also makes it pretty easy to throw MRTG out and switch to something else.

I had already settled on Grafana and InfluxDB for a large part of the new monitoring infrastructure with most of the data being collected via collectd running on all my physical and virtual hosts. I am monitoring containers with cAdvisor which also feeds into InfluxDB, so I very much wanted to keep data going into InfluxDB yet I needed something to bridge the gap to the SNMP monitoring that the switches and UPSes in my network require. Enter Telegraf.

My only complaint is that the configuration for the SNMP input module in Telegraf is garbage. It took a bunch of trial and error to figure out the most efficient way to get everything in and working. I do very much like the results though...


Setting up Telegraf as a SNMP agent

There are a number of blog posts kicking around with fragments of information and copy/paste chunks of configuration files but not much in the way of well written documentation. I guess I'll just pile more of the former on.

I deployed Telegraf as a Docker container, though the configuration is largely the same if you deploy directly on a host. I did install all the SNMP MIBs I needed (in Debian, the snmp-mibs-downloader package covered most of them, I added the APC PowerNet MIB for my UPSes and the Synology MIBs for my work NAS) on my Docker host so I could mount them into the container. I pulled the official container and extracted the default configuration file.

docker run --rm telegraf telegraf config > telegraf.conf

With that in hand I set about killing the vast majority of it, leaving only the [agent] section. Since I am only doing SNMP collection the only change I made there was to back the interval off to 120s instead of 10s.

I then configured Telegraf to send metrics to InfluxDB

# Configuration for sending metrics to InfluxDB
  urls = [ "http://influxdb:8086" ]
  database = "telegraf"
  skip_database_creation = true
  username = "[REDACTED]"
  password = "[REDACTED]"

This just left the SNMP input configuration, which I'll break up and describe a bit inline.

  agents = [ "" ]
  community = "[REDACTED]"
  version = 2

This is pretty self-explanatory, the basic information to poll the agent. You can pass a list into agents and it will use all the same configuration for all of the targets. You can have multiple inputs.snmp stanzas.

  name = "hostname"
  oid = "SNMPv2-MIB::sysName.0"
  is_tag = true

This collects the value of the SNMPv2-MIB::sysName.0 OID and makes it available as a tag.

  inherit_tags = [ "hostname" ]
  oid = "IF-MIB::ifXTable"

This is the meat, it walks the IF-MIB::ifXTable and collects all the leaf OIDs as metrics. It inherits the hostname tag from above.

      name = "ifName"
      oid = "IF-MIB::ifName"
      is_tag = true

      name = "ifDescr"
      oid = "IF-MIB::ifDescr"
      is_tag = true

      name = "ifAlias"
      oid = "IF-MIB::ifAlias"
      is_tag = true

These specify additional OIDs to use as tags on the metrics. The difference between this and the hostname tag is that these are scoped to the index in the walk of the IF-MIB::ifXTable, so if you are looking at index 0 in IF-MIB::ifXTable, it will fetch IF-MIB::ifName.0 and use that. I put the configuration and a docker-compose file in Puppet and let the agent crank the wheel and was rewarded with a happy stack of containerized monitoring goodness.

Telegraf, InfluxDB and Grafana in Containers

The compose file is below, but I'll leave the configuration management bits up to you, dear reader.

version: '2'
      - MIBDIRS=/usr/share/snmp/mibs:/usr/share/snmp/mibs/iana:/usr/share/snmp/mibs/ietf:/usr/share/snmp/mibs/syno
      - grafana_backend
      - /var/local/docker/data/telegraf/telegraf.conf:/etc/telegraf/telegraf.conf:ro
      - /usr/share/snmp/mibs:/usr/share/snmp/mibs:ro
      - /var/lib/snmp/mibs/iana:/usr/share/snmp/mibs/iana
      - /var/lib/snmp/mibs/ietf:/usr/share/snmp/mibs/ietf

      name: grafana_backend

Gluing it to Grafana

The last piece was updating the links to the new graphs. Happily if you setup a variable in a dashboard you can pass it in the URL to the dashboard so I was able to simply change the URL in the template file and regenerate the page.

Graph Homepage

In my case the new URL was

https://[REDACTED]/grafana/d/[REDACTED]/switch-statistics?var-host=[SWITCH NAME]&var-port=[PORT NAME]

Hopefully this makes it a little clearer if you are trying to achieve a complex SNMP configuration in Telegraf.


December 04, 2018 @11:00

I'm trying to figure out a way to balance the lack of surprise and schadenfreude I have at Tumblr/Verizon's decision to paint all sexual content with the regressive and transparent 'but think of the children' brush. Tumblr grew largely thanks to the alternative and adult communities that found its permissive and accepting nature welcoming. It became what it is today because of the LGBTQ+ and sex worker communities, and now it has decided to break up. Their post paints a pretty picture full of platitudes, inclusiveness, acceptance, and love of community but it is obvious to the most casual of observer that it is just a sham. Tumblr is breaking up with the people that helped it grow because it is easier than trying to actually make the service a better place.


I feel bad for the users that are being displaced, some of whom I have followed for close to a decade. I admit I do feel a bit like pointing and laughing at the management of Tumblr who just signed their product's own death warrant, but most of all I feel like this is yet another billboard for retaining control of your community, and your content. In an era with filled with social media companies promising to help you build communities and share content it is more important than ever to remember that at the end of the day they all will betray you eventually because you aren't their customer, you are their product. At some point your needs will clash with theirs and they will without remorse chose themselves every single time.

Anyone who creates anything on the Internet needs to relentlessly protect their community by ensuring that they have control. I am sure it sounds a bit bizarre to some but if you are going to use services like social media to engage people (which you basically have to right now) you need to act like any day you could wake up and find them gone. You need to ensure that people can find you again, that your content and community doesn't just disappear and that you can move on to whatever comes next.

Tumblr will die, the Internet will move on. In a couple years it will be another story the olds tell the kids these days, but hopefully... we learn. In the mean time, I'm firing up grab-site and archiving the people I have followed on Tumblr for the last 10 years. Hopefully we will cross paths again.

To The Archive Mobile!

For creators, find a way to root your community in something you control. Go pay Ghost or a similar host to house your blog. Domain names are cheap these days, I like Gandi but there are many places that will sell you one. Resist the urge to get a free blog with a free url. Being is no less risky than being It's not expensive, or difficult anymore to maintain a presence online where you are the customer. It isn't perfect, but at least when you own the domain name if you need to change providers your address stays the same and your community can follow you to your new home. Link everything to your blog and link your blog to everything. Make it the clearing house for all that you are doing, make it easy for your community to follow you when the inevitable happens.

For members of communities, and followers of creators, if it isn't clear where to go next reach out to the creators. Many of them are scrambling to find a place to land or to let all their followers know where else they can be found. If you don't know ask, and politely suggest they think about creating a place they own to anchor their community if they havne't already..

November 29, 2018 @10:57

I have been running a FlightAware / FlightRadar24 ADS-B feeder for almost 4 years now. It is an older Raspberry Pi B with a RTL-SDR stick running dump1090 at its core. These days it is mounted in my garage with the antenna on the roof. When I built it I stuffed a Honeywell HIH6130 temperature and humidity sensor in the enclosure. At the time it was mounted on a fence in my back yard so it would be in full sun for much of the day so I hooked it up to Icinga to alert me if it ever got too hot or too wet inside.

asdb-feeder in August 2014

Lately I've been investigating ways to get more information into a central location for my infrastructure as a whole. I have a lot of one-off, largely custom built systems to collect and aggregate system status data. While this has worked for the last 12 years, it is most certainly starting to show its age. At the moment I'm working with a stack that includes collectd, InfluxDB, and Grafana. The latter two run as Docker containers, while the former is deployed by Puppet to all my physical and virtual hosts.

I wanted to pull together some additional monitoring information from the ADS-B feeder to see just how far I can go with this setup. Luckily the dump1090 web interface works by reading JSON files from the receiver daemon, so all the interesting statistics are available on disk to read.

dump1090-fa web interface

I was able to pull together a quick python script that loads the JSON and emits the statistics to collectd (which forwards them onto InfluxDB for Grafana to work with). I need to get the script into git somewhere but for now, here is the currently running copy.

#!/usr/bin/env python3
''' (c) 2018 Matthew Ernisse <>
 All Rights Reserved.

Collect statistics from dump1090-fa and send to collectd.  Uses the collectd
Exec plugin.

Redistribution and use in source and binary forms,
with or without modification, are permitted provided
that the following conditions are met:

    * Redistributions of source code must retain the
      above copyright notice, this list of conditions
      and the following disclaimer.
    * Redistributions in binary form must reproduce
      the above copyright notice, this list of conditions
      and the following disclaimer in the documentation
      and/or other materials provided with the distribution.

import json
import os
import socket
import time

def print_aircraft(stats):
    ''' Parse and emit information from the aircraft.json file. '''
    aircraft = len(stats.get('aircraft', []))
    messages = stats.get('messages')
    if not messages:
        raise ValueError('JSON stats undefined')

    m = "PUTVAL \"{}/dump1090/counter-messages\" interval={} N:{}".format(

    m = "PUTVAL \"{}/dump1090/gauge-aircraft\" interval={} N:{}".format(

def print_stats(stats):
    ''' Parse and emit information from the stats.json file. '''
    counters = [

    gauges = [

    values = stats.get('local')
    if not values or not type(values) == dict:
        raise ValueError('JSON stats undefined')

    for k in counters:
        value = values.get(k)
        if not value:
            value = 'U'

        m = "PUTVAL \"{}/dump1090/counter-{}\" interval={} N:{}".format(

    for k in gauges:
        value = values.get(k)
        if not value:
            value = 'U'

        m = "PUTVAL \"{}/dump1090/gauge-{}\" interval={} N:{}".format(

if __name__ == '__main__':
    interval = float(os.environ.get('COLLECTD_INTERVAL', 10))
    hostname = os.environ.get('COLLECTD_HOSTNAME', socket.getfqdn())

    while True:
        with open('/var/run/dump1090-fa/stats.json') as fd:
            stats = json.load(fd)

        stats = stats.get('total')

        with open('/var/run/dump1090-fa/aircraft.json') as fd:
            stats = json.load(fd)


I also wanted to pull the temperature / humidity sensor readings, that ended up being a similarly easy task since I already had written a script for Icinga to use. A quick modification to the script to emit the values in the way that collectd wants and that was flowing in. I created a user for the i2c group so the script can use the i2c interface on the Raspberry Pi.

The script currently looks like this.

#!/usr/bin/env python3

import os
import socket
import time
import smbus

def read_sensor():
    ''' Protocol guide:
    bus = smbus.SMBus(0)
    devid = 0x27

    # writing the device id to the bus triggers a measurement request.

    # wait for the measurement, section 3.0 says it is usually
    # 36.65mS but roll up to 50 to be sure.

    # data is 4 bytes
    data = bus.read_i2c_block_data(devid, 0)

    # bits 8,7 of the first byte received are the status bits.
    # 00 - normal
    # 01 - stale data
    # 10 - device in command mode
    # 11 - diagnostic mode [ ignore all data ]
    health = (data[0] & 0xC0) >> 6

    # section 4.0
    humidity = (((data[0] & 0x3F) << 8) + data[1]) * 100.0 / 16383.0

    # section 5.0
    tempC = ((data[2] << 6) + ((data[3] & 0xFC) >> 2)) * 165.0 / 16383.0 - 40.0

    return (tempC, humidity)

if __name__ == '__main__':
    interval = float(os.environ.get('COLLECTD_INTERVAL', 10))
    hostname = os.environ.get('COLLECTD_HOSTNAME', socket.getfqdn())

    while True:
        retval = read_sensor()
        print("PUTVAL \"{}/hih6130/gauge-temperature\" interval={} N:{:.2f}".format(

        print("PUTVAL \"{}/hih6130/gauge-humidity\" interval={} N:{:.2f}".format(


The collectd plugin configuration is pretty easy, the dump1090 files are readable by nogroup so you can execute that script as nobody. As I said I made an i2c user that was member of the i2c group so the Python SMBus module can communicate with the sensor.

LoadPlugin exec
<Plugin exec>
    Exec "nobody:nogroup" "/usr/bin/"
    Exec "i2c:i2c" "/usr/bin/"

Once the statistics were flowing into InfluxDB, it was just a matter of putting together a dashboard in Grafana.

Summary Panels

Host Status Panels

SDR Status Panels

The JSON from Grafana for the dashboard is here, though it may require some tweaking to work for you.

So far I'm pretty happy with the way this all went together. I still have a bunch of network equipment that I'd like to bring over and a stack of ancient MRTG graphs to replace. Hopefully it will be a similarly simple experience.


November 26, 2018 @15:30

While I was waiting for new tires to be put on my car today I was able to watch the landing of Mars InSight which was relayed via the MarCo A&B experimental interplanetary cube sats.

Misson Control as touchdown was confirmed

Since everything worked so well we even got back a picture from the lander mere moments after landing was confirmed.

Hello, Mars

Congratulations to everyone involved in this mission, I'm excited to see what we learn not only about our friend the red planet but also about the continued feasibility of the cube sat program. Maybe we'll see something like the PlanetLabs Dove cube sat streaking towards Mars someday.

MarCo Relay Animation from NASA/JPL-Caltech

November 25, 2018 @23:40

I know it's not particularly uncommon for web sites these days to drastically change things and in fact most people consider this a feature. The fail fast mentality is great and all except that it means you are often failing more than not and the general consensus seems to be that it's perfectly acceptable to do it in public to the detriment of your users.

There are a few patterns that I really wish would die. One of the worst offenders are single page "apps" that hijack the history navigation controls of your browser to keep you from having to download the 20MB of JavaScript on every page load making it next to impossible to refresh the page after the inevitable crash of the script somewhere in the dark mess of the minified JavaScript source. The other is the continued fad of styling the native browser video controls and overriding the functionality that is built in to provide "a consistent look and feel across platforms." This wouldn't be quite so annoying if it didn't almost always break in some unique and fun ways and omit features that the developer didn't have on their personal laptop. I don't think I've seen a single video site that skinned the HTML video element that provided a native full screen view or picture in picture on macOS. I find those features really useful and Apple worked very hard to optimize them for performance and battery life so it would be great if people would just leave them alone.

To move from a general rant to something slightly more specific here are a few examples from the latest YouTube redesign that really drives home the level of amateur hour that keeps infesting the web.

This video was not vertically letter boxed

No, this video was not letter boxed

I know CSS is hard, but come on Google...

I guess even Google doesn't get CSS

Yes, I always wanted to scroll left and right to see my whole video

Why would you ever scroll part of a video off screen?

Insert sad trombone here.

November 07, 2018 @23:30

A little over six and a half years ago I left the Linux as a desktop community for the Mac community. I replaced a Lenovo Thinkpad T500 for an Apple refurbished late 2011 MacBook Pro and honestly have not regretted it.

Over the years I maxed out the memory, went from the 500G SATA HDD to a Crucial 256GB SSD, then put the 500G HDD in the optical bay, then upgraded to a Samsung EVO 512GB SSD with the optical drive back in there. I replaced the keyboard twice, the battery twice, and had the logic board replaced out of warranty for free by Apple thanks to the recall for an issue with the discrete graphics. Through all that it quite happily chugged along and for the most part just worked. Even now it's in better shape than most of my old laptops, but the lid hinge is starting to get weak (it will often just slowly close itself on me), it needs yet another new battery, and the inability to run the latest macOS lead me to conclude that it is time to look for an upgrade.

Old Laptops

It ended up being a bit of a challenge to decide on an upgrade, though. I really like the 13" Retina MacBook Pro I have for work, I really like the portability of the MacBook, and the new MacBook Air looks like a great compromise between the two. I fought with myself for quite some time over what would come next for me and finally settled on a 15" Mid-2015 Retina MacBook Pro. Essentially the bigger brother of what I have for work.

Hello, Kitsune

Now I won't blame you if you are wondering why I'd pick a 3 year old laptop over the latest and greatest. In the end it was almost the only choice. I wanted a 15" class laptop because I spend most of my time computing sitting on the couch. The 13" is really nice and portable but it's actually a little too light and a tad too small to comfortably use on my lap. That basically ruled out the lighter and smaller MacBook and MacBook Air. As for the newer 15" MacBook Pro, I almost exclusively use the vi text editor so not having a hardware escape key is just not something I feel I can get used to. I've also heard many people at work who do have the new MacBook Pros complain loudly about the keyboard so that was another nail in the coffin of the new models.

Given all that, the last non-touchbar 15" MacBook Pro is... the Mid 2015. I found a nice example with the 2.5GHz i7 and the Radeon R9 on eBay for a real good price after a few weeks of looking and snapped it up.

Since this is the second Mac I've ever had as my primary workstation it was the first time I got to use Migration Assistant. I have previously used recovery mode to recover from Time Machine which works a treat so I had high expectations. In the end I'd say the experience lived up to them. The only real problem I had seems to be related to how I have my WiFi configured. I use WPA2 Enterprise (so username and password authentication) on my main SSID which I configure using a profile in macOS (which also serves to disable Siri, a bunch of iCloud stuff I don't use, sets up my internal certificate trust settings, and my VPN). Every time I started up Migration Assistant it would drop off the WiFi with no explanation. After flailing around a bit it looks like that was because it couldn't access the authentication information after logging me out to start the migration, so I figured I'd use Ethernet. That would have worked except that the laptop showed up on a Saturday and the only Thunderbolt to Ethernet adapter I own was at the office. Thankfully my guest WiFi uses WPA2 PSK and that actually appears to work just fine.


It took about 4 hours to transfer the 210GB or so of data, but afterwards the new Mac looked just the same as the old Mac. A quick run through my customization script to catch the few settings in the new version of macOS, the automounter, and the applications I have installed via homebrew, I have not had to go back. Sunday evening I shut off the old laptop. I do plan on re-deploying it as a test workstation if I ever get around to building a dedicated test environment, but for now it is sitting in a drawer under my desk.

It's been a good laptop and this new one has big shoes to fill.

Goodbye, Aramaki


November 06, 2018 @13:40

It's probably too late to change anyone's mind, but I saw a particularly salient Twitter come across this morning.


Also particularly poignant for me is this morning's post over at McMansion Hell

Nub Sez Vote So please, vote. This is basically the bare minimum required of all citizens in this republic other than paying your taxes (maybe). In any case, it is our only chance to directly influence the policies of this nation and is a right that thousands died to embody us with. Your voice counts, but to get it heard you have to show up.

I Voted 2018

🇺🇸🍻 🎉

October 31, 2018 @21:50

Getting Started

Tor in Containers

I have been looking for reasons to try Docker on one of the random stack of un-used Raspberry Pis that I have laying around and thought it might be fun to build a little travel router. Somehow that morphed into lets get Tor working on here and then well if I can get a client, and a relay, why not an onion service?

Getting the Tor relay / proxy working was pretty easy. The entrypoint script is a little bit long because I wanted to allow for a fair bit of configuration flexibility in how the container is deployed.

You can find the container in the 'tor-relay' directory of my git repo.

I chose to also put polipo in a container to provide a HTTP proxy front-end. This made it pretty easy to get on the Tor network from a machine anywhere on my LAN. I even threw together a docker-compose.yml to bring up both the Tor client and polipo. You can find that in the tor-proxy-bundle of my git repo. Then I decided to go exploring, err researching.

Tor works!

The "Dark" Web

"Onion" services, often times called "hidden" services are addresses that end in .onion and allow operators to provide services over the Tor network without having to disclose their location or IP address. There is no centralized directory of these services like there is with DNS on the 'regular' Internet so discovering what is out there is a bit tricky. After some searching I found that much like the regular Internet there are various directories of links, and search engines available. The big difference in search engine tech is that they seem to start by crawling the regular Internet looking for .onion addresses and then primed with that they can start crawling and indexing links to other .onion addresses just like any Internet search engine would.

My favorite so far is Fresh Onions because I can sort by 'last seen' and just keep poking at whatever it crawled most recently. Things seem to come and go rather frequently on the Tor network so when I was looking at the link directories I kept finding that something like 60% of the links were dead so this provided a better experience.

After a day of poking around I came to realize that as with most technology the general understanding of the dark web is pretty far from reality. The idea that the dark web is awash with black markets, and hit men for hire, and hackers appears to be about as true as it was when people talked about the Internet back in the early 1990s. In fact the reality is that the dark web even looks an awful lot like the 1990s Internet.

Some Examples

What a nice retriever we have here!

No really, that's the name of the site

There are a lot of sites like this, though this is probably the cutest... The HTML is very rudimentary, quite literally the minimum you need to get an image on the screen.

Placeholder, Placeholder, Everywhere.


Lots of placeholders too... Sometimes not even a page but an empty directory listing from a freshly configured webserver with nothing on it.

Under construction, but no .gif.. yet

Under Construction

If you remember the Internet of the 1990s you almost certainly ran across (or maybe even used) one of the many under construction animated gif images that were out there. While I have yet to see one of those pop up on the dark web, there are lots of pages that purport to be under construction.

Sign my Guestbook!

No really, please sign it!

If you remember the under construction gif you probably also remember guestbooks. A rudimentary precursor to the blog comment box, these let visitors leave public notes for the site owner. Often times these devolved into... well what you might expect from an anonymous board where anyone can post anything...

Turns out those exist on the "Dark" web too.

"Dark" Thoughts

Design aside the "anonymity" of the dark web is very similar to the feeling back in the 1990s and early 2000s Internet. Before advertizers could track you all across the Internet it had a "wild west" feel in places. There were lots of aliases (the hacker nom-de-plume or 'handle'), and strange usernames (I was mee156 at one point thanks to a particularly uncreative corporate IT department) and often they were ephemeral. There were plenty of sites purporting the same sort of potentially illegal (often fake) products and services attributed to the dark web all because by and large you were anonymous (sorta). In a way as someone who grew up in those early days it is actually sort of heartening to see a bit of a renaissance so that maybe the kids today will get a shot at making some of the same mistakes I did and not have that follow them forever.

Hidden service in a (pair of) container(s)

Docker, Tor, Raspberry Pi

There are a lot of reasons people might run an onion service. Nefarious purposes aside if you aren't just using it for research or as a way to provide a link back into your private network then you are probably concerned about anonymity. There have been a few good DEFCON talks about anonymity and security on Tor and how people often screw it up.

While not a silver bullet it seems like putting your service and the Tor client that provides your .onion address into isolated containers are reasonable first steps towards operational security. By isolating the network to just the two containers you can reduce the attack surface and information leaked if the service is compromised. You can also develop the service in isolation (say on your laptop) and then transport it to another machine to deploy it providing an airgap. Beyond that wrapping this into containers makes it simpler to deploy just about anywhere. You could even put them up on a public cloud provider (if you can get an anonymous account), or since this works on a Raspberry Pi you could hide the Pi somewhere other than your home or work and simply snag an open Ethernet port or WiFi network (obviously with permission from the owner...).

Similar to the proxy and relay stuff, you can see an example docker-compose.yml, hidden service client, and Apache instance over in my git repo (under onion-service-bundle, onion-service, apache-rpi respectively).
The example onion service that I have on my Pi right now is available here if you are interested.


Containers bring a lot of interesting possibilities to systems like Tor, where you are essentially creating an overlay network that you are then isolating and keeping largely ephemeral. The onion service keeps a little state (public/private key pairs) but for the most part there isn't anything that needs to be kept around between container runs. There are also other ways to create tunnel connections from inside a container to the world, opening up many different possibilities.

The other interesting thing is that while there are a lot of sites claiming services like what you read in typical reports about "The Dark Web", the vast majority of what is out there are either legitimate attempts to provide anonymous services (eg: The New York Times via Tor, SecureDrop to pass sensitive tips to journalists, and publications or collections of written works like zines), or research / experimentation like the examples above (and my own test service). There is even a streaming radio service over Tor out there.

I think demystifying things helps normalize them. There are plenty of people who use Tor to be able to access the free and open Internet in ways that those of us in countries that don't censor the Internet take for granted, and people who live under regimes so oppressive that reading certain things or posting certain opinions can earn them real jail time. It is important for more people to use Tor for the usual everyday things, provide relays, and run onion services to ensure that the people who are under real threat have more noise to hide in.

Final Thoughts

Peeking around Tor onion services did leave me with one other piece of advice I'd like to pass along. If you have not already I'd urge you to do two things.

  1. Use a password manager. I generally recommend LastPass, but people I know and trust like 1Password as well. This is your best defense the next time some service gets breached and your data ends up out there (more often than not it is found on the regular Internet).
  2. Sign up for Troy Hunt's very good and free service Have I Been Pwned. This will alert you when your data has been found in a data breach.


September 27, 2018 @11:30

I was making some firewall changes last weekend and while watching the logs I discovered that every now and then some host would try to connect to on port 80. This was peculiar since I don't use the IPv4 link local addresses anywhere in my network. It seemed to be happening randomly from all of my Linux hosts, both physical and virtual.

If the processes originated on my firewall which is running OpenBSD I'd be able to track down the process that was doing this by adding a more specific rule with "log (user)" to pf.conf(5) but it seems that Linux dropped this ability from Netfilter sometime back in the 2.6 time frame. 😢

The part that makes this a bit unique is that this is a connection that will certainly fail. It trips a rule in my firewall that blocks bogons outbound on my WAN interface which means that the normal tools like netstat(1) and lsof(8) will only reveal anything if I somehow catch the process between the execution of the connect(2) system call and it failing. What I need to be able to do is log in real time what is happening, which I could do with something like strace(1) but I'd need to know the PID of the process and that is what I'm trying to find out.

So off I went looking for other things that might be helpful and stumbled upon the Linux kernel audit system. The audit system has been around for a while and lets you ask the kernel to communicate the details of syscalls as they happen. There is a filtering mechanism built in so that you don't end up dumping too much information or dramatically impacting performance and the raw data is sent to a userland program via a netlink socket. By default most distributions ship auditd, which listens on that netlink socket and dumps all the messages into a log.

Since I am looking at an attempted TCP connection the connect system call is the one I am interested in. I don't know much else about it though so it turns out a pretty simple filter rule is what I was looking for.

$ sudo auditctl -a exit,always -F arch=b64 -S connect

This asks the kernel to log upon exiting of the syscall any calls to connect(2). This immediately started flooding the audit log with entries like:

type=SYSCALL msg=audit(1538057732.986:13752): arch=c000003e syscall=42 success=yes exit=0 a0=12 a1=7ffc987e28c0 a2=6e a3=7f20abf93dd0 items=1 ppid=9803 pid=19584 auid=4294967295 uid=33 gid=33 euid=33 suid=33 fsuid=33 egid=33 sgid=33 fsgid=33 tty=(none) ses=4294967295 comm="apache2" exe="/usr/sbin/apache2" key=(null)
type=SOCKADDR msg=audit(1538057732.986:13752): saddr=01002F746D702F7...
type=CWD msg=audit(1538057732.986:13752): cwd="/"
type=PATH msg=audit(1538057732.986:13752): item=0 name="/tmp/passenger.PIT9MCV/agents.s/core" inode=1499146 dev=fe:01 mode=0140666 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL
type=PROCTITLE msg=audit(1538057732.986:13752): proctitle=2F7573722F7362696E2F617...

OK, so I'm getting closer but obviously some of the data is coming out in some packed hex format and the things I want aren't all on the same line so I need to figure out how to decode this. While searching for the format of the messages in hopes of writing a quick and dirty parser I found ausearch(8) which includes the handy -i option.

I fired up tcpdump(8) on the pflog(4) interface and waited for one of the packets to be dropped. That let me find what I was looking for in the audit log... the culprit.

ausearch and tcpdump to the rescue

It turns out it was a puppet agent run. Now I know none of my modules try to talk to that address but puppet does a lot of things including running facter to get information about the system and the environment it is running in. I know some cloud infrastructure has standardized on that address as a location for guest agents to pick up metadata from so I suspected some default module trying to see if we are running on a cloud provider. A quick locate(1) and grep(1) later and it turns out that the built in facter ec2 module does in fact try to pull metadata from

apollo@10:16:49 1.8T ~ >locate facter
[ ... many paths elided for brevity ...]
[ ... many more paths ...]
apollo@10:16:53 1.8T ~ >grep -R /usr/lib/ruby/vendor_ruby/facter/*
/usr/lib/ruby/vendor_ruby/facter/ec2/rest.rb:      DEFAULT_URI = ""
/usr/lib/ruby/vendor_ruby/facter/ec2/rest.rb:      DEFAULT_URI = ""
/usr/lib/ruby/vendor_ruby/facter/util/ec2.rb:      url = ""
/usr/lib/ruby/vendor_ruby/facter/util/ec2.rb:  # GET request for the URI  If the
/usr/lib/ruby/vendor_ruby/facter/util/ec2.rb:    uri = "{version}/user-data/"

So in the end, the Linux audit system is our friend. There is a lot of other cool stuff in there, I ran across a post from the slack engineering team that talks about how they use the audit system and how they leverage this information to alert on and challenge user actions in real time. It is also a cautionary tale that good network hygiene is important since you never know what random things you might leak out onto the Internet (or your ISP's network) if you aren't careful.


Subscribe via RSS. Send me a comment.