DOT4J

I've recently been quite interested in Graph languages, not for Graph Databases, but for representing information, in code that is essentially "things" and "relationships". For example, one perspective on Organizations is that they are simply representable as people, systems and the information that flows between them. Those three building blocks are, in theory, enough to describe anything from a small business to a government, if you have an appropriate way to describe all the information, ensure the connections make sense, and report on it. If you further layer on the idea of systems of systems, you have the ability to describe different levels of government, business units, departments and so on. Public and private flows of information allow models that accurately represent the fact that no organization is fully meshed: the systems only expose certain information.

The reporting on it is quite interesting, and led to be becoming interested in DOT as a way to describe reports resulting from queries. I didn't find a small, embeddable Java library that enabled me to read and write DOT files, so I ended up writing DOT4J.

The writing of DOT files, is trivial. The reading however, ended up requiring a lexer-parser since DOT is, essentially a programming language of its own. The grammar came from ANTLR, of course.

DOT4J is here. The Github page has the Maven co-ordinates for published binaries.

OpenAPI

I recently had reason to get to know OpenAPI at work, so I decided to become familiar with it. I have a HomeSeer home controller at home, which exposes a JSON API, so I decided to write a Java OpenAPI server to expose the JSON API over OpenAPI.

The net result is hsOpenAPI. It exposes most of the HomeSeer API and uses hsClient, the Java wrapper I had previously built for the HomeSeer API.

Every house needs a Service Bus

After all, now that we have wireless thermostats, door locks, outlets, dimmers, alarms, garage doors, and motion sensors, it only makes sense to publish events onto a Service Bus. In my case I used Mosquitto in a FreeBSD jail as my broker.

My home controller is a HomeSeer. There are MQTT plugins available, but I preferred to use my own hsClient library which is a Java wrapper for the HomeSeer JSON API. I ended up writing hsMQTT, an MQTT bridge for HomeSeer which polls the HomeSeer API and publishes JSON messages onto an MQTT topic when device values change.

Typical change messages look like these:

{"deviceRef":107,"value":0.0,"lastChange":1594530001541,"location":"Living Room","location2":"Main Floor","name":"Living Room Dimmer","status":"Off","type":"Z-Wave Switch"}
{"deviceRef":111,"value":0.0,"lastChange":1590035156557,"location":"Study","location2":"Upstairs","name":"Study Light","status":"Off","type":"Z-Wave Switch"}
{"deviceRef":104,"value":18.88888888888889,"lastChange":1594534463932,"location":"Kitchen","location2":"Main Floor","name":"Furnace Thermostat Heating  Setpoint","status":"18.9 C","type":"Z-Wave Heating  Setpoint"}

You can download the source here

Building a personal search engine

I recently got a call from a friend who has a library of thousands of PDF documents. He wanted to know there was a such thing as a personal search engine that could index his library of documents, and allow full text search inside of them.

I have a giant document library too, including PDF documents, Office documents, source code, email and so on. I decided I needed one too, and it should be based on Elastic Search.

The net result of the weekends work is "kSearch", a personal search engine. kSearch include a threaded file system indexer, automatic maintainence of the search index as the file system changes, and a web search UI.

You can see screen shots and source code on the github project.

Home Automation DevOps

Home automation is coming into the mainstream a with recent offerings from the big tech companies, and I'm interested in it too. I've had a number of home automation controllers, most recently an H3 Pro SEL from HomeSeer.

I've also become quite interested recently in DevOps monitoring, using InfluxDB, and GrafanaSensu, Telegraf and Prometheus are great too, but not for this project.

So, I decided to import home status information into InfluxDB and graph it. The first step was getting the information from HomeSeer.   There is a JSON-over-HTTP API, but no existing Java client.  So, a simple HomeSeer H3 API client in Java was written: hsclient

The next step was getting the data into influx.  The best way to do this would be to write a Telegraf plugin.  However, I wanted a chance to learn a little more about Java closures and lambas, so I wrote hsinflux a threaded command-line app which connects to HomeSeer on a schedule, downloads the device status data and drops it into InfluxDB. hsinflux is here.

The net result, with Grafana, is the ability to see my house temperatures, humidity, door and window status, and thermostats.  Here are some examples.

House temperatures this week on 5 minute intervals

At the time of writing, its -17C outside, and my server room temperature is showing a steady decrease, down to -6C. The increase on the far right is the result of opening the server room door to get some warm air from the house.

Dimmer Status

And since I have a TSDB, why not record the time it's taking to collect data from HomeSeer?

Since I can, I'm also querying battery charge on the devices that have batteries. It's not an interesting graph, but down the road I can use Grafana to alert me when the batteries need to be changed.

Overall, a great opportunity to become familiar with the technologies. Once the data is in Influx, building arbitrary graphs to visualize data is both simple and powerful. The graphs I've built are simple, however for IT professionals wanting insight into complex infrastructure, tools such as Influx and Grafana represent a step change in visibility and analysis.

From an operational perspective, TSDB's give DevOps engineers an opportunity to answer a question we've been trying to answer for decades:

What changed the hour before it crashed? The week before it crashed? The month before it crashed?

FreeBSD on AWS with Ansible

I often have a need for a fresh FreeBSD install, to host a server, or perhaps just experiment with. My usual solution is FreeBSD jails. However, I've recently gotten interested in Ansible and AWS.

There are quite a few advantages to installing infra with AWS, however the one I'm most interested in is automating the simple stuff and getting it right the first time. For example, remembering to:

  • Add the service accounts
  • Ensure that all files are added by the right accounts
  • Add startup and shutdown scripts
  • Configure pf
  • Ensure logging is "just right"

AWS also has the notable advantage that with VPC, I can configure not only the OS and applications, but build infra.

With the help of these two pages (1,2) I've been able to provision a simple FreeBSD-12 host, in a VPC. Code is here.

K-REPL

A couple projects I've been working on have a need for a REPL style console.  I had written a simple framework for creating REPL's, which is open-sourced here.

 

Quadrigacx API

I've recently become very interested in blockchain, and that, naturally led me to Ethereum and Bitcoin.  From there, I got a little interested in online trading, but since I prefer not to have to think about trading, I started thinking about a trading bot.  Part of the work I needed to do was write a Java API for Qaudrigacx.  The API is here and is free to use under the GPL v3 terms.

 

FreeBSD Jail filesystems

I've been using FreeBSD jails for a while, and was looking for a simple way to create jail filesystems.   After some reading, I came up with this simple script which creates full FreeBSD filesystems.   The filesystem can then be used via jail.conf.

#!/bin/bash
JAILNAME=$1
echo "kjail "$1
JAIL_HOME=/tank/kjail/
FTP=http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/11.1-RELEASE/
# 
# get packages
#
if [ ! -f $JAIL_HOME/packages/doc.txz ]; then
        wget $FTP/doc.txz -O $JAIL_HOME/packages/doc.txz
        wget $FTP/base.txz -O $JAIL_HOME/packages/base.txz
        wget $FTP/kernel.txz -O $JAIL_HOME/packages/kernel.txz
        wget $FTP/lib32.txz -O $JAIL_HOME/packages/lib32.txz
fi
#
# make jail
#
mkdir -p $JAIL_HOME/$JAILNAME
tar zxvf $JAIL_HOME/packages/doc.txz -C $JAIL_HOME/$JAILNAME
tar zxvf $JAIL_HOME/packages/base.txz -C $JAIL_HOME/$JAILNAME
tar zxvf $JAIL_HOME/packages/kernel.txz -C $JAIL_HOME/$JAILNAME
tar zxvf $JAIL_HOME/packages/lib32.txz -C $JAIL_HOME/$JAILNAME