Simple Household Energy Monitoring

I was inspired by Alan Abbott’s excellent article about his home setup. https://alanabbott.me/posts/2024/07/my-diy-data-center-how-i-built-enterprise-it-skills-at-home/

When the power company put in smart meters, I wanted to see the data too. I wanted to know

I started looking at how I could get the data.

I’ll describe my first failed attempt at getting data. Then, how I successfully got the electricity usage and power source data. I’ll talk about the database and visualization software that I used and the scripts I wrote to capture the data.

Finally, I’ll talk about the current project to automatically store and graph which appliances are running and what power they are using.

House Layout

My house is a typical mid-century California ranch house. It was built in the 1950s and expanded/updated in the 1980s. As was typical at the time, there is no central heat or air. It has natural gas wall heaters in two rooms and a gas fireplace. 

When the house was remodeled and expanded in the 1980s they put in a swimming pool and wall unit air conditioner. In the early 2000s when I moved in, I replaced the single pane aluminum windows and the tar and gravel roof with modern dual pane windows and a polyurethane foam roof. With our climate, it’s comfortable. 

Our gas and electricity utility is PG&E. They installed smart gas and electricity meters about 10 years ago.

False Start

My original plan was to do it all myself. I had this idea that I could use a Software Defined Radio (SDR) dongle to snoop on the wireless signals that PG&E was sending from the smart meter, decode, and store them. I was interested in trying out SDR anyway.

The frequencies that PG&E use are published. However, the electricity data is encrypted and it’s not really feasible. (Zigbee protocols). 

I was able to get Gas Meter data though it was not terribly useful. Unlike the electricity meters that report data multiple times a minute, the gas meter was once a day or more. It wouldn’t be useful for analytics purposes. I abandoned the idea of getting gas meter data and concentrated on getting the electricity data

Good Data

I found a reasonably affordable (US$100) device that PG&E supports that will capture the data from their smart meters. I bought a Rainforest Eagle-3 https://www.rainforestautomation.com/rfa-z114-eagle-200-2/ 

It is a little box that connects wirelessly to the smart meter and communicates with it via the Zigbee interface. You have to give the details of the device to PG&E via their web site to connect it. It was eventually pretty straight-forward.

I say eventually because my first attempt to connect it did not work, even with PG&E’s tech support. I got busy with other projects for a while and didn’t try again for several months. In the second go around, it worked without any hassles. 

I wasn’t planning on using the Rainforest site for storing and visualizing my data, but it was nice to see that it was operational.

Data Storage and Visualization Overview

There’s a bunch of ways this could be done. I’ve been using Raspberry PIs for my home weather station and it was logical to just expand on it. I have this Raspberry PI set up as a headless server and I connect into it via SSH, but mostly I connect via the Grafana web interface.

I use InfluxDB to store the time series data with Grafana for charts and graphs. These are both well supported solutions that have free, open source versions that run well on PI. I have them both running on a Raspberry PI 4.

I’ve considered moving this to containers/Kuberetes or virtualized solutions, but haven’t had the need yet.

Electricity Usage Data

The Eagle box has two published APIs. One is a local API for getting data from within the local network from the box and a cloud API where you connect to their cloud service. I chose to use the local option as I will be communicating within my local network and behind my firewall. If was going to use a hosted solution, like running it on AWS, Azure or GCP, I would have used the cloud API. 

It took a little bit of trial and error to figure out how the API worked, particularly that I needed to include cgi-bin/post_manager/ to the URL to get it to respond. You can find my script on Github. https://github.com/PeterQuinn925/CrateGrafanaWeatherdata/blob/main/eagle.py

I turned this script into a daemon that runs continuously.

Once the data is in InfluxDB, it’s super easy to write queries and display it. I used the GUI to do it, but as text it ends up as:

SELECT mean("instantuse") FROM "electricuseage" WHERE $timeFilter GROUP BY time(1m) fill(previous)

Power Source Data

I found a Python library that made this all easy. https://pypi.org/project/isodata/

It’s a library that does all the heavy lifting and all you have to do is call the API and it will give you the current power mix for various regions. I wrote a script which runs every few minutes and updates the database.

Appliance Disaggregation - Work in Progress

I can pretty much tell from looking at the graphs which appliances are running. It’s not difficult for a human to see the patterns. What I’m currently working on is how to do this automatically. I found a number of resources - namely the Non-intrusive Load Monitoring Toolkit https://github.com/nilmtk/nilmtk?tab=readme-ov-file#nilmtk-non-intrusive-load-monitoring-toolkit

I’m also learning about Hidden Markov Models. I might implement/train a model on my dasta without using the NILMTK implementation. I’m still figuring it out.

I want to implement one of these algorithms and convert it to handle streaming data.

The alternative is to invest in one of the meters that you clamp on each circuit and measure the power usage directly. I may end up doing that. I’m going to try to do it with a single meter and software first. 

I can kind of figure out the cost per hour of running the A/C or the pool pump, but I’d like to be able to do it automatically using one of these ML methods. I’ll update this when I have made some progress.