Project Logs

Collapse

• PCB by Steve Bowles

  Solenoid • 09/16/2018 at 14:43 • 0 comments

  Steve Bowles was interested in this project and decided to draw up a PCB for it. He basically laid out my perfboard circuit on a PCB, which makes assembly so much more convenient. You can find his files in the files section of this project.

• Refactoring with Node-RED and MQTT

  Solenoid • 09/09/2017 at 22:35 • 0 comments

  I've recently learned how to use Node-RED and MQTT and I wanted to try them out in. I decided to refactor this project as it was difficult to maintain the way it was made. I was pleasantly surprised, actually this was exactly what I had imagined from the start:

  My goal was to build a system that I could check whenever I wanted, from any platform and that did not require any additional infrastructure or services. That's why I aimed at a self-contained system to log, serve a webpage and publish data to the cloud.
  

  Doing all this on an ESP8266 was possible, but challenging. Indeed, it did not handle concurrent web requests or large data transfers well, uploading data to Google Spreadsheets was riddled with obstacles (because of SSL), logging methods and space were limited and testing was bothersome.

  Node-RED allowed to unload the hard work form the ESP, actually it was simply a matter of publishing MQTT messages to a broker and the rest of the program could be "written" on the Node-RED server:

  This had many advantages:

  • The front end could be served to multiple clients at the same time
  • The widgets offer a clear visualisation of the power consumption
  • Data uploading to the cloud is easily handled by a computer
  • New data processing and visualisation features could be added without having to change the ESP code
  • Virtually no size limit on the locally logged data
  • Data can be stored in actual databases
  • Less complex code on the ESP8266

  As I had a Raspberry Pi running 24/7 to handle various things around the house it was simply a matter of installing Node-RED and the MQTT broker and point the IoT Power Meter to the server.
  

  A final note on this log: this project has made me much more aware of my energy consumption, how much certain things around the house consume and what it costs me. I started this project as a learning opportunity and a fun way to do something potentially useful. I now realise its true potential is to give me an incentive on getting the lowest numbers possible.

• The IPM in the wild!

  Solenoid • 07/08/2016 at 10:23 • 0 comments

  Peter contacted me recently about the Internet-of-Things Power Meter he built with the help of the instructions here, it's kind of a milestone for me to see it in the wild. It shows the true power of open hardware and the advantage of using a perfboard as component support.

  He did not place the parts the same way as I did, from this I take that the hardware design is not that crucial as long as the schematics are provided.

  Peter had trouble getting the IPM to connect to his Wi-Fi access point, after suspecting a faulty ESP chip and ordering new parts it turned out the access point itself was the issue.

  I'm truly grateful for his feedback and hope getting updates in the future to make this project better.

  If you have built an IPM yourself please send me some pictures of your setup and suggestions on how to improve the software, hardware, building instructions...

• IoT Power Meter deployed

  Solenoid • 05/28/2016 at 07:26 • 0 comments

  The IPM is finally deployed and counts the blinks. Everything seems to be working well and there's already some things happening that are worth mentioning.

  Robust blink reading

  The household power meter is in the basement, there is no parasitic light sources that could make the light detector go off without a blink. I tried shining a flashlight on it and it didn't count extra blinks, so pretty robust.

  More useful than ever

  Immediately after deploying it I saw the difference with the old system, the minute resolution allows to clearly see when the heat pump is on.

  I tried turning on and off various household appliances to see the difference in live power usage, it works! Basically this device can tell you how much power an appliance uses.

  The dryer had a peak power usage in the end of its cycle, I suspect there's something else that consumed power as well... I need to dry more clothes to confirm that. The other appliances, such as the fridge, computer, lights... are drowned in the ground hum of about 400-600Wh (7-10Wmin).

  Useful log system
  

  The log system has already paid off: the IPM missed one hour on Google Spreadsheets, the log indicated that the upload failed, now it's a matter of finding out why, but that's already much better than the first version where I had no idea what was going on.

  Thanks to the data on the SD card I was able to manually copy the missing data to the spreadsheets, however this is typically something that could be automated in the future.

  Debounce issue fixed

  As with the first version the IPM tends to read one LED pulse multiple times. I used the same debouncing technique. The multiple reading is because the signal has a ringing to it, perhaps due to some weird optic or electric effect that I don't care enough about to investigate.

  The debouncing works, but now the system has a fundamental limit: a light pulse from the LED takes 100ms, after the first rising edge pulses are ignored for 200ms to avoid reading the same pulse multiple times. This means the maximum instantaneous power reading can only be:

  The limit is pretty high, the highest number, during one hour, recorded by the old system was 6.6kWh, so there's some margin. There may be peaks over 18kW however and this might not be a good solution for everybody. There are other ways to debounce a signal, but I'm not going to mess with the counter now that it has been deployed.

• Code updates

  Solenoid • 05/26/2016 at 09:49 • 0 comments

  No OTA updates

  It seems it will be impossible to implement over the air (OTA) firmware updating because the code is too big. To update the SPI flash memory over the air the internal memory is split in two: one partition runs the current code, the other partition gets the updated code and upon reboot the code in the other partition is executed. During the next OTA update the first partition is rewritten, etc.

  This essentially means that the available memory is divided by two and currently the code already uses about 70+% of the SPI flash memory.

  Google Spreadsheets SHA1 fingerprint changes

  Google seems to change its SHA1 certificate depending on the server you land on, this means the hardcoded SHA1 fingerprint in the code will fail from time to time. It doesn't affect the code submission as the IPM can just ignore the mismatch and still send the data, I removed the SHA1 verification so that the requests would still go through, a bit unfortunate though.

  Token for Google script

  I added a token check for the Google script, this way even if the Google script URL is disclosed the token will prevent somebody else messing with the entries.

  To use it define the googleSpreadSheetsToken variable in the config.h file and enter the same token for the TOKEN variable in the Google script.

  Event logger

  I was interested in seeing how the IPM was doing over a long period of time. I made a logger function that saves some events such as Wi-Fi reconnections, data uploading tries and the total daily power usage to log.txt file in the root of the SD card. This file can be viewed by logging in the IPM and opening it in the file browser.

• Uploading data to Google Spreadsheets

  Solenoid • 05/23/2016 at 11:23 • 0 comments

  The ESP8266 being an IoT platform it would be suitable to make it upload the power usage data to the cloud.

  Uploading data to Google Spreadsheets is not the most straight forward task to however: a script (written in JavaScript) must receive, process and save the data in the right cell, under the correct date and time.

  Here are the steps to get your IPM to upload hourly data to Google Spreadsheets, you can use the same method for any kind of IoT logger of course.

  I chose to upload hourly data instead of minute data because the columns in Google Spreadsheet are limited to 256, the IPM would need 3 + 24 * 60 = 1443 columns if every minute was stored in a separate cell. The minute data remains accessible on the SD card.

  Google Spreadsheets

  Assuming you have a Google Drive account create a new Spreadsheets document:

  You will be presented with a blank spreadsheet, the sheet should be formatted as follows:

  The first column is for the date, second column is a sum of columns D to AA and third column is the cost of electricity that day.

  The cost formula depends on what kind of plan you're on, I happen to have a double profile tariff where off-peak kWh price is cheaper (22h-6h and weekends), so the cost formula depends on the hour of the day and what day it is.

  Rename the current sheet from "Sheet1" to "Day" from the bottom of the screen. This is important for later.

  Get the spreadsheet ID from the URL bar above, this is needed by the script code to know which spreadsheet it should update. The ID is the highlighted part of the following picture (yours will be different of course):

  Finally make a script by going to Tools -> Script Editor... in the top menu:

  You should now be presented with a blank canvas where you can write your code. You can simply use my code as a base, the only thing you need to change is the variable named SPREADSHEET_ID, the very same you got from the step before.
  

  Once you have made your changes you need to publish the script as a web app so the IPM to call it, for this go to Publish -> Deploy and web app... menu:

  Next choose the following configuration, otherwise it will not work:

  • Execute the app as: Me (<myemail>@gmail.com)
  • Who has access to the app: Anyone, even anonymous
    

  It will ask you to authorise the code execution on your behalf, simply click Allow. You only need to do this once.

  You'll be presented with the URL you can call for the script to be executed:

  Obviously you shouldn't disclose this address, however even if somebody does get their hands on this URL there's not much they can do with it.

  Note: if you ever want to change anything in the code you need publish the script again as a new version (select Project version: New) or it will not have any effect, I lost so many hours figuring that one out...
  

  To test the script you can call something like:

  https://script.google.com/macros/s/SCRIPT_ID/exec?time=1464325345&power=1234

  This should create a new row with a date:

  Note: the timezone and daylights savings time are set in the script, you only need to modify the TIMEZONE variable to an IANA time zone value, you can find the complete list of timezones here. The region configures the daylight savings time as well.

  Google Spreadsheets is a pretty powerful tool, you can draw graphs using the data you have, for example you could visualise the hours your household uses the most electricity, the average power usage every month, total power usage over the year...

  IoT Power Meter

  The only modification the IoT Power Meter code needs is the script ID in the config.h file:

• Web user interface (front end)

  Solenoid • 04/09/2016 at 10:38 • 0 comments

  The web user interface currently consists of a live power counter, a total day counter and a graph showing daily used power with a minute resolution.

  An example of the current state of the front end, these are dummy numbers:

  It is based on jQuery Mobile as it's simple, pretty and quickly implemented, it's also touch-screen friendly making it usable on smartphones and tablets. The graph is generated by the C3 library. The libraries are hosted by Google which make loading times faster and less work for the ESP, although it requires Internet connectivity.

  The administration page is an unmodified version of what's coming with the SDWebServer example of ESP8266 code for Arduino IDE. It can show all the files on the SD card and allows uploading and downloading files.

  The files can be updated by uploading a file with the same path and name.

• Upcoming features

  Solenoid • 04/04/2016 at 21:31 • 1 comment

  The IPM is not finished yet, I would like to add a number of additional features, if time and motivation allow for them:

  • Over-the-air (OTA) programming using the web page
    • Currently the front end can be updated by uploading a new index.htm via the web page, but the flash memory cannot be updated yet, this is technically possible however
  • Configure Wi-Fi and access credentials with access point (AP) mode
  • Make something useful with the short and long presses of the button, for example:
    • Short press: toggle between the stats and an "analog style" counter on the OLED screen
    • Long press: Wi-Fi configuration mode
  • Design and order proper printed circuit boards
    • The perfboard is kind of challenging to assemble, but the eBay components would stay as they're annoying to source individually

  I welcome any contributions to the source if anybody's motivated, maybe I can motivate contributions with free PCBs, hmm...

• Show me the source code!

  Solenoid • 04/03/2016 at 09:25 • 0 comments

  The IPM files, including source code, are now published on GitHub, the project is not yet finished, but can already be used at this stage. The documentation and project progress logs stay on hackaday.io while it's still in development.

  To configure the IPM one must currently edit the credentials in config_dummy.h file and rename it to config.h:

  1. Install Arduino IDE 1.6.8
  2. Install GitHub version of ESP8266 Arduino core
  3. Select Tools->Board->Generic ESP8266 Module
  4. Connect the IPM to the computer using an FTDI programmer (set jumper to 3.3V), but do not connect the 5V yet (red wire)
     
  5. Edit the config_dummy.h file, enter the proper credentials and change the authentification information
  6. Rename config_dummy.h to config.h
  7. Hold the button down
  8. Connect the 5V (red wire)
  9. Click upload in Arduino IDE

  Every time it has to be reprogrammed the steps 7-8-9 must be repeated. If the button is not pressed before power is applied it will simply start the program and the upload will fail.

• Licencing

  Solenoid • 03/30/2016 at 21:04 • 0 comments

  The project got this far thanks to the free tools like Arduino IDE, examples, community behind the ESP8266 and bits and pieces all over the internet. I'd like to share this work so that it could be modified, adapted and used by other people as well.
  

  Software licence

  The software of this project is based on a number of libraries, it must follow the licensing rules stated by them first. However different parts were licensed with different licences, which is very annoying.

  As an engineer legal text is out of my domain and frankly I wouldn't want to mess with it as much as possible, however to make my work available to others this is a necessity for some reason.

  The table on the ESP8266 Arduino GitHub page lists all the licences that their work is based on and since the IPM software, in turn, is based on theirs it must comply with them:

  • Arduino IDE is used to program the code (GPL)
  • Arduino ESP8266 core libraries are used (LGPL)
  • Xtensa toolchain compiles the code (GPL)
  • Esptool uploads the code to the ESP8266 (GPLv2)
  • Espressif SDK is needed to compile the code (Espressif MIT License)
  • umm_malloc is probably used somewhere in the code (MIT)

  The licences have different requirements, made relatively easy to understand thanks to this page.

  The current software has to comply with LGPL because the code borrows a lot from the SDWebServer example and it requires derivative work to adopt the same licence. LGPL also permits to publish under GPL (but not vice versa).

  GPL has a "bigger restriction" than LGPL (stronger copyleft) in that all derivative work, be it used as a library or not, must adopt the same licence. I like the fact that all derivative work based on this project needs to published it under the same very permissive licence, so I'm publishing it under the GNU General Public License.

  The L/GPL also requires documentation, which I think is just as important as the project itself.

  Hardware licence

  I don't really care about people selling this project, or derivatives of it, and making a profit. However I very much care about making these works just as available as this project, so that is why I decided to go with Creative Commons Attribution-ShareAlike 4.0 International License for the hardware.

View all 20 project logs