Close
0%
0%

The Water Watcher

Monitoring the pilot light on my water heater.

Similar projects worth following
This project consists of two parts (or three parts if you count the MQTT server through which they communicate, or four parts if you count the wifi router, or, hey, just stop counting).

One part is a sensor that monitors the status of the pilot light and main burner in my water heater. The other part is small LED matrix that graphically displays that status. The monitor broadcasts the status over MQTT every 30 seconds, and the display is updated based on the status messages.

This is all done locally on my wifi network. No cloud services are involved.

My home's water is heated by storage tank water heater fueled by natural gas. Even though the water heater is fairly new, it uses a pilot light to light the main burner when it needs to heat the water. The advantage of that is that the operation is completely free of electricity. It's all driven by water pressure and some miraculous combination of thermal and mechanical stuff.

One of the disadvantages of using a pilot light is that it is always burning a little fuel, 24/7/365. It's amazing to me that systems like that are so widespread, but there we are. (My gas range, oven, and furnace do not have pilot lights. They use some kind of electrically-powered lighter to ignite the main burners when needed.)

At my house, there is another weird disadvantage: a few times a year, the pilot light goes out. I discussed that with a "water heater guy", and he immediately told me that he noticed on his way up my driveway that the exhaust outlet was not tall enough above the roof. According to him, the wind can hug the roof and sometimes blow back down the exhaust stack and blow out the pilot light. Is he right? I don't know, but I do know that something puts out the pilot light.

The pilot light only goes out 3-4 times a year. When it does happen, which is almost always noticed first thing in the morning, after I've waited in a stupor for an incredibly long time for the shower water to get hot, I trudge from the upstairs bathroom down into the basement. Then I do a kind of acrobatic headstand while using all three of my hands to go through the pilot light re-lighting procedure.

Sure, sure, I should talk to some more water heater pros and get this thing resolved for good. Someday. In the meantime, I've cooked up this remote monitoring system so I can at least know at a glance that the pilot light is out and the water is cold. My family's admiration for my level of technical geniosity more than makes up for the occasional tepid shower.

Even though this project is essentially complete, I'm going to give most of the details in the Project Log section since that's the best way to describe the several discrete pieces of information. So, start at the bottom and read upwards.

To make things immediately exciting, here is a short video of the Water Watcher in action:

The things you see in the video are the same things you see in the still images in this project's gallery, though the video gives a better look at the intentional flickering effect. The main picture in the gallery is the Water Watcher on duty in the master bathroom.

waterbug.yaml

This is the esphome config for the device that reports on the water heater flame status.

application/x-yaml - 5.30 kB - 01/11/2021 at 01:22

Download

waterwatcher55.yaml

This is the esphome config file for the monitor/display device.

application/x-yaml - 13.27 kB - 01/11/2021 at 01:22

Download

waterbug-OLD.yaml

Previous version, before ESPHome supported TSL2591. You probably don't need this.

x-yaml - 5.62 kB - 08/19/2021 at 02:24

Download

  • 1 × Any ESP8266 board
  • 1 × M5 Atom Matrix A small ESP32 gadget with an integral 5x5 RGB matrix
  • 1 × BME280 environment sensor on an I2C breakout board temperature, humidity, pressure
  • 1 × TSL2591 light sensor on an I2C breakout board
  • 2 × USB power supply

View all 9 components

  • Back and in color

    WJCarpenter09/28/2021 at 03:27 0 comments

    I finally got around to fixing the colors of the LED animations on the display device. For a long time, I thought that something had changed subtly in the color model used by ESPHome and that it would be a lot of bother for me to get back to the colors I wanted (and that show in the videos on this project). It turned out to be something simpler, though still an infrastructure change somewhere along the way.

    To give the totally awesome "world on fire" look to the display, I'm using the ESPHome fastled component. Specifically, I'm using the addressable flicker effect. It has a tunable parameter for how much the flickering can vary from the base color of the LED. I had been using 25%, which looked good in the original incarnation with an older ESPHome release. I cranked that "intensity" value way down and, after some experimentation, settled on 7% variation. It now looks pretty much like I wanted.

    I have updated waterwatcher55.yaml to reflect this change (and to correct one other unrelated typo.)

    This is probably the last log entry for this project. That is, unless I think of something else. :-)

  • ESPHome supports the TSL2591

    WJCarpenter08/19/2021 at 02:29 0 comments

    ESPHome release 2021.8.0, just released, includes my PR to directly support the TSL2591 sensor. My previous waterbug.yaml depended on loading support as a custom component, along with the Adafruit TSL2591 library. The ESPHome support uses the ESPHome I2C abstraction and does not use the Adafruit TSL2591 library at all. A nice side-effect of that is no longer needing to configure a fake "spi:" entry because of the Adafruit standard sensor library's dependency.

    Changes to waterbug.yaml to use the ESPHome support for TSL2591 are minimal and are mostly removing things. I updated the YAML files on this project to reflect the changes.

  • I see the light

    WJCarpenter08/06/2021 at 21:10 0 comments

    Well, that was silly. I didn't pay close attention to this before, but what I have been calling "lux" throughout is not lux at all. It's still a useful value (for this scenario), but it's just not lux.

    ESPHome supports the obsolete TSL2561 device but not the replacement TSL2591 device. Over the last few days, I undertook adding support for the TSL2591 device to ESPHome. (Here are the pull requests, if you are interested: https://github.com/esphome/esphome/pull/2131, https://github.com/esphome/esphome-docs/pull/1370). It was only while working through the details of that that I started paying more attention to what I was measuring.

    What I have been calling "lux" is actually just the raw value of a 16 bit ADC for one of the sensors on the device. To get actual lux values, you have to do a calculation based on the ADC readings of both sensors. The calculation involves the configured gain, the configured integration time, and some empirically-determined coefficients. It is fairly complicated and scenario-specific to tune the calculation, but there is a calculation implemented in the Adafruit TSL2591 library. (Actually, there are 3 different calculations there, but 2 of them are commented out.)

    In my ESPHome integration, I decided to make the raw ADC values available in addition to the calculation provided by the Adafruit library. That way, anyone who wanted to do their own calculation would have the inputs they needed (well, they also need a handful of physical characteristics that are out of scope for how much work I am willing to do for my own feeble efforts).

    What does this mean for this project? Not much, in practice. I'm going to continue using that raw ADC read-out ("full_spectrum" in the graphic below), even given my better understanding of what it is, because it suits the coarse-grained conditional logic needed here.

    I did not have an intuitive feel for real world lux values (like I do for furlongs, fortnights, and shillings and pence) until I looked at the Wikipedia article about lux. It has a nice table of a few examples. It tells me that what I see when the pilot light is on the order magnitude of some kind of moonlight on a clear night. Fair enough.

    Here are some sensor readings that show the ADC values (600ms integration time, 428x gain) as well as the lux calculation from the Adafruit library:

  • Sense and sensor ability

    WJCarpenter07/06/2021 at 01:18 0 comments

    A couple of weeks ago, readings from the light sensor went crazy. Starting at a certain point, the only two values ever reported were 0 and 64k. The latter (actually 0xFFFF) indicates saturation of the sensor. That was very bad for the use case, so I unplugged the upstairs display while I worked on it.

    This started happening remarkably soon after I applied an ESPhome update and reflashed the code on the ESP8266 hosting the light sensor.  Since the driver for the TSL2591 is a custom component in ESPhome's point of view, I thought maybe there was an incompatibility in the update. I spent a lot of time horsing around with trying to get  an older version of ESPhome and matching PlatformIO working. That's probably a pretty simple thing for someone more familiar with that ecosystem, but I was basically in dependency hell. I eventually gave up and just ordered a new sensor in the hope that it wasn't a software problem after all.

    Lo and behold, with the new sensor in place and firmware from the latest ESPhome, things were completely back to normal. So, I guess the sensor was bad after all. I haven't had a chance to play around with the CQRobot sensor to see if I could figure out what made it angry. I hope it's not the case that the heat coming through the viewing window eventually destroys the sensor. It would be a disappointment to have to replace it once or twice a year.

    For the replacement sensor, I switched to Adafruit's board for the TSL2591. Like the CQRobot part, it has a nice socket for an interface cable, though it's a different size (a bit smaller in all 3 dimensions). I got lucky on shipping, and it arrived from New York City to my Seattle-area home in just 3 days. W00t!

    The readings for the 3 interesting states of the gas flame are not the same with this board. That could be related to manufacture of the board, factory calibration of the sensor, or something as simple as my placement of the board onto the viewport window. (Since this board has connectors at both ends, it sits naturally on the glass at a different angle than the CQRobot board.) Even though they are not the same values, they were still just fine for the thresholds I had already put into the firmware.

    (I was a little worried because I tested the board by shining a flashlight onto it. The readings were the 64k saturation value, and I feared it was a software problem after all. But I guess that just shows the high quality of my flashlight. Once I installed it on the water heater viewport, I got suitable values with no further adjustments.)

  • A longer "dazzled" period

    WJCarpenter01/24/2021 at 19:35 0 comments

    I had another one of those "extended dazzle period" burns today. I'm not sure if it's a characteristic of the way the water heater burns or if it's something to do with the behavior of the light sensor I'm using. In either case, I've decided to "cure" the situation (really, a workaround) by increasing the "dazzled" timeout (FULL_FLAME_DAZZLE_PERIOD in waterbug.yaml) to 12 minutes. That covers the period of the two incidents I have observed, and it won't really make much difference in an actual case of the pilot light going out.

  • Just when you thought it was safe to ...

    WJCarpenter01/18/2021 at 23:31 0 comments

    Well, here's a little mystery. Earlier, I included a graph of a typical water heater burn cycle, including the "dazzled" period of 3 minutes or so. Just today, I saw another graph shape that I had not seen before. It's mostly about the shape of the graphs, not the absolute durations.

    Normal cycle:

    Weird cycle:

    I don't really know how much of the shapes of these graphs is due to the intrinsic intensity of the burner flame and how much is due to variation in the illumination sensors. In a normal cycle, the flame stops abruptly. In the weird cycle, the flame seems to taper off gently. The "dazzled" period was about 7 minutes long. It was long enough that I saw a few minutes of "pilot_off" animation on the LEDs. There was no significant wind going on. Before I got down to the basement to check it out, the display had returned to "pilot_on" animation. The next burn cycle, a few hours later, was back to the normal picture.

    I'm hoping this is just some unaccounted-for characteristic of how the water heater burns gas and isn't some kind of wearing out of the TSL2591 sensor. The latter would be a bummer since it's only been watching the flame for a few months.

  • Retrospective and the future

    WJCarpenter01/17/2021 at 23:23 0 comments

    Having the Water Watcher display in the master bathroom does at least solve part of the problem. Decoupling the Water Bug from any listeners via MQTT messages means that I can change the Water Watcher and add any additional listeners without necessarily having to update the Water Bug or the existing listeners.

    Here is the original Water Watcher patiently standing by in the master bathroom. Oh, look, the pilot light is working correctly.

    There is another bathroom with a shower where I might put another clone of the Water Watcher. It's trivial to do with another M5 Atom Matrix. M5 makes a couple of other variations on the M5 Atom.:

    • The M5 Atom Lite is a couple dollars cheaper. It has only a single RGB LED instead of a matrix. 
    • The M5 Atom Echo, still about US$10, also has a single RGB LED, but it contains a microphone and speaker. That gives the possibility of some kind of audible alert. (Now, if I could just work out a way to give a voice command to re-light the pilot light from the cozy comfort of my warm bed. Hmmm.)

    There are lots of available MQTT clients for Android phones, so I can consider getting some kind of PagerDuty-like wake-up when the pilot light goes out. Although I'm not that crazy about getting up at 3am to reboot the server, uh, I mean to re-light the pilot light, it's all-in-all better than discovering that it's needed at exactly the time when the hot water is needed.

    I guess I could train the family cat to re-light the pilot, but he's not allowed to go down into the basement. I could train him to recognize the "pilot_off" display and wake us up, but it would be hard to distinguish that from the other conditions where he wakes us up in the middle of the night ("Timmy fell down the well!", "are you still alive?", "I'm hungry/thirsty/lonely").

    OTOH, this cat looks pretty obedient: Meow — the Slack Bot with the smart paw

  • The 20% solution

    WJCarpenter01/17/2021 at 20:50 0 comments

    M5 recommends not using a brightness level above 20% when using the FastLED library (which is the library behind ESPhome's FastLED component). I'm pretty sure the reason for the recommendation is because of potential damage from heat generated by the LEDs.

    I found that when I used the recommended 20% brightness and the "flicker" effect, the LEDs would sporadically turn themselves off after a few seconds. With a brightness of 25%, that doesn't happen. so that is the value that I settled on using. The exception is the "pilot_off" animation, which uses all of the blue LEDs in a blinking pattern and which might be on for several hours. It uses 20% brightness.

    I originally used brightness values in the 30s for most animations, and the photos and video in this project were taken with those higher values. A side-effect of switching to 25% is that the orange color is generally a bit more red, with some orange coming in occasionally during the flicker. I could experiment with different RGB values to try to get back to more orange, but those kinds of experiments wear me out pretty fast. The colors shown for RGB values on any one of a million web page charts bear only a mild resemblance to the colors produced by M5 Atom Matrix.

    The original brightness values in the pictures and video are:

    •   startup: 30%
    •   full_flame: 35%
    •   dazzled: 35%
    •   pilot_on: 25%
    •   pilot_off: 50%

  • The display animations

    WJCarpenter01/17/2021 at 02:32 0 comments

    Like just about every other artistic effort in my life, things looked a lot better in my mind than in reality. There is only so much you can do with a 5x5 display, even if it is RGB. As every civilized person knows, an RGB LED is just 3 LEDs in the same housing. With the M5 Atom Matrix, you can sometimes see the individual components of whatever RGB color you've selected. From a distance, it's OK, but up close it's a bit, uh, discrete. In the end, I had to simplify my grand designs so that the displays were more symbolically meaningful and less literally artistic. I didn't have enough pixels for good pointillist artwork. (I had one graphic that was intended to show a match being lit into a flame by an unseen force, but it turned out that I had to explain what it was to my test audience. After I explained it, the reaction was something like, "yeah, I guess it is".)

    Itt also occurred to me that I wasn't sure how I was going to orient the M5 Atom Matrix device in its final position. Even the otherwise reasonable "lots of flames shooting up" would look a bit wrong if the tops of the flames went sideways or down. In the end, I decided to use designs that were horizontally and vertically symmetrical, and I achieved some animations using built-in lighting effects provided by ESPhome's FastLED component.

    Here are the animations that I finally used. To see the actual animation effects, you should watch the video I provided earlier. These still photos are just for easy reference. My code reacts to button presses of the front face of the M5 Atom Matrix by stepping through the different animations. That's my bony finger doing just that in the video.

    The Water Watcher has one additional flame status state that the Water Bug doesn't send. The "startup" state is used when the Water Watcher boots up but before it has received its first MQTT message. I just wanted something visually distinctive, so I used the ESPhome "addressable rainbow" effect.

    The "pilot_on" animation is displayed almost all of the time. There are only a few minutes a day when something else is displayed. I could have chosen to just have a blank display, but I wanted to have some indication that the Water Watcher was working and not dead. I settled for flickering dots (using the ESPhome "addressable flicker" effect) in each corner of the 5x5 matrix to imply "a little bit of flame" to suggest the pilot light.

    Analogous to the "pilot_on" animation, the "full_flame" animation is meant to suggest "a lot of flame". It uses the same RGB orange color and ESPhome "addressable flicker" effect, but in this case all of the LEDs are involved. (I get to see this display when I step out of the shower.)

    I chose to have the Water Watcher give s distinctive display for the "dazzled" state. It's just these five RGB orange pixels in the center of the display, again animated with the ESPhome "addressable flicker" effect. I don't have any illusion that this suggests to anybody else what's going on during the "dazzled" state, but it makes me nerdly happy to be able to see it as a distinct state.

    Finally, the reason we're all here: the "pilot_off" state. The animation for this is a lot of blinking and flashing of RGB blue. I wanted it to be very distracting and unavoidable. Even though I told the household members what it means, I expect the normal reaction to seeing it will be to ask me what the heck it is; mission accomplished!

    Again, I recommend you watch the short video to see the actual animations. The flickering effect for the flames is fairly effective (and it took some experiments with parameter tuning to get it to be so).

  • Flame on!

    WJCarpenter01/17/2021 at 01:42 0 comments

    I experimented with several different things, both on the Water Bug sensor board and the Water Watcher display board. I ultimately decided that the Water Bug sensor board would send a simple text message to indicate the status of the water heater flame. As I mentioned earlier, the lux values read by the TSL2591 aren't calibrated, so it's not especially meaningful to push those around for consumption. They do happen to go to my Home Assistant server, and I also send the lux values over MQTT to be displayed in the debug log output of the Water Watcher. The latter is for my own convenience so that I didn't have to monitor both devices to see what was going on during troubleshooting.

    The Water Bug reads the lux value from the sensor every 30 seconds. Based on the experimental observations I mentioned earlier, it reduces that to "pilot_on", "full_flame", "pilot_off", or the special case of "dazzled". I keep track of the last time the Water Bug saw "full_flame". If it sees "pilot_off" within a short time after that, it sends "dazzled" instead of "pilot_off". The message recipient can decide what it wants to do about "dazzled" messages.

    ESPhome provides a mechanism for knowing the wall clock time, either from the Home Assistant server or from an NTP server. It makes it available in a convenient object form that measures seconds since the Unix epoch. I use the wall clock time as part of the display for my various sensor boards that have a built-in OLED display. The Water Bug doesn't have a display, and it doesn't otherwise need to know what time it is. The ESP32 provides a high-precision timer for elapsed time since boot-up, and ESPhome makes that available in its "Uptime" pseudo-sensor. It's granularity is in seconds, but that's fine for this particular use, and it let me avoid the need to deal with the ESP32's timer in native code.

    Once it has decided which piece of text to use for the flame state, the Water Bug sends that as a message on a particular MQTT topic. It then sends the raw lux value with the prefix "lux: " on the same MQTT topic. The only thing the Water Bug has to remember from earlier iterations of the 30 second loop is the last time it saw "full_flame".

    The Water Watcher listens on that MQTT topic and reacts to the messages. It logs every message it sees to the debug log, but it only actually reacts to flame status messages that it recognizes. And, in fact, it only reacts to recognized messages if they are different from the last recognized message that it saw. The reason for that is to avoid some visually distracting repainting of the LED matrix when there was no actual status change. The reaction takes the form of some kind of display on the LED matrix. Those are animated and keep running until a different reaction is triggered.

    The animations in the display and the rapid succession of the flame status and lux messages arriving over MQTT led to some kind of timing problem. I found that the listen-and-react model was not completely stable. I hypothesize that MQTT message strings were changing out from under my code that was trying to react to them. Since I don't know the internals of ESPhome's event looping all that well, I changed my code that listens to the MQTT topic so that it immediately copies the string and pushes it onto a FIFO queue as quickly as possible. The reaction code, which is configured to be single-threaded, then reads from that FIFO queue and triggers the display update when appropriate. That at least decouples the timing and gets rid of the instability.

View all 15 project logs

Enjoy this project?

Share

Discussions

[deleted]

[this comment has been deleted]

WJCarpenter wrote 03/14/2022 at 20:06 point

What kind of help are you looking for?

  Are you sure? yes | no

Adam Quantrill wrote 01/22/2021 at 16:08 point

Nice! I was thinking of doing the same for my boiler (furnace in the US) which is old, the thermocouple wears out once a year or so. However I was thinking to use an extra K-type thermocouple in/near the pilot flame instead of the light emissions, as the observation window gets very hot when the boiler is on full. Also it could be fooled by ambient light. Secondly, instead of a display, I'll have a broadcast alert on the home WiFi that can be picked up by any device and displayed.

For your graph observations, would the explanation lie in the combustion, if it's complete and a blue flame then not as bright as incomplete and a few yellow bits?

  Are you sure? yes | no

WJCarpenter wrote 01/23/2021 at 03:30 point

I think the burner area of my water heater is completely sealed from at the bottom, though it's obviously open at the top for exhaust. So, for my case, there isn't a simple way to get any kind of sensor inside. 

I also worried about ambient light, but it seems to not be an issue. The entire area is normally covered by a cottony fiberglass insulation layer, and that in turn is covered by a loosely-fitting metal plate.

I probably never will properly figure out the flame wavelength stuff. But maybe someday the right piece of inexpensive test gear will show up so I can actually measure it. I read a couple of papers about the characteristics of natural gas flames, but they didn't even agree with each other. :-)

  Are you sure? yes | no

WJCarpenter wrote 01/24/2021 at 19:41 point

I had another look at my water heater and the manual for it. It's only sealed in the sense of a plate held on by some screws and sealed with a gasket. So, I think I actually could  have used your idea of a thermocouple near the pilot light. At this point, I'm reluctant to do that since I have a working system. I also have plenty of experience with "improving/fixing" things and actually ending up making them worse, so I don't want to mess with disassembling that part of my water heater's innards.

  Are you sure? yes | no

Adrian wrote 01/19/2021 at 16:34 point

I'm impressed by the amount of energy and detail you put into this :) But I can really feel why you're doing this and how it feels nice to "beat" a problem like this with technology.

  Are you sure? yes | no

WJCarpenter wrote 01/20/2021 at 02:06 point

Thanks. It's one of those cases where I know I should just get the root cause problem fixed, but once I started imagining what a notification system would be like, well, I wanted to build it.

  Are you sure? yes | no

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates