Soil moisture monitoring in a flower garden

When do your flowers have enough water, and how do you know?

Similar projects worth following
A network of Bluetooth connected sensors and ESP32 modules to track the soil moisture and nutrients in a flower garden.

My wife and I transformed the (rather) small yard in front of our house into a flower garden.  It is full of roses, lavender, tulips, dahlias, and shrubs.  The open areas are covered in bark chips to keep down the weeds.

To make sure that the flowers stay properly watered, I put some Bluetooth capable sensors out in critical spots in the yard.

To my surprise, it turned out that I hardly need to water things at all - or do I?  Some times the moisture is at the low end of the suggested range, and stays there for hours.  Is that bad?  Did it maybe drop below the recommended levels at some times and places?  I don't know, but I want to find out for sure.

The sensors in the critical spots seem to show that there's always enough water in the soil under the 4 inches of bark that I didn't need to water things at all.

The soil moisture varies over the course of the day.  The soil dries out somewhat during the day and recovers during the night.

During the driest part of the day, the sensors showed that the flowers still had enough moisture available - if the charts provided by the sensor manufacturer can be trusted and if the measured spots truly represent the state of the whole garden.

The handful of sensors I used couldn't give me a very detailed picture of the situation.  I tracked things as well as I could with the original software from the sensor manufacturer, and cursed it every day.

The software is only intended to track the moisture and nutrients for single plants - it cannot combine data from multiple sensors.

Even for a single sensor, it has painful limits.  You can't display the last 30 days so that you can track things easily.  Nope.  It only shows you the last month - on the first of the month you have an empty chart.

I started planning a better system with my own software.

The current idea is to space sensors on a 1 meter grid through the front yard (and selected spots in the backyard) to provide a comprehensive picture of just what the water in the dirt is doing.

The sensors will monitored by several solar powered ESP32 modules which will transmit the collected sensor data via WiFi and MQTT to a Django program on a Raspberry Pi.

I want to generate an animated "heat map" style chart of the soil moisture to see how it develops over the entire spring and summer, and I want to use the daily charts to tell when (or if) I need to water things.

  • 21 × Garden sensors (Xiaomi or compatible)
  • 3 × ESP32S
  • 3 × Solar charged battery banks
  • 1 × Raspberry Pi

  • More Progress and a Change of Plans

    Joseph Eoff4 days ago 0 comments

    I posted some pictures last time around showing some of the data from my yard.

    There were some bugs, though, and I also wasn't happy with the representation of the area of the yard.

    I looked into what was causing the bugs, and decided that I was doing something wrong with the SciPy interpolate.interp2d function.  I had originally intended to use the SciPy interpolate.griddata function, so I switched to that and got the results I expected - mostly.

    This is a plot from a couple of days ago:

    The shape of the yard is now correct.  The bogus data is gone - it was never in the database, it was all in an incorrect interpolation.  The griddata interpolation gets it right.

    That picture shows a moment when it was raining here.  The blue(ish) areas have around 70 percent soil moisture. The green is around 50 percent.

    The animation part works, but I don't have it rigged to render to video yet.  I did a screen recording of an analysis run instead. 

    Here's a video of the soil moisture in my garden for last 14 days:

    That's fourteen days, at 24 plots per day, compressed down to just a few seconds.  Sixteen sensors contributed data through two control nodes.  That's over 5300 data points interpolated into a video.

    That bunch of rambling, flickering colors has been in my mind for over a year now.  It's nice to finally see it in real life, reflecting real data.

    From the video, you can see that things started out a little dry a couple of weeks ago.  We had some warm weather and sunny days for a while, and things dried out a bit.  It turned cooler after a few days, so the soil moisture stayed a little more stable.  Towards the end we got some rain, resulting in times where the soil was temporarily wetter than is really good for the plants - that's the blue areas.

    All of the parts are in place, now.  It's all "just" improvements from here on out.

    There's still lots of improvements to be made, though:

    1. Convert the interpolation to translate from WGS84 coordinates to meters by way of UTM before interpolating.  Degrees are only linear over the very short distances.  Anything larger than my yard would be very distorted.
    2. Implement a "compare" mode to show two sensor types simultaneously.  It'd be nice to correlate soil moisture with temperature  or light to see what's driving the variations.  Playback the main display, and have the secondary sensor track it  - synchronized animation of the two data sets.
    3. Implement a Date/Time display for the animation.
    4. Fix the "non-existent date" bug - there's one hour on the day of the daylight savings time switch over that doesn't exist. The animation generator hits that one hour and causes a date/time exception.
    5. Implement the overview page.  I need to see which sensors are showing low moisture and where they are.  I also need to see stats on the batteries so that I can replace them as needed.
    6. Implement "maximum, average, and minimum" functions for the animation to make it easier to see where there's not enough water.
    7. Save and recall the color settings for the various sensor types so that I don't have to adjust the histogram manually.
    8. Probably lots more that I can't think of right now.

    With a bare minimum of analysis functions working, I think it is time to submit this to the "Data Loggin'" contest.

  • Heatmap - Nearing the Goalline

    Joseph Eoff04/09/2021 at 20:58 0 comments

    The pieces are slowly coming together.  I got my first heatmap view of the data this evening.

    That's the distribution of soil moisture in my garden from 1 April, just before noon time.

    This is the same day, but between 3 an 4 AM:

    As you can see, the moisture changes over the course of the day.

    I can also plot the other types of data:

    That's the temperature between 3 and 4 AM on 1 April.

    For comparison, the same day just before noon:

    There's still quite a bit to do, but it is getting there.

    I've got to group the data for the animation steps, and lots of other things.

    It turns out that the pyqtgraph ImageView control that I'm using can accept stacks of images for time sequences - that'll make the animated displays easier.

    I've got to find the source of some bugs first, though.

    Things like this:

    That's a soil moisture plot from this evening.  It runs from -40 percent to way over 100 percent - but only on the heat map.  The data behind it is all within the proper range (0 to 100 percent.)  There's something tricky going on with the interpolation.  I think maybe missing values cause the interpolation to freak out.

    I'll look into that later this weekend.

  • First Glimpse at the Data

    Joseph Eoff04/07/2021 at 19:28 0 comments

    I spent a little time implementing the sensor data plotter this evening.  I can now plot the data for single sensors over any time period.

    A quick look at the data collected over the last week delivered some surprises.

    First off, CR2032 coin cells do not like cold weather.  The sensors are powered by coin cells, and the sensors report the battery capacity along with the other values.

    We had some warm weather when I put the sensors out, with days around 20 to 25 degrees Celsius (mid 70s for Fahrenheit types.) The nights were much cooler, though.

    Here's the battery capacity plot of a typical sensor:

    This is the temperature plot for the same sensor:

    The lowest capacity matches the lowest temperatures on each day.  The last three days have been cold, with temperatures staying below 10C (50F) all the time.  The battery capacity has stayed low, as well.  If it doesn't warm up, I may end up having to replace all the sensor batteries.

    This is all about the soil moisture, though, so lets have a look.

    This is a rather typical sensor:
    The moisture dropped each day around noon time when it was warm.  Since the temperatures have gone down, the moisture has stayed more stable during the day.

    This sensor shows rather more concerning data:

    It dropped more drastically during the day, and still had drastic variations after the weather cooled down.

    When I implement the "Daily" step size, I'll have to include a maximum, minimum, and average for each day.  The average alone isn't much use when you have such variations all the time.

    Besides the battery, temperature, and moisture, the sensors also measure soil conductivity and light.

    Here's a typical plot of the light:

    Brightest during the daytime (duh) and zero at night.  The street light isn't bright enough to register at night.

    Conductivity is a proxy for soil nutrients.  More conductivity corresponds to more nutrients in the soil.

    Conductivity also varies with the soil moisture. The nutrients have to be dissolved in water to conduct electricity.  When the soil dries out, the conductivity drops.  You can see how it varies over the course of the day just like the moisture varies.

    I implemented a sort of "pseudo-sensor" when I wrote the software for the control nodes.  The control nodes report a "readtries" value for each sensor after reading the data. This value says how many attempts it took to read to sensor data from the sensor.  It should ideally be 1 - read correctly on the first try.

    I find that the temperature (and therefore battery capacity) has an effect on the readtries - it takes more attempts to read a sensor when it is cold.

    Here's the "readtries" plot of one sensor where the battery capacity has dropped to below 30 percent from the last couple of cold days:

    The closely spaced lines over to the right correspond with the battery capacity dropping below 30 percent, and they correspond to the temperature staying below 10C.

    Given how much the plots change over the course of the day and how different the measurements are for the individual sensors, I expect the heat map view will be rather lively.

    I'll get started on that soon - right after I finish the "Daily" step size for the sensor data plot.

  • Turning a Sketch into Reality

    Joseph Eoff04/06/2021 at 20:48 0 comments

    I haven't posted anything in the last few days, but I haven't been (only) loafing over the Easter holiday.

    I spent some time on Saturday putting together the framework of a PyQt5 project to do the data analysis.

    I got back to it this evening, and implemented parts of the GUI.

    I started with the somewhat simpler single sensor plot.

    It follows the sketch fairly well, expect that I added some additional features.

    The "Sensor ID" and "Value Type" fields are filled from the Mud-Py database - I've already implemented that part. The API is somewhat limited right now, but it just needs a few more methods to get the data for the plot.

    The "Date/Time Range" block is a custom control that I built.  I'll need it on most of the other tabs, so I went ahead and made a control rather than putting all of the bits and pieces on each tab.

    I like PyQt5, but getting those few items to look like they should while still automatically adjusting to the window size and the content was a pain.

    At any rate, it is making progress.

    25 April is coming up fast, though, so I'm going to have to get moving or else miss the cutoff date for the "Data Loggin'" contest.

    As with all the software for this project, the analyser got its own GitHub repository.

  • Data Analysis - Start With a Sketch

    Joseph Eoff04/01/2021 at 18:29 0 comments

    With my control nodes quietly collecting data, it is time to start doing some analysis.

    The entire point of this project has been to visualize the distribution of the soil moisture in my yard.

    I want a heat map, or topographic map of the yard showing the moisture, and I want it animated to show how the moisture changes with time.

    I don't know of a standard program that can do what I need (Veusz could probably do it, but I'd have to export the data to csv or something.)

    The simplest thing for me is to (again) use the Mud-Py Django project as a library, and write my own analysis software.

    That sounds like a lot of work, but it isn't, not really.  At least, not if you use the right tools.

    The right tools for this are Python, NumPy, PyQtGraph, and PyQt5.

    • Python gets me access to Django and the database.
    • Numpy gets me easy access to complex math stuff.
    • PyQtGraph gets me easy access to powerful, good looking  charts.
    • PyQt5 lets me join all the parts in an easy to build GUI.

    I'm sure it is possible to do all of the needed things completely inside Django using Leaflet for the graphing, but that would require more contact with HTML and Javascript than I want to subject myself to.

    Before I start on GUI projects, I usually make a rough sketch.  There are two this time around:  One for the heatmap/topographic chart, and one for a line chart.


    Line chart:

    I did say they were rough sketches.

    The heatmap has a selection for the zone to plot and the time range.  The idea is to pull the data by hours or days from the Mud-Py database.  Each sensor is then averaged by the time step (day or hour) and used to plot a heatmap using the NumPy interpolate function to make a smooth surface out of the individual peaks.

    The line plot will pull data for a single sensor for a specified time range and plot is as a simple line.  It also has a selection for hourly or daily averaging.

    That's what I'm going to implement as a start.  There'll be more bells and whistles added as I go along, but that's the basic thrust.

  • Up and Running

    Joseph Eoff03/28/2021 at 19:24 0 comments

    I got out in the yard Friday afternoon, late, and set out the sensors.

    I used a long tape measure and a yard stick to put them in something resembling a regular grid.

    Thereafter followed the better part of two days of cursing and pulling of hair.

    The control nodes simply would not work properly.

    Sometimes they'd read some sensors and quit.  Sometimes they'd read some sensors, reboot, and start over again.  Maddening.

    The worst part was that they only misbehaved when actually out in the yard.  The control nodes can't "see" my yard from the work room, so they'd just time out trying to connect to the sensors.  The errors were happening after the connection was made.  I couldn't debug the problem because I couldn't reproduce it when the nodes were connected to my computer for debugging.

    I tried several things blindly - correcting things that could be mistakes then trying them  out.

    In the end it was something I didn't do that was the problem, and something I accidentally did while troubleshooting that kept me from finding it.

    You have to call "delete" on the Arduino Bluetooth client (BLEClient from the BLEDevice library) after you call "disconnect" on the BLEClient.

    It wasn't that hard to figure that out, but while doing other things I managed to accidentally paste a line of code in the wrong place.  I wanted to return a nullptr at a particular spot when the Bluetooth service connect failed, and put it in the wrong spot - it always returned a nullptr, even when it worked properly.

    I spent much of today trying to figure out how a delete and a try/catch could cause all connection attempts to fail. It can't, of course.  I didn't find the mistake until I tried a nearly line by line eyeball comparison of the version on GitHub and the version I was working on.

    At any rate, I got it running (the updated software is in the control node repository.) 

    The first complete run showed that the front yard needed two control nodes rather than just one.  The sensors furthest from the control node took forever to connect. 

    The control node software reports back how many connect attempts it took to reach each sensor. The sensors furthest from the control node hit the maximum (4 tries) and timed out.

    I have two nodes out there now.  They've gathered data for all 16 sensors on each run every time they've gone through.

    While I was trying to fix the problems, I changed the software so that the control nodes can only read a maximum of 30 sensors.  From all of the errors, I found that it could easily take an hour to read 30 sensors if every one of them timed out.  Since I want to read from the sensors every hour, there's not much point in trying to read more than you can handle (worst case) in an hour.  When it all works properly, a node can read 15 sensors in just a few minutes.

    The nodes now read all the sensorIDs from the server and put them in an array before reading the sensors.  That reduces WiFi traffic while the Bluetooth stuff is going on.  It also reduces fragmentation of the memory somewhat.

    From a quick look at the raw data, I found a couple of interesting things:

    1. I'm going to have to buy batteries for the sensors much sooner than expected.  Half of the "brand new" batteries in the sensors are reading less than 50% capacity right out of the gate.  About 1/3 of the sensors show 100%.
    2. There's a noticeable temperature gradient across the yard.  Something like 3 degrees celsius between warmest and coolest.
    3. The nutrient content is much more even than I expected - around 200 microsiemens per centimeter conductivity, with a single sensor showing over 300.
    4. The soil moisture (which is what this project is all about) varies considerably.  It ranges from 18% to 51% from different sensors.

    Now that I've got good data coming in, I'll get started on the analysis section this week.


    I may order some...

    Read more »

  • Quick Update

    Joseph Eoff03/25/2021 at 22:18 0 comments

    I  now have all the parts together.

    1. 21 sensors - I just finished gathering the Bluetooth IDs and marking them so I can tell them apart.  I also found one sensor where the LED doesn't work.
    2. 3 solar powered control nodes.
    3.  1 Raspberry Pi running the Mud-Py data collection software.

    Here's the hardware ready to go out in the yard:

    Here's the sensor management page with all the sensors:

    All that's left is to position the sensors in the yard and park the control nodes in their jars - and then write the software to analyze the collected sensor data, of course.

    I spent a little time this evening expanding the Mud-Py software to make it capable of a couple of things I KNOW are going to happen.

    • I'm going to probably have to change the position of some of the sensors - I probably won't get them set right in the yard or I might get them out of place in the map.
    • I'll probably have to replace a sensor or two along the way. There'll be 15 of them in the front yard in amongst the flowers, and it is a sure bet that I'll step on one sooner or later.

    To handle that kind of stuff, I've made it so that each sensor data entry has a location and the zone the sensor belonged to at the time the measurement was made. Done that way, I can select sensor data by zone and date/time and get a correct set of data no matter if the sensors were moved around.

    I am still utterly fascinated by the way Django works with mobile devices.

    Here's the sensor list as shown on my phone:

    I didn't have to do a thing to the software at all to get it looking good on the phone.  Django just does it right.

  • Mud-Py on a Pi

    Joseph Eoff03/23/2021 at 22:33 0 comments

    It has taken me two evenings in a row, but I got the Mud-Py Django site running on my Raspberry Pi.

    It was trickier than I expected.  I ended up following several tutorials in order to get the Mud-Py site to run under NGINX and to start when the Pi boots.

    Here's the Mud-Py Zone editor served by the Pi:

    Yeah, it looks just like the Zone editor served on my computer - but that's what it is supposed to do.

    I went ahead and setup the Mud-Py MQTT Bridge as a service on the Py and pointed one of the ESP32 nodes at it.

    The control node connects, no problem, and reads data from the sensor:

    Almost there.  I've got a few small things to clean up in the software (a configuration file for the bridge) and then get all the sensors and nodes out in the yard.

    Documenting how you get all the pieces to play together is going to be FUN (not.)  That'll almost be a project by itself.  The setup and a list of all the associated libraries will go in a wiki in the Mud-Py GitHub repository.

  • More Show and Tell

    Joseph Eoff03/20/2021 at 17:52 5 comments

    I managed to get a functional version of the  ESP32 control node software working this week.  I spent several rather intense evenings, flipping back and forth between ESP32 documentation, Arduino documentation, MQTT PubSubClient library documentation, C documentation, and StackOverflow.

    The control node works as planned - sort of.  I ended up changing plans a bit.

    The idea has always been for the node to register with the MQTT server and report in.  The bridge then sends it a list of all the Bluetooth sensor IDs it should scan.  The node then reads them and reports the sensor values back to the bridge.  The bridge stores the data in the Mud-Py database. Once the node has done all the sensors, it tells the bridge "done" and the bridge tells it to take a nap for X seconds then check in again.

    The outline stayed, but the details changed.

    My original concept had the node dropping the WiFi connection while scanning the sensors.  I've noticed interference between Bluetooth and WiFi before, and wanted to avoid that.

    That would have required the node to store the list of Bluetooth IDs to scan, and also to store the results.  That turned out to be more complicated than I thought - memory management is no fun in the Arduino environment.

    I ended up leaving WiFi active the whole time, and just doing things "live" - send the sensor IDs one by one as MQTT messages and let the node read and respond to each command as it comes in.

    That made a lot of things much simpler, and is the reason I have a functional control node software ready to run.

    You can't see bits and bytes, though, so I'm going to show the progress on the hardware side.

    I finally got around to buying some hot melt glue to finish up the control node adapter boards and the modifications to the power banks.

    Insulated control node adapters:

    I put hot melt glue over the back of the adapter boards.  Most of the bare spots back there are connected directly to the battery - I wanted them insulated to avoid accidentally testing the PTC fuses I put in the power banks.

    I put some more hot glue in the hole in the power banks where the wires come out.  That's more to provide a little strain relief and keep bugs from crawling in and making homes in the power banks.

    Here's the first completed power bank and a control node doing the first ever battery powered update:

    I also did some preparation for putting the nodes out in the yard.

    My original plan was to go full out redneck and just stick the nodes on a post and plop a big, empty soda bottle over the whole shebang to keep off the water.

    My wife was not amused by the idea of a soda bottle on a stick in our front yard, so I changed plans.

    Lots of people around here put glass decorations (colored globes and what have you) in their yards.

    I bought some (cheap) decorative glass jars that will fit over the post and the electronics.  The jars look like a (somewhat plain) version of the glass thingies that
    folks use for decorations.

    Here's the approved "jar on a post" design:

    The solar cell is charging in there - the green "charge" LED is visible.  There's also a blue "progress" LED lit, but it is hard to see.

    I think I have all of the pieces together now.

    The software is functional.  It doesn't have all the features yet, but I can add those in while collecting data.

    The hardware is ready.  I've got a pile of sensors, and all the stuff it takes to put the nodes outside.

    I've got a big pile of sensors stacked up here in my work room.

    All that's left is updating the Raspberry Pi with the finished software, and putting all the hardware out in the cold.

    I'll do that piece at a time over the next few days.

    I have to mark the sensors so that I know which is where, and that'll take a while.

  • Back to Boring

    Joseph Eoff03/17/2021 at 20:21 0 comments

    I've started on the ESP32 control node software.  I'm writing an all new program rather than using the existing MiFlora MQTT bridge as I originally planned.

    The changes I'd have needed to make were so massive and invasive that you couldn't have called it the same program any more - I decided I didn't want to try to untangle licensing requirements on something mangled beyond recognition.

    I have a basic structure set up, with the ESP32 capable of connecting to WLAN and reporting its battery status via MQTT.

    I was surprised at how quickly the ESP32 connects to  WLAN and then the MQTT server.  From reset to the MQTT message arriving at the server takes less than two seconds.

    I have things set up to use the ESP32 watchdog to reset the ESP32 if something goes wrong.  I have it set to trigger if it isn't reset every 60 seconds.

    If something gets stuck (WLAN or MQTT server connection right now,) then the ESP32 will go into deep sleep and reboot after 5 minutes.

    I've started work on decoding the data from the sensors, but haven't gotten that to a state that I want to commit it to the repository.

    I don't normally work with C++ or whatever mix of C and C++ the Arduino IDE uses, and I had forgotten how hairy it can be to do things like return an array from a function.  That's caused me the stop and rethink the structure I had in mind for the message decoding.

    I'm going to sleep on it, and see if I have a better idea tomorrow evening.

    I think I'm just going to make a static decoder that just accepts the raw data and has a bunch of methods or properties with the decoded values in them.  Maybe two decoders - one for the sensor data, and one for the sensor battery data.  The flora sensors have two queries you can make - sensor data and version/battery.  A decoder for each query seems like the best bet right now.

View all 21 project logs

Enjoy this project?



PB wrote 03/31/2021 at 12:42 point

That is a nice setup.

Is it possible that the sensors leach toxic substances into the soil?

Have you taken into consideration the sensor materials and  corrosion/electrolysis by-products?

  Are you sure? yes | no

Joseph Eoff wrote 03/31/2021 at 13:14 point

I'm not using homemade sensors. 

I'm using proper sensors that measure moisture via capacitance so the moisture sensors won't corrode the way most homemade one will. 

The only thing exposed to the soil is a couple of stainless steel buttons (for conductivity) and the fiberglass of the sensors prongs.

I have six sensors that I'm re-using from last season as well as 15 brand new ones of the same type.  The ones that spent all of last spring, summer and fall in the dirt show zero corrosion and work as well as the new ones.

  Are you sure? yes | no

newblicious wrote 03/04/2021 at 00:04 point

Nice. looks like a highly useful project

  Are you sure? yes | no

Joseph Eoff wrote 03/04/2021 at 00:07 point

I'm working on it.  There's still loads to do.

It's more out of curiousity than need.

I want to know what's going on.

  Are you sure? yes | no

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates