Well well

Two bugs have eluded me since installing the first prototype. The first appeared to be a reset caused by a glitch on the AC power-line, but I could not rule out that there was a software bug that short circuited the power rails causing the processor to reboot and consequently clearing the short. The other bug seemed to be with the Android app, it would just simply stop updating the display with data from the device. Stopping and restarting the app did not help, only restarting bluetooth would do the trick.

Checking my parts heap for a suitable substitute wall-wart was fruitless, so I hooked the device to my homemade bench power supply via several pairs of alligator clips. Two weeks later, neither bug has been seen.

Now, wanting to return the bench supply to its' place on my desk, I had to clear a space for it first. While doing so, I found a power cube that would be worth testing. Back to waiting for something/anything/nothing, to happen. The original power supply delivered a mere 4.5V to the WellWell device. A 10% tolerance, these days? Of course, the bench supply was set to 5.0V for testing. Conveniently, because I also want to test a higher input voltage, the supply in testing now has its' output right at 5.25V.

Several days later, I've lost count, nary a beep has been heard from the device. No signs of the lost communication bug, nor the spontaneous reboot bug. I think that I'll be switching the power supply suppliers.

Between spontaneous reboots and false alarms, the device that I'm testing has been quite noisy lately. The false alarms were caused by a variety of voltage or current spikes at either pump start-up or shut-off. Now these are all on event counters, so a spike will result in a small event count and a problem will have a large count. It is much quieter already.

The power grid has also been rather noisy too resulting in the monitor rebooting at random times of the day. Or, was it a power problem? For now, the watchdog register's initial value is being reported to the WellNow app in place of one of the other monitored values. Most of the time it was reporting that the power was indeed the culprit, but sometimes it was the watchdog timer causing the reboot. Eventually, I found that it re-setting the time after a power outage was causing the watchdog to time-out. Changing the clock would cause the next watchdog refresh to be delayed. I've chosen to set the watchdog timeout interval to a bit over two minutes rather than invoke the refresh sequence when the clock is changed.

Fixing the power problem may be a bit trickier. First, I tried adding a 220uF capacitor to the 5V input. That worked for all of about three hours. Ok, 1000uF. Within a few days, "Beeeep!". Another power glitch reboot. Fine, 3300uF, but no more. It's been silent for about a week, but a better solution is needed. I'm going to increase the input voltage a bit. No, I'm not. The MCP1700 that is installed in my prototype has a 6V maximum input. Well, there is the MCP1703, same footprint and can handle 16V. But increasing the voltage to a linear regulator will increase the power dissipated by the regulator, and in a SOT-23 case, there is only a little leeway. There is just enough, though. On a hot day, it will be a warm chip, but well within parameters. Time to order some more parts...

Testing the device, means that I'm playing a videogame. There's not much that can I do while waiting for the well to cycle. Sometimes, though, I am looking over the code for potential bugs. I did find a pair of minor ones, and I corrected one I considered a show stopper. One other bug still eludes me. While looking for it, a possible power saving idea came to me, turn the on-chip crystal driver off. Since the device is using an external oscillator, this should not be a problem, right?

Wanting to test the bug fixes, I flashed the new code in to a device. Maybe it flashed, I don't recall the exact error message that came back from the dev studio but the device seemed to be dead. A few more unsuccessful attempts to re-flash the device convinced me to build a new wire harness to connect the board to the debug system. With that in hand I resumed the quest.

Still no luck. Could it be that I killed the oscillator, or something else, while poking around the board with the voltmeter? Warm up the o-scope, one probe on the Z8's clock input and the other on the debug input. There's the 20 MHz signal, the Z8 should be running. Oddly, there was no activity on DBG. Poke, prod, oopsie, just shorted two pins with the probe. What? The status LEDs just lit up. That could mean that my code is executing. How odd. Didn't I read something about...

The actual clock signal used inside the Z8 is derived from the signal on the XOUT (Crystal Output) pin, not the XIN (Crystal Input) pin that the oscillator is attached to. Shorting XIN and XOUT forces a clock signal in to the chip, allowing everything to function and re-flashing the chip possible. Connecting two outputs got the job done!

Now that all of the non-functioning devices have been reflashed and the new code is monitoring my well, I'm again waiting for the pump to cycle. I think I should go slay some rodents of unusual size and leave the option bits alone.

The device and software has been working here for a few weeks. Not quite flawlessly, but well enough that I have been considering enlisting my neighbors as beta testers. The first oppurtunity arrived a few nights ago, when my friend's well either ran dry or froze and I was asked to diagnose the problem. By the time I arrived to test the system, the condition had cleared itself and results were inconclusive. A dry well could have recovered in the time it took me to get to the house, while a frozen line could have cleared after my friend had turned on the heat in the space that the water line traversed.

Now that water was flowing and life could resume, we powered up the WellWell and started the WellNow app. The power-on self test ran, and then nothing. No data appeared in the app, something that I feared would happen. Something in the data timing of the well monitoring device has changed and I need to drag out the reverse engineering tools again. This time, however, I have a device that is practically purpose made for the task, it just needs appropriate software. Introducing WellWhat, an infrared to VCD tool chain. Load this firmware in to a WellWell device, pair it to something that can capture serial data. Run the captured data through a few lines of a Ruby app to get a VCD file for Pulseview to display. There are refinements to make in the code, but as things are I can now get the data I need to see what has changed.

This brings us to the meaning of the title of this entry. While developing the new firmware, this time without the assistance of DMA hardware, I ran in to a huge bottleneck. Data transfer between the Z8 and HC-05 is over a serial UART running at about 113kb/s. That is close enough to 115kb/s that it hasn't been a problem until now. Sending just the two byte timestamp values can take longer than than the bit time of the well monitor, leading to lost data and unhappiness. Why not turn up the baud rate, you ask? The next higher baud rate I can use, 250kb/s, is not supported by the HC-05. The '05 does support 230.4kb/s but there would be a more than 8% timing error, a little too much to ensure reliable data transfer. If I were to change the 20MHz oscillator to one running at 18.432MHz, I could turn the baud rate all the way up to 1.15Mb/s but that might require better signal routing and possibly terminations for the signal paths. I think I'll try 460kb/s, when the parts order arrives.

This project is putting nearly everything on the Z8's die to use. Nearly all of the RAM and Flash ROM are being used. Both timers are busy, as are the UART and SPI ports. All of the I/O pins have something to do. Because I installed a Z8F022 device, the ADC and its' input multiplexer is idle. For now the Watch-Dog timer is also idle, but I intend to put it to work soon. Finally there is the I2C controller, it was idle until yesterday. 

I have been wanting to benchmark several aspects of the firmware, but was unable to due to the lack of an available timing source. Or, so I thought. Eventually, I recalled that one or another of the serial communication devices could be used as a simple timer somehow. I reread the fine manual to find that all (UART, SPI, I2C) of the serial devices could be used in this way. The I2C hardware is now the benchmark timer!

Using the timer mode of the I2C port as an event counter exposed a mis-handling of one of the interrupts. Correcting the interrupt's behavior freed up enough CPU cycles that it is time to beta test!!

Soon after building the Z8 dev-board based prototype built, I made a Python+Tcl/Tk app to display the data it gathered. Time passes and it became clear that a more mobile display would be quite useful. A rainy spell during the summer permitted me to dig in to developing a phone application. Well, an iPhone app was a non-starter, as I don't have one and have no interest in purchasing something that proprietary. That seems to leave Android. When I tried it on my new-ish Moto, it was awful! Within a few minutes of the first boot of that phone, I began the process to install LineageOS. With that done, let's write some code!

Most, if not all, 'droid development appears to be done with Java or Kotlin. I've heard of Java, even remember a little. Ok, start with the includes. Err, no. Ok, guess I forgot how to Java. Again.

Finding a 'Hello, world" app that made use of things I already have installed and know how to use got things underway quickly. Five weeks later I have an app that is substantially complete, AND the boards have arrived from OSH Park! (I ordered the boards and parts when the app was functioning reliably.) A few boards built and tested later, time to kick some more app. Three months of testing, modifying, and rewriting things everything looks ready to move to beta-testing! (I still need to find an enclosure however, as my test device is taped in place.)

I'll admit that the first version of software I flashed to the WellWell device was a bit of a mess. When an invalid voltage value kept appearing, I became convinced that it was time for a major re-write. (There was also the false alarm and crash at midnight. Beeeeeeeeep!) Now that the re-write has been been debugged, the testing has begun! (There was a bit flag that I failed clear. It really messed things up.)

Testing certain parts of the code requires the well pump to run, but my being loathe to waste water makes this slow going. Fine, I'll make an emulator for the infrared transmitter. Push a button, get a simulated pump cycle. Fun!

Some of the changes made during the rewrite shrank the size of the compiled code to the point where I could add a few more features. There's now a factory reset built-in, more thorough power-on diagnostics, and a special data-only mode when the pump is running.

So, here we are. One last thing to check and the alpha test can begin.

< a few days pass >

Update: The test ended at nearly the same time it started. Tests showed that the main loop was just a little too slow to continuously update everything and not drop the occasional bit of incoming data. I am now putting the finishing touches on a new burst oriented, and interrupt driven data transmitting routine. Stay tuned.

Wow. Lots of changes later, and we're running again. Rather than sending one data item at a time, they're now bundled and sent in groups. There's now a data limiting system in place so unnecessary data packets are not sent. Even found an interrupt bug, where there were 2.5 times the number of expected interrupts being generated. Onward!

The Z8's first task is to initialize its' I/O pins. That done, it goes on to determine the presence or absence of the HC-05 module, as I would like to have radio-silence available as an option. Even though the module is present, for some reason, it is not being detected.

Shall I skip over the trials and tribulations of coaxing the updated module in to operation?

Sure, because in the end, the difference was mostly how the reset and "key" pins operated.

The module also handles paired devices a little differently, allowing me to remove the bit of code that facilitated un-pairing a device.

Now having a working bluetooth interface, it's time to put the device in front of the pump monitoring device.

Printers.

When I finished a very DIY toner-transfer friendly circuit board layout, I hit the print button. Beeeeep, Magenta cartridge empty. I took it out to give it a shake, in a futile attempt to dislodge the last few toner particles. Ordered a refilled/refurbished cartridge, but life intervened and I took too long to discover that it wasn't. Purchasing another cartridge wouldn't fit in the budget, so this project went on hold. A incident with a neighbor's well, stoked the urgency of getting this done, but what to do?

Again, Hack-A-Day comes to the rescue! OSH Park is mentioned favorably. Their standard price is based on the dimensions of the board. Hold-on, they can make vias. Those will save some space! Time to re-design. The original layout came in at a bit over 4 square inches (55x55 mm). The updated board shaved an entire square inch, and now measures 43x48mm. That's quite a discount and will fit in my budget better than yet another toner cartridge.

A few days later, the boards are in my unstained by ferric chloride hands, and they look great! Assembling the first board took a little longer than I might have expected. Seems that my SMD hand soldering skills were a bit rusty. The closest thing to a flaw in the layout was my footprint for the HC-05, it was off by a small amount. Not by enough, at the moment, to bother adjusting.

Powering up the assembled board was unexciting, nothing exploded, smoked, beeped or even blinked. Let's load the code!

The device I intend to build has one job: sound an alarm at the slightest sign of trouble. I'll leave it to Lassie to let me know when Timmy is stuck in the well, but that should alert as line blockage. We have voltage, current, and duration data available. Let's extend the timing component to track time of day. Now we need to set the time, or do we? Not really, what is needed is when the night-time cycle begins. A pump running overnight can be a sign of trouble. Add a button, press it when the night cycle should begin. (If we power up at noon, the clock would have the right time.) Now that there's a button, it can be used to silence the alarm signal. If there was an alarm, it would be useful to know why. A few LEDs to show a binary code should suffice. The power source can be a unused 5V phone charger. A battery backup is not really necessary, since a power outage will also shut down the power hungry pump. That was easy.

Scope creep

There is quite a lot of unused potential, even on an 8k Z8. I know, add a bluetooth adaptor! There's an HC-05 laying around, unused. That'll do. Since there is now bi-directional data transfer, there should be read-modify-write for threshold values. If the remote device has graphics capability the pump's operating current can be dispalyed as a graph, a valuable diagnostic tool.

Let's go!

Hold on, that part is not available until ... When???

Redesign a little bit.

Let's use a piezo beeper in place of the 555 and speaker. Forget the TCXO, use a standard oscillator. Only 4 left in stock? I plan to build more than that just for the beta test. Keep looking.

More scope creep

A conversation with a neighbor led me to the posibility of calculating the water depth in the well shaft based on the pump's performance. I may have to invite Math.h to the party.