Close

No Pain, Just Memory Gain:Using the Pro Mini XL with The Things Network

A project log for Searching for a better Pro Mini

I LOVE the Pro Mini! But it has limitations. To overcome some of them, I’m designing the Pro Mini XL & Pro Mini nRF52. What do you think?

andyAndy 07/04/2019 at 00:380 Comments

TL;DR

I got the Pro Mini XL working with the Arduino-LMIC library. It’s performing OTAA to The Things Network over LoRaWAN. And it’s reporting temperature and humidity measurements from an SI7021 sensor - with 80% program memory to spare!

(Note: There’s no SI7021 sensor connected in the central photo - I added it in later when my extra female pin headers finally arrived!)

In Brief

I’ve been running out of memory.

Whilst trying to develop/debug a Pro Mini 328P-based low-power temperature/humidity sensor node on The Things Network over LoRaWAN, I’ve been reaching the 32KB program memory limit and 2KB dynamic memory (SRAM) limit.

Then it occurred to me.

Why not test the pin-compatible, drop-in replacement benefit of the Pro Mini XL?

Essentially, all I needed was more program memory and SRAM.

So I updated the pin-mappings, replaced the 328P for the XL, recompiled.

Shazam!

Everything just worked!

And now my existing Arduino sketch was only using 20% of the program memory!

Now to add more sensors: GPS. Altimeter. Barometer.

Hmmm, what else could I fit, I wonder...?!

Gettng Started

I began tinkering with all things LoRaWAN back in 2016 during a commercial farm monitoring project.

Farmer "Dave" needed to monitor the temperature and humidity of a 120+ acre farm.

But it was approx. half an hour’s drive away from the main HQ.

Initial research into low-power, long range radio transceivers led me to LoRa and LoRaWAN.

I soon discovered that I had just missed out on a Kickstarter campaign for The Things Network (back in 2016), but I pre-ordered myself a Gateway, Node and Uno anyway.

Whilst I waited for the official development devices to arrive, I did what any impatient "Tech Tinkerer" would do...

I DIYed my own.

Using a Raspberry Pi 2 and LoRa concentrator board for the gateway (more on that below!) and a nice supply of Pro Mini 328P clones with Hope RFM95 radio modules for the nodes, I was all set.

So I set up a LoRaWAN system of solar/battery powered nodes and a gateway to provide the environment visibility Farmer "Dave" needed.

All was ticking along nicely.

Until one day a blaze of fire consumed one of the barns and almost all of the tech perched on top of it!  (Thankfully, no one was injured and the fire was contained quickly.)

So I was invited back to upgrade and reinstall a new one.

The (Main) Problem

In a word: freezing.

Farmer "Dave’s" farm sits at over 4,000 feet up in the mountains, so it gets cold.

Very cold.

Especially at night.

It can get so cold that it will freeze the fruit buds when they’re just starting to emerge… and then that’s it.

Your entire crop is gone - for another year!

So how to stop the buds from freezing?

You can only resort to defensive measures (e.g. water sprinklers, etc.) when you’re almost positive a freeze is imminent.

That's because it might actually cost tens of thousands of dollars to do so.

Especially in a state with severe drought like California.

In a perfect world you would monitor as much about the local environment as possible to help you predict when freezing might occur.

Temperature, humidity, altitude, barometric pressure, wind speed, etc. all play a factor to a greater or lesser degree.

You might also want some sort of location information from a GPS perhaps, to tie all that environment information to a specific area.

On large scale farming, you could get micro-climates specific to one area of a field that differ from another area - depending upon the geography and topology.

However, using the Pro Mini 328P as the microcontroller for the LoRaWAN nodes had previously constrained the number of sensors I could comfortably hang off it.

With only 32KB program memory, I ended up limiting myself to an SI7021 to monitor temperature and humidity.

But now I was revisiting the problem with my freshly minted Pro Mini XLs.

I wanted to see how easily I could replace the existing Pro Mini 328Ps and how much room I would have for anything (...or everything!) else.

So I dug out a breadboard, Pro Mini 328P clone and radio module and set to work.

Testbench Tangents

Initially, I breadboarded everything in the usual way.

And of course, wires were sticking out everywhere like a rat’s nest. 

Debugging bad connections on the breadboard soon began to eat up precious time.

So I decided to make a strip-board testbench instead with female headers for the microcontroller.

In theory, this would enable a more repeatable like-for-like comparison when I later compared the different Pro Minis.

I wanted to test whether I could literally just swap out the 328P-based Pro Mini for my Pro Mini XL, and even possibly the Pro Mini nRF52.

Since I was also going to make an upcoming move between the continents (UK to US...and back!) and needed to respect the frequency changes (e.g. 868 to 915 MHz) in the LoRaWAN spectrum, I also added female headers for the radio module too.

Of course everything needed a common power source, but I wanted to see first if I could get away without using a regulator and run direct from a battery.

So I went with two male header pins and a standard 3v supply from two fresh 1.5v AA Duracell batteries.

And last, but by no means least, was debugging over the UART.

Three more male header pins for GND, TX and RX connecting the side UART pins of the Pro Mini boards.

This is what all of the above looked like:

  1. CurrentRanger with OLED Display
  2. 2x AA Batteries with 3D Printed Cam-Switch
  3. 80mm x 50mm Strip Board with various wires and 2.54mm Male/Female Header Strips
  4. GorillaBuilderz LoRa Adapter PCB for RFM95 Modules
  5. HopeRF RFM95W LoRa Module (Only 868 MHz Shown)
  6. 868 MHz Antenna with SMA Male-to-Female Adapter (Can't remember where I bought this...)

And here it is again with the radio module plugged in and (almost) ready to go:

(Note: Pro Mini 328P on the left in blue, Pro Mini XL with ATmega1284P on the right in green.)

Low Power Mods

The (very) eagle-eyed among you will notice some components missing from the Pro Mini 328P.

And you’d be right:

You’ve probably seen mods like this to reduce the power consumption of the standard Pro Mini 328P.

My initial reference for this idea was this one from back in April 2013.

Patrik had removed the regulator and power LED from his 5v, 16MHz Pro Mini 328P board and reduced his overall average current down from 476uA to 136uA.

Quite a saving!

In the same manner, Andreas had modified the same kind of Pro Mini 328P clone as I was using and had further profiled the power consumption of just the sleeping microcontroller down to 20.9uA.

If you’d like to learn more about power saving techniques for microcontrollers like this, then I highly recommend Nick Gammon’s lengthy but awesome posts on the topic from back in 2012. 

Possibly the genesis for the above experiments?  I don't know...

In my case, instead of removing the power LED, I simply removed the 1K resistor next to it accomplishing essentially the same thing.

I’ve found 0603 SMD LEDs a bit tricky to remove without damaging, especially using either hot air or soldering iron.

Removing the 1K resistor instead meant that I could always restore previous function later if I needed - since 1K 0603s are practically ubiquitous.

Nothing unique about this variation-on-a-theme, but I hope maybe helpful to the unfamiliar.

The voltage regulator, by comparison, came off with copious amounts of flux, wicking and tightly directed hot air.

Wiring Connections

Wiring connections between the microcontroller and radio were straightforward.

SPI connections are standard pin-to-pin, as are VCC and GND.

I took advantage of pin and track alignment on the strip board to connect DIO0 and DIO1 from the radio module to A2 and A3 respectively and finally wired RST to A0.

[Insert Wiring Diagram - I'll try to do this when I get a moment...]

You'll see the pin mappings in the Arduino code below too.

A DIY LoRa Gateway

All well and good, but at this point all I’ve made here is a (potential) LoRaWAN Node.

What about the LoRaWAN Gateway you ask?

After all, what is going to receive and forward all these LoRaWAN packets to The Things Network?

I couldn’t use my pre-ordered (and now delivered!) gateway from The Things Network, since I had subsequently deployed that elsewhere for another project.

And I couldn’t afford to buy another just yet at this phase of development.

So I had to improvise…

Behold:

[Insert Photo - Still need to photograph this one...]

This is very much a DIY set-up right here - but it works!

[Note:  If you're not really interested in all this LoRa Gateway stuff, please skip ahead to 'The Baseline' section below, since I go off on a bit of a tangent here...]

Basically, I’ve got a Raspberry Pi 2 with one of Multitech’s (old) LoRa mCards inserted into a modded USB-to-Mini-PCIe converter.

I actually made this back in 2015 from Nestor’s fantastic wiki guide right here.

He really deserves all the credit for reverse engineering Multitech’s hardware and getting a software solution working.

Essentially, you’ve got a triplet of Semtech LoRa chips (1x SX1301, 2x SX1257) interfacing via SPI to a standard FT232H configured in USB-to-SPI mode in what (I understand) was quite a common LoRa “Concentrator” design at the time.

Nestor’s hardware modification is simply to power the mCard from the RPi 2’s 5v supply and interface.

You’ll probably need to remove the RPi 2’s built-in (software) current limiter to get the concentrator to start up correctly (like Nestor demonstrates below) or power it externally.

On the software side, it was less straightforward.

Nestor needed to configure the original lora_gateway code to use a different FTDI USB to SPI converter PID, reset the SX1301 by toggling one of the FT232H's GPIO pins and finally trace the right clock source from the SX1257s to the SX1301.

Phew!

Again, the details are in the excellent wiki and all I did this time around was reconnect my original hardware and update/re-compile/execute the software from Nestor’s git.

And everything just worked.  (Thank you Nestor!)

Sadly, Multitech discontinued the original LoRa mCard used in the above tutorial and replaced it with a new SPI only version in 2017 - i.e. no USB to SPI (through FTDI) bridging anymore.

Therefore, the new hardware is NO LONGER compatible with Nestor’s tutorial.

However, it might be possible to still get it working by applying some patches from here.

Take a look at the “lora-gateway-usb” and “lora--packet-forwarder-usb” patches in particular.

I’ve not looked into this myself since I don’t possess the new SPI-only mCard, so I’m afraid I can’t assure you of any success - sorry!

There are lots of other LoRaWAN Gateway cards in this mPCIe format though as well as standalone boards, modules, devices and, of course, The Things Network’s own offerings.

Personally, I think the next time I take a look at this, I might try building Will’s OpenSource version!

It looks awesome and is designed to be in a Raspberry Pi hat format.

Although it can’t current do LBT (see below) - take note Korea!

So take your pick, but back to the Multitech tangent...

Old vs. New Hardware

The reasons for Multitech upgrading their mCard were probably many.

Here’s my guess at a few of them - please correct me if I’m wrong!

First, the original LoRa mCard and associated software implemented what looked to my eyes to be a very basic downlink queue - “very basic” as in a queue depth of 1.

Yeah...

Basically, any downlink packets sent to the gateway were either immediately sent out or risked being overwritten by any subsequent downlink packets.

One option to delay the inevitable was to increase the RX delay to, say, 5-8 seconds or so which might allow sufficient time for the downlink message to be received and pushed to the packet forwarder before another came along.

But as you can imagine, it wasn’t a solution.

I understand that Semtech subsequently released a version of the packet forwarder that implemented a basic software queuing structure.

But in the end even they decided to deprecate support for FTDI in version 2.2.0.

Moving to a purely SPI-version of interfacing with all the SX chips provides the most flexible approach to queuing and inter-chip communication anyhow.

FPGAs and LBT

Second, Semtech’s AP2 reference design for their SX1301/8 chips calls for an FPGA to implement the Listen Before Talk (LBT) feature in spec. v1.5.

The idea behind this feature is to make it at least possible for multiple transmitting nodes to share the same transmitting channel.

Nodes would continuously monitor the channel(s) to determine which channel(s) are free and when, and then only transmit when a channel is not in use by any other node.

Semtech’s recommended implementation calls for an FPGA chip, like the iCE5LP2K that Multitech use in their updated mCard:

Thermal Solutions

Looking at the backside (see below) of Multitech’s updated mCard, suggests that the new version might have a better heat dissipation profile too.

The original version had the FT232H on (almost) the exact opposite side of the SX1301 - see the above board layout.

Under load, the metal cover on my DIY gateway would become too hot to touch and, under heavy load, felt like I could fry an egg on it!

The Things Network guys in Leiden went with a passive heatsink

...a very big heatsink!

Since the new version uses SPI only, there is room on the backside of the SX1301 for a heat dissipation pad.

My gateways tended to become unpredictably unreliable (worst of both worlds) when they got too hot - and I don’t think I was the only one.

Better cooling always made the “problem” disappear, so hopefully this improved board layout and chip placement will go some way to mitigate that issue somewhat.

The Baseline

Now that all those testbench tangents were out of the way, I moved on to (re)establishing some baseline results with the original Pro Mini 328P.

Whenever I revisit a past project, I invariably spend half a day getting back to where I left off.

It can be painful.

But it’s usually always profitable too.

Improved Uploading

The first profitable improvement I made - that I think is worth mentioning here - is updating the standard Pro Mini Optiboot bootloader to run UART0 at 250Kbps for uploading.

Why?

Well for one, it’s faster at uploading code.

Your typical Pro Mini 328P will often come with a bootloader running at either 38.4Kbps, 57.6Kbps or at the most 115.2Kbps.

But these are simply a standard practice.

Buried deep in the recesses of the ATmega 328P datasheet, on pages 196 to 199 (of 662!), you see tables showing (among other things) how UART speed, microcontroller frequency and error rate compare.

Notice that when running the microcontroller at a frequency of 8 MHz and the UART at a baud rate 250KBps the error rate is still 0%.

In fact, you’ll notice it’s still 0% right up to 1Mbps!

So why not run it at 1Mbps?

Well, I might, given a specific scenario, but at least at 250Kbps, I can run the Pro Mini at 2, 4, 8, 16 and even 20 MHz - see the other tables.

I ran a few initial tests and uploading was solidly reliable.

Again, this idea isn’t original to me.

The first person I read about who’d pulled it off was Charles Hallard with his awesome ULPNodes.

Charles designs great boards and I’ve bought his stuff before, everything from WeMOS gateways to NodeMCU gateways.

(Check out his new one that uses a TPL5110 as a power gate and has written his own bootloader to boot!)

Improved Sketch

The second improvement I made was to re-write my basic wake-on-interrupt sketch.

I wanted to see just how much I could successfully hang off the Pro Mini XL.

But at the same time, I wanted to easily use the same sketch for both the Pro Mini 328P and Pro Mini XL.

That meant being able to configure which peripherals were attached to testbench.

Enter the #define.

For the unfamiliar, wrapping code sections with #ifdef and #endif allows the compiler to ignore whole sections of your code if a #define is not present.

So in the screenshot above, if line #33 (#define SENSOR_SI7021) was commented out or not present at all, the compiler would simply ignore lines 40-56.

This was the first time I’d intentionally used them as a way of making “configurable” code to be used in a sort of “cross-platform” way and it worked quite well.

Please see my Github project for a repository of the code I used during this project log.

Results: Pro Mini 328P

With everything compiled and uploaded, I fired up the CurrentRanger and watched…

In short, the video shows the microcontroller booting up, joining TTN and dropping into low power wake-on-interrupt sleep.

I then short the interrupt pin to wake it, whereupon it sends another data payload and drops back into sleep.

Nothing fancy.

Just a proof-of-concept to give you an idea of the general code size required to accomplish a basic node operation.

But look how much of the program memory on the Pro Mini 328P it uses up:

This (very) basic sketch uses up 76% of the available program memory and uses a further 81% of the available dynamic memory (SRAM).

That doesn’t leave much for adding more sensors or a GPS - not to mention simply expanding the main code beyond this basic operation of wake-send-sleep.

In fact, even just increasing the LMiC debug verbosity by one level in the existing libraries, i.e. no new sensors, etc. just the LMiC and SI7021, blew through the Pro Mini 328P's SRAM completely:

But back to the original compile above, it is also worth noting the red warning message there too:

As your code size creeps ever closer to the maximum available in your microcontroller, stability issues start to rear their ugly heads!

Your code might run fine initially, then appear to lock up or reset the microcontroller for no apparent reason.

Not great when you’re trying to protect 120+ acres of fruit farm from a crop-threatening freeze!

Or some other security or safety sensitive operation.

So what can you do?

At this point, I would usually turn to Bill Earl’s excellent Adafruit article “You know you have a memory problem when…” to harvest what was low-hanging.

See especially the section on Solving Memory Problems where Bill deftly breaks down some helpful tips and tricks on how to optimize program memory, SRAM and make better use of EEPROM.

Not this time though; I needed to “earn more”, not try to “save my memory rich”.

But how straightforward would that be…?

Results: Pro Mini XL

I took one of my Pro Mini XLs and set it up as close as I cared to mirror the 328P.

I set it’s fuses to use the internal 8 MHz oscillator, uploaded an Optiboot bootloader to use UART0 at 250Kpbs and swapped it for the 328P in the testbench.

Then I literally took the same wake-on-interrupt sketch and just changed the pin-mappings to those of the Pro Mini XL.

Finally, I needed to add the ATmega1284P as a new compile target in the boards menu.

Although this was a multistep process, it was also a straightforward one which I detail in the “Getting Started Guide” for the Pro Mini XL.

You add another Board Manager URL, download through the Board Manager window and then select the newly added Board options.

So with the new target selected, I re-compiled, uploaded and this is what I saw:

Yes, you read that right.

The same sketch uses up only 20% of the available program memory and only 10% of the available SRAM!

I now had almost 100KB more still to play with - more than 3x ALL the program memory of the original 328P.

But did the same code still run as before?

See for yourself:

The microcontroller boots up, joins TTN and drops into low power wake-on-interrupt sleep.

I then short the interrupt pin to wake it, whereupon it sends another data payload and drops back into sleep.

Exactly the same.

And that’s the point!

The Pro Mini XL proved its purpose as being a drop-in replacement for the original 328P.

I only needed more program memory and SRAM and that's exactly what I got.

With the bare minimum of fuss and effort on my part.

No code optimization.

No library modification.

No bootloader shrinking.

Just more memory.

Having said that, the more observant among you will have noticed the slightly lower power consumption during sleep at approx. 630nA.

I’m not entirely sure why that is, but during testing it did appear to be consistently lower.

Epilogue

This log entry was an attempt to cover my journey as I revisited an old project and did something new.

By using the Pro Mini XL, I was able to solve my previous memory problems without creating any new ones.

And the beauty of this board is that it’s also pin compatible, so I can replace the 328Ps in my other nodes without having to make any further modifications.

The reduced sleep current at least appears to be a nice bonus too!

Finally, a MASSIVE THANK YOU is definitely in order!

Any hardware is only as useful as the software makes it to be.

The only reason why any of this "just worked" is down to the awesome effort of everyone involved with the software and libraries.

I'm afraid I'll have to avoid naming individual names, since I'll inevitably leave someone out, but in an attempt to cover my bases:

MASSIVE THANK YOU to everyone who helped shape the MightyCore Library into what it is today.

Thanks to everyone who helped get the 1284P core working with Arduino.

Thanks to everyone who helped with the underlying core code, supporting libraries, bug fixes, etc.

This project would not have been a success without you!

One Last Thing...

Curiosity, got the better of me.

Would the Pro Mini nRF52 work just as well?

After all, it's pin-compatible too and has 16x the program memory of the 328P.

To cut an already far too long story short, it certainly appears to...

If you'd like to hear more about that, please let me know in the comments section below.

Also, any other questions, comments or suggestions - I'd really appreciate hearing from you.

Thanks!

Discussions