Close

Saving precious EEPROM space

A project log for µPower (beehive) SD logger

Let's observe an emerging beehive throughout the year 2018! (And create an logger from scratch for that purpose)

janJan 08/05/2018 at 10:374 Comments

Tl;dr: Use this trick if you need to save space and don't need more than one decimal precision. Save >60% more logs using the same EEPROM.

Intro

Okay, using a 512Kbit EEPROM isn't exactly the reason to save space. After all it stores 3120 data entries which consist of timestamp and 4 float values. Reminder: the internal EEPROM of the Atmega328P is 1024 bytes only. Using that, the following might come in handy.

But can we get more out of it? Yep:

It's a small trick which works if we need only precision to one decimal e.g. 25.3°C or 67.9% RH.

Floats use 4 bytes (but are much more precise of course), integers do need 2 bytes in EEPROM.

Let's assume we just shift the decimal point of 25.3(°C) to the right. We get 253, which is easily stored as an integer. Same with negative values. I use the following function to store my floats as integers (and convert them back to float):

#define floatToInt(x) (int)(x*10)
#define intToFloat(x) (float)(x/10.0)

 Here's my data structure I use to save log entries:

typedef struct
{
  uint8_t tag;
  uint8_t monat;
  uint8_t jahr;

  uint8_t stunde;
  uint8_t minuten;

  int temp1;
  int temp2;
  int temp3;
  uint16_t feucht1;
} logEintrag;

logEintrag LOG;

 When I get my values from the sensors I just write them to the struct like this:

LOG.temp1 = floatToInt(25.3);

Of course 25.3 is some variable I access directly.

When reading back the logs to write them to the SD as a CSV string I j´convert them back on the fly:

intToFloat(readLOG.temp1)

 Works like a charm, with negative values as well. It takes a few processor cycles more but now the EEPROM holds 5041 instead of 3120 logs, which is 61,5% more!

Want to save even more logs?

Other things to optimize would include:

  1. saving the year-value only once and then again, when it changes
    1. savings: 1 byte
  2. same with months
  3. (...)

This would need more code to handle that when creating the CSV entries which are saved to SD. Doing this with year and month would get you close to 6000 entries or over two months of data in EEPROM (and thus not switching on the power hungry SD in two months).

Discussions

K.C. Lee wrote 08/05/2018 at 15:04 point

You can do some simple compression. e.g. storing the delta values between readings and this could take fewer bits.

  Are you sure? yes | no

Jan wrote 08/06/2018 at 10:13 point

You mean if say temp. reading 1 is 26.5°C and temp. reading 2 after 15 minutes is 27.0 I just store the difference? This would mean that I needed 2 bytes for reading 1 (265 stored as uint16_t) but only one byte for reading 2 (5 = uint8_t)?
But then I´d need to allocate space dynamically?! Converting it back to the "real" value for the CSV string totally depends on first value. If that gets lost or corrupted all my readings are worthless?!

  Are you sure? yes | no

K.C. Lee wrote 08/06/2018 at 12:27 point

You can store the full value once in a while to limit the data loss.  e.g.every 8 or 16 bytes would have one 16-bit value.  7 readings to 8 bytes or 15 to 16.   This still provides significant savings.

  Are you sure? yes | no

K.C. Lee wrote 08/05/2018 at 14:48 point

I use scaled integers for most of my projects.  When your inputs are integers and most of the time, the outputs can simply be scaled.  I keep my math in mV or whatever and only scale the outputs.  Unless you are doing some fancy DSP code you rarely need the dynamic range outside of 32-bit integer: 10 orders of magnitudes (-2147483648 to  2,147,483,647) and more significant digits because none of the bits are needed for the exponents.

There are integer fraction that can be used for approximation for float constants.  They can be good to 8-9 significant digits easily.

  Are you sure? yes | no