I'm going to be storing a lot of data, so I need to develop a protocol for storing and handling that data. I touched on it briefly, but this is actually the central, critical part of the project. The goal here is to be able to store any kind of data, and have the system (or user) be able to pull the data they need to make decisions.

Currently, I'm looking at three memory modalities. The first, very short-term memory, is happening within the programs themselves. Certain sensors you want being polled continuously, and either reporting a min/max, an average, or not reporting unless certain conditions are met (Or a combination of these traits). I don't care that the door is closed, 30 times per second. But, I want to know when it opens (and I may want to know for how long it opened), and I definitely would like to be able to send a request to make sure it is closed.

For this reason, many sensors (if not all) are going to be set-up to be continuously polled. In testing I was able to poll a BMP180 for temperature and pressure over 30 times per second. If I can continue polling air pressure that quickly, then I stand a good chance of detecting door opening events through that mechanism. For temperature, i've already written a delta alarm. If the temperature raises by more than a set amount, an alert condition is triggered. This is in addition to a floor and ceiling alert (my house never needs to be warmer than 35c, or colder than 5c).

Now, the sensor itself is currently logging temperatures and pressures every minute. Eventually I'll drop that down to every 5 or 10 minutes. I can do that because they system is still polling the sensors continuously. If an alert condition is met during those 5 or 10 minutes, the system is immediately going to kick out an alert to the logging system, and that alert will run through a rule based process to trigger notifications.

Currently, when the system logs temperatures and pressures every minute, it is logging it both locally (in a file on the sensor node), AND over the network to a main logging node. Smart systems will be in place to truncate the locally saved data, so that it isn't saving days of temperatures, minute by minute.

However, there are going to be times where it will be useful to know trends with more resolution than once every 15 minutes. In this case, the main logging node (or user) will be able to reach back to the local sensor node and request higher resolution trend data.

It is hoped that eventually the systems will be smart enough to recognize specific event data (for example: The barometric profile of a door opening and closing, or what it "looks like" when I wake up in the morning, etc). In that case, the sensor node can also report an event to the logging node, with raw data verification/transmission upon request. Days down the road, it won't matter what the barometric profile looked like, as long as the event "Door opening and closing detected" is logged at the right time.

So, three memory modalities: Very short term memory within the program loop itself, looking at no more than a couple of minutes. Short term memory maintained on the Pi itself, lasting no longer than a couple of days. Then long term archival storage on a main logging node, but with the lowest resolution.