Yesterday's post was big, and ties directly in to this one. Today I'm going to describe the actual logline contents themselves, along with an explanation for the entry (when needed).

First, it's important to realize that we're logging each event in two places. There's a local log file kept on each node, and a log kept on the main sensor node. This is both for backup, but primarily to allow for higher resolution sensor logs to be kept locally, and automatically purged as they age. Keeping dozens of nodes' high resolution data for weeks to months is going to get prohibitively expensive in terms of network bandwidth and memory.

For the most part, the logs on the sensor node are similar to the logs sent to the main logging node. This is because the process is essentially the same. The sensor node is constantly polling sensors, and either kicking out a log line when it detects an alert, or when it reaches a timing threshold (a minute since the last report, five minutes since the last report, etc).

Each sensor is it's own programmatic construct (object), that reports to another programmatic construct (object), the Node. The sensor object decides when the conditions are right for it to report, and then it hands it's data to the Node. The Node takes the sensor data (typically the name of the sensor board, sensor type, sensor value), and passes it to the Node. The Node takes this sensor data and packages it with Node specific data (Node Name, IP), and also appends the send time. All of this data is packaged in a Python Dictionary, which is ultimately Pickled (cPickled, to be precise) before it is both saved locally, and sent over the network to the logging Node.

Once the message is received at the logging Node, the message is unPickled, and the receipt time is appended. This is done so that time in transit can be (possibly) measured. This may give an indication of network performance issues down the line (and the logging node could create a sensor looking at this data).

Since the loglines are stored as Python dictionaries, parsing them is pretty quick. I've already written some commands that can be used recursively to pull a sub-set of the logs, and create a pseudo-log that can then be re-parsed for a different criteria (Show me all loglines from yesterday that contained temps).

Times are recorded in epoch time because that is MUCH simpler to deal with when programming. Python makes it very easy to put in an epoch time and spit out a human readable format.

Because the logging protocol uses Python dictionaries, with all of that flexibility, I can log all kinds of events. When a Node is booting up, it is logging the boot-up process, including configuration and connectivity. When the system is "finalized", we'll be able to write a log parsing file that re-creates configuration files, or even archives them automatically when a change is noted in the configuration. This feature will really help novice users when they are building their own system. Make a change and it breaks things, execute a couple of simple commands, and the system automagically reverts back to a working version.

During boot-up, the system is checking the I2C bus, and logging active I2C addresses that it finds. This is another nice feature that will help users get their sensor network up and running. It's not a big issue, but for a newbie, every little obstacle seems like a massive hurdle.

I'm working on bringing on my other project, a Raspberry Pi MAC/IP monitor, over to this project. The goal of that project was to detect and later classify when new systems came on to the network. I'm hoping that I can leverage this as a "presence" sensor, and detect when cellphones are in the residence/sensor net. The next effort, on that part of the project, is to get a network sensor node up and operational, and then have it report back to the main logging system using the already developed Node architecture. Eventually I'd like to be able to detect when Bluetooth devices are in range of nodes, but that's been fraught with difficulty.