Close

Autonomous Small nodes BigData solution - concept of arhitecture for future IoT support

A project log for Raspberry Pi 2/3 and Odroid XU4 5x compact cluster

Design for compact cluster and case that can be used for housing up to 5x Raspberry Pi2, Pi3 or Odroid XU4 (SoC) computer nodes.

JovanJovan 07/07/2016 at 01:200 Comments

We all knows that Small nodes based on SoC computers can bring excellent power efficiency and low cost to BigData system solution domain, bit it (still) luck much needed high computing performances that rack-mount PC based servers provide.

What I research recently was how to help BigData databases to obtains > 1MSPS reading from analog (industrial) sensor sources in real time. Making of log-files from ADC reading can not be considered as "real-time" solution.

Any way think that we need to make >1M database transactions (1 row insert) that will record all conditions that follow particular ADC value, and you will find out that even bigger among Big Data systems will "hiccup" on this demand.

Why is this important to save ADC values as unique values and not as log them and use after you will ask, and all I can say is because in a next few years M2M communication among +20 Billion Sensor/IoT systems and SoC/MCUs will demand processing in real time exponentially greater amount of information compared with today's volume which is generated by human social networks and web.

In concept that I proposed in my academic research Small nodes SoC systems and clusters will be first line of data acquisition and pre-processing. In that way BigData system will have by order of magnitude less demanding tasks to perform regarding pre-processing with much lower network load and power consumption.

In next illustrator I will sheer with you my vision of concept that need to enable reading (ADC) of fast sensor systems and real time pre-processing that will be sent to Small Nodes Big Data system.

(With a big apologies in advance because my original illustration of the Serbo-Croatian

language and for details you will be needing translator.)


Any way you can easily recognize main block Sensors (Physical to Analog electrical values), block with ADC and MCU development board systems ( from Atmel, Microchip, ST), block with SoC computers that use GPIO ports, USART (SIP, I2C...) and USB communication to get ADC values and pack them to a form with all other needed ingredients that can be sent via ETHERNET LAN (asynchronous packet communication) to Small nodes Big Data system on pre-processing or even to Big Data system in some far data center on regular processing.

Discussions