Close

Cluster Processing Work in Progress

A project log for Portable NAS Storage System with USB Drives

ArOZ Online >> A portable cloud storage and multimedia streaming station using Raspberry Pi zero w

tobychuitobychui 06/09/2019 at 12:280 Comments

When mentioning about Raspberry Pis, you might think of those low processing power mini SBCs. But not with ArOZ Online with Raspberry Pis!

We are working on the latest ArOZ Online System (or we sometime call "aCloud" as its name is a bit long) with Clustering functionality.

In simple words, you can choose clusters by "names" to work on the same task together.

So this is how it works. First of all, in your home networking, you got a router which act as an DHCP server. Each time a computer (Network interface connection initiated and request DHCP Discover to be precise), your router gives that devices a new IP address. Usually start with 192.168 follow by two more numbers and one dot. This is usually the nightmare of clustering as cluster require stationary end points for reaching other computers.

In aCloud system, UUID is used to replace IP address and for each cluster tasks starts, it resolves the ip address in real time by local area network scanning.  So, you can choose which nodes to help process the current task with a simple selector like this one.

We are still working on the communication pipelines between nodes but I am sure developers can use this with a bit of learning. Let me give an example for how this can speed things up.

For example, you got a 3 minutes video that you want to convert from mp4 to mkv. What you usually do is that you will feed it into ffmpeg to do the conversion. Let say a Raspberry Pi can perform 1 minute of video conversion using 10 minutes in real time. Using clustering, developer can split the video into 3 parts, 1 minute each, feed them into 3 Raspberry Pi which is linked as cluster and the video will be split using 1 minute, converted using 10 minutes and join back with 1 extra minute. (So total 12 minutes. Much better than 30 minutes.)

In case you are not interested in clustering, we also did some updates on Custom IME (Input Method Editor) . You can simplify call it with ao_module API wrapper and you can "take over" other module's keydown event and handle it on a separated window.  Fun huh?

Discussions