A project log for Terrible Cluster

5 Raspberry PI Zeros. One custom USB hub. Endless disappointment.

ajlittajlitt 09/28/2017 at 02:424 Comments

I spent the last couple of weeks trying to figure out the best way to deploy an OS to the nodes over USB.  I spent a few late nights trying to roll my own minimal Raspbian with debootstrap, and a few more scratching my head over how to use the USB boot and an initramfs to write the SD card on first boot or when directed to reformat and reinstall.  And I was hesitant to write anything to automate any of this, since I have my heart set on using Ansible for deployment and I'm still an Ansible noob.  I was busy yak shaving and yet I still didn't have a working way to copy an OS out to all the nodes over USB.

And then I remembered what I named this project.

So I went for the easy way out.  Stick with the stock Raspbian Lite image.  Modify the image so that it doesn't boot over SD and falls back on USB.  Use the mass storage mode in rpiboot to write the Raspbian SD image to each node.  Use rpiboot to serve different cmdline.txt bootfiles based on the USB hub port so that they each get their own unique USB networking MAC address.  And use shell scripts for image generation and SD writing instead of making this another Ansible lesson.  None of this is how I'd imagined it working, but at least it's working now.  Later I can move the process to Ansible and different OS images and deployment schemes.

Three evenings later and the cluster can now write a Raspbian .img to all four nodes very slowly (30 minutes or so for all 4), boot them very slowly with a unique MAC and IP (about 5 minutes for all to come up), and take remote SSH logins from the host Pi.

It's hard (and boring) to show any of this in action, so here's a session showing a network between the head node and all compute nodes at once:

See?  The Terrible Cluster lives up to its name.


Thomas wrote 10/07/2017 at 17:29 point

Have you thought of using Docker on the Raspberry Pi?

  Are you sure? yes | no

ajlitt wrote 10/08/2017 at 13:54 point

That's one of the things I'm planning to use this for.  We are using Docker at work and it's about time I got some experience with it.

  Are you sure? yes | no

Thomas wrote 10/08/2017 at 16:53 point

Docker on x86-64 is easy, especially on Debian derivatives, and using DockerHub. Docker on ARM or x86-32 hosts is "a bit" more difficult, especially if you want to create a Swarm. I've never tried that, and I'm curious about your approach.

  Are you sure? yes | no