close-circle
Close
0%
0%

Terrible Cluster

5 Raspberry PI Zeros. One custom USB hub. Endless disappointment.

Similar projects worth following
close
Terrible Cluster is a toy computing cluster for experimenting with distributed computing and software deployment tools. The Terrible Cluster backplane board uses USB 2.0 to tie together five Raspberry Pi Zeros in a compact package.

The author does not expect that this will be of any benefit to society. I mean, maaaaybe someone could learn something about cluster computing, networking, USB, or hardware design. But only as a cautionary tale.

Please keep any advice about cluster simulation with multiple VMs, dumpster diving for free PCs, alternative interconnects, and how I'm contributing to the decline of humanity to yourself. I don't call you out on idling your car for ten minutes in the drive through waiting to purchase overpriced coffee every morning, so please return the favor.

"If you were plowing a field, which would you rather use: Two strong oxen or 1024 chickens?" -Seymour Cray

Terrible Cluster is possibly the worst example of a computing cluster imaginable.  Four Raspberry Pi Zeros each offer up their puny decade-old ARMv6 core and meager 512MB RAM for computation.  Meanwhile, a Raspberry Pi Zero W head node tries in vain to direct traffic across an inappropriately named "fabric" using USB 2.0, host a distributed filesystem, and control power to the compute nodes.  The Terrible Cluster backplane board powers and connects all five Pis together using a Cypress USB 2.0 hub and four USB power controllers.  Through hole micro USB vertical plugs allow 6mm spacing between nodes which results in a high density low volume system that fits easily in the e-waste recycling bin of your local electronics retailer or under the tire of a passenger vehicle.

I created Terrible Cluster to further my experience working with cluster systems and to play with strategies for cluster deployment of firmware and applications.  This hackaday.io project is a fantastic opportunity to strengthen the above self-delusion through accumulation of Internet Points in the form of views and skulls.

  • Boring bring-up pt. 2

    ajlitt09/17/2017 at 05:37 0 comments

    I soldered the rest of the board. Realized I had ordered the wrong part for the USB power connector and bodged it on anyway.  I'll update on functionality in a few days.

    But for now, enjoy some photos of the assembled board and 5 Pi Zeros:

  • Boring bring-up pt. 1

    ajlitt09/14/2017 at 05:59 0 comments

      The boards came in today, so naturally I had to get soldering despite being tired from working late last night.


      I populated everything but the USB power jack and the four downstream node USB plugs.  I didn't want to waste the connectors until I checked out the upstream end of the hub.

      I first smoke tested the board with my bench PSU and checked that the oscillator was doing its thing.  Once satisfied that I didn't have any surprise dead shorts, I hooked it to my PC through a micro USB jack to USB-A plug cable going to an intermediate sacrificial hub.  I was able to see the hub enumerate under Linux, and checked that it's reporting as having per-port power control.


      I also did a test fit of a Pi Zero W, and since I'm now confident it's not going to do any damage I powered it up.  The Pi's LED did its blinking thing as usual as Linux boots.

      Stuff I learned:

      1.  The hole dimensions for the USB vertical plug are too big to fit snugly.  I had to "justify" the upstream plug with one edge of the board.  I'll need to remember to align the other four the same way to keep the Zeros spaced uniformly.
      2. The Cypress hub IC shuts off the 12MHz oscillator when nothing interesting is happening, probably to save power.  I didn't catch this detail in the datasheet.  I spent a good half hour wondering why I'd see a fleeting 12MHz on the scope on power-on and nothing afterwards.  I suspect it will stay on once I get downstream devices connected.
      3. My thermal reliefs aren't that relieving.  I soldered in one of the electrolytics backwards, and it was a pain to desolder and clean the hole on the grounded leg.  I should have used thinner spokes on the reliefs, and made sure that there was no additional ground traces hiding under the plane fill.  It would also help if I replaced the solder sucker I broke a few years ago.
      4. Just because silkscreen art looks good in the KiCAD 3d render doesn't mean it will look the same on a finished board.  For grins I converted a photo of a picture one of my kids drew in school to a KiCAD symbol.  What was supposed to be a skull looks more like an upside down diseased pear.  Elecrow and other PCB manufacturers have a resolution limit to their silkscreen.

      Next I need to get the rest of the connectors in, test downstream power control, make up a test cable for the downstream ports (more later), and maybe plug in some more Zeros.  But that will have to wait for the weekend.

  • The sad story so far

    ajlitt09/05/2017 at 16:04 0 comments

    Last week I submitted the Terrible Cluster backplane PCB to Elecrow for manufacturing, and this weekend received notification that it has shipped.  Elecrow is nice enough to take a photo of the boards before shipping, and I can see that the plated slots for the microUSB plugs and the power jack were done correctly.  The graphic on the back, a skull my son drew last year, doesn't look so hot.  But I'm not losing any sleep over that.

    I still haven't ordered parts.  I have a BOM on DigiKey that's ready to go once I see the boards make it through customs.  I'm guessing it's a couple of weeks before I can get down to soldering.

    Meanwhile, let me catch you up.

    A few weeks ago at work I had my first experience using Ansible.  I needed to automatically build firmware for a legacy product that has very specific host requirements.  At the suggestion of a coworker, I used Ansible and Vagrant to assemble VMs with the right build environment and see the build through to completion.  While that was fun, it didn't give me an opportunity to use Ansible for parallel deployment.  I had a pile of Raspberry Pi Zeros, an interest in HPC, and an itch to make another PCB.  And so I decided to make a simple cluster.

    I first did a mock-up using a Pi3 and three Pi Zeros connected using USB device mode to gauge network performance.  I lost the numbers as they were unimpressive, but with all Zeros doing bidirectional transfers the upstream perfomance was in the 10Mb/s range and downstream was around 80Mb/s.  That's going to drag down any tightly coupled compute jobs, but since I plan to use it for playing with deployment and management I think it will be fine.

    I then looked at packaging.  I ended up choosing a through-hole soldered USB micro vertical plug from Hirose rather than the "dock" SMT one used by other similar projects so that I could fit the Pis closer together.  The SMT pins on the dock connector live on the long edge of the connector, so hand soldering would be near impossible unless I spaced them 20mm or so apart.  Cost is comparable, and the through hole is narrower which gives me more board space to work with.  For some reason, this connector and others like it require a board that's under 1mm thick.  Fortunately Elecrow can do 0.8mm boards.  I will have to take board flex into account when I make a case for this since 0.8mm could flex too easily under the strain of the vertically mounted Pi Zeros.

    I ended up laying out a board with the Cypress CY7C65632 4 port USB 2.0 high speed hub controller.  I chose this part for its on chip 3.3V regulator and the ability to configure operating modes without an external EEPROM.  I added an AP2191DWG USB power controller and a big 220uF cap to each downstream port so that the head node can control power for each port.  Each power controller is driven by the hub, and the overcurrent signals are fed back to the hub to handle any unpleasantness.  The AP2191 can handle 1.5A per port, and I don't expect any of the nodes to go above 0.7A.

    I strayed pretty far from Cypress's design guide, using a two layer board instead of four, and putting the smaller decoupling caps around the periphery of the hub chip instead on on the back side.  I think I did an ok job on trying to keep loops small for power pins and maintain USB differential impedance, so I'm hoping for the best.

    Todo:

    • 3D printed enclosure with strain relief for the backplane
    • Build and bringup v0.1 backplane
    • Shell scripts for USB port status monitoring and power control
    • Set up head node to USB boot the compute nodes
    • IP assignment based on USB port

View all 3 project logs

Enjoy this project?

Share

Discussions

Owen Trueblood wrote 09/06/2017 at 17:24 point

This project looks like fun.

How'd you make that nice 3D model of the PCB assembly?

  Are you sure? yes | no

ajlitt wrote 09/06/2017 at 17:30 point

I exported the board from KiCAD as VRML, then used Meshlab to turn it into STL.  I then wrote some OpenSCAD that uses the backplane STL and a Pi Zero model STL from Thingiverse to make the mockup.  The OpenSCAD and link to the Pi model are in the github in the sidebar.

The alignment of the USB connectors on the backplane and Zeros is eyeballed, so it's likely that the boards aren't aligned right.  It should be enough to make a first pass at an enclosure while I wait for boards.

  Are you sure? yes | no

Owen Trueblood wrote 09/07/2017 at 01:50 point

I've played with the 3D export feature of KiCAD but got stuck trying to use the VRML output. Converting with MeshLab is a useful tip, I'll have to give it a try. Thanks for the detailed answer!

  Are you sure? yes | no

Peabody1929 wrote 09/05/2017 at 16:07 point

I have a Pi 3 with a Cluster Hat and four Pi Zero's.  It's main purpose is to look good.  That can be said about many things in today's world.

  Are you sure? yes | no

ajlitt wrote 09/05/2017 at 21:35 point

The Cluster Hat partially inspired this project, particularly to improve on a few things like cost and size.  At some point I'll have a small 3D printed case for the system to make it a little prettier.

  Are you sure? yes | no

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates