Close
0%
0%

Raspberry Pi Ceph Cluster

A Raspberry Pi Ceph Cluster using 2TB USB drives.

Robert RouquetteRobert Rouquette
Following
Liked Like project

Just one more thing

To make the experience fit your profile, pick a username and tell us what interests you.

Pick an awesome username
hackaday.io/
Your profile's URL: hackaday.io/username. Max 25 alphanumeric characters.
Pick a few interests
Projects that share your interests
People that share your interests

We found and based on your interests.

Choose more interests.

OK, I'm done! Skip
Similar projects worth following
902 views
0 comments
7 followers
View Gallery
I needed an easily expandable storage solution for warehousing my ever growing hoard of data. I decided to go with Ceph since it's open source and I had slight experience from work. The most important benefit is that I can continuously expand the storage capacity as needed simply by adding more nodes and drives.

Current Hardware:

  • 1x RPi 4 w/ 2GB RAM
    • management machine
    • influxdb, prometheus, apcupsd, apt-cacher
  • 3x RPi 4 w/ 4GB RAM
    • 1x ceph mon/mgr/mds per RPi
  • 10x RPi 4 w/ 8GB RAM
    • 2 ceph osds per RPi
    • 2x Seagate 2TB USB 3.0 HDD per RPi

Current Total Raw Capacity: 36 TiB

The RPi's are all housed in a nine drawer cabinet with rear exhaust fans.  Each drawer has an independent 5V 10A power supply.  There is a 48-port network switch in the rear of the cabinet to provide the necessary network fabric.

The HDDs are double-stacked five wide to fit 10 HDDs in each drawer along with five RPi 4's.  A 2" x 7" x 1/8" aluminum bar is sandwiched between the drives for heat dissipation.  Each drawer has a custom 5-port USB power fanout board to power the RPi's.  The RPi's have the USB PMIC bypassed with a jumper wire to power the HDDs since the 1.2A current limit is insufficient to spin up both drives.

  • 1 × Raspberry Pi 4 w/ 2GB RAM
  • 3 × Raspberry Pi 4 w/ 4GB RAM
  • 10 × Raspberry Pi 4 w/ 8GB RAM
  • 14 × MB-MJ64GA/AM Samsung PRO Endurance 64GB 100MB/s (U1) MicroSDXC Memory Card with Adapter
  • 14 × USB C Cable (1 ft) USB Type C Cable Braided Fast Charge Cord

View all 10 components

  • Rebalance Complete

    Robert Rouquette • 3 days ago • 0 comments

    The cluster rebalance is finally complete.   The final distribution also appears to be tighter than before the rebalance.

  • Rebalance Progress

    Robert Rouquette • 10/13/2020 at 21:01 • 0 comments

    The rebalance is still progressing.  It's particularly slower because I also changed the failure domain on my cephfs EC pool from OSD to HOST which increased the number of PGs that needed to be remapped.  Aside from that, the rate of progression appears fairly typical.  The two gaps in the plot are power outages caused by Hurricane Delta.

  • Added 8 more HDDs

    Robert Rouquette • 10/07/2020 at 22:51 • 0 comments

    I added 4 more RPi 4's and 8 2TB drives to bring the overall disk count to 20.  The cluster rebalance looks like it's going to take a few days, but at least the I/O performance isn't too severely impacted.

  • Grafana Dashboard

    Robert Rouquette • 09/29/2020 at 20:52 • 0 comments

    I added a dashboard to my grafana server to show the telegraf data from the ceph cluster.  You can see the disk io and cpu load from the cluster rebalancing from 10 osds to 12 osds.

View all 4 project logs

Enjoy this project?

Share

Discussions

Similar Projects

5 Raspberry PI Zeros. One custom USB hub. Endless disappointment.
Project Owner Contributor

Terrible Cluster

ajlittajlitt

Exploring the possibility of a simple portable wallet with QR scanning capability for making easy transactions.
Project Owner Contributor

Portable Raspberry Pi Wallet

sfdogecoinsfdogecoin

#SolarPunk, Software Defined, Flexible Power Routing USB-C + DC Hub
Project Owner Contributor

PD OmniTool

Jaunty WunderkindJaunty Wunderkind

Shaped like a USB stick, ideal for a Raspberry Pi or Jetson. Connect up to 28 devices to a Pi.
Project Owner Contributor

StickHub - The Space Friendly 7-Port USB 2 Hub

rbtscorbtsco

Does this project spark your interest?

Become a member to follow this project and never miss any updates

Going up?

About Us Contact Hackaday.io Give Feedback Terms of Use Privacy Policy Hackaday API

© 2020 Hackaday

Yes, delete it Cancel

Report project as inappropriate

You are about to report the project "Raspberry Pi Ceph Cluster", please tell us the reason.

Send message

Your application has been submitted.

Remove Member

Are you sure you want to remove yourself as a member for this project?

Project owner will be notified upon removal.