Really compliments! From the photos you should be a fan of r/cableporn ! :)

Słabo z chłodzeniem.

Moim zdaniem lepszym sposobem była by tuba. Zobacz jak projektowano cray-1 albo nowy mac

płyty ułożone dookoła wytwarzają naturalny obieg powietrza jak w kominie. Do tego można spokojnie wstawić człodzenie wewnątrz i na zewnątrz (np. pojemnik z wodą) 

From my own observations of the hardware temperatures over time, the cooling is already adequate.  While none of the hardware is running cold, it's still certainly well within acceptable operating limits.  More aggressive cooling would simply increase the power consumption and increase the TCO.

Hello, Do you or is there a way to set thiss up? I'm having a hard time trying to figure this out.

I wold be nice if there was a video tutorial on how to setup the raspberry pi cluster for this.

I'm not particularly inclined to create a video tutorial.  There are already countless online tutorials available (not counting the official Ceph documentation) that walk through the planning, creation, and stand-up of Ceph clusters.  There are no special steps or tricks to the RPi OS installation either, and any distro that supports the RPi will work.

Hello i understand. I just don't know where to start in software after setting up all the hardware. Where to begin.

This is so overkill for what it does. Technology should be used to simplify a need or a process, not pervert and over complicate it!

I agree with your sentiment about over complication. However, after many years of maintaining RAID arrays both large and small, and having watched as Ceph and other distributed storage systems have matured, I've come to appreciate that the distributed storage approach is significantly better in terms of failure tolerance and data integrity.  In terms of data warehousing, which is the principal purpose of this project, distributed storage offers the best long-term viability compared to RAID systems in terms of both data loss probability and capacity expansion.  I have personally experienced instances where rebuilding a failed drive on a RAID array triggered secondary drive failures.

I do ultimately agree that for the average person, this type of storage would be inappropriate, but in a situation where substantial continuous growth is expected, conventional single-machine approaches have their limits.

The more I look into this the more sense it makes and like you I have decades building and working with RAID systems and this does look like a more flexible way to work with storage.

I'm not familiar with Ceph but from what you described (based on the 8GB RP4 requirements) and what's written online, it seems like the RAM is being used as a cache and Ceph distributes the data to the disks. That would make writing extremely fast but reading is still bound to the read speed of the drives. Or do I misunderstand the concept?

The data is distributed on the client side among the OSD service instances.  The OSD in-memory cache fulfills the same purpose as a kernel disk cache would, but allows the OSD daemons more nuanced access for it's internal operations and data integrity verification.

In terms of IO performance, the caching does help repetitive reads, but, for the most part, the performance is strictly limited by the RIO performance of the physical storage media.  Ceph write operations block on the client side until the data is actually committed to disk.  The only exception is CephFS where there is some limited client side buffering to allow for data chunking on large block writes.

I've been looking at doing something similar, but the cost of 4TB drives looks to be better per GB. Is there any reason for going for 2x2TB drives per Pi, e.g. performance, cost? Could you go 1x4TB, or even 2x4TB per Pi? 8TB per OSD would still technically meet the "~1GB RAM per 1TB" rule of thumb for Ceph.

I went with 2TB drives because they seemed like a good trade off between capacity on rebuild time.  There's also the fact that Ceph performance is directly tied to the number of OSDs.  More OSDs gives you more throughput especially with low performance hard drives.

That being said, there's no reason you can't stretch the OSD memory that far, but it's important to keep in mind that the full 8GB on the RPi is not truly usable by the OSD daemons.  The kernel carves out its own dedicated chunk, and you need to leave some for the kernel disk cache.  I would not use more than 7.5GB of the 8GB if you the RPi to remain stable.

I was thinking of doing almost the exact same thing, except instead of using RPI 4's I was going to use Orange Pi +2E boards.  Does anybody have any comments on how this change will affect the feasibility of the project?

The main limitation with that approach would be the lack of USB 3 and memory. The +2E only has USB 2 and 2G RAM.  It would certainly be possible to run an OSD daemon, but I don't think you would  get usable read/write performance.

You know, you're absolutely right... without USB 3 it's going to be practically unusable.  But I just had another thought.... what about if, instead of using a USB-connected SSD, I use a large SD flash card for the "user storage" and use the eMMC and/or SPI (still trying to work out the differences and capabilities of each) for the OS?

Thanks for your answer. I have expected that you are going 64 bit. (sure, 8GB Pi´s I saw now)
I use a ceph cluster for some years at home too. 

Started with 3 nodes and ceph hammer. First I used odroid XU4, but because of USB-HD issues changed to odroid hc1/hc2. Disadvantage is that they are only 32 bit.
But I love the flexibility of ceph.
Now my cluster is a little mixed: 3 mon/mgr/mds with octopus 64 bit, 12 osd mimic 32 bit. With armhf I found no possibility to go more than mimic. The octopus version for armhf in ubuntu 20.04 is broken.

What kind of throughput are you getting when you're transferring files to/from cephfs?

Cephfs can fully saturate my single 1 Gbps NIC indefinitely.  I don't currently have any machines with bonded ports, so I don't know the tipping point.

Since seeing your project two weeks ago I couldnt get it out of my head and now I'm trying to find a usecase that would necessitate building a cluster ;)

Have you ever benchmarked your setup? Red Hat has a go-along written up on  https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/4/html/administration_guide/ceph-performance-benchmarking

I'd be quite interested how performance increases in such a large scale compared to the other rpi clusters found on the net which are mostly 3-5 units only. If you have a client with multiple ports it'd be interesting to see how much LA increases speeds... Maybe that can circumvent the client as the limiting factor.

Looks very nice.

I think you are using the raspies in 64 bit mode because of limited support of ceph for armhf?
Which version of ceph is running?

The rpi4's are running Ubuntu 20.04 aarch64, and I'm using Ceph Octopus (15.2.7) from the Ubuntu repos (not the Ceph repos)

Looks great. Just a couple of questions:

- Which install method did you use for Ceph?

- What kind of data protection do you use? Replication or EC? How has the performance?

I used the ceph-deploy method.  It's simpler and makes more sense for lower-power embedded systems like the Raspberry Pi since it's a bare-metal installation instead of being containerized.

I use 3x replication for the meta, fs root, and rbd pools.  I use 5:2 EC for the majority of my cephfs data.

20TB raw capacity per shelf is great. How are you finding heat? It would be very cool to wire in two temperature probes to your management Pi's GPIO and monitor temperature by the exhausts.

Have you found single gigabit per node to become a bottleneck?

Awesome project, makes me very jealous.

The CPU and disk temperature tends to stay below 55 C.  The Gigabit ethernet hasn't been a bottleneck so far, and I don't expect it to be.  The disks don't perform well enough with random IO to saturate the networking, and filesystem IO on CephFS is typically uniformly distributed across all of the OSDs.  As a result the networking bottleneck is almost always on the client side.

This is a sweet looking project!  I noticed that you look to be playing it safe with how you distributed your Pi's based on available ram, are you able to provide any details on what the memory consumption has looked like when idle and under load?

The OSDs are configured for a maximum of 3 GiB per service instance and they tend to consume all of it.  The comes to 6 GiB per 8 GiB RPi just for the OSD services.  The kernel and other system services consume a minor amount as well, so they tend to consistently hover around 20% memory nearly all the time.  The extra "unused" memory is necessary as padding since there is no swap space.  (Adding swap on an SD card is simply inviting premature hardware failure.)

Awesome, thank you.