OSD Memory Consumption

A project log for Raspberry Pi Ceph Cluster

A Raspberry Pi Ceph Cluster using 2TB USB drives.

Robert RouquetteRobert Rouquette 11/20/2020 at 00:442 Comments

Per the inquiry from @Aaron Covrig , here's the memory consumption over the past week:

The three series with nightly jumps are the mon instances, and the jumps are the mon services shutting down for the backup process.


Robert Rouquette wrote 01/25/2021 at 19:32 point

The backup process is a systemd timer service that stops the mon process, tarballs the mon data directory, and the restarts the mon process.  The tarballs are copied to an independent rpi for archival.

The OSDs are using 1.5 GiB/TB of ram, which comes to 6GB total for the two OSDs.  Using a 4G pi would make the OSDs unstable during heavy I/O or recovery.

I haven't noticed any appreciable wear on the mon SD cards.  The cluster spends most of it's time clean and idle, so there's not much activity most of the time.

  Are you sure? yes | no

for.domination wrote 01/25/2021 at 19:17 point

Hi, I really like this little cluster. Could you elaborate on "the backup process" a little more please? Do you mean you're stopping the mon processes completely, backup the mon('s system) and restart? I'd stagger the backup points so not have them occur all at the same time. What folders exactly are you backing up? Or is it simply the whole system? And what is the green line up there with the mons?

Am I reading it right that the osds aren't even utilizing only around 1 gb/tb even while rebalancing? That should be comparable to rebuilding memory usage. So judging from that for 2x2 TB disk per pi you could've gone with 4 GB ram pi 4s without impacting performance too much after all.

On another note I read that the mons tend to write quite a bit of logs to their system disk and can wear out sd-cards and flash drives in a matter of months. Do you have any insight regarding the volume of logs your mons write?

  Are you sure? yes | no