I use a sliding bearing. This keeps costs low and ensures the prism is kept in a given plane. A challenge is that the prism falls if this sliding bearing is flipped 180 degrees.  This is fixed by adding a ferrite sheet on the back of my PCB.

Materials used flexible ferrite sheet MHLL5040-200 Laird, Magnets MKSA-8x5-ZW-N45 i.e. 8x5 mm, neodium 45 with a black epoxy coating.
I think that ferrite instead of neodium is possible if the whole contraption was made better.  Ferrite is typically 2-7 times less strong. If the distance would be decreased a bit, which is still possible, ferrite should work.