I came up with a strategy to mount the rod onto the prism.
A very viscous putty is placed between the rod and the prism which becomes very hard and green.
The rod hole tolerance is low and the prism rotates over its glass substrate.
You might argue that some of the putty sticks out and this affects the rotation. I cut a small groove in the final version.
Friction seems low,
I will need to check measurements for the cross scan error.
Most interesting would be to compare distorting coming from the prism facets with those coming from the bearing.
In the figure below you see a quartz prism, a sinter bronze sliding bearing and green putty