I will add a presentation to clarify,

I reduced the pole length to 10 mm. I now end up with a cross scan error of 180 microns (60 pixels at 3 microns wide). The pattern is very constant over time and seems to be due to the prism.
I measured seven prisms and arrived at an error ranging from 35 pixels to 45 pixels.
The prism in this image had an error of 41 pixels. As a result, I think 19 pixels are due to "vibrations".

The camera distance to the prism is 36 mm (prism facet parallel to CCD chip and distance between prism and CCD chip). Note that this is very similar to my earlier measurement with the Ricoh motor (PCB motors suited for laser scanning). Here I arrived at 160 microns.