To increase the friction of the stepper and the connector, I added a 4mm diameter acrylic cylinder extension to the stepper, then wrapped that acrylic cylinder with coarse sandpaper. Coarse sand paper are not easy to bend, therefore, I cut them into small slices and glue them to the cylinder.

Also a good structural glue is quite handy in projects like this. Try to find ones that would cure at room temperature within 72h.