I tested against the omega2+ and the Raspberry pi versions of keychain backup. Both ran for more than 500+ interrupted backups without failing the test. I never saw any filesystem corruption on the USB drive filesystem.

With those kinds of results, I am extremely confident that removing the USB drive without "Safely Removing USB" will not have an adverse effect on the USB drive.