I realized today that I'm not using the correct part for the ATtiny85. I have been using an ATtiny85-20PU. This is rated for up to 20mhz clock and a minimum VCC of 2.7v. I'm switching to ATtiny85V-10PU. It's rated to 10mhz, I'm currently running at 8mhz, and down to 1.8v VCC if I drop the frequency down to at least 4mhz. The lower voltage support is needed for the power fail circuit as in my testing it dropped to about 2v before the save was complete.
I guess that means I'll be running experiments as too how low I can take the clock frequency.
A side benefit is that the slower the clock is the less power the ATtiny85 will use while its not in power down mode!