I have the chance for another decent chunk of development time for the cube today. The big changes where I upped the internal clock in the MCU from 1MHz to 8MHz and did a lot of fiddling with interrupt timing. In the end I have the whole cube running at an acceptable brightness at close to 60Hz refresh rate with individual 4-bit dimming for each LED. From the scope capture below you can see that the whole cube cycles every 17.5ms which represents a 57Hz refresh rate.
Zooming into a single pulse, the LEDs are on for 269µs. This means that I have a duty cycle of (64*269µs)/17.5ms or 98%. As I suspected from the zoomed out scope capture in the last post this is pretty high. It means that I'm doing all of my computation in between changing LED states which is exactly what I wanted.
Because of the increased system clock I had a lot more program cycles which allowed me to implement 4-bit dimming for each LED. To get any smoother I would have to pass off animation computation off to another device and/or reduce the LED count, but for now 16 steps of intensity is plenty. The dimming test can be seen below. The camera doesn't pick up the dimming curve very well, but you get the idea. It looks much better in person.