I used an ATMega8 as the brains for this project. Typically with a matrix like this people use shift registers or other ICs to drive the display, but I found there was no need since the mega8 has just enough I/O pins to get the job done.
The mega8 drives the LED anodes directly with one I/O port. Since each pin only drives one LED at a time, it's OK to drive the LEDs directly from the I/O pins. For the common cathodes of the LEDs, I used 2N2222 transistors. Since there may be up to 8 LEDs worth of current going through each common cathode, a transistor must be used instead of directly connecting to an I/O pin.
With 8 anodes and 8 cathodes being driven by I/O pins, that left exactly 4 pins for me to use for inputs, which I wired up to 4 pushbuttons.