It's been a few months since I've touched this project. Primarilly that's due to the difficulty of the previous board design (the board had way too many parts and just didn't suit the goal very well overall). After some thinking and a lot of digikey browsing I think I've come up with a decent way to handle the project.
The primary difficulty of this project is the requirement for a large amount of IO (I'm still sticking with the ATMega2560 for this) and the overall current requirement. The Atmega2560 has a maximum output current of 50mA which is a significant problem since each row of the matrix has 16 LEDs drawing (ballpark) 10mA each. To handle the LED driving I've decided to use the 74ABT245B on the highside (controlling the RGB anodes) and some simple n-channel mosfets on the low side. The overall goal being to reduce the total number of components.
Here's the current schematic, the design is greatly simplified since each IO port on the Atmega has its own 74ABT245B buffer, which then connects to the header to drive the LED's directly. The low side switch block consists entirely of cheap n-channel mosfets to get used in the multiplexing. Finally this design also has a micro-sd card available, this should allow for much more complex animations since everything doesn't need to be stored directly on the flash memory of the micro-controller.
I'm on break at the moment, with the semester over I hope to push ahead on a few projects while I still have the time. See you in the next log.