so I started this wiring mess, and I think for doing this I should reassign the pins first for better connecting. So whenever I remember to print this, I'll update and upload the designs. Just looking at it, I could remove the A6 and A7 pins out of that messy area. This setup would leave me with two analog inputs and a serial connection. I also ordered some IC sockets for the displays, 12pins where hard (read: not) to find, 24 where to special for my taste - so I ordered 8pin IC sockets that are chainable.
After some awesome iterations and mouse clicks of other persons (see comments for credit) - we got this result: