Unfortunately my latches are of the ALS family that can sink quite well (24 mA) but only have an official very measly 2.6 mA source capability so I realized that it might be a good idea to see if the display is bright enough when 1:6 multiplexing it using two 74ALS573 latches.
One of the latches will be connected to the anodes (segments) connected in parallel on all the displays, and the other latch have 6 of its outputs connected to the common cathodes of the displays.
It adds up quite well, the first latch might have to source current to up to eight segments @ 2.6 mA = 20.8 mA and then the second then will have to sink it and should be able to do that with its max 24 mA ports.
Of course these current limits are when the chip still can maintain VOH and VOL within their specs. If you don't care about the actual voltages then the chip can do better but not much due to internal resistances.
But the internal resistances in the drivers can work to my advantage. It allows me to hook up the display without any external current limiting resistors. This is *not* proper engineering and should not be done on any real products that other will use, but since this is just an internal hobby project I frankly don't care if I by torturing the drivers (and possibly the display) will shorten their lifespan a lot nor not. For the time being it works and the display is bright enough to be usable in indoor lightning and with 1:6 multiplexing the display at about 100Hz.
To test it I hooked up the displays/buttons combo and the driver IC bus with aa bunch of alligator clips and hooked some of the inputs of the drivers to an Arduino with even more alligator clips.