I didn't get the milled PCB to work, I triple checked everything but couldn't find a root cause. Technexion is taking a look at my schematic so lets see what they have to say.
In the meantime I just went ahead with a new design as I think it probaly is just a fluke issue that I just haven't noticed yet. I ordered a PCB this time to cast out any issues arising from a missing solder mask.
My main goal is still to only get it to boot but I broke out a bunch more stuff in case I want to test more. I don't have high hopes for HDMI and DSI as I did not order a impedance controlled PCB this time and haven't taken much care to aproximate the impedance with this stackup but it will be interesting to see if it would still work, just to see how much you can abuse these high-speed busses before they completely crap out on you.