Decided to go over all the grounding on the circuit & separate any more digital grounds from the analog grounds. Lions were previously good at separating motor ground & speaker ground from signal ground, but it seems there was never a notion of separating the signal ground into analog & digital grounds. After much effort separating the 5V into analog & digital sections, the ground was assumed to be an infinite pool of electrons since it wasn't moving anything.
Past ADCs were used for gyros, battery voltage, & photodiodes. It was good enough to ground them to the digital ground. The luck was over with 24 bit audio. Even sharing 2cm of a wire with digital ground introduced noise. All hifi gear has separate digital & analog grounds. Despite flood filled ground planes, much effort is spent separating grounds, making sure the ground paths don't have any bottlenecks.
There were a few places where a digital component had multiple grounds & shared an analog ground path on 1. The STM32 had 1 more digital ground sharing an analog ground. With these all moved to the star, the meter dropped right near -90. The final problem was the wavy DC offset. It was too low in frequency to ruin a datasheet, but it would greatly limit the amount of possible gain in software without clipping.
Suspected the op-amp ground was drifting away from the ADC ground, so put a 4.7uF in the audio wire. It greatly reduced the DC drift. The ADC datasheet says it already has a highpass filter in its audio input, but it's rated at a 1Hz cutoff.
Anyways, the audio editor has an ancient tool for measuring sound level. It showed the ADC having -85dB noise peaks & -99dB RMS noise. 13.8 bits & 16 bits based on the ENOB equation. To be sure, 2 channels are averaged & only sampled at 48khz to get those figures.
The formal calculation isn't as warm & fuzzy as counting the pixels in the waveform. If the noise spans 12 pixels in the 1/262144 waveform, just divide 262144/12 & convert to binary to get the number of bits it can resolve. It can resolve 14 bits + some fraction of a 15th bit. The problem is the ENOB equation takes away 1.76bits for a quantization handicap.