I hacked together a charge pump on a little SOT23 breakout board and wired that into the prototype. It sounds really good. I certainly can’t hear any artifacts from the charge pump in the audio. And the chip I used had quite a low oscillation frequency. I want to use the MAX1721 in the final design so that hopefully the switching frequency will be too high to impact the audio.
I tested with my AirPods Max so as to try to maximize my ability to pick up any noise or distortion, but didn’t hear anything untoward. That’s not a great test, of course, but it’s a start.
I also tested against the other two TOSLINK DACs I have handy - a $10 one from Amazon and the one built with the CS4334. The other two sound distinctly flat to me by comparison. I am not sure if it’s a placebo effect or not (and I’m not sure I would be able to by definition), but the more expensive DAC does seem to make a difference.