Thanks to Ted Yapo for responding in the comments to a previous log. He solved the mystery.

He said that the LM4871 LPF feedback cap (C20) was the problem. That got me to recalculate LPF rolloff and it was 24 Hz. Way, way too low. I had wanted something more like 2.4 kHz.

First I removed the cap entirely and it really did jump right up.

I had replaced the 20kΩ feedback resistor (R9) with 56kΩ, so the LM4871 gain is now officially 7 dB instead of 3 dB. I added back in 1000pF just to give it some pro-forma rolloff, but I can't really hear a difference.

When the next boards come, I can experiment a bit. I can replace the pre-amp's feedback resistor with 0Ω and leave off the rest of the preamp components and it will be as if nothing is there. Or I can populate the pre-amp and see which way is better.

As for the speakers, the little tiny ones I originally bought won't cut it. Trying to get enough gain for the voice output to sound decent makes it clip badly. The larger speakers, unsurprisingly, sound a lot better. If there's a downside, it's that the case will wind up having to be much larger (still not at all sure what I'm going to do there).

It turns out that a new board rev was going to be necessary anyway because of the previously noted errors. In addition, I swapped the poles of the two gain trimmers so that the directionality correctly matches ordinary expectations (CW for higher gain).