I ended up modding the mains power supply to produce ~5.5V instead of ~5.0V so that downstream of the Schottky the supply rail was about 5V. Now, when the main supply is pulled, the Pi no longer immediately restarts. Instead, it appears to initiate the shutdown as intended. There are still some problems though:
- the 'debounce' part of the dtoverlay doesn't seem to be working, as it begins shutting down immediately rather than after a couple of seconds
- the GPIO pin that is meant to drive low after shutting down doesn't appear to do anything, so the power never gets turned off
- in normal state, the dc-dc converter isn't quite fully off (the EN line is at about 1.3V) and seems to be sucking ~1mA instead of the ~0.2uA I get when EN is pulled to 0V properly. That will flatten the batteries in no time... [edit: EN needs to be less than about 0.9V for the standby current to be around 0.2uA]
Still more to be done then!