It was the power supply all allong! Never ignore a first hunch.
More testing on the connection revealed packet losses when the roomba was docked. I wasn't expecting I could simply ping the ESP8266, but apparently you can. Whenever the roomba docked, ping times would increase and packets would be lost.
This would ONLY happen when the roomba was docked and would be resolved immediately when it's removed from the dock.
So what's the difference between being docked and undocked? Power! It will get incoming serial communication in both scenario's. And all previous testing pointed to something other than software. I even put a hard ESP.restart(); in the onMqttDisconnect callback without proper result.
And providing the ESP with a antenna did help a bit. So it's time for something more substantial. Whenever in trouble or in doubt... add a big ol' C.
My hypothesis is that the raw battery voltage supplied by the roomba will fluctuate quite a bit when docked and hooked up to the charging circuitry. The buck step-down power converter needs some time to adjust and stabilise. On the other hand we have the ESP8266 with it's WiFi radio which is very sensitive to unstable power.
And one of the antennae that arrived in the post today.
The black stick-on with quite a long wire so I have placement options:
And that's it! Boom! Stable as lead. I could immediately see better signal strength and bigger throughput on my router:
And monitoring the connection for a few hours made me quite confident.
It's been running for a full 24 hours now, with 1 MQTT-reconnect. Over The Air updates also works stable now.
I hereby solemnly declare this problem resolved!
Guess I have to start on the hard bit and the math now ey?