Over the past month, a lot of work has gone into testing out various communication methods between the master and slave nodes. Basic UDP, mulicast UDP and TCP have all been tried out, with each having various issues. Digging deeper I went back to some of the ESP-IDF examples that test performance and found that even for the basic examples, something is not right with the Sparkfun ESP32 Thing dev boards I'm using.
Apparantly, the ESP32 Thing has a non-standard crystal frequency of 26 MHz, whereas the vast majority of the other ESP32 products have a 40MHz crystal frequency. The ESP-IDF seems to be tested against products with 40 MHz crystals and there appear to be a lot of bugs when the examples are run on the ESP32 Thing. Like getting very slow UDP speeds (0.4 Mbit vs 130 Mbit) and for TCP, the connection disconnects after a few seconds.
I'm going to contact Sparkfun about getting some tech support about this, but I don't really have any expectations that it'll get resolved. In the meantime. I'm going to order a pair of Adafruit Huzzah32 boards and try those out.