After I posted the previous log, I kept working on trying to figure out the data rate figures, the worrying one being iperf performance when the wESP32 was the server, which was really poor.
Taking a step back, the reason I was doing these tests in the first place was to validate the hardware. Before I go to production, I need to have confidence that the design is sound, and in the case of this test, that there aren't any circuit or layout issues that are making the signal path performance poor, such as interference from the PoE power system or impedance mismatches causing reflections and affecting the signal to noise ratio. Such issues would result in high packet loss, making retransmission necessary, which in turn is observed by poor data rates.
So from the previous tests, it looked like the outgoing signal path had been validated as working well (iperf client), but the incoming signal path was suspect (iperf server). So I started looking at the layout to see if there was anything obviously different between the incoming and outgoing signal paths. Not really, both the TXP/TXN and RXP/RXN were laid out as impedance matched differential pairs. In the picture below, you can see them as the 4 fatter traces under the "Wired ESP32 with PoE" writing:
While I was perusing the LAN8720A phy datasheet to see which pair was suspect, I looked at the "Architectural Overview" block diagram on page 5, and I noticed something I had forgotten: that this chip implements "Auto-MDIX", a feature that automatically switches the TXP/TXN and RXP/RXN pairs as required to work correctly with either straight or crossover Ethernet cables.
This gave me an idea: if I used a crossover cable, the incoming and outgoing data would switch on which differential pairs they were travelling, and the phy would internally switch them back so things would still work. If the problem moved from incoming to outgoing when I did this, then I'd know one of the data pairs was performing poorly. If nothing changed, and the performance on the outgoing data test was still good, I'd know both data pairs were performing well.
After locating a crossover cable, I ran the test, and the results were identical to the ones from the test with the straight cable. This means both data pairs are performing equally well and the data path is sound! Yay! :)
So, the question remained: why was the iperf server test working so poorly? Even though I had pretty much validated the hardware, this still bugged me so I kept trying to see if I could figure out the issue. I decided to play with some of the settings offered in 'make menuconfig'. In the process of doing this, at some point I got the client performance to >80 Mbit/s, but the server performance remained poor. Eventually I did find something that made the TCP test work: the option "CONFIG_EMAC_L2_TO_L3_RX_BUF_MODE".
The docs explain that "when Ethernet MAC doesn’t have any unused buffers left, it will drop incoming packets (flow control may help with this problem, to some extent)". Still, the docs recommend to keep this turned off if unsure, because it involves copying data instead of passing pointers, which decreases performance. Still, the iperf test was apparently running into the problem that the MAC didn't have buffers left, because turning it on made a big difference. Here are the results of the iperf client test with this config:
And here are the results of the server test, which now performs well:
That's when testing with TCP. Oddly, the UDP server test still performs very poorly: I only get around 1 Mbps.
Obviously, this is some other configuration issue. Since TCP was working well, I called it a day.
Overall, I'm very happy with the performance of these first prototypes: the power output beats my expectations and the spec, the attainable data rate is very good and beats my expectations as well. With these prototypes working as well as they do, this design is ready to go to production! Most components are on order already. Stay tuned!