Put more capacitance on power line to ESP module, on both inputs and outputs of regulator. ESP takes a lot of juice, what more, in impulse manner. I'd use at least 100uF or 220uF or so, to be on safe side.

I've noticed that with the second board I used without the regulator on, running purely off of the rPI converter. I was under the impression that it would be okay, thought that ajlitt said something alike. But adding caps seems reasonable, do you have any real life numbers on the max current, other than those 170mA from the datasheet? I ran the ESP in this configuration on my rPI zero just fine getting power from USB3, but being battery powered is a whole different story, as it seems. 

Yep, the 150-200mA-ish figure is what I measured with my DMM, though this is average consumption (refresh rate of DMM is usually in order of 10E-1s). The real consumption will look like impulses with much higher value and frequency few orders of magnitude higher - so DMM does not catch it, just displays average value. Relatively slow regulator, PCB tracks resistance and inductance will make voltage drops and probably cause ESP brown-outs - those are hard to catch bugs, often looking like software/configuration problems. Capacitor will help to source current for this impulse demands.

Powering from rpi may work somehow - the onboard regulator is quite beefy to power the Broadcom SoC, but you are sharing the load between SoC and Wifi - that may be quite unpredictable, though my gut feeling tells me it's somehow better than linear regulator with insufficient capacitors.

Anyway, relying on marginal conditions isn't the best idea. Be on safe side, rather than surprised :-)