Based on a tip from @amorim317, I found this article on via stitching that helped to explain why my original numbers were off. Turns out I was not factoring in my FR4 substrate into the equation. The original equation that I found only looked at the wavelength of the RF frequency in a vacuum, not its propagation through the FR4.
Based on the info in the above article, my via fence spacing in meters should be:
|= Wavelength in a dielectric medium|
|= Speed of light (299,792,458 meters/second)|
|= Frequency (In this case 2,450,000,000 cycles/second)|
|= Permittivity or dielectric constant of material (In this case 3.66)|
The frequency I'm concerned with is 2.45 GHz (2,450,000,000 cycles/sec). The dielectric constant of the 4 layer FR4 that OSH Park uses is 3.66 @ 1 GHz. If you look at the data sheet, it says 3.67 @ 2 GHz. I'm just going to use 3.66 because the difference is small, I'm lazy, and it's what I've already used to calculate my 50 Ohm transmission line.
Give the above, for my RF via fence we end up with a max distance of:
S(via) = 299,792,458 m/s / (20 * 2,450,000,000 * 1.913) = 0.003 meters * 1,000 = 3 millimeters
The article referenced above also recommends containing RF at board edges by using via stitching at wavelength/8 instead of wavelength/20. This means my edge via stitching should be separated by a max distance of:
S(via) = 299,792,458 m/s / (8 * 2,450,000,000 * 1.913) = 0.0079 meters * 1,000 = 7.9 millimeters
All of the above is very helpful in understanding the calculations involved in via stitching/fencing. If you're still reading, here's your reward: An easy to use calculator for TEM wavelength calculations! Take the result of the calculator divided by 20 to get RF trace via fencing max distance. Take it divided by 8 to get board edge via stitching max distance.
All of the above is great for validating your via spacing decisions. However, it doesn't factor any error or material differences or variability into the equation. I'm going to err on the side of extreme caution on my RF trace fence and use 1 mm spacing and cut down my board edge stitching to 5 mm.