So it turns out for some reason on the nrf52 Feather, sampling the ADC prevents the device from going to sleep afterwards! Even if you're only sampling once! I even tried to turn ADC off manually like so:

  NRF_SAADC ->ENABLE = 0;

But no dice, once you sample once, the current increases by about 4mA -- not cool.

Alright so, the temporary fix is to disable all battery / power features. ie. battery left, charge status etc. and then reach out  to the guys at Adafruit and/or Nordic to see if anybody has a fix for this.  

The worst part is since I already cut my traces, I'm not even sure if using the TX/RX pins was preventing the USB chip from sleeping! 

Ok, now some good news. The power consumption without all the other features and at 7ms delay (ie. 8ms debounce) is 2mA (without power button LED) or in other words, using a 3,000mAh LiPo and using the keyboard 12 hours a day straight would last 87.5 days. This made the whole jump from i2C to SPI well worth it.