Well, I did some more investigation into the ISP issues, and while I made some progress I still don't know exactly what is happening.

I ended up soldering a new controller board with just the AVR, crystal, and 8pF caps, and programmed the bootloader + fuses successfully. I then soldered the rest of the components, and tried ISP again, and it didn't work. Something in the circuit is interfering with the ISP programming.

The confusing part to me is two-fold:

1. The circuit is very simple (the ISP pins are shared with the encoder headers, and the only components are a pullup resistor, a small capacitor, and a TVS diode; all programming testing was done with the encoders not installed).
2. Obviously the capacitor / pullup could have something to do with this issue, so the first time I encountered it, I removed them: it still didn't work.

Now, to be fair, I had some clock issues the first time I saw this problem, so it may have been two issues coinciding that produced the confusing results. Regardless, I am a bit confused still. I will have to look into options for fixing this in future revisions (anything from a simple resistor to separate the ISP from the encoder, to perhaps a DIP switch to switch between the two).