After the problems that I was having when using an LM317 as the adjustable current regulator, I decided to scrap that approach and go back to basics. I swapped out the current limiting LM317 for a TIP102 (generic NPN darlington transistor). The base of the transistor is driven by a high gain op amp which is comparing the current set poitnt with the current output. Lo and behold, it actually works! I can get all the way down to 20mA (i.e. low enough that I can drive a single low power LED without any extra resistors).

Some other changes include using an op amp in instrumentation amplifier mode rather than differential amplifier mode, and adding some more decoupling capacitors for stability.

The updated schematic is below. Thanks to all of you who have given suggestions on how to improve things!

Cheers