Silicon sensors are sensitive up to more than 1100nm so in order to limit the effect IR light has on photos cameras have an IR filter that cuts off around 650nm. Since I'm using IR LEDs in the target this was a problem so I removed that filter from the camera. This worked fine but too much visible light could confuse the camera. I found on Aliexpress IR bandpass filters for 940nm, just right for the LEDs I'm using.

Superglued the filter on the Raspberry Pi camera lens holder and done! Luckily it's thin enough (0.55mm) to not interfere with the 6mm lens.