After some probing around the flex cables going to the cartridge carriage it was clear which of the pins are for the optical sensor.

Pins 15+16 on the bottom flex (look at the photo of the main board) are the optical sensor pins.

Output is the quadrature signal with the frequency of ~750Hz.

The end of the encoder strip seems to be sensed as the end of one the quadrature signals, with the next one ending ~10 cycles later.