After spent some time unsuccessfully looking for precise information about the Mouse protocol used by MSX computers I got a subrom binary to disassemble and take a look.
Mouse reading is performed by NEWPAD subroutine at address 0x1ad but the reading effectively occurs after addres 0x3509:
The Mouse routine reads 6 nibbles (0..5) in sequence, cadenced by the (toggling of) pin 8. Small delay loops are inserted between readings. Both the Mouse and the Trackball reset their internal counters after one edge of the Pulse signal.
The 5h nibble read might be the High order nibble for the Mouse or the delta Y value for a trackball.
Given the typical DPI resolution of the MSX mouse no displacement above 16 steps should be observed, then the expected value for the 5th nibble should be 0b0000 for positive displacement or 0b1111 for negative displacement.
The trackball is a much less resolution device and uses solely 4 bits, nevertheless as the mouse only a small amount of displacement should be observed (zero or one count up/down).
One of the curious aspects of the trackball is that it uses an inverted sign bit, which means "0" displacement is represented by 0b1000, and that is the key for the NEWPAD routine to differentiate a trackball from a mouse.
; Nibble 5: Trackball: Delta Y, inverted Sign bit ; (7) 0111 (-1) minus one ; (8) 1000 ( 0) zero ; (9) 1001 (+1) one plus ; ; Mouse: Delta X high nibble, normal sign bit ; ( 0) 0000 ( 0) Zero ; (15) 1111 (-1) Minus one xor 08h ; Turn values 7, 8, 9 into 15, 0 and 1 sub 02h ; Turn 15, 0 and 1 into 13, 14 and 15 cp 0dh ; Trackball should greater or equal 13 jr c,returnMouse ; values less than 13 ; returnTrackBall: ....
The brief positive pulse issued after the 6th nibble is read has the purpose of reset the internal Y counter of a mouse, as registers should be read in pairs (4 nibbles).