Trying a different approach, I took a whole bunch of raw sensor readings whilst holding/moving/rotating different coloured bricks in front of the sensor, at a variety of distances from 5mm to about 20mm. Then imported the readings into Matlab so that I could scatterplot them in 3D (rgb-space) and observed that all the points relating to a certain brick formed a straight line. One end tends towards the origin (black, i.e. poor illumination) but the other end does not tend towards (1,1,1) i.e. white. This means that the shade or hue⁽¹⁾ of each brick does not change with distance or brightness of illumination. We humans might see a white, specular flash from a shiny face, but the sensor just sees brighter red (or whatever colour). I'm sure this effect is obvious and well-known, just not to me.

I think it means, as I suspected, that I can classify the bricks' colours by normalising the rgb values to account for brightness of illumination, resulting simply in hue and ignoring brightness. The down-side is that black, grey and white bricks all get classified the same, and cannot be separated. Maybe that's not a problem, at least for now.

It's also surprising how much 'desaturation' there is, e.g. the green and blue channels show significant responses when shown a 'red' brick. Perhaps it's correct, or perhaps I'm getting light from the illuminator LED scattering directly onto the sensor by accident.

^{(1) Not really 'hue', since it includes all levels of saturation including grey, so really it's a mix of hue and saturation.}