Software - Vision subsystem

The job of the vision subsystem is to keep the sequencer updated as the arrangement of the blocks is changed.

The software continually grabs video frames from the webcam, identifies the block positions and maps them to cells on a 16 by 16 virtual grid. These cells are then converted to an array of notes for loading into the sequencer.

To identify the blocks the "SimpleBlobDetector" class from Emgu.CV is used. As the name suggests this identifies blobs in an image and outputs a list of the blob centroids (the coordinates of the block centres). The blob detector can be configured to only accept blobs in a certain size range, which can be optimised by trial and error for the particular blocks used.

Once the block coordinates are obtained they can be mapped to the nearest grid cells and then to an array of notes, the cell row and column giving the pitch and order of the notes respectively. The note array is then loaded into the sequencer.

This process runs independently of the sequencer so differences in frame rate or blob detection time do not affect the timing of the audio output.

The video sub system also displays the captured images and overlays some markers showing the positions of detected blocks and the current grid column being played.