Project description

Stereo camera aka binocular camera shield for Arduino portenta H7. Shield is build on pair of MT9V034 monochrome imaging sensors with global shutter and high dynamic range (HDR) operation. Image processing is performed by Gowin GW2AR FPGA with embedded SDRAM.

Global shutter allow to capture images not affected by motion blur. All this make camera module perfect for professional machine vision applications. Gowin GW2AR FPGA have enough resources not only for simple image stitching, but also for comlex preprocessing.

RISC-V soft-cpu inside FPGA is used for communication with image sensors and Arduino Portenta board.

Demonstrations:

Blob detection based distance measurement

Here is simplified example of distance measurement based on blob detection algorithm, implemented in OpenMV framework:

 1.mp4 

TensorFlow object detection and distance measurement

The principle is to detect a human figure using a neural network on the left side of the image, search for the corresponding object on the right side of the image using correlation, and calculate the conditional distance to the corresponding objects

for i, detection_list in enumerate(net.detect(img, roi=NET_ROI,thresholds=[(math.ceil(min_confidence * 255), 255)]) ):
    if i == 0:
        continue  # background class
    if len(detection_list) == 0:
           continue  # no detections for this class?
    ...
 
   img_template = img.copy(1.0, 1.0, ([x-10, y-10, w+20, h+20]), copy=True) 
   r = img.find_template(img_template, 0.7, step=4 , roi=[WINDOW_FRAME_WIDTH, 0, WINDOW_FRAME_WIDTH, WINDOW_FRAME_HEIGH])

The size of the doubled frame in this test is 960x240. The object is searched for on the center of left half-frame, inside 240x240 pixel area. When three objects were found on the right frame and, accordingly, three correlation calculations, the processing speed dropped to 4 frames per second.

Scene view :

Output :

Frame difference mode

The output frame is obtained by subtracting the pixels of the current frame from the previous one.

Morphological Transformations

No filtering:

Both filters switched in erosion mode to visually "separate" nearby objects:

Gamma correction

No correction:

Gamma = 1/2.2:

Luma inversion:

Frame difference, morphological transformations and gamma correction performed on FPGA and not affect Arduino resources and FPS.

Features

• MT9V034 image sensors (monochrome, global shutter, HDR)
• Gowin GW2AR FPGA with RISC-V soft-cpu core inside
• SD card
• ISP with frame-difference, gamma correction and morphological transformations are implemented on FPGA

Operation

Shield is controlled by I2C interface. 16-bitwidth address and 16-bitwidth data is using. There is 4 address spaces inside:

• 0 - internal FPGA processor registers. Using for control common camera functions
• 1 - direct access to MT1 image sensor i2c registers
• 2 - direct access to MT2 image sensor i2c registers
• 3 - simultaneously write to both MT1 and MT2 sensors (not tested yet)

Number of address space passed in high 8 bits of i2c address, number of register to write is passed in low 8 bits of address. For example, to write data to MT2 register 0x35(analog gain) user should provide i2c address as 0x0235.

Internal FPGA processor registers :

• Writing to CAM_FRAME_WIDTH or CAM_FRAME_HEIGHT updates internal registers. This action also configures the necessary registers...