Nithin PS
- The robot uses an Arduino Nano RP2040 Connect as motor controller. It is connected to jetson nano over usb. The arduino firmware has common functions to drive motors and leds connected to it. For example, sending "blinkledwhite_1" over serial to the arduino turns on the white led.
- Three leds (white, green and blue) connected to the arduino shows the current state of the program. The white led blinks when the program is ready to accept input. The green led blinks when a green blob is detected in the input and the program tries to recognize the input. The blue led blinks when a blue blob is detected in the input and the program tries to learn the input. When both the green and the blue led blinks it indicates that both green and blue blobs are present in the input.
- A speaker and microphone is connected to jetson nano over usb. A Raspberry pi camera module v2 is connected to csi camera port of the jetson nano. Wireless connectivity nor internet is needed to run the program but is used for setting up jetson nano.
Patterns
The program can handle both green and blue patterns simultaneously. These different combinations of green and blue patterns can occur,
1. Only green patterns are present in the input.
In this case only recognition happens, either in 'text', or 'audio' channel. Channel having higher weightage is selected.
2. Only blue patterns are present.
In this case only learning happens. Since there are no green patterns, all the learning happens in 'audio' channel only. If there are multiple blue patterns, each constituent blue pattern has to be learned seperately in 'audio' channel before learning the pattern as a whole.
3. Input contains a combination of green and blue patterns.
In this case both learning and recognition happens. The blue patterns are learned with the symbol recognized from reading the green patterns. Both recognizing green patterns and learning blue patterns takes place in channel 'text'.
Pre-training
Letters of the English alphabets are pre-trained to provide a vocabulary for learning more complex patterns. The pre-trained letters can be relearned later to update the knowbase with different variations of the pre-trained letters as required.
Motors
The motors are controlled by four keywords in channel 'text'.
1. 'fw' for moving forward.
2. 'bw' for moving backward.
3. 'sc' for spinning in clockwise direction.
4. 'sa' for spinning in anti clockwise direction.
These keywords can be reassigned later to any arbitrary pattern you like by relearning.
Channels
When an image is learned its signatures are stored in channel 'text'. When a recording is made the signatures are stored in channel 'audio'. During recognition, different symbols may be recognized for each channels. The symbol from channel with higher weightage is selected.
Matikas
Roni Bandini
Gintaras Valatka
Nick Lim