Close
0%
0%

Gesture/Pattern Recognition Without Camera : TOF !

Grumpy Hedgehog is a connected device that allows you to read and interact with movements. Its LCD screen displays the Hedgehog's expression

Similar projects worth following
We can easily create our own movements and patterns to transmit actions through several communication channels (USB HID, Wifi, IR...) or simply use its relay and servomotor

1.1 What problem are we going to solve?

Today we are used to have physical contact with the objects that surround us. This applies to switches, touchscreens, remote controls... The current crisis forces us to question these habits and to enter a new digital era where physical contact with certain objects must be reduced to a minimum.

While speech recognition allows a complete approach for transmitting information, it is still quite complex for machines and humans to grasp. A child is usually able to understand and imitate many signs long before his or her first words.

Why not use signs to create simple interactions that can convey more information than ON/OFF?

Once you send a movement or a pattern, GrumpyHedgehog can :

  • Control a computer by sending keyboard command (movie pause, volume...)
  • Track the number of person in a shop on your smartphone
  • Safely unlock a computer
  • Send encrypted informations to a server
  • Control light or any power components
  • Send IR command to your TV
  • Move a servo motor

1.2 Key Elements

In addition to offering a modular solution with its different interfaces Grumpy Hedgehog is particularly distinguished in 4 aspects:

  • It uses a VL53L1X Time Of Flight Sensor allowing complex motion recognition with 16x16 SPAD sensing array, it can run on most micro-controllers such as SAMD21 (MKR Wifi 1010) and Atmega328p (NANO/UNO board).
  • The second aspect is the management of the LCD screen which is optimised to store in FLASH and display a moving image. The first tests on my Arduino UNO are promising!
  • The third differentiating aspect is its design. The challenge here is to create a decorative object that you will gladly leave on display in your living room or office.
  • It uses the ATECC508 secure element to create a strong security layer based on Public/Private Key to authenticates node IDs, ensures the integrity of messages, and supports key agreement to create session keys for message encryption. (This part isn't fully finished)

So you will understand that the GrumpyHedgehog with his thirst for learning, his gestural communication, his cute look and his grumpy character will quickly become your new companion.

  • 1 × Arduino MKR Wifi 1010
  • 1 × VL53L1X ST TOF
  • 1 × TFT LCD 1.44
  • 1 × Single Relay
  • 1 × SB90 Micro servo Motor RF, RFID Hardware / Antennas

  • 1
    How it Works ?

    By default GH detects three movements: Left, Right and Up. The left and right movements allow you to send commands depending on the actual mode:

    To switch from one mode to another, simply leave your hand within 15cm of the sensors for 3 seconds. You don't have to use all the mode, just comment the modes you are not using on the top of Arduino Program.

    In addition to the different modes, you can always use Pattern Recognition with the upward movement the upward. GH will detect on a 4x4 pixel grid the shape of the object in front of it. The Arduino program allows to easily add Pattern and match it to an action.

    To add security, a Pattern and a Keyboard Password can be saved in the secure element ATECC508. With the right configuration, it will be impossible to recover the Password unless the Pattern match. See the ATECC508 section for more informations. (This part isn't finished yet)

  • 2
    Time Of Flight Sensor and Algorithm

    The use of VL53L1X TOF sensor is really interesting. But what makes it different from many competing proximity sensing technologies? (Ultrasonic, Sharp IR angle, LIDAR...)

    Beyond the fact that it allows a detection up to 4 meters, a 50Hz refresh rate and a good immunity to light and surfaces reflexion. What I was particularly interested in is called SPAD Array (Single Photon Avalanche Diode) and ROI (Region Of Interrest).

    This is because VL53L1X not only detects proximity, it can also target this information to a detection field. The SPADs thus form a 16x16 pixel grid like this :

    There is however some restrictions, the detection area cannot be less than 4x4 pixel. By using a reading of each area without overlapping, one can therefore have a reading of 4 pixels by 4 pixels, i.e. 16 detection zones. The other restriction is speed. Using 16 pixels at 50Hz per reading is barely 3Hz for a full reading. You can find an Arduino/Processing Example on Git repo.

    The mode reading Pattern read the 16 ROI and print everything on the LCD.

    After several tests, the configuration that seems optimal to detect 3 movements (left, right, up) is with 3 ROI. As soon as a SPAD detects an activity, it trigger a detection_flag to 1. The last SPAD to detect something while this detection_flag is active considers it as a movement in this direction.

    In theory we could also do a DOWN motion detection with this pattern. But my algorithms are not yet robust enough and the refresh rate in relation to the motion doesn't make it easier.

    To further optimise, I improved the detection by reading all the SPAD at 50Hz. When it trigger something, then I set a detection_flag to 1 and then I start reading my 3 ROI at 17Hz.

    There is a lot of testing to be done to optimise ROI for movements detection. If you have any advice, please comment or send me a message.

  • 3
    LCD, emotions and optimisation

    One of the challenges on this project is to run on both SAMD21 and Atmega328p. With some modules like TOF or relay, this won't be too much of a problem, the LCD is here a real problem for 2 reasons:

    Problem 1 : Memory

    With its 32kb of Flash, the Atmega328p can't store a lot of information onboard. Let's imagine that we want to store an image encoded on a RGB565 (2 bytes) of 128x128 pixels. We have : 128x128x2 = 32768 bytes. A single image on a small screen would already completely overload the memory!

    Problem 2 : Refresh Rate

    To refresh this screen of 16384 pixels it will be necessary to pay also attention to speed. The Atmega328p has only one SPI. If you want to use, for example, an SD card to overcome the memory problem. SD and LCD will be on the same bus. So you will have to do the same on the SPI: get the image from the SD and send it to the LCD...

    I found someone who have a solution to this problem HERE, but it implies a RAW storage of the images, which can be quite difficult to stay flexible.

    Implementation

    To start with, I imagined how to represent my hedgehog with simple shapes that I could generate with the Arduino GFX library. Here, my screen displays the eyes (2 circles) and the nose (1 circle + 3 mini circles) of my hedgehog. I can thus simply use GFX without having to load an image in memory. In addition, I can draw the circles without reloading the whole screen, which allows me to gain a lot of speed.

    In order to validate the shapes and movements, I started by a simulation on Keynote

    On the LCD, to implement a movement simply, I redraw the previous shape in black and then I redraw it with color to its new position.

    On the Arduino MKR 1010, I can take advantage of the speed and the extra memory to make my animations more fluid and complex.

    UNO = 16MHz ---------- MRK 1010 = 48MHz

    UNO = 8 bit instruction ---------- MKR 1010 = 32 bit instruction

    UNO = 32Kbit Flash ---------- MKR 1010 = 256Kbit Flash

View all 10 instructions

Enjoy this project?

Share

Discussions

jean.perardel wrote 07/20/2021 at 13:05 point

Thanks a lot :)

  Are you sure? yes | no

Dan Maloney wrote 06/01/2021 at 17:04 point

That's a pretty clever idea. And the hedgehog case is kind of adorable. Nice stuff!

  Are you sure? yes | no

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates