Gesture Recognition Wearable

Using Machine Learning to simplify Twitch Chat interface for my tvhead and OBS Scene Control

Similar projects worth following
All files are in the Github repo.

Atltvhead Gesture Recognition Bracer - A Tensorflow gesture detector for the atltvhead project and an exploration into Data Science

This repository is my spin on Jennifer Wang's and Google Tensorflow's magic wand project.

Using accelerometer data and a CNN model, detect gestures during live streams. The gesture data is sent through Twitch chat to my tvhead project, changing what animations are available for the chat to control.

Machine learning model interplays with Twitch chat. Simplifying tvhead chat API.
**EDIT** Gesture Recognition now control OBS scene settings for live editing during streaming!

I run an interactive live stream. I wear an old tv (with working led display) like a helmet and backpack with a display. Twitch chat controls what's displayed on the television screen and the backpack screen through chat commands. Together Twitch chat and I go through the city of Atlanta, Ga spreading cheer. 

As time has gone on, I have over 20 channel commands for the tv display. Remembering and even copy pasting all has become complicated and tedious. So it's time to simplify my interface to the tvhead.

 What are my resources?
During the live stream, I am on rollerblades, my right hand is holding the camera, my left hand has a high five detecting glove I've built from a time-of-flight sensor and esp32, my backpack has a raspberry pi 4, and a website with buttons that post commands in twitch chat. 

What to simplify?
I'd like to simplify the channel commands and gamify it a bit more.

 What resources to use?
I am going to change my high five gloves, removing the time-of-flight lidar sensor, and feed the raspberry pi with acceleration and gyroscope data. So that the Pi can inference a gesture performed from the arm data.

Here is my game plan!

  • 1 × ESP32 Thing Plus by Sparkfun
  • 1 × LSM6DSOX + LIS3MDL - Precision 9 DoF IMU - STEMMA QT / Qwiic by Adafruit
  • 1 × Push Button
  • 1 × Raspberry Pi 4

  • OBS Gesture Control

    atltvhead09/01/2020 at 17:48 0 comments

    This is more of an add-on type of feature. For live streaming, I use OBS with the websockets plugin and lorianboard as a scene controller. Lorianboard is pretty awesome including features that allow you to trigger multiple sounds, scenes, and sub-menu controls. At its most complicated, it can even create a simple state machine control over obs based on twitch chat, channel points, and other non twitch related triggers.

    Since my gestures already trigger a chat message: !wave_mode !random_motion_mode !speed_mode !fist_pump_mode. It is rather easy to trigger a button in lorianboard. While editing a deck, choose a button and select Edit Twitch Triggers.

    Click the plus and add chat command, type in the command and the role associated. You are done. With triggering the effect.

    Coding the effect itself is a little more complicated.

    In OBS, create a scene for you to change to. Add in whatever sources you'd like.

    For me, I recreated a green screen effect using machine learning and the yolact algorithm. So I included the source window generated from python and yolact. I also rewrote part of yolact to display only the objects detected or the inverse. I also provided a glitchy vhs background. Back In Lorianboard I created a button with a "stack" or queue for effects. It looks like the following. An if statement with variable to the corresponding effect, a bash script executed by lorianboard, and scene changing commands based on timers.

    The bash script runs my yolact python file with line arguments.

    python --trained_model=weights/yolact_resnet50_54_800000.pth --score_threshold=0.15 --top_k=15 --video_multiframe=20 --greenscreen=True --video="rtmp://"

    Notice my video is actually an rtmp video stream. I have an nginx-rtmp video server running in a docker container, which my phone broadcasts to using Larix broadcaster.

    But Anyways the EFFECTS! Let's check out what I have created so far.

    Pretty good for AI.

    Inspired by Squarepusher's Terminal Slam video and some others work to recreate it. I tried my own varient.

  • Smaller Time Window

    atltvhead08/11/2020 at 19:09 0 comments

    I shrank the time window in which a gesture can be preformed from 3 to about 1.4-1.5 seconds. It is working well and feels much more fluid than the original. There is still a little room for optimization on gesture timing, but I am happy with its current state.

    This time when comparing models, the 2nd and 3rd cnn were better trained than the first. However, the larger, 2nd model preformed better than smaller 3rd one. I proceeded with using cnn_model2_half.h5

    Some improvements need to be made when using pandas however. Since the dataframes memory is not released each gesture sampling will consume more and more memory. Once solution is to use the multi processing library to run the gesture sampling as in a separate process. When a processes completes it returns its memory to the operating system. This should allow for true continuous gesture reads without filling up the memory on the raspberry pi.

    Still working towards deployment on the esp32, but just making sure there is plenty of room for other bits of code and processing.

    The files have been update are reflected in the GitHub.

  • New Control Scheme

    atltvhead07/24/2020 at 19:58 0 comments

    Using both Gesture modes, and channel commands the following animations are broken up in equation style.
    Gesture A + command "1" = Animation A

View all 3 project logs

  • 1
    Choosing a Sensor

    To briefly explain my sensor choices.

    Lidar: I've used "lidar" Time of flight sensors to detect high fives, in the previous version of my wearable. However, it cannot detect arm gestures without complex mounting a ton of them all over one arm.

    Stain Sensor: Using the change in resistance stretch rubbers I can get a sense of what muscles I'm actuating or general shape of my arm. However, they are easily damaged and wear with use.

    Muscle Sensors: Something like an MYO armband can determine hand gestures, but require a lot of processing overhead for my use case. They are also quite expensive.

    IMU: Acceleration and gyroscope sensors are cheap and do not wear out over time. However, determining a gesture from the data output of the sensor requires a lot of manual thresholding and timing to determine anything useful. Luckily machine learning can determine relationships in the data and even can be implemented on a microcontroller with tflite and TinyML.

    I chose to go forward with an IMU sensor and Machine Learning. My sensor is the LSM6DSOX from ST on an Adafruit Qwiic board.

  • 2
    Arduino Setup

    In the Arduino_Sketch folder of this projects Github repo is the AGRB-Traning-Data-Capture.ino. A script to pipe acceleration and gyroscope data from an Adafruit LSM6DSOX 6 dof IMU out of the USB serial port. An ESP32 Thingplus by SparkFun is the board I've chosen to use due to the Qwiic connector support between this board and the Adafruit IMU. A push-button is connected between ground and pin 33 with the internal pull-up resistor on. Eventually, I plan to deploy a tflite model on the esp32, so I've included a battery.

    Every time the push-button is pressed, the esp32 sends out 3 seconds of acceleration and gyroscope data over usb.

  • 3
    Build A Housing

    I used fusion 360 to model my arm and and a square case like housing. It took some revisions to get it to fit perfect and is actually rather comfortable even on a hot day in Hotlanta. All the cad files are in the linked github on this project.

View all 10 instructions

Enjoy this project?



Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates