Exploring Alternative Computer Input Devices: IMU Joysticks

A project log for 2020 HDP Dream Team: UCPLA

The 2020 HDP Dream Teams are participating in a two month engineering sprint to address their nonprofit partner. Follow their journey here.

Kelvin ChowKelvin Chow 07/06/2020 at 02:052 Comments

For this project, we want to first brainstorm and explore different input devices that can be used as universal remote controllers.  For individuals with cerebral palsy, mechanical joysticks are commonly used for controlling motorized wheelchairs.  For the first proof of concept, I wanted to explore the use of joysticks as an input controller.  Individuals with CP have difficulty using a mouse and keyboard, so I tried to use joysticks to control my computer.

Since I didn't have a joystick, I used inertial measurement unit (IMU) sensors to act as a "floating" joysticks.  I had a few microcontrollers/IMU sensors lying around, so I wanted to do a simple demo using these IMUs to draw using the computer program, Paint.  

Below is a high-level overview of the architectrue.  Two M5StickC watches were used as the inputs.  An ESP32-WROOM dev board  was used as the primary communication hub, receiving IMU data from the M5StickC watches through the ESP-NOW protocol, which was then processed and translated into mouse and keyboard data and relayed to control the computer via Bluetooth. 

Most of these components were used out of the box with no soldering/wiring (except for adding an input button to disable the joystick controls).  Below are the 3 controllers. 

To briefly summarize how this works, Joystick 1 controls the mouse movement functionality.  I only used the accelerometer data to detect the tilt angle in two directions (Forward-Backward and Left-Right).  Similarly for Joystick 2, I hotkeyed a few mouse/keyboard commands based on tilting the IMU in one direction.  Below is an image showing different angles of Joystick 2 mapped onto different mouse/keyboard commands. 

With this setup, I attempted to draw a few images using the Paint program on my laptop. The first one was the Supplyframe logo operating both joysticks with my hands.

The next thing I tried was to try to control Joystick 1 (mouse movement)  with my head.  I wore a hat and placed the joystick in the hat.  

This was much more difficult to control and after about 10 minutes, my neck was sore.  Part of this exercise was helpful to put myself in the shoes of others and experience using input controllers without hands.  


Tom Hartley wrote 07/06/2020 at 08:34 point

Hi there! Interesting work - I thought you might like to check out this project that some friends of mine worked on:

The short version is that they designed a system where you can attach two sensors to a person, and the relative movement between the sensors acts as the control signal. It was designed for amputees, but you may get some inspiration! Good luck!

  Are you sure? yes | no

Kelvin Chow wrote 07/07/2020 at 00:36 point

Really cool, thanks for sharing!

  Are you sure? yes | no