For this project, we want to first brainstorm and explore different input devices that can be used as universal remote controllers.  For individuals with cerebral palsy, mechanical joysticks are commonly used for controlling motorized wheelchairs.  For the first proof of concept, I wanted to explore the use of joysticks as an input controller.  Individuals with CP have difficulty using a mouse and keyboard, so I tried to use joysticks to control my computer.

Since I didn't have a joystick, I used inertial measurement unit (IMU) sensors to act as a "floating" joysticks.  I had a few microcontrollers/IMU sensors lying around, so I wanted to do a simple demo using these IMUs to draw using the computer program, Paint.  

Below is a high-level overview of the architectrue.  Two M5StickC watches were used as the inputs.  An ESP32-WROOM dev board  was used as the primary communication hub, receiving IMU data from the M5StickC watches through the ESP-NOW protocol, which was then processed and translated into mouse and keyboard data and relayed to control the computer via Bluetooth. 

Most of these components were used out of the box with no soldering/wiring (except for adding an input button to disable the joystick controls).  Below are the 3 controllers. 

To briefly summarize how this works, Joystick 1 controls the mouse movement functionality.  I only used the accelerometer data to detect the tilt angle in two directions (Forward-Backward and Left-Right).  Similarly for Joystick 2, I hotkeyed a few mouse/keyboard commands based on tilting the IMU in one direction.  Below is an image showing different angles of Joystick 2 mapped onto different mouse/keyboard commands. 

With this setup, I attempted to draw a few images using the Paint program on my laptop. The first one was the Supplyframe logo operating both joysticks with my hands.

The next thing I tried was to try to control Joystick 1 (mouse movement)  with my head.  I wore a hat and placed the joystick in the hat.  

This was much more difficult to control and after about 10 minutes, my neck was sore.  Part of this exercise was helpful to put myself in the shoes of others and experience using input controllers without hands.