The 2020 HDP Dream Teams are participating in a two month engineering sprint to address their nonprofit partner. Follow their journey here.
To make the experience fit your profile, pick a username and tell us what interests you.
Initial Project Proposal Presentation Slides
presentation - 17.07 MB - 09/17/2020 at 18:51
Initial Project Proposal -2 page summary
document - 65.00 kB - 09/17/2020 at 18:51
Project Final Presentation Slides
presentation - 18.87 MB - 09/17/2020 at 18:49
Previous project developments have heavily focused on physical hardware prototyping of a keyboard and motion-based remote controller. One of the tasks that haven't been explored in depth is the peripheral digital devices that we can control with our prototypes. We have discussed some ways to control a Roku media player as a proof-of-concept starting point and in this log, I will detail one method using an ESP32 microcontroller through Home Assistant to emulate a Roku remote.
While we are working on a more developed keyboard prototype, I created a low-fidelity one for the purposes of debugging. This is shown below. It is 8 tactile switches connected to an input pin on the controller. The uploaded sketch publishes a message (UP, DOWN, HOME, etc.) to an MQTT broker, which will be used as a trigger to perform an action on the Roku.
There are many possibilities on how we can communicate from a microcontroller to the Roku. I chose to use Home Assistant because it was simple to setup, easy to learn, and has integrations with the majority of smart devices. This last point is important because it makes it easy to translate from controlling a Roku to controlling another device like an Amazon Echo or Google Nest.
The setup to install Home Assistant on a Raspberry Pi was followed from the Home Assistant website. After, the Roku and MQTT broker integrations were setup for this specific example.
The next step was to establish a connection between Home Assistant to the Roku media player. Tests scripts were written where a button on the Home Assistant dashboard could be used to send Roku commands.
Once that was done, the ESP32 microcontroller needed to send messages to Home Assistant. With the MQTT broker configured, short messages were published to HA, which could be seen from the Home Assistant dashboard.
Finally, an automation routine was written to combine the trigger from the microcontroller and to perform actions on the Roku media player. Two simple automation scripts are shown below.
The video below shows the ability to use the 8 buttons to emulate Roku remote to control a Roku media player.
Currently, we've been using the M5StickC platform out of the box as our first iteration of our IMU-based motion sensing remote. At its price point of 10 dollars, it included a lot of components in a small package, great for wearable device development. Below, I summarized a table of the different features of the M5StickC, highlighting the useful components for an IMU remote.
|6-AXIS IMU (MPU-6886)||YES||main feature of this universal remote|
|Real Time Clock (RTC) module (BM8563)||YES||Important for deep sleep functionality of ESP32 chip|
|Power Management Unit (PMU) (AXP192)||YES|
|3 TACTILE SWITCHES||MAYBE||Difficult for the user to press these buttons|
|IR LED||PROBABLY NOT|
|RED LED||MAYBE||Can be used as an indicator light (i.e. low battery)|
|HEADER PINS||MAYBE||modularity, add more things custom for each user|
|LED DISPLAY (0.96")||NO||small display not suited for this population, high power consumption|
|LIPO BATTERY (85 mAh)||YES/NO||Battery capacity needs to be increased to be useful|
The first prototype was helpful in gaining useful insight for our first user testing session. The second iteration will include some additional features to address making the device wireless by having a larger battery, incorporate haptic feedback, and adding a non-contact button for turning the device on/off.
My initial thought was to use the header pins and attach a second housing to the orignal M5StickC platform, which is seen from some of the hats from their website. With this approach, I tried to imagine what the updated device would look like and could not envision a compact design. I also didn't like the lack of robustness of using the header pins for connecting the additional features.
My next idea was to open up the device and see what the guts looked like. I ended up breaking some parts including the LED display and one of the PCBs. Regardless, there's a picture of the main PCB below.
This main PCB had all the main components except including the battery connection and decided to just mount a second board with all my additional features onto the header pins, similar to how the red PCB was attached. This implementation will be shown at the bottom of this log.
The current 85 mAh battery allowed a runtime of about 20 minutes, not really useful for any applications. Thus, for our first prototype that we shipped out, we instructed our users to keep the device plugged in to a wall/computer.
I measured the current consumption of just sending IMU signals to a receiver to be about 120 mA. I hooked up a 400 mAh battery to the M5StickC and got just over 3 hours of runtime, which correlated to the current measurement. When I set the device to deep sleep mode, it drew about 10 mA, which is higher than the 10 uA of current draw in deep sleep mode of the ESP32 development board. This is something I still need to look into. I also measured the current while the device was turned off and was surprised to find it drew about 7 mA of current. After looking at some online forums, it seems like this wasn't a mistake and is a bug of the platform. For this application where I'm not requiring a device to run for months or years, I think the current consumption at off/deep sleep states are good enough for now.
I ended up choosing a 1,000 mAh battery which would extrapolate to about 8 hours of runtime of just sending IMU signals. However,...Read more »
Last week, we had our first user testing session with UCPLA where we got our first chance to see our prototypes in the hands of our target audience. We were lucky enough to see this firsthand remotely through Zoom and got some valuable learning lessons. A few screenshots from this testing session are shown below.
This was the first time our devices were in the hands of other people, and it gave us valuable insight into some considerations we didn't focus on. Early on, we asked a few questions to help us better understand our audience, but watching someone try our devices allowed us to better grasp some of these issues.
The last couple of weeks were busy preparing for prototype shipment and I kind of forgot to post log updates, so this entry will cover a few topics. While this upcoming week is focused on user testing and receiving feedback on our keyboard and joystick prototypes, this log will focus on the development of an intermediate prototype of a wireless Bluetooth button.
Currently for this project, we were focusing on delivering two prototypes: the keyboard and the IMU-based joystick. I wanted to add an intermediate milestone to create this wireless button prototype, which could potentially be a third deliverable but more importantly serves as learning lessons for future development of other prototypes. Below, I describe 5 discussion topics regarding this one button.
A lot of buttons and switches that are catered towards use as "assistive technology" are in my opinion, really expensive. A big red button can cost about 65 dollars. A mechanical keyboard switch that we are using for our keyboard input device is more than an order of magnitude less that at $1.06. Going one magnitude further, a tactile switch costs $0.10. From an electrical/hardware perspective, they are all just momentary push button switches. Looking at a previous Hackaday project, the Clunke Button, that project addresses this issue by creating a DIY assistive button using a keyboard switch, an audio cable, and some 3D printed parts.
Now taking a look at wireless Bluetooth "assistive technology" buttons, the cost is also expensive with this model priced at $195. I looked at this button, and challenged myself, could I make a button by only having $19.50 to spend on parts? Below are the list of electronics from Digikey to make a wireless Bluetooth button, with the subtotal coming in at $19.36.
|Quantity||Part #||Description||Unit Price||Extended Price|
With these parts, 3D printed parts, and some wires and solder, a Bluetooth button with an stereo jack to connect another "Clunke" style DIY button can be made at almost 1/10th of the cost of a commercial alternative, and 1/3rd the price of a just a button. Below is a snapshot of the CAD model with a section cut out and an image of trying to get all the electronics to fit inside the 80x65x52.5 mm housing.
So with $20 worth of electronic components, we can make a $200 button that has Bluetooth connectivity to be used with our platform we're developing. This leads to the next topic, why have just a simple button instead of the other cool devices we're developing?
The keyboard and IMU-based input devices we're developing aren't necessarily commonplace in the assistive technology market. Even if we're designing these devices that we think have added functionality or an improved user experience, it doesn't mean our devices are going to be adopted by potential users. My guess is that there will be a large population...Read more »
We had a meeting to go over all of the content and have Aragna set everything up to be sure it was all working. While we did have everything he needed in the box, I soon discovered that I should have labeled all of the cables.
Our goal for the final version is to eliminate all of the cables.
Another lesson learned was that I should have double checked the address before shipping. At second glance I would have noticed that the address was missing some information (apt #). Even so we lucked out and the package was still delivered in time for our meeting.
Also, I need to do better at tacking screen shots during a session.
Other than the slight confusion of which cables to use where (The joysticks came with three USB-C cables), The demo went well. Once challenge when showing a demo is trying to align the camera to give your viewers a good view of the items you are using.
One way we were able to kind of work around that was by having me screen share my remote connection to the display unit. That way the team could see what the device was doing as Aragna was testing it out.
Over all I say it was a success!
We already have a meeting lined up with our first user this Monday.
At the moment these will be used to gather more user data, to help design a full on wireless joystick.
Running our Universal Remote Demo Software to show the concept of how we plan on assigning many actions to only a few buttons. The UI will match the button layout, then each button on the custom keyboard will correspond to button in the UI. (To test this with a keyboard, use the following keys
It is simply a custom keyboard with only 14 keys. The design is more about giving our potential users a functional test kit, so they can feel the difference between the keys and decide on which button type, size, trigger force, and recess level they prefer. We also modified two of the buttons by adding a much stiffer spring. (I didn't get the chance to measure the actual force so the values are guesstimates.)
Early in my college years I came across this very useful and web based animated circuit simulator, falstad.com/circuit. Since I'm a hands on and visual learner, this tool really helped me grasp the behavior of electronic and logic circuits.
While I was working on the firmware for our concept prototype keyboard, I thought it would be fun to create an animation of how keyboard matrix scanning works.
As you can see the micro-controller drives the rows low one at a time and reads the columns as inputs. If both the row and the column are low, then a key-press is detected.
This allows us to read key presses without needing a single input for every single key! In this example we are able to detect key presses for 15 keys with only 8 GPIO pins. Almost all keyboards or TV remote controls follow the same principle.
In the first log of this project, I experimented with IMU sensors as alternative input methods to large buttons and mechanical joysticks. Along with the smart remote prototype we are working on delivering to UCPLA for testing, we want to also deliver an IMU-based input controller as well. In this log, I will discuss the software updates making the joystick more robust and usable for different users.
In the first iteration, the code was quickly written to make things barely work enough to demonstrate a concept. For example, raw IMU signals (accelerometer only) was used and threshold values were hardcoded specific to how I wanted to use it. Three improvements are made. First, the raw IMU signals were processed to smooth out the signals to give better results. Second, a calibration routine was programmed and a new thresholding technique was used to detect different position states. Last, a two joystick controller was shown to demonstrate modularity and the ability to expand input capabilities.
The hardware did not changed from the previous iteration. All of the processing is done with a receiving unit, an ESP32 based board. Two M5StickC units ( with an IMU sensor, button, and screen) send signals to the receiver. In this iteration, one M5StickC unit is intended to be worn on a user and the other unit is intended for the caregiver for handling calibration of the system.
Lightweight Signal Processing for Smoothing Data
The IMU in the M5StickC is a 6-DOF IMU, with 3 accelerometer signals and 3 gyroscope signals. With this IMU, all I am trying to do is measure tilt of the unit. The first iteration measured tilt only using accelerometer signals, which is not the most accurate way to measure tilt angles. The problems with measuring tilt from accelerometer signals is any sort of movement including vibration will cause noisy signals.
Tilt angle can also be measured using signals from a gyroscope, with signals measuring the angular velocity. Calculating orientation from gyroscope can be done by integrating the gyroscope signals. The issue with doing the angle drifts from the correct orientation over time.
The most common technique to calculate tilt angles using IMUs is using a Kalman filter to combine accelerometer and gyroscope signals. However, I opted to use a complementary filter which is stupid simple to implement and doesn't need a lot of computation. It combines both methods together to mitigate both noisy signals and IMU drift. Some more detailed info on complementary filters and IMUs in general is found here, which I followed to implement for this system.
The image above compares the three methods to measure angles based on accelerometer and gyroscope data from one of the IMUs.
Adding a Calibration Routine
The next improvement that was necessary was adding a calibration routine to make this system more robust. Considering that each user with cerebral palsy has different limitations, they won't all be able to use this input device the same way. Thus, a calibration routine is necessary to stay within their motion limits. In my head, the plan that makes sense is to involve a caregiver for setting up the IMU. The IMU will be strapped onto an extremity with decent range of motion. The caregiver will then instruct the user to move in different body positions with each position corresponding to a different button. The caregiver will have a second M5StickC unit, which will give them prompts on a display and a physical button to press to help with calibration. Once calibrated, the information will be written onto the flash memory of the receiving microcontroller to store until the IMU is re-calibrated again.
Once calibrated, the IMU is ready to use immediately. ...Read more »
As Ruben alluded to in a previous log entry, one of the first proof-of-concept prototypes we want to deliver for testing is a smart remote with an array of 15 buttons. Depending on how a user decides to hit a button, a parameter we want to play with is how much force or effort it takes to hit a button. For me typing on a keyboard, I have pretty good control of all my fingers and won't mistakenly hit the wrong keys. In my case, I would want something with low resistance and low force. If we are designing for somebody with less motor control, mistakenly hitting wrong buttons (false positives) is a concern with regular remote control designs. One way we can avoid hitting wrong buttons is changing the button sensitivity to make them harder to press.
The main button type we have keyed in on are mechanical keyboard switches. Advantages of using these buttons are they are extremely common to find, come in a standard size, can withstand millions of cycles, and have a relatively low profile compared to other options we were considering (arcade buttons, limit switches, etc.). We were also thinking of making our own switches if we wanted a low profile button, but at least for this first prototype, it's not the highest priority.
For a mechanical switch where the button is either on or off, how do we change sensitivity? If we had an analog "button" like a force sensor, we could change the sensitivity on the software side by simply adjusting the pressure threshold. Similarly for a mechanical switch, the way we would do this is by changing the spring stiffness inside the key. Cherry MX sells a bunch of different types of key switches with varying parameters such as actuation force (sensitivity) and key travel distance. We have ordered 9 different types of buttons to use in our test panel to vary some of these parameters, including button sensitivity. However, after thinking about it some more, the actuation force of different key switches really didn't vary too much. From their website, all of the keys' actuation force lied within the range of 45-80 gF. Even though we ordered a bunch of different keys, the actuation force didn't span a broad enough range. We decided to first open up a key switch to see how feasible it was to replace the spring with one that was stiffer.
It turns out these keys are really easy to open and are relatively simple to disassemble with only 4 components, one being the spring. By replacing the spring with another with a different spring stiffness or length, the hope is that we can have much stiffer keys switches that is at least an order of magnitude higher than the original switch. As a quick test, two springs were pulled out from ballpoint pens. Below is an image of two longer and stiffer springs, compared to the shorter spring from the key switch.
Each of the springs were installed inside the key switch to test the actuation force required with each spring To estimate this force, a kitchen scale was used. The first test was with the original short spring that came with the key switch and it required about 50 gF to actuate. The rated force from the website is 45 gF, so this was pretty good validation for using this kitchen scale for measuring force.
The next two videos show testing of the springs from the ballpoint pens. For the longest spring, I had trouble closing the housing, meaning I probably exceeded the fully...Read more »
Become a member to follow this project and never miss any updates