The 2020 HDP Dream Teams are participating in a two month engineering sprint to address their nonprofit partner. Follow their journey here.
Early in my college years I came across this very useful and web based animated circuit simulator, falstad.com/circuit. Since I'm a hands on and visual learner, this tool really helped me grasp the behavior of electronic and logic circuits.
While I was working on the firmware for our concept prototype keyboard, I thought it would be fun to create an animation of how keyboard matrix scanning works.
As you can see the micro-controller drives the rows low one at a time and reads the columns as inputs. If both the row and the column are low, then a key-press is detected.
This allows us to read key presses without needing a single input for every single key! In this example we are able to detect key presses for 15 keys with only 8 GPIO pins. Almost all keyboards or TV remote controls follow the same principle.
In the first log of this project, I experimented with IMU sensors as alternative input methods to large buttons and mechanical joysticks. Along with the smart remote prototype we are working on delivering to UCPLA for testing, we want to also deliver an IMU-based input controller as well. In this log, I will discuss the software updates making the joystick more robust and usable for different users.
In the first iteration, the code was quickly written to make things barely work enough to demonstrate a concept. For example, raw IMU signals (accelerometer only) was used and threshold values were hardcoded specific to how I wanted to use it. Three improvements are made. First, the raw IMU signals were processed to smooth out the signals to give better results. Second, a calibration routine was programmed and a new thresholding technique was used to detect different position states. Last, a two joystick controller was shown to demonstrate modularity and the ability to expand input capabilities.
The hardware did not changed from the previous iteration. All of the processing is done with a receiving unit, an ESP32 based board. Two M5StickC units ( with an IMU sensor, button, and screen) send signals to the receiver. In this iteration, one M5StickC unit is intended to be worn on a user and the other unit is intended for the caregiver for handling calibration of the system.
Lightweight Signal Processing for Smoothing Data
The IMU in the M5StickC is a 6-DOF IMU, with 3 accelerometer signals and 3 gyroscope signals. With this IMU, all I am trying to do is measure tilt of the unit. The first iteration measured tilt only using accelerometer signals, which is not the most accurate way to measure tilt angles. The problems with measuring tilt from accelerometer signals is any sort of movement including vibration will cause noisy signals.
Tilt angle can also be measured using signals from a gyroscope, with signals measuring the angular velocity. Calculating orientation from gyroscope can be done by integrating the gyroscope signals. The issue with doing the angle drifts from the correct orientation over time.
The most common technique to calculate tilt angles using IMUs is using a Kalman filter to combine accelerometer and gyroscope signals. However, I opted to use a complementary filter which is stupid simple to implement and doesn't need a lot of computation. It combines both methods together to mitigate both noisy signals and IMU drift. Some more detailed info on complementary filters and IMUs in general is found here, which I followed to implement for this system.
The image above compares the three methods to measure angles based on accelerometer and gyroscope data from one of the IMUs.
Adding a Calibration Routine
The next improvement that was necessary was adding a calibration routine to make this system more robust. Considering that each user with cerebral palsy has different limitations, they won't all be able to use this input device the same way. Thus, a calibration routine is necessary to stay within their motion limits. In my head, the plan that makes sense is to involve a caregiver for setting up the IMU. The IMU will be strapped onto an extremity with decent range of motion. The caregiver will then instruct the user to move in different body positions with each position corresponding to a different button. The caregiver will have a second M5StickC unit, which will give them prompts on a display and a physical button to press to help with calibration. Once calibrated, the information will be written onto the flash memory of the receiving microcontroller to store until the IMU is re-calibrated again.
Once calibrated, the IMU is ready to use immediately. ...Read more »
As Ruben alluded to in a previous log entry, one of the first proof-of-concept prototypes we want to deliver for testing is a smart remote with an array of 15 buttons. Depending on how a user decides to hit a button, a parameter we want to play with is how much force or effort it takes to hit a button. For me typing on a keyboard, I have pretty good control of all my fingers and won't mistakenly hit the wrong keys. In my case, I would want something with low resistance and low force. If we are designing for somebody with less motor control, mistakenly hitting wrong buttons (false positives) is a concern with regular remote control designs. One way we can avoid hitting wrong buttons is changing the button sensitivity to make them harder to press.
The main button type we have keyed in on are mechanical keyboard switches. Advantages of using these buttons are they are extremely common to find, come in a standard size, can withstand millions of cycles, and have a relatively low profile compared to other options we were considering (arcade buttons, limit switches, etc.). We were also thinking of making our own switches if we wanted a low profile button, but at least for this first prototype, it's not the highest priority.
For a mechanical switch where the button is either on or off, how do we change sensitivity? If we had an analog "button" like a force sensor, we could change the sensitivity on the software side by simply adjusting the pressure threshold. Similarly for a mechanical switch, the way we would do this is by changing the spring stiffness inside the key. Cherry MX sells a bunch of different types of key switches with varying parameters such as actuation force (sensitivity) and key travel distance. We have ordered 9 different types of buttons to use in our test panel to vary some of these parameters, including button sensitivity. However, after thinking about it some more, the actuation force of different key switches really didn't vary too much. From their website, all of the keys' actuation force lied within the range of 45-80 gF. Even though we ordered a bunch of different keys, the actuation force didn't span a broad enough range. We decided to first open up a key switch to see how feasible it was to replace the spring with one that was stiffer.
It turns out these keys are really easy to open and are relatively simple to disassemble with only 4 components, one being the spring. By replacing the spring with another with a different spring stiffness or length, the hope is that we can have much stiffer keys switches that is at least an order of magnitude higher than the original switch. As a quick test, two springs were pulled out from ballpoint pens. Below is an image of two longer and stiffer springs, compared to the shorter spring from the key switch.
Each of the springs were installed inside the key switch to test the actuation force required with each spring To estimate this force, a kitchen scale was used. The first test was with the original short spring that came with the key switch and it required about 50 gF to actuate. The rated force from the website is 45 gF, so this was pretty good validation for using this kitchen scale for measuring force.
The next two videos show testing of the springs from the ballpoint pens. For the longest spring, I had trouble closing the housing, meaning I probably exceeded the fully...Read more »
UCPLA was able to review our concepts over the weekend and we are delighted to be moving forward with Proposal 1 that was pitched during our Concept Presentation last week. Since users mobility is so unique to each user, we will be focusing on Variation 1.
Now the fun can begin!
While we have a concept idea, we still need to design the prototype. This will require all hands on deck as we only have 10 days to design and build a concept prototype.
There are two parts to this design, the display, and the input. The goal is to have several different options for input, but for now we will focus on the button grid as one of the options.
The key point here is that the teams goals are beginning to align. Sure we all had the same general goals when starting the project and first applying with the team. One example is that I feel we all have a genuine desire to help those in need of assistive technology. However, our approaches were drastically different and so our personal agenda were also. But now that we have pitched our individual (but surprisingly complimentary designs) to UCPLA, we are anxiously waiting their response help guide us in which path to focus on for this project.
After three weeks of understanding the problem, needs analysis, and brainstorming concepts, we have gotten to the next stage where we can start doing some real work. To summarize our ideas, we made this high-level figure showing our complete platform solution to the problem.
There's a lot of exciting ideas and it would be great if we could work on all of them to deliver an all-encompassing solution for any user with disabilities. However, given our timeframe and team bandwidth, it is impossible to do everything proposed at a high level. With the help of the team at UCPLA, we hope to narrow down the most promising concepts that will deliver immediate benefit to improve users.
This upcoming week is the prototyping phase, where we hope to develop an initial proof-of-concept prototype(s) and prepare to send it over to UCPLA for some user testing. One unique circumstance of this project is 3 different people simultaneously prototyping while never meeting in person. Communication will be key and I look forward to learning valuable lessons. Can't wait to get started!
In case you missed it - last Friday we presented our set of three concept ideas for a feedback session to UCPLA team as well as broader Hackaday community! Check them out here! Please do not hesitate to leave your questions, feedback and suggestions in Comments section!
Also big thank you to everyone who tuned in on Friday!
As some of you might have seen - last Friday we presented the initial set of three ideas to both UCPLA team as well as to Hackaday community for feedback and to help narrow down our directions!
Based on the discussions and feedback from UCPLA, we decided to open up our survey to accommodate any physical challenge within our survey, and not only Cerebral Palsy (CP). Please note, that the questions of this survey are supposed to be answered only by people with ANY physical challenges (Cerebral Palsy - CP, ALS, SCI - of course, with the help of their caregivers if needed).
If you are a caregiver of a person with any physical challenge, or you work in a center or hospital supporting people with physical challenges or simply know someone who has a physical challenge, please share this survey with them! Thank you!
As a reminder, the purpose of this survey is to understand the needs and experiences of people with ANY physical challenges in regards with the devices they use and want to have as well as accompanying digital tools.
Here is the link; it is the same as in our previous post, so do not worry if you have already shared/answered it, please go ahead, if you have not yet! We need your input! Thank you!
Currently, our team is in the phase of coming up with preliminary abstract ideas that we present to UCPLA for feedback. Part of this time is spent looking for inspiration from other sources that would be helpful to implement for our project. In this log, I want to focus on Apple's Force Touch trackpad.
MACBOOK FORCE TOUCH TRACKPAD
In 2015, the Macbook trackpad design was completely overhauled. However to the user, nothing really changed. in previous iterations, the trackpad was a cantilever beam design, hinged near the top and mechanical buttons near the bottom for the left and right mouse clicks.
The new trackpad introduced was a stationary plate with no mechanical buttons. At the four corners of the trackpad were 4 strain gauges that could detect the how much pressure user's were exerting on the trackpad. Above a certain pressure threshold, the computer would process it as a mouse click. In normal trackpads with physical buttons, there is auditory and tactile feedback from the button pressing and releasing. The new stationary trackpad replaced this feedback sensation with what they call a "taptic engine". The taptic engine is a vibrating motor that would activate when a user pressed hard enough and felt no different than a mechanical button. This is a useful for a variety of reasons, which I will try to explain how it could be applicable for our context.
SHRINKING FORM FACTOR
First, the force touch refers to different force thresholds that could detect how hard a user pressed. While not moving finger position, a user could do a light press for a regular mouse click and a hard press to activate Force Touch for secondary functions. In our context for the CP population, big buttons are often used to compensate for the lack of fine motor control, leading to large, clunky designs as seen in the remote control below. To shrink form factor, can we have one physical button that represents three digital buttons by distinguishing between a light, medium, and hard press? Apart from force sensing, can we use this concept using a different method?
MOVING TO A NON-MOVING DESIGN
For able bodied individuals, operating a joystick that moves provides an intuitive feedback through spatial awareness and/or joystick resistance. Healthy individuals would have good control and can easily learn how far a joystick can be pushed in any direction. For an individual who may suffer from tremors or have a delayed feedback response between the hand and the brain, it becomes more difficult to consistently stay within the joystick limits. Thinking about the Macbook trackpad, would it be a more durable solution to have a 4-direction pressure sensing "joystick" that doesn't move (basically a solid object that users can comfortably grip), and replace feedback mechanisms using alternatives such as vibrating motors?
I couldn't really find any existing product that resembles this idea, so it's likely a gamble to go this direction. Watching how individuals struggle to use existing joysticks by having to change their hand position to push in different directions makes me think a joystick that doesn't move would help. From discussions with UCPLA, we learned joysticks have a tendency to break., which makes me think a stationary, sturdy "joystick" would be an improvement upon current joystick controls.