Audio feedback for the brain-injured who need help learning to balance again.
Firstly, a helpful man called David suggested an alternative volume control IC, the TI PGA series. Thank you David.
Next, I had the honour of sitting next to the insane @phoenix perry who made me realise that we need to cross the audible barrier into haptics, and soon!
We're hopefully going to work on something in collaboration but in the meantime I listened again to her recent appearance on the embedded.fm podcast which I heartily recommend to electronics and microcontroller fans. That repeat listen gave me the tip of looking into velostat 3M sheet, a force sensitive material that, only this morning, Marco Reps posted his video about starting to make a full pressure measuring mat for physiotherapy using sheets and sheets of the stuff. @James Bruton might want to look at getting some for his exo-skeleton robot control system as it appears way cheaper than purpose made FSRs and these can be made into custom sensors.
A SoleSense current prototype update will be coming soon.
@oshpark were cool enough to put a call out for designs to be loaded on their trial run of a flexible PCB service. I applied and was accepted by the very helpful @Dan Sheadel. There's no guarantee that the boards will come back in a fit state to ship to me but here's a peek at what they might look like. The front and rear of the foot were split into two parts to improve nesting efficiency on the PCB "panel" - if you can use that term for flex PCBs: "sheet"? The images from the OSH Park online viewer aren't to scale, relative to each other, below.
Imagine the fabrication simplification of a developed device if this works!
I was just trying to make a pun. And I'm sure someone has smashed audio and Arduino together in a clone-style board before. This log is about the requirements, challenges and solutions I found in generating stereo audio, using Arduino, for brain injury balance rehabilitation.
The brain injury specialist physiotherapist I'm working with tells me that brain injury patients' hearing abilities are among the most likely senses to survive brain injuries intact or less impaired than other senses. They would like to see whether by providing continuous, as opposed to error/boundary-based feedback, the patients can make better progress at learning to balance for standing and walking. I presume that this is because you can tell earlier if you are veering the wrong way if you have continuous sound signals, rather than waiting to receive a beep or buzz if you cross an error threshold in one direction.
I discussed a variety options, including pulse width modulation, beep frequency but in the end, the specialist thought that the best option to start with would be to use pitch to represent weight shifting forward and backward (backward/over the heels would be low pitch, forward/over the toes would be high pitch). We are initially going to use volume to represent a shift between left and right foot (if relatively more weight is through one foot it will be louder and the other one will be quieter).
The classic and most obvious way to produce an audio signal is to use the tone() command in the Arduino IDE. I knew that this relied on timers but I hadn't realised that this function cannot produce a tone on more than one pin at a time, let alone tones of different pitches simultaneously, which is what I want to do.
I searched and searched through various timer libraries in an attempt to generate simultaneous and different varying tones from a single Arduino and came across @Jeremy Blum 's code for 5 speakers simultaneously. Perfect.
I coupled the pitch to the capacitive sensor readings and took it for an early demonstration to the rehabilitation specialist, who said that the audio didn't sound smooth enough. I have to agree that it sounded very "8-bit" and "bleep-bloop-bloop-bleep" as it progressed up and down the frequencies. I thought this was to do with the counter limits I was using in the interrupt service routine and this was generating large frequency jumps. However, it appears that it was more to do with refresh rates, which I was surprised by.
Human persistence of vision allows us to watch rapidly changing static images and perceive them as moving images. At around 20-30 Hz this update rate appears imperceptible (unless you rapidly turn your head). I increased the audio refresh rate so that frequencies were being updated every 25ms (40Hz) and the effect was still clunky. I then updated every 20ms (50Hz) and could still perceive a rough progression in pitch. Finally, at 15ms updates (~60Hz) I found the pitch progression to be smooth. Seems like hearing is less easy to fool than sight...
I have spent many an hour with the headphones continuously droning high pitch into one or both ears while I write, test and adjust code and I can empathise with victims of noise torture. I decided to try different frequency ranges and although there's no scientific basis for my results, I did start looking in the vicinity of Middle C (~262Hz) and played with around an octave of range. The frequency range I have settled on, because I found it large enough to tell that I was moving from front to back foot loading and because I didn't want to disturb patients with an overly screechy high pitch, is 200Hz to 480Hz.
The volume needs to be controlled by the microcontroller, so I initially tried pulse width modulation and varied the duty cycle to try and represent a volume change, using one of my piezoelectric discs as a crude speaker (they were still hanging around from the sensor experiment)....Read more »
The MPR121 is a really nice IC, which is unfortunately now a legacy chip that you need to order in large quantities if you want to mount it on your own board. If we get to any kind of production status we'll need a full review of this component in the Bill of Materials but for now, it's a wonderful device to use on the various break-out modules available for a few £ each.
One of the characteristics of capacitive sensing is that pretty much any conductor will act as part of the capacitive plate if electrically connected. In the case of #SoleSense, there were clear differences in the readings that were obtained by each sensor when no-one was near the copper tape. The jumper wires bundled together were definitely contributing to the varying readings. They also affected readings when they rubbed past each other. I knew that I needed to make a pair of insoles that would allow comparison of the patient's weight being transmitted through each foot, so I needed to make the readings more repeatable.
The first thing I decided was to use a uniform pad layout between each foot. To ensure repeatability, I took a photo of my foot out line with some notional pad areas over key sections of the foot.
I imported this photo into Inkscape and drew round it. Then I exported to dxf into Onshape and made a thin insole with raised pad sections.
You can see that I have left minimal material under the arch of the foot to allow flex, and there are recessed cable runs to try and get the same length and route of wire between each pad and the MPR121 board. Of course, once this is in CAD, it is easy to mirror to make the other foot. The only change is that the other end's terminals are used (11,10,9,8,7 as opposed to 0,1,2,3,4) on the MPR121. This needs adjusting in code, which is trivial.
I printed this out (in hot pink PLA) and ended up with suitable physical versions.
I also printed out a cutting template for each pad so that I could make uniformly sized copper tape pads.
I tested soldering the wire (printer cable style ribbon wire this time) to the copper tape once it was stuck to the PLA and to my pleasant surprise it was possible to make a good solder connection without damaging the PLA (noticeably).
The MPR121 module is held in place with a piece of double sided foam tape. I soldered a female header to the other side of the module but for now, I'm happy that I have got a physical design with a much better chance of producing repeatable and comparable readings between left and right food, regardless of the cable routing of the power and I2C bus to the Arduino.
This project presents several challenges but the first one I grappled with was a good way to cheaply sense the distribution of body weight to the ground through the different parts of the foot. I was sent this paper which is a good literature review of different sensor methods.
Initially I looked at using the small piezoelectric discs that are often pressed into service as low profile speakers or vibration / knock sensors in hobbyist Arduino projects. I was quite excited to find the alli-GAIT-or project here on Hackaday.io and that it had been entered into the Hackaday Prize 2016. However, when I tried to replicate foot pressure readings using piezoelectric discs I found that they would only produce a voltage that could be sensed during loading or unloading. It appears that because you need a high value resistor to dissipate the charge across the leads, it quickly drops when movement stops. This may be ok for the dynamic analysis of the #alli-GAIT-or analysis project but this application needs to compare static relative loading of areas of the foot.
After that I checked out force-sensitive-resistors but they look really expensive when you want several per foot.
Then I remembered the MPR121 capacitive touch sensor breakout I had in the parts trays. I wasn't sure whether it would work but I remembered that you can configure it to supply the raw capacitance readings to the Arduino, as opposed to the "touched" / "released" signal. It has up to 12 plates/sensing pads so well in the region of the numbers discussed in the literature review. I wondered if you can use an elastic compression/deformation of a layer between the foot and the capacitive pad to approximate the relative weight being transmitted to the floor at that point of the foot. I also remembered that soldering to aluminium tape wasn't going to work but that I had some copper foil tape somewhere.
The first success was soldering jumper wires to the copper tape. I used a flux pen and lead-free solder. then I hooked up the MPR121 board to the arduino and copper pad and managed to get raw readings from the copper pad. I drew around my foot on a pizza box and taped several soldered pieces of copper tape to various parts of the foot shape that I had outlined.
Then I placed neoprene foam sheet, about 2-3mm thick on top.
Then I stood on top and varied my weight over different parts of the sole, rocking backwards and forwards and swaying from one side to the other. I used the Arduino serial plotter to see if I was getting usable variations in the readings between different sensors and whether the readings would stay static if movement stopped.
Although I can't exactly remember which pads were which trace on here, there are some really nice swings that represent my shifts in weight distribution well.