• Refining the audio "picture"

    Simon Merrett11/07/2017 at 02:50 0 comments

    The soundscape that SoleSense is painting is quite dependent on e.g. the weight of the patient and the "no pressure" readings that are generated when the insoles are in close proximity to the patient's feet but they aren't bearing any weight through the feet. The algorithms to arrive at a tone and volume value for each foot are fairly simple. In this log, I want to quickly log the changes to the volume algorithm.

    The original algorithm was taking the average reading across all five pads under the foot and mapping these values to an acceptable volume range, before appending the value to the radio payload for transmission by the NRF24L01 to the SoleSense audio unit. This had a major shortcoming in that the patient could be weigh bearing through only the toes or back of the heel on a single foot but because the weak readings on the other four pads reduce the volume value down, it sounds like the patient isn't weigh bearing on that side. 

    I tried to think of a few strategies to overcome these likely scenarios, including treating the stronger readings as more significant. If using a lookup table for this I think it could have led to some jarring progressions (although volume appears to be less refresh-rate-sensitive than tone). If we'd used some (slow) maths functions and power laws to produce a non-linear response, that might not have supported the frequent tone updates we require to keep on top of the bleeping. 

    Instead, for now, I have opted to drive the volume based on the strongest reading out of the individual pads To try and see what is going on, I plugged the insole into the Arduino serial plotter to look at the streams of sensor readings (slowed down with judicious 100ms delay() to prevent an unmanageable stream of data). This was really helpful as, even by calibrating the "no pressure" scenario, it didn't account for the individual differences in the sensor "start readings". I implemented an averaging for each sensor plate in the calibration routine and then created an offset from the combined mean across all five sensor plates. Then, the offset is used to "level" the readings of the sensor pads, before arriving at a suitable volume.

    The other part that I could determine from the serial monitor was the strongest reading that a sensor would pick up. With one 2-3mm layer of neoprene/EVA foam between the heel and the heel pad copper tape, this reading tended to dominate the high volume when loading my full body weight through it. Whereas, the toes didn't  as easily create full volume when standing on them alone. To remedy this, I have added a second layer of foam under the heel on each insole.

    From watching the serial plotter, it looks like the volume matches the loading of each foot much better and it certainly sounds loud when you are on tip-toes or rocking on your heel. There may be some benefit of smoothing the volume using a running average, to avoid volume spikes which don't seem critical but may be a feature to work on in the clinician's eyes.

    I look forward to the Verostat I have ordered arriving so that I can start to characterise it. I hope that it is easier to refresh and calibrate against than the capacitive approach has turned out to be! 

  • Audio and Calibration Improvements

    Simon Merrett11/01/2017 at 02:57 0 comments

    The dreaded bleeps 

    So, I have had a constant battle to try and make the sound (tone/frequency progression) "smooth". There just seems to be a low refresh rate "bleepiness" to it. I thought this was coming from delays in one or several of the following:

    1. Reading the MPR121(s)
    2. Calculating the frequency and volume to send in the radio packet
    3. Sending the radio packet with retries and acks etc
    4. Decoding the received packet in the audio unit and parsing into the frequency and volume components
    5. Updating the "tone counter" in the audio unit interrupt service routine (I know, unlikely but I was getting desperate)

    I took one insole and started littering the audio unit and insole sketches with timestamp variables, to spit out over the serial port every two seconds (ie like a blink without delay - up-to-date, but without holding up the main loop). 

    I managed to identify that the radios were able to pass data ridiculously fast - so they probably weren't the issue. But that took time and I now know much more about the RF24 library (thank you so much for that library TMRh20). 

    The MPR121 read took less than a millisecond, so that wasn't the issue. But neither was the calculation or parsing of the radio packets. And the tone counter just couldn't be the issue. I was starting to lose confidence when I thought rather than counting how many radio packets I was transferring, I'd count the number of changes to the frequency I was receiving. Around 30 every two seconds. 15Hz? No wonder it sounded bleepy.

    So I checked the insoles - yes, the readings were spitting out in less than a millisecond but the values didn't change more than 17 times a second. This couldn't be the MPR121 I2C bus could it? I searched around for the maximum read/report rate of the MPR121 and, in the datasheet, there is this:

    See the red? That's what the code I copied had set it to:

    // ESI=100 
    mpr121Write(0x5D, 0x24); // CDT=001 

    So I tried setting it to the blue:

    mpr121Write(0x5D, 0x21); // CDT=001 & ESI = 001 (2ms period)

    And I now have buttery smooth audio tone progression! It's just a shame that the MPR121 isn't necessarily a good choice for scaling this up, should we get that far with the trials.

    Silence is golden less distracting

    The patient who will hopefully benefit from this device needs the audio to completely disappear if they aren't putting weight on the insole, so we need to adjust the volume to account for the fact that even the proximity of the patient's foot will cause a change in the readings, whether they are standing on it or not. I intend to make this calibration routine available to the therapist by command from the audio unit at some point soon, but for now the calibration routine runs when we start the Arduino:

    void calibrate() {
      maxRead = (value1 + value2 + value3 + value4 + value5) / 5 ; // set the initial reading of maxRead to "seed" the loop below with a realistic value
      for (int i = 0; i < 10; i++) { //this for loop takes a number (10) readings of the sensors and averages them to arrive at the "no pressure" reading
        maxRead = (maxRead + (value1 + value2 + value3 + value4 + value5) / 5) / 2;
      maxRead = maxRead - 3;

     The last line just ensures there's no "wobble" and that the mute setting is selected on the digital pot.

    We're also learning lots about IPR and open source licensing in order to satisfy the requirements of the trial authorities. Perhaps a log on that will be due soon.

  • Protoboard and bus-wars

    Simon Merrett10/24/2017 at 20:05 0 comments

    I decided to use prototyping board as the next version of SoleSense, to reduce the clutter of the jumper cables and make a reasonably compact unit at each subsystem. For the insoles, this was relatively simple and consisted of a board with female pin headers to accommodate the Arduino nano and the NRF24L01 module. Then a thin four wire cable with a JST 1mm connector in the middle was twisted and then soldered between the board and the MPR121 module, carrying power and I2C signals.

    The audio unit was significantly harder to do on prototyping board because as well as the NRF24L01, it needed the digital potentiometer that controls volume, the trim potentiometer that adjusts left-right ear volume balance and the headphone jack.

    Once I had soldered everything, I also wrote code for the insoles that not only calculated the weight distribution from front to back but also took an average reading to indicate the proportion of the patient's weight which was being transferred through that foot. Using bit shifting, I combined both the tone and the volume for each insole into a single 16 bit integer variable for the NRF24L01 to transmit.

    Unfortunately, the resulting audio was patchy, not to mention sounding bloopy, like an early video game. I had a feeling that the audio Arduino was struggling to handle all its tasks so I had to go into fault-finding mode. Firstly, I tried generating the audio tone with a separate Arduino and feeding it into the DS1801 digital potentiometer to see if the burden of generating two individual audio tones from a timer interrupt was a bridge too far for the Arduino, on top of listening to the NRF24L01 and commanding the DS1801 on the SPI bus. Sadly, this didn't improve the quality of the audio.

    Then I wondered if the problem was to do with the NRF24L01 and the DS1801 sharing the same SPI bus. Although the DS1801 has a non-standard logic, I thought that this would only affect the Chip Select line behaviour and therefore it shouldn't affect the NRF24L01 on a different CS pin. Nevertheless, once I took the DS1801 out of the circuit (and bridged between the wipers and ends of the resistors) the tone was produced continuously, although still low resolution. I remembered that the NRF24L01 can use a softSPI library, where you select the replacement MOSI, MISO, SCK, CS etc pins in a config file, after installing the Digital.io Arduino library and including it in the sketch. I hastliy soldered some female headers along the spare digital pins of the Arduino and used some jumper wires to break out the NRF24L01. With the softSPI both volume and tone were responding as they should - Hooray! - and then I rerouted the protoboard wires to accommodate this setup, remove the jumper wires and allow the use of the original 2x4 NRF24L01 header socket. But audio was still bloopy...

    PS, by this time I was getting fed up maintaining a left insole and right insole version of the insole code, so for the first time I used the #if defined C preprocessor (according to Google) to define the different pin mapping between the left and right MPR121 modules, along with a #define leftFoot line that was included or commented out to compile for each side. Definitely a massive time and sanity saver and I recommend it if you haven't tried it before.

    I was a bit concerned about this apparently low refresh/update rate on the tone (I couldn't tell if it was there on the volume). My radios were being told to transmit new readings every 20ms, to match the 50Hz I had experimentally found was acceptable in a previous log. Finally, I remembered that I was using code where the NRF24L01 was being used in a transmit and receive configuration, so after it sent something it would go into a period of listening on receive and only after a set period would it exit the routine with a timeout. It turned out that I had upped the timeout from 2ms to 200ms in an earlier stage of development *for the audio module, not the insole module* but had...

    Read more »

  • Quick Update: profusion of ideas

    Simon Merrett09/19/2017 at 08:26 0 comments

    Thanks to @Jenny List, @Mike Szczys, @Elliot Williams, RS Design Spark and others for throwing an excellent Hackaday UK Unconference, I now have plenty of ideas for taking SoleSense forward.

    Firstly, a helpful man called David suggested an alternative volume control IC, the TI PGA series. Thank you David.

    Then, Saar Drimer from Boldport told me I should check out his Linux based graphic PCB tool, PCBModE. Thanks Saar!

    Next, I had the honour of sitting next to the insane @phoenix perry who made me realise that we need to cross the audible barrier into haptics, and soon! 

    We're hopefully going to work on something in collaboration but in the meantime I listened again to her recent appearance on the embedded.fm podcast which I heartily recommend to electronics and microcontroller fans. That repeat listen gave me the tip of looking into velostat 3M sheet, a force sensitive material that, only this morning, Marco Reps posted his video about starting to make a full pressure measuring mat for physiotherapy using sheets and sheets of the stuff. @James Bruton  might want to look at getting some for his exo-skeleton robot control system as it appears way cheaper than purpose made FSRs and these can be made into custom sensors.

    A SoleSense current prototype update will be coming soon.

  • Dabbling with flex PCB production possibilities

    Simon Merrett08/31/2017 at 21:51 4 comments

    @oshpark were cool enough to put a call out for designs to be loaded on their trial run of a flexible PCB service. I applied and was accepted by the very helpful @Dan Sheadel. There's no guarantee that the boards will come back in a fit state to ship to me but here's a peek at what they might look like. The front and rear of the foot were split into two parts to improve nesting efficiency on the PCB "panel" - if you can use that term for flex PCBs: "sheet"? The images from the OSH Park online viewer aren't to scale, relative to each other, below.

    Imagine the fabrication simplification of a developed device if this works!

  • Cracking up: sub-system boundaries

    Simon Merrett08/31/2017 at 21:21 0 comments

    This is a short project log about starting to divide up the sensor reading and audio generation and control. It has taken me some time to to post this log, which reflects the state of progress as it was when I presented at the UK #HackadayUncon in Sep 17.

    With the pink insoles I made a cable that went between the two insoles and then up to an Arduino at the waist, where it was reading both MPR121s. Be aware that as they were both on the same I2C bus they needed to have unique addresses. As I was using a female headers on the MPR121 break-out boards, I could just put a jumper across from the unused ADDRESS header to the unused 3.3V header, this changes the I2C address of the second MPR121 from 0x5A to 0x5C. Note that I'm using MPR121 breakout boards with 5V regulators and logic level shifiting but other boards might not have this.

    I physically now had the whole setup connected together but there is no way in the world that this could be used in trials. At least I could take data from two MPR121s simultaneously (figuratively).

    I then set about adding, in slow time due to other projects, a pair of NRF24L01 2.4Ghz radios, to allow me to have a completely wireless link between one foot and the other, with one foot being wired to the audio. I also separated the role of one arduino into three; one for each foot and one to deal with the audio.

    The separation between the feet was a vast improvement but for the sake of a third NRF24L01 and saving myself from coding a serial connection, I got rid of the wire between the right foot and the audio arduino.

    So now, we have three arduinos, three radios, three usb lithium battery banks (small ones based on a single 18650 cell) and two MPR121 modules. The feet are sending a reading based on the subtraction of the heel reading from the big toe reading (just for tests, the actual calcs will be different) with an offset to make sure there's a positive value. The audio arduino is receiving tone commands from both feet and can tell which one is sending which value. 
    The next stage is getting off the breadboard. 

  • Aud/*u*/i/*n*/o

    Simon Merrett08/31/2017 at 21:17 0 comments

    No, it's not a typo...

    I was just trying to make a pun. And I'm sure someone has smashed audio and Arduino together in a clone-style board before. This log is about the requirements, challenges and solutions I found in generating stereo audio, using Arduino, for brain injury balance rehabilitation.

    Why audio?

    The brain injury specialist physiotherapist I'm working with tells me that brain injury patients' hearing abilities are among the most likely senses to survive brain injuries intact or less impaired than other senses. They would like to see whether by providing continuous, as opposed to error/boundary-based feedback, the patients can make better progress at learning to balance for standing and walking. I presume that this is because you can tell earlier if you are veering the wrong way if you have continuous sound signals, rather than waiting to receive a beep or buzz if you cross an error threshold in one direction. 

    What kind of audio?

    I discussed a variety options, including pulse width modulation, beep frequency but in the end, the specialist thought that the best option to start with would be to use pitch to represent weight shifting forward and backward (backward/over the heels would be low pitch, forward/over the toes would be high pitch). We are initially going to use volume to represent a shift between left and right foot (if relatively more weight is through one foot it will be louder and the other one will be quieter).


    The classic and most obvious way to produce an audio signal is to use the tone() command in the Arduino IDE. I knew that this relied on timers but I hadn't realised that this function cannot produce a tone on more than one pin at a time, let alone tones of different pitches simultaneously, which is what I want to do.

    I searched and searched through various timer libraries in an attempt to generate simultaneous and different varying tones from a single Arduino and came across @Jeremy Blum 's code for 5 speakers simultaneously. Perfect.

    I coupled the pitch to the capacitive sensor readings and took it for an early demonstration to the rehabilitation specialist, who said that the audio didn't sound smooth enough. I have to agree that it sounded very "8-bit" and "bleep-bloop-bloop-bleep" as it progressed up and down the frequencies. I thought this was to do with the counter limits I was using in the interrupt service routine and this was generating large frequency jumps. However, it appears that it was more to do with refresh rates, which I was surprised by. 

    Human persistence of vision allows us to watch rapidly changing static images and perceive them as moving images. At around 20-30 Hz this update rate appears imperceptible (unless you rapidly turn your head). I increased the audio refresh rate so that frequencies were being updated every 25ms (40Hz) and the effect was still clunky. I then updated every 20ms (50Hz) and could still perceive a rough progression in pitch. Finally, at 15ms updates (~60Hz) I found the pitch progression to be smooth. Seems like hearing is less easy to fool than sight...

    I have spent many an hour with the headphones continuously droning high pitch into one or both ears while I write, test and adjust code and I can empathise with victims of noise torture. I decided to try different frequency ranges and although there's no scientific basis for my results, I did start looking in the vicinity of Middle C (~262Hz) and played with around an octave of range. The frequency range I have settled on, because I found it large enough to tell that I was moving from front to back foot loading and because I didn't want to disturb patients with an overly screechy high pitch, is 200Hz to 480Hz.


    The volume needs to be controlled by the microcontroller, so I initially tried pulse width modulation and varied the duty cycle to try and represent a volume change, using one of my piezoelectric discs as a crude speaker (they were still hanging around from the sensor experiment)....

    Read more »

  • Uniform readings

    Simon Merrett08/31/2017 at 19:58 0 comments

    The MPR121 is a really nice IC, which is unfortunately now a legacy chip that you need to order in large quantities if you want to mount it on your own board. If we get to any kind of production status we'll need a full review of this component in the Bill of Materials but for now, it's a wonderful device to use on the various break-out modules available for a few £ each.

    One of the characteristics of capacitive sensing is that pretty much any conductor will act as part of the capacitive plate if electrically connected. In the case of #SoleSense, there were clear differences in the readings that were obtained by each sensor when no-one was near the copper tape. The jumper wires bundled together were definitely contributing to the varying readings. They also affected readings when they rubbed past each other. I knew that I needed to make a pair of insoles that would allow comparison of the patient's weight being transmitted through each foot, so I needed to make the readings more repeatable.

    The first thing I decided was to use a uniform pad layout between each foot. To ensure repeatability, I took a photo of my foot out line with some notional pad areas over key sections of the foot.

    I imported this photo into Inkscape and drew round it. Then I exported to dxf into Onshape and made a thin insole with raised pad sections.

    You can see that I have left minimal material under the arch of the foot to allow flex, and there are recessed cable runs to try and get the same length and route of wire between each pad and the MPR121 board. Of course, once this is in CAD, it is easy to mirror to make the other foot. The only change is that the other end's terminals are used (11,10,9,8,7 as opposed to 0,1,2,3,4) on the MPR121. This needs adjusting in code, which is trivial.

    I printed this out (in hot pink PLA) and ended up with suitable physical versions.

    I also printed out a cutting template for each pad so that I could make uniformly sized copper tape pads. 

    I tested soldering the wire (printer cable style ribbon wire this time) to the copper tape once it was stuck to the PLA and to my pleasant surprise it was possible to make a good solder connection without damaging the PLA (noticeably).

    The MPR121 module is held in place with a piece of double sided foam tape. I soldered a female header to the other side of the module but for now, I'm happy that I have got a physical design with a much better chance of producing repeatable and comparable readings between left and right food, regardless of the cable routing of the power and I2C bus to the Arduino.

  • Challenges in sensing foot pressure

    Simon Merrett08/31/2017 at 19:29 0 comments

    This project presents several challenges but the first one I grappled with was a good way to cheaply sense the distribution of body weight to the ground through the different parts of the foot. I was sent this paper which is a good literature review of different sensor methods.


    Initially I looked at using the small piezoelectric discs that are often pressed into service as low profile speakers or vibration / knock sensors in hobbyist Arduino projects. I was quite excited to find the alli-GAIT-or project here on Hackaday.io and that it had been entered into the Hackaday Prize 2016. However, when I tried to replicate foot pressure readings using piezoelectric discs I found that they would only produce a voltage that could be sensed during loading or unloading. It appears that because you need a high value resistor to dissipate the charge across the leads, it quickly drops when movement stops. This may be ok for the dynamic analysis of the  #alli-GAIT-or analysis project but this application needs to compare static relative loading of areas of the foot. 

    After that I checked out force-sensitive-resistors but they look really expensive when you want several per foot.

    Then I remembered the MPR121 capacitive touch sensor breakout I had in the parts trays. I wasn't sure whether it would work but I remembered that you can configure it to supply the raw capacitance readings to the Arduino, as opposed to the "touched" / "released" signal. It has up to 12 plates/sensing pads so well in the region of the numbers discussed in the literature review. I wondered if you can use an elastic compression/deformation of a layer between the foot and the capacitive pad to approximate the relative weight being transmitted to the floor at that point of the foot. I also remembered that soldering to aluminium tape wasn't going to work but that I had some copper foil tape somewhere.

    The first success was soldering jumper wires to the copper tape. I used a flux pen and lead-free solder. then I hooked up the MPR121 board to the arduino and copper pad and managed to get raw readings from the copper pad. I drew around my foot on a pizza box and taped several soldered pieces of copper tape to various parts of the foot shape that I had outlined. 

    Then I placed neoprene foam sheet, about 2-3mm thick on top.

    Then I stood on top and varied my weight over different parts of the sole, rocking backwards and forwards and swaying from one side to the other. I used the Arduino serial plotter to see if I was getting usable variations in the readings between different sensors and whether the readings would stay static if movement stopped.

    Although I can't exactly remember which pads were which trace on here, there are some really nice swings that represent my shifts in weight distribution well. 

    Sensing concept proved!