Close
0%
0%

Earswitch: assistive technology switch & new HCI

assistive technology interface & games controller etc - an earphone switch controlled by voluntary contraction of a middle ear muscle

Public Chat
Similar projects worth following
Description : Ear switch as an assistive communication and control device / gaming switch and Human Computer Interface

In this hack I have used a USB otoscope camera to detect voluntary movement of a middle ear muscle (tensor tympani), to control assistive technology software (Grid3), the Microsoft On-Screen Keyboard , and as an "left-click" selection as an adjunct for eViacam head- tracking software.

>=75% of people recognise that they can contract their tensor tympani (a rumbling sound or muffling of the hearing eg when yawning/ tightly closing eyes etc). >=20% could generate the contraction in isolation to other movements

I have shown with a USB auroscope from Amazon that this causes movement of the eardrum, and I have detected this movement with motion detection software and successfully used it to control (handsfree); asssistive software (Grid3), the On-Screen Keyboard and to function as a "left click" for head tracking.

I hope this can help p

This is to allow communication for people with severe communication disabilities, and can help with control of other assistive technologies. I developed this as an assistive switch but also could be used as an accessory games controller switch (if both hands are already fully occupied!), and any other user interface. It's likely to work as a general new Human Computer Interface and may even be able to control smart earphones/ connected smart phones and hearing aids if the idea is incorporated into general consumer electronics.

I attach a background giving more details of the tensor tympani and the reasoning for the project, Proof of Concept videos, and instructions on how to set up the prototype, but would value any help/ advice. Link to the original prevalence survey is; https://www.surveymonkey.com/results/SM-PQ559JZW7

It would be great to help people communicate - so please let me know if you have any success in setting this up and using it.

I'd be grateful of any help or advice on the best way to do this.

I am hoping that if we can prove that the  Earswitch can help people communicate , or  it's taken up as a popular games  controller switch, that the tech companies will take this up and develop a mass produced/ wireless miniaturised version so that people with assistive needs can benefit from the function in cheap off-the-shelf earswitches.

Since I originally posted this project I have UK patent granted for the earswitch in view of the potential for general consumer use, but am hopeful that the current hack can help people communicate now ( as I've been able to type with the earswitch and its a shame that others dont have the opportunity to use this now!)

 We are also collaborating with Bath Uni with NHS (NIHR) funding for research into the prevalence and validation, and early protyping for the Earswitch for assistive needs.

Can you replicate this? & have you helped people communicate with this?

Can you help us find out how many people may gain from The Earswitch:

https://bathreg.onlinesurveys.ac.uk/earswitch-h-v13-final

Also - are you interested in a virtual Hackathon on Assisted Living at Herriot Watt Uni (Edinburgh):

⁦‪@earswitch‬⁩ attending: ⁦‪@NRobotarium‬⁩ and ⁦‪@ProductForge‬⁩-five day online "mashup" event from the 10th - 14th May 2021 bringing together health & care professionals, academics and assistive technology providers to prototype new solutions to support the care sector: 

https://vi.to/hubs/robotics-care-mashup 

Thanks for your help

Nick

hackaday set up of earswitch prototypes updated 30.3.2021.docx

Updated basic demo of easily available otoscope used to show the effect of the tensor tympani (“ear-rumbling”) on the ear-drum, and to connect to software to control target practice software and On-Screen keyboards.

document - 2.34 MB - 03/30/2021 at 22:58

Download

document - 9.90 MB - 10/25/2020 at 18:55

Download

Typing with Earswitch and Facetracking with and without predictive text.mp4

Video of handsfree typing on Microsoft On-Screen Keyboard using head tracking (iViaCam), Earswitch (as "left click") and iSpyconnect software; to show potential benefit of Earswitch with eye-tracking.

MPEG-4 Video - 2.11 MB - 01/05/2020 at 17:24

Download

Hackaday Earswitch general introduction v3 25.12.19.pdf

General description of tensor tympani and earswitch control

Adobe Portable Document Format - 4.26 MB - 12/26/2019 at 20:37

Preview
Download

Software setup info for Earswitch as Left Click v3 26.12.19.pdf

Use in conjunction with hardware set up noted in "How to set up Grid3" file

Adobe Portable Document Format - 1.48 MB - 12/26/2019 at 20:28

Preview
Download

View all 10 files

View all 8 components

  • Earswitch as Drum Controller and Gaming Switch

    Nick GPx05/19/2020 at 21:48 0 comments

    To show potential benefit as a general interface; The Earswitch prototype can be used with Switch ViaCam software to control Drumkit software https://www.microsoft.com/en-us/p/drumkit/9wzdncrfhw5v , as a handsfree drum control.

    The SwitchViaCam can output definable keystrokes and Drumkit can be configured to accept these keystrokes to generate a drum beat handsfree; I guess like a bass drum with a pedal (clearly I am no musician!)

    SwitchViaCam could also be used in a similar way for gamers ; for an extra handsfree control - its been suggested that could be used to replace “Push to Talk” and “Aim Down Sight” view, or “Crouch” position (although , again I am not a (recent) gamer)

    In SwitchViaCam (MUST open as Adminstrator)

    Config> Select USB Otoscope Camera (needs to be plugged in)

    Enable Sensor - select box

    Drag and expand selection box (blue box) to cover the middle of the eardrum (not shown here)

    Decide on output keystroke in “Event type”

    Alter Time Filter/Motion Detection Threshold (see below - alter to get required sensitivity)

    Open -  DrumKit or Game - & alter the keystroke required for the control input

    Clearly it may be hard to get hold of the otoscope currently but if anyone lets me know how they have used this, I would be very grateful

    Thanks

  • Update 21.2.2020: Use of Switch ViaCam as movement tracking software

    Nick GPx02/21/2020 at 23:06 0 comments

    Switch ViaCam is an image detection and keystroke output by Cesar Mauri which can be used instead of the combination of iSpy/Autohotkey script.

    Switch ViaCam Link

    It needs to be in the Foreground in Windows for it to work with Grid3, but it is a lot more responsive that the iSpy/Autohotkey combo.

    Some configuration of the camera settings may be needed for the Teslong otoscope, although the newer smaller 4x20mm otoscope worked with the default settings.

    For Teslong I needed to change the camera configuration to the following to switch to the Teslong otoscope to prevent it crashing:

            640x480 , capture speed 5 & a time filter of 500ms approx.

    Below is a screen shot of the set up using the 4x20mm otoscope, the Decibullz moulded earplug and Switch ViaCam to type on Grid3 (Grid3 configured to use a "Tab" keystroke as the switch access)

  • Update 17.2.20: success - anchored moulded earplug sensor

    Nick GPx02/17/2020 at 07:54 0 comments

    Using the heat moulded Decibullz Custom Earplugs as the anchor for the smaller 4x20mm otoscope, and a silicon earphone tip; this gives a stable Earswitch that doesn’t move with speech or head movement but can control Grid3 with tensor tympani control. 

  • Updates from 14.1.2020

    Nick GPx01/14/2020 at 23:02 0 comments

    Success with earshell and smaller otoscope

    Attached shows drilled out Ear shell to fit the 4mm otoscope & below is a picture of the drum showing that the otoscope is directed in the correct position (although the camera was upside down!):

    Although shows proof of concept it’s not currently stable in the ear as the wire is too inflexible and moves the shell out of position 

    I’ll try with a longer silicon ear tip as this may secure it better and protect the end of the otoscope .

View all 4 project logs

  • 1
    General instructions

    see “How to set up Grid3 trigger with Earswitch” and “Software setup Earswitch as Left Click” files

  • 2
    Alternative prototype: eartips for earphones

    It might be more comfortable to use silicone earbud eartips for earphones rather than the auroscope speculum on the end of the auroscope.

  • 3
    Smaller otoscope options​ & wireless adapter

    A smaller otoscope options:

    20x4mm otoscope 

    - but needs some form of permanent anchoring. Awaiting the Decibullz Custom Molded Earplugs to see if they are suitable .

    And wireless transmitter that comes with the newer “wireless” larger Teslong otoscope:

    The wireless adaptor transmits via its own WiFi network to an app eg on iPhone, and provides a rather delayed feed that would not be suitable for real-time switching. 

    I haven’t found a way to transmit the output to a PC. Any ideas would be helpful.  Thanks

View all 3 instructions

Enjoy this project?

Share

Discussions

thepenguinmaster wrote 02/25/2020 at 00:50 point

Cool concept. Would love to experiment with the concept, but too bad your parenting it and it's not open source.

  Are you sure? yes | no

Nick GPx wrote 04/24/2020 at 19:34 point

Thanks for your comment. Sorry for my delay in replying. The concept is OpenSourced here and so available for any one to gain from it’s use as an assistive technology switch or gamers switch, I sincerely hope it helps at least someone communicate.

I developed this for Assistive technology users, but then realised its a useful new Human Computer Interface & hence the patent and trying to get big tech companies to incorporate it into off-the-shelf earphones and hearing aids for general use for everyone;  which would have the advantage that assistive technology users could  get the benefit of mass produced/ high tech & cheap general consumer devices.

  Are you sure? yes | no

Jeroen.V wrote 02/24/2020 at 08:45 point

Hi Nick,

Very interesting!

Last year I built a prototype using sound. It took a lot of time to tweak because of the noise surrounding a contraction but now I am able to form words on a screen.

Your method seems more solid and indifferent to noise. Very cool!

(link with video https://geromy.com/tensor-sympathy/ )

  Are you sure? yes | no

Nick GPx wrote 02/25/2020 at 11:26 point

Thanks - your project looks great. Your "radar" selection of letters looks a much faster selection basis than some of the assistive technology keyboards. Have you thought of suggesting it to some of the assistive technology companies eg Smartbox (Grid3) as there may be users out there who would gain from that as an input option?

  Are you sure? yes | no

Jeroen.V wrote 02/25/2020 at 21:00 point

Tnx Nick.  Yes, the rotor works quite fast, especially after some practice. Im considering to develop it further. The code is available on Github. Might work well with Earswitch too.
Best of success!

  Are you sure? yes | no

Nick GPx wrote 02/25/2020 at 23:51 point

Thanks Jerome

Sorry -I'm not familiar with GitHub or code in general!- is it possible to use a keystroke input as a switch for the rotor (as thats the output that I'm triggering with the Earswitch ---- or alternatively iSpy can execute an .exe program file

Thanks

Nick

  Are you sure? yes | no

Jeroen.V wrote 02/26/2020 at 08:54 point

It would be possible to use a keystroke as input but the code must largely be rewritten. Or alternatively, start from scratch and import the 'rotor' functions from the existing script.  And yes, the application can be exported as an executable for Windows or Linux.

  Are you sure? yes | no

Nick GPx wrote 02/26/2020 at 21:44 point

thanks Jeroen.  I’ll have a look at that 

All the best. Nick

  Are you sure? yes | no

Constantin Wolf wrote 01/03/2020 at 08:26 point

We all know the sound, no one knew it's a muscle contracting. Thanks for that info.

I'd also suggest to explore other less power hungry sensors:

- EMG (sense muscle activation)

- Impulse response change of the ear (microphone)

- Sound of the muscle (maybe the mic hears the same as we do)

- Time of flight distance sensors

- Project Soli or other radar sensors (in pixel 4 smart phone)

- LSS by Rambus (tiny camera with no classic optics)

(Last two aren't that accessible, but maybe they like that unexplored HC interface idea so they give you a sensor or SDKs)

Many ideas pop to my mind: morse code translator (inverse babelfish), skip songs/mute/take calls wearing ear pods, hearing aid detecting seizures e.g. epilepsy.

Awesome work.

  Are you sure? yes | no

Nick GPx wrote 01/03/2020 at 21:17 point

Thanks Constantin. That's a great summary of potential sensor modalities.

I guess the important qualities are that its small/ cheap/ not susceptible to "noise" from speech/external noise/jaw movement/ and is specific to TT movement 

-EMG:   I dont think it would be possible as it would need an electrode attached to the TT which is the other side of the eardrum

-tympanometry (used to detect compliance changes of the eardrum in response to sound); needs a sealed ear canal and constant emitted sound

-microphone ; risk of  triggering by external noise/ speech etc 

       (I think probably the sound of the TT contraction will be minimal; sound heard is due to the movement  directly propagated to the cochlear by the TT moving the ossicles)

-time of flight sensors-

Is this similar to laser dopplar vibrometry/ optical coherence tomography- which have been used to detect drum movements? Are there any small enough to fit within the ear canal?

-Soli; radar sensors

I did try and contact their project team but didn't get any reply and did not pursue this as I felt the health implications of radar close to the brain / inner ear might cause concern

-LSS by Rambus 

Sounds fascinating - not come across this before but their website suggests that its no longer in production?

Also

--Ultrasound 

- IR camera

Benefit of not needing light source (though I cant find one thats small enough?)

LED and photodiode

- suggested by Luke (comment below)

I guess the benefit of CMOS cameras is that they are small enough / relatively cheap (although the small 1mm sq ones aren't!), and easily available. Also an option of a fibre optic camera would also work.

I'd be really grateful of your ongoing thoughts

Thanks

Nick

  Are you sure? yes | no

Constantin Wolf wrote 01/06/2020 at 09:19 point

- A potential TOF sensor could be the "VL6180X". I somehow remember a 3x3 depth sensor chip, but couldn't find it. With 1D depth value this is probably not going to work, as you'd need to calibrate via the orientation of the ear bud (pointing at the edge, that moves, to get a difference in depth), 3x3 would be able to autocalibrate, i guess.
- having in mind it might be used in parallel with music, optical seems the way to go

- i'd check out the LED/photodiode method (IR SMT proximity sensor), as from the videos i see, the change in brightness might be sufficient, maybe add additional photodiodes to scan different sections of the ear.

- light fibers may be a good way to get the light in/out the narrow ear channel in the prototyping phase

- For IR (Near IR) you still need a light source. If you mean far IR, you don't, but then you left 'cheap'-land. Maybe, really maybe, you can hack a PIR sensor (motion sensor) to see a change. but they aren't too small and far IR needs special optics.

- Pulsing the LED in an adequate frequency (Nyquist–Shannon sampling theorem: minimum double  the frequency/speed you can use the TT) you should be able to pull down power consumption. But that already 'optimization'-land :)


All the best for your endeavor.

  Are you sure? yes | no

Nick GPx wrote 01/07/2020 at 18:33 point

Thanks again Constantin

I cant see how to reply below your second post - so have added this here:

ToF sensor "VL6180X" looks amazing -- it seems this is a miniature version of the lasar dopplar vibrometry thats used in some of the research papers to detect ear drum movement; & even with a single point measurement may give enough detail if the sensor can be reliably directed down the canal at the end of an earplug/phone.

LED/photodiode method (IR SMT proximity sensor): again looks really hopeful - particularly the miniature 1x2mm : TMD2635 IR Proximity Detection Module

"Far IR" -- thanks, I wasnt clear and hadnt realised that not all IR sensors are configured to pick up a thermal image; I guess ideally it would need a "thermal image camera" type sensor to pick up the movement eg 

"LWIR MICRO THERMAL CAMERA MODULE; Lepton® " - but looks a little large. 

The pulse idea is really useful ; thanks

If you or anyone has identified any other models of these types of sensors  (or others) that you can recommend as small enough - I'd be very grateful for the advice.

Thanks

Nick 

  Are you sure? yes | no

ZoomZoomLuke wrote 01/02/2020 at 23:39 point

Interesting project Nick! I now have rather tired middle ear muscles :)

This is very new to me, I always thought that loud noise that I have been making in my ears since a child was the result of a larger muscle or muscles or my jaw or something, how fascinating! I can control my right muscle independently from the left but have yet to move the left muscle without affecting the right... more practice required.

While observing on the microscope I noticed the movement of the ear drum was quite substantial and wonder if an led and photodiode might be a more appropriate sensor in terms of power requirements, cost and simplicity. Have other sensor options been explored?

  Are you sure? yes | no

Nick GPx wrote 01/03/2020 at 20:43 point

Thanks. It becomes a bit of an obsession when you know you can control it! Dual control could really add to the potential . Constatin's comment (just after yours) actually outlines a lot of the thoughts already, & I'll post a comment back to him in a bit as well, 

I'd be really interested in how the LED/photodiode set up would work?

Thanks for your interest

Nick

  Are you sure? yes | no

ZoomZoomLuke wrote 01/05/2020 at 21:44 point

The LED (or maybe laser diode) / Photodiode setup would I presume consist of a light source, photodiode,
a trans impedance amplifier and a DSP circuit to detect the specific tone reflected back off the eardrum.

If an earpiece with this setup be developed it would have the form factor and power requirements of an in-ear hearing aid and could provide a wireless link via bluetooth.

You could possibly hack an existing hearing aid by replacing the mic with the amplified photodiode input. These modern hearing aids possibly may even already have the DSP capable of doing what's required. Many have bluetooth and debugging is possible (at least from my limited experience with doing analysis of Oticon configuration software).

  Are you sure? yes | no

Nick GPx wrote 01/07/2020 at 18:46 point

Thanks Luke (relying to your 2nd comment)

Really interesting re hacking existing hearing aids; that would get round the issues of suitable mountings/  anchoring the sensor in the ear canal & positioning for direct vision of the drum:

Would the LED/photodiode; 1x2mm : TMD2635 IR Proximity Detection Module be suitable for this hearing aid hack? 

Or the TOF sensor "VL6180X" that Constantin mentioned in his post (although rather larger at approx 2.8x 4.8mm)?   

Thanks again 

  Are you sure? yes | no

Steven wrote 01/02/2020 at 20:46 point

I'm one of your 17%  I have full control on the right side, and partial control on the left.  

  Are you sure? yes | no

Nick GPx wrote 12/29/2019 at 18:03 point

thanks for your support

  Are you sure? yes | no

Dan Maloney wrote 12/27/2019 at 17:13 point

That's fascinating, I didn't know anyone had voluntary control over the middle ear muscles. I knew there was involuntary control over them in response to loud sounds. Interested to see how this develops. Nice work!

  Are you sure? yes | no

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates