Cool concept. Would love to experiment with the concept, but too bad your parenting it and it's not open source.

Thanks for your comment. Sorry for my delay in replying. The concept is OpenSourced here and so available for any one to gain from it’s use as an assistive technology switch or gamers switch, I sincerely hope it helps at least someone communicate.

I developed this for Assistive technology users, but then realised its a useful new Human Computer Interface & hence the patent and trying to get big tech companies to incorporate it into off-the-shelf earphones and hearing aids for general use for everyone;  which would have the advantage that assistive technology users could  get the benefit of mass produced/ high tech & cheap general consumer devices.

Hi Nick,

Very interesting!

Last year I built a prototype using sound. It took a lot of time to tweak because of the noise surrounding a contraction but now I am able to form words on a screen.

Your method seems more solid and indifferent to noise. Very cool!

(link with video https://geromy.com/tensor-sympathy/ )

Thanks - your project looks great. Your "radar" selection of letters looks a much faster selection basis than some of the assistive technology keyboards. Have you thought of suggesting it to some of the assistive technology companies eg Smartbox (Grid3) as there may be users out there who would gain from that as an input option?

Tnx Nick.  Yes, the rotor works quite fast, especially after some practice. Im considering to develop it further. The code is available on Github. Might work well with Earswitch too.
Best of success!

Thanks Jerome

Sorry -I'm not familiar with GitHub or code in general!- is it possible to use a keystroke input as a switch for the rotor (as thats the output that I'm triggering with the Earswitch ---- or alternatively iSpy can execute an .exe program file

Thanks

Nick

It would be possible to use a keystroke as input but the code must largely be rewritten. Or alternatively, start from scratch and import the 'rotor' functions from the existing script.  And yes, the application can be exported as an executable for Windows or Linux.

thanks Jeroen.  I’ll have a look at that 

All the best. Nick

We all know the sound, no one knew it's a muscle contracting. Thanks for that info.

I'd also suggest to explore other less power hungry sensors:

- EMG (sense muscle activation)

- Impulse response change of the ear (microphone)

- Sound of the muscle (maybe the mic hears the same as we do)

- Time of flight distance sensors

- Project Soli or other radar sensors (in pixel 4 smart phone)

- LSS by Rambus (tiny camera with no classic optics)

(Last two aren't that accessible, but maybe they like that unexplored HC interface idea so they give you a sensor or SDKs)

Many ideas pop to my mind: morse code translator (inverse babelfish), skip songs/mute/take calls wearing ear pods, hearing aid detecting seizures e.g. epilepsy.

Awesome work.

Thanks Constantin. That's a great summary of potential sensor modalities.

I guess the important qualities are that its small/ cheap/ not susceptible to "noise" from speech/external noise/jaw movement/ and is specific to TT movement 

-EMG:   I dont think it would be possible as it would need an electrode attached to the TT which is the other side of the eardrum

-tympanometry (used to detect compliance changes of the eardrum in response to sound); needs a sealed ear canal and constant emitted sound

-microphone ; risk of  triggering by external noise/ speech etc 

       (I think probably the sound of the TT contraction will be minimal; sound heard is due to the movement  directly propagated to the cochlear by the TT moving the ossicles)

-time of flight sensors-

Is this similar to laser dopplar vibrometry/ optical coherence tomography- which have been used to detect drum movements? Are there any small enough to fit within the ear canal?

-Soli; radar sensors

I did try and contact their project team but didn't get any reply and did not pursue this as I felt the health implications of radar close to the brain / inner ear might cause concern

-LSS by Rambus 

Sounds fascinating - not come across this before but their website suggests that its no longer in production?

Also

--Ultrasound 

- IR camera

Benefit of not needing light source (though I cant find one thats small enough?)

LED and photodiode

- suggested by Luke (comment below)

I guess the benefit of CMOS cameras is that they are small enough / relatively cheap (although the small 1mm sq ones aren't!), and easily available. Also an option of a fibre optic camera would also work.

I'd be really grateful of your ongoing thoughts

Thanks

Nick

- A potential TOF sensor could be the "VL6180X". I somehow remember a 3x3 depth sensor chip, but couldn't find it. With 1D depth value this is probably not going to work, as you'd need to calibrate via the orientation of the ear bud (pointing at the edge, that moves, to get a difference in depth), 3x3 would be able to autocalibrate, i guess.
- having in mind it might be used in parallel with music, optical seems the way to go

- i'd check out the LED/photodiode method (IR SMT proximity sensor), as from the videos i see, the change in brightness might be sufficient, maybe add additional photodiodes to scan different sections of the ear.

- light fibers may be a good way to get the light in/out the narrow ear channel in the prototyping phase

- For IR (Near IR) you still need a light source. If you mean far IR, you don't, but then you left 'cheap'-land. Maybe, really maybe, you can hack a PIR sensor (motion sensor) to see a change. but they aren't too small and far IR needs special optics.

- Pulsing the LED in an adequate frequency (Nyquist–Shannon sampling theorem: minimum double  the frequency/speed you can use the TT) you should be able to pull down power consumption. But that already 'optimization'-land :)

All the best for your endeavor.

Thanks again Constantin

I cant see how to reply below your second post - so have added this here:

ToF sensor "VL6180X" looks amazing -- it seems this is a miniature version of the lasar dopplar vibrometry thats used in some of the research papers to detect ear drum movement; & even with a single point measurement may give enough detail if the sensor can be reliably directed down the canal at the end of an earplug/phone.

LED/photodiode method (IR SMT proximity sensor): again looks really hopeful - particularly the miniature 1x2mm : TMD2635 IR Proximity Detection Module

"Far IR" -- thanks, I wasnt clear and hadnt realised that not all IR sensors are configured to pick up a thermal image; I guess ideally it would need a "thermal image camera" type sensor to pick up the movement eg 

"LWIR MICRO THERMAL CAMERA MODULE; Lepton® " - but looks a little large. 

The pulse idea is really useful ; thanks

If you or anyone has identified any other models of these types of sensors  (or others) that you can recommend as small enough - I'd be very grateful for the advice.

Thanks

Nick 

Interesting project Nick! I now have rather tired middle ear muscles :)

This is very new to me, I always thought that loud noise that I have been making in my ears since a child was the result of a larger muscle or muscles or my jaw or something, how fascinating! I can control my right muscle independently from the left but have yet to move the left muscle without affecting the right... more practice required.

While observing on the microscope I noticed the movement of the ear drum was quite substantial and wonder if an led and photodiode might be a more appropriate sensor in terms of power requirements, cost and simplicity. Have other sensor options been explored?

Thanks. It becomes a bit of an obsession when you know you can control it! Dual control could really add to the potential . Constatin's comment (just after yours) actually outlines a lot of the thoughts already, & I'll post a comment back to him in a bit as well, 

I'd be really interested in how the LED/photodiode set up would work?

Thanks for your interest

Nick

The LED (or maybe laser diode) / Photodiode setup would I presume consist of a light source, photodiode,
a trans impedance amplifier and a DSP circuit to detect the specific tone reflected back off the eardrum.

If an earpiece with this setup be developed it would have the form factor and power requirements of an in-ear hearing aid and could provide a wireless link via bluetooth.

You could possibly hack an existing hearing aid by replacing the mic with the amplified photodiode input. These modern hearing aids possibly may even already have the DSP capable of doing what's required. Many have bluetooth and debugging is possible (at least from my limited experience with doing analysis of Oticon configuration software).

Thanks Luke (relying to your 2nd comment)

Really interesting re hacking existing hearing aids; that would get round the issues of suitable mountings/  anchoring the sensor in the ear canal & positioning for direct vision of the drum:

Would the LED/photodiode; 1x2mm : TMD2635 IR Proximity Detection Module be suitable for this hearing aid hack? 

Or the TOF sensor "VL6180X" that Constantin mentioned in his post (although rather larger at approx 2.8x 4.8mm)?   

Thanks again 

I'm one of your 17%  I have full control on the right side, and partial control on the left.  

thanks for your support

That's fascinating, I didn't know anyone had voluntary control over the middle ear muscles. I knew there was involuntary control over them in response to loud sounds. Interested to see how this develops. Nice work!