Close
0%
0%

SonoSight: The AR glass for hearing impaired

SonoSight uses the perceptual phenomenon of Synesthesia to map 3D sound as visual elements through AR.

Similar projects worth following
Inspired in part by the amazing work of David Eagleman with the Sensory Substitution wearable vest, We were inspired to come up with a wearable AR glass for hearing impaired which can project physical sounds to holographic projections through the magical technology of Augumented Reality. Our hopes is to come up with an HUD - Heads Up Display which can help the hearing impaired identify important sounds both indoor and outdoor. The HMD, through the advances in mobile connectivity use real time voice to text translation.

Problem:

• There are over 43 million worldwide with a total hearing loss.

• The Cochlear Implant is expensive an requires a surgical procedure to setup

• Sound location awareness is not provided by existing devices and requires a special hearing dog to aid the user identify important sounds around them. Hearing dogs are expensive and the training between user and hearing dog extensive.

• They often require high precaution while venturing outside due to the lack of sound location awareness

• Most products in this domain seem to target the speaking and communication aspect of sounds and not the general day to day context provided by the ear.

Background:

I, Manoj Kumar along with my partner Vignesh have always found assistive technology devices to be interesting due to the real impact the devices have on the people using them. I admire the artistic potential of modern technology coupled with elegant design. Sensing in particular draws my attention due to feeling of augmentation of my existing perception. The visually impaired live a normal life by embracing the additional information provided by their remaining senses. After seeing a talk from David Eagleman on his hearing assistance vest, I wanted to work on adapting positional sound to an Augmented Reality headset.

Solution:  
SonoSight- The AR glass for hearing impaired

(Solution image, Sketch)

Our solution is an affordable Augmented Reality headset with multi-microphone setup to assist the hearing impaired user identify various sources of sounds around them in an unobtrusive and intuitive manner. SonoSight uses a low cost holographic projection setup based on the DLP2000 projection engine. The various sound sources would be identified using acoustic triangulation and mapped accordingly while taking the user's position and orientation into account. We propose the use of FFT to determine the various frequency bands to identify the source of sound and to isolate speech. Other than the sound source awareness conveyed to the user, We propose near real time speech to text translation by sending an audio signal to the google speech API which can help the user assimilate information from people who cannot use sign language. We aim to go beyond simple voice to text translation while we use voice for communication in daily life, Hearing goes beyond that to things we take for granted. It gives us spatial awarness and often alerts us to things in the environment (Eg: Doorbell, walking along a road or being called by someone from behind)


Design Goals:

  • Real-time voice to text: Provide automatic speech recognition for voice around the person once the person enters the sphere of normal hearing and provides quotations for environmental sounds. We might even be able to provide translations for multiple people if all goes well.
  • Localize sound: The visualizations should provide clear and minimal information about the location of the various sounds around the person.
  • Glanceable: The directional information should be easy to-understand at a glance.
  • Responsive: The visualization should be real time and respond to the user moving their head.
  • Coexist: The visualizations should work in tandem with the hearing impaired person's own techniques like lip reading and should be harmonious and adaptable.
  • Spherical sensing: The visualizations should provide a spherical spatial acoustic mappings of information. 

Block Diagram:


(Block Diagram of the final system)


The components which would make up the initial system are:
1) DLP2000 projector system
2) Beagleblone Black
3) BNO055 IMU
4) 4x Microphones (I2S or Analog)
5) USB Wifi Adapter

Holographic AR Projection:

For our holographic projection system we went with a beam splitter assembly, With the DLP microprojector behind us. The prototype had to be confined to this tight space due the space constraints of the beaglebone cape. We have been in talks with supplier who can hook us up with a compact SPI protocol DLP2000 module including the optical assembly. ...

Read more »

  • 1 × DLP2000EVM A 99$ 20-lumens microprojector for the AR headset (http://www.ti.com/tool/DLPDLCR2000EVM)
  • 1 × Beaglebone Black The ever classic amazing linux SBC
  • 4 × ADMP401 MEMS Microphone Breakout MEMS microphone for sound location trilateration
  • 1 × USB WiFi dongle for Beaglebone Cloud based real time voice to text
  • 1 × Dual USB powerbank To power the setup

  • Holographic Projection System

    Vignesh Ravichandran10/15/2017 at 17:37 0 comments

    For our holographic projection system we went with a beam splitter assembly,  With the DLP microprojector behind us. The prototype had to be confined to this tight space due the space constraints of the beaglebone cape. We have been in talks with supplier who can hook us up with a compact SPI protocol DLP2000 module including the optical assembly. 

    But without further ado here is the build log of our custom projection system for the HMD.


    (Figure: Manoj wearing the AR HMD setup)

    (Figure: HMD in action)

    The HMD did work well with the right angle beam splitter assembly (Despite some ghosting issues). The ghosting issues however dont show up when viewed in first person by the user.

    (Figure:Side view of the projection setup)

    Now over to how we did it, We used some very simple components to pull this off. 

    1. Polycorbonate sheet (3mm thick) (15cm*8cm) 

    2. DLP2000 projection cape with beaglebone black

    3. Double Corrugated cardboard box

    Now over to the actual nitty gritty build process, We take the Polycardbonate sheet and we cut it to a 15cm*8cm rectangle using a strong pair of shears. 

    We also need to make a hinge mechanism to adjust the viewing operation of the device. For this we use a CD case as a hinge. We bind it to the cardboard cutout using some E7000 adhesive. 

    The beaglebone black with the projection unit and LiPo boost circuit is placed on the top of the cardboard box. A convex lens from the google cardboard setup is used to bring the focus near to the viewer.  I'll share a bit about the optics in a later post but for now, Here are some images and demo of our first version.

    (Figure: Setup with the holographic projection system.)

  • Automatic Speech Recognition

    Manoj Kumar10/01/2017 at 12:49 0 comments

    Speech recognition is the process of converting an acoustic signal, captured by a microphone to a set of words.  Some speech recognition systems use "training" where

    an individual speaker or user reads text or isolated vocabulary into the system.  Voice User Interface are included in Speech Recognition off late in all smartphones with applications like

    voice dialing (e.g. "Call home"), domestic appliance control, call routing (e.g. "I would like to make a collect call") , search, simple data entry. With recent efforts in multi layer deep learning the accuracy has gone way up and is now offered as a service.
    Image result for google voice recognition block diagram

    (Figure: Block diagram of typical speech recognition software)

    This application is based on using the Google voice and speech API’s which google provides for free. The voice is recorded using the Microphone. Then using Google speech API’s it is converted to text. The text is compared with previously configured commands in
    configuration command file. We also set several control command keywords which would control the interface. If the text matches with any of the commands, thus it performs the specific task. After finding the required commands the bash command for particular task is executed.

    (Figure: Our implementation for the voice to text)


    To setup speech api, an account is created with the google cloud and a json file is obtained which is an authentication file. To export the authentication file the following command is used . 
    export GOOGLE_APPLICATION_CREDENTIALS "pathname"

    export GOOGLE_APPLICATION_CREDENTIALS "pathname"
    

    Once installation is done, recognition can be obtained by simple api calls to the cloud. 
    --For examples please have a look at this. ----

    Since the recognition we need is a real time which is like listening to a audio and simultaneously it should be able to update. There should be some way of getting the inputs from the mic and storing it and sending it to the cloud in a proper way. To do this Pyaudio module is used. 

  • Augumented Reality: A primer

    Manoj Kumar09/30/2017 at 18:52 0 comments

    Augumented Reality is all the rage today thanks to the constant battle between the giants Apple, Google and Microsoft. Despite each of them trying to come up with their own AR glasses along with startups like META there is still no wide market AR glass. While starting this project we looked for affordable compact AR displays that were ready to buy and we found none. So thanks to the work chip manufacturers like TI are putting in, We can make our own AR glass. But before we get into the nitty gritty of AR, Let's discuss AR.

    Augmented reality in it's simplest definition actually a mixture of real life and virtual reality, A mixed reality. Augumented Reality is a gateway to making all our cyberpunk wishes a reality by projecting it to our retinas. AR has already found a loyal following from artists, designers and gamers. With a bigger push it is sure to reach a wider audience with day to day consumers. The push is on it's way, The reason why no big name consumer lever AR glass is out is because of the cost involved with the nascent technology. Microsoft Hololens costs 3000$ (Granted it comes with a micro PC in the headset) and the standalone Meta dev kit costs 1500$. The big giants are hard at work with the photonic lightfield optics technologies the AR break has yet to arrive yet. 
    Image result for augmented reality projection system glass(Figure: AR glass projection system)

    While low cost options would enter after the high end ones come out, We needed to get a budget AR system out. Somewhat like a godsent, Texas Instruments got out a 99$ DLP pico projector beaglebone cape. 
    Image result for dlp lightcrafter display 2000 evm(Figure: DLP pico projector)

    By making a custom optical system, We could channel the DLP2000 projection to Augumented Reality display and possibly even stereo 3D?



  • Origin of Idea

    Manoj Kumar09/30/2017 at 15:20 0 comments

    I came across this idea after seeing amazing assistive technologies being developed for visual impairment. I was also inspired by the amazing work Dr.David Eagleman did at Stanford on Synesthesia. 


    Synesthesia is a unique condition a few people have around the world by simulating  two or more sensory perception and ties one to another. (Eg: Hearing colors, Visual tactile). While I found the whole concept very interesting and appealing, I saw some potential in the way we could treat assistive devices. While people with sensory disability in sight, hearing or smell often find it difficult at start a lot of them begin to rely on their other senses to guide them through daily life. People with visual impairment often rely frequently on sound to assist them in crossing a road or using smell to identify people they regularly meet. A recent research found using Magnetic Resonance Imaging that other sensory processing regions in people afflicted with blindness were more active and showed evidence of remapping (link). I personally found the concept to be beautiful and representing the resilience of the brain. 

    When I conversed about it with my friends Vignesh and Bala, We eventually struck on an idea to represent external sensory input in sensory impaired people in a way providing Artificial Synesthesia. The concept while appealing from a neuroscience perspective also had wide implications for people with such impairments. We decided on developing an artificial synesthesia system for the hearing impaired by mixing external sounds and visual stimulus. The reason was that existing efforts which took a similar idea in a different perspective.  David Eagleman's group had designed a vest to solve a similar problem by combining external audio signals with mechanical actuators to give a tactile feedback. 


    VAC

    (Figure from Scott Novich and David Eagleman)

    While we did love the concept and the amazing work behind it, We wanted to take a new approach which would combine Augumented Reality based overlay with external sensors. 
    The benefits of such an approach are:

    1) Ability to provide quasi real time voice to text service around the hearing impaired
    2) Lower cost due to the advent of low cost DLP projectors
    3) Additional services provided by the AR glass
    4) Information overlay would be intuitive and would require limited training.

    While there are concerns already about the ethics of cochlear implants in the hearing impairment community, We want to make it clear that we see our device as an addition to amazing deaf community and their intricate sign language. Assistive technologies often walk on a tight rope so as to not take away from the amazing culture the communites have built and we would be willing to hear feedback to this end on how we can adapt in addition with it.

View all 4 project logs

Enjoy this project?

Share

Discussions

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates