Microphone arrays that use multiple microphones are widely used, for example, in smart speakers to detect the direction of the speaker. In recent years, inexpensive microphone array boards that incorporate MEMS microphones onto PCBs have also been released. The device we created combines a commercial microphone array with a high-performance microcontroller, and is an AR device that can display real-time estimated sound source positions on video images.
The mechanism for estimating the sound source location:
Microphone arrays are geometrically arranged on a plane. Sounds emitted from the source propagate through the air and reach the microphone array.
When the sound reaches the microphone array vertically to the PCB where the microphone array is mounted, the timing for the sound wave to reach each microphone is almost the same. On the other hand, when the sound wave arrives from a direction with a certain tilt angle relative to the microphone array, there is a time difference between when each microphone detects the sound.
By detecting and analyzing this time difference, it is possible to estimate from which direction the sound came.
This device is equipped with a camera, a microphone array, and a microcontroller that can also be used for edge AI.
The microcontroller analyzes the sound information obtained from the microphone array and converts it into a color map, which is then overlaid on the image data captured by the camera and displayed on the back display in real-time.
Since the color map of the estimated sound source position and the field of view (FoV) of the camera are significantly different, the camera lens is replaced with a wide-angle lens and the color map is cropped to adjust the display image.
This project can be used to visually confirm the location of sound sources for people who are deaf or hard of hearing.
It is useful not only for people who are completely deaf, but also for those who have lost hearing in one ear and cannot detect the direction of sound. It was developed as a portable device powered by four Ni-H batteries of size AAA.
In addition, it can also be used for cicada catching by people with normal hearing. With this device, the sound of cicadas can be visualized, so even children who are not good at catching cicadas can always find the location of the cicadas that are chirping.
The device was renamed to "Semi-tori Meijin" in Japanese, which means "Cicada-catching Master", because it makes catching cicadas so easy.