Close
0%
0%

PionEar: Making Roads Safer for Deaf Drivers

PionEar provides early warning to deaf drivers of an approaching emergency vehicle

Public Chat
Similar projects worth following
PionEar is a smart, small-sized sensor that helps alert hearing-impaired drivers in time to approaching emergency vehicles. It provides a bright visual alarm in the driver’s field of view, enhancing driving comfort and reducing the risk of an accident.

The PionEar sensor utilizes artificial intelligence to analyze external sound patterns. It leverages Syntiant’s TinyML development board to run machine learning algorithms in an extremely energy-efficient manner. This design allows for the sensor to operate for extended periods without charging, or alternatively, it can be powered via a solar panel, eliminating the need for recharging altogether.

Upon learning about the challenges deaf drivers face with emergency sirens, I reached out to the hearing-impaired community and discovered a positive demand for such a device. I aim to share this idea with the maker community and bring this device to real users.

The PionEar sensor concept is based on Syntiant TinyML board.  TinyML board is the ideal platform for building low-power voice, acoustic event detection (AED) and sensor ML applications. Equipped with the ultra-low-power Syntiant® NDP101 Neural Decision Processor, the TinyML board packs native neural network computation for the most demanding applications in the lowest power envelope.

The onboard microphone and BMI160 sensor enable easy configuration for any speech, AED or 6-axis motion-and vibration-related application. Trained models can be easily downloaded on the TinyML board through a micro USB connection without the need for any specialized hardware.

The below picture shows how the TinyML board is interfaced with other external circuitry. The external custom-designed PCB contains RGB LED for logo illumination, DC to DC power supply, and a phototransistor acting as a light sensor. The light sensor provides a signal (AIN) that is sensed by the TinyML board, which controls the brightness of the logo. This prevents the driver from being dazzled at night but enables good visibility in the daylight.

The high-temperature li-Pol battery cell (Zoncell LP602530T / 500mAh) is connected directly to the TinyML board battery connector. In parallel to the battery, a solar charger module AEMLION and solar panel IXOLAR SM101K07TF are connected as well. The whole system draws less than 2mA so the solar panel is able to deliver enough power to cover the power consumption of the system. This is valid in any daylight condition not necessarily in the sunlight. It is still possible to recharge the battery via a micro USB port which is present on TinyML Board.

I plan to reduce average current consumption even more by leveraging a motion sensor (MEMS accelerometer) and an additional MOSFET switch. It will sense if a car is moving – if not, it will cut off the power supply from the system. This feature aims to prevent battery discharge during periods when the car is idle and lacks adequate solar charging, such as nighttime or when parked in a garage. I'm confident that once this feature is in place, the PionEar sensor might operate without the need for any charging.

Fig. 1) PionEar sensor HW block diagram

In order to deploy the machine learning model into a TinyML board the Edge Impulse platform can be preferably used. Edge Impulse provides easy to understand interface where you can create and manage your datasets, extract features, train ML models, test and finally deploy into numerous of supported hardware platforms. The TinyML board is fully supported so I have leveraged this advantage.

First, it is necessary to have a suitable dataset of emergency siren sounds and other road noises. One can find many public datasets – I have used sireNNet (license: CC BY 4.0) as a basis for my ML model. I have used other datasets of noises and speech to mix with the original dataset. My dataset including the ML model is publicly available on the Edge Impulse platform – you can find it here. I expect, that I will further evolve it as I will continue with the prototype testing.

I have already tested the latest ML model in real driving conditions. It can safely detect ambulance sirens and exhibits a low amount of false positives when exposed to road noises while driving. There are still some weak points for example music when listening to the radio in the car. However, music was not part of the dataset because I expect that hearing-impaired people typically do not listen to the radio in a car 😊

Fig. 2) ML model accuracy for testing dataset

License_documentation.docx

List of SW sources and linked licenses

document - 14.05 kB - 09/24/2023 at 15:34

Download

TinyML_extension_LED_board_eeBOM.xlsx

Electrical BOM file for extension LED board

sheet - 11.12 kB - 09/24/2023 at 12:28

Download

PionEar_car_holder_adapter_v3.step

3D model of adapter for standard magnetic car holder used eg. for smartphones. This is needed to avoid blocking the microphone port.

step - 95.32 kB - 09/24/2023 at 12:09

Download

PionEar_box_solar_v3.step

Complete mechanical model of housing and internal parts

step - 665.34 kB - 09/24/2023 at 12:08

Download

ExtensionBoard_Schematic_V2.pdf

Schematic for extension board

Adobe Portable Document Format - 126.60 kB - 09/24/2023 at 11:58

Preview
Download

View all 9 files

View all 13 components

  • Project introductory video

    Jan Říha10/08/2023 at 18:59 0 comments

    I'm sharing a video where I'll introduce my project in more detail, discuss the motivations behind starting it, and share my plans for the future.

  • Manufacturability Assessment and Cost Estimation:

    Jan Říha10/08/2023 at 18:48 0 comments

    As I have indicated in my project entry video, I plan to turn the PionEar sensor into mass production. Of course, this can be done after the careful evaluation of the “market size” and annual sell volume estimation. This process is currently ongoing. However, I would like to demonstrate possible scenario in what way the PionEar sensor can be produced in larger quantities.

    First of all, it is necessary to realize that the current proof of concept prototype - PoC is mostly built on development boards. This is good for building the first concept, performing various tests and evaluations, or even providing it to the first customers to receive valuable feedback. Using the evaluation boards saves a lot of time that would be needed for the development of custom-made designs. However, it is not very effective in terms of cost and size/fit factor. For this reason, I plan to develop a custom-made PCB where all the electronics will be placed. This will greatly simplify the sensor assembly process as the wiring will be minimized to a few connections (Battery, solar panel) With a custom-made PCB one can also expect cost savings on the component/material level.

    The next part to consider is the manufacturing process of the plastic housing. Currently, the whole housing is made with a 3D printer. The 3D printing process is again very suitable for prototyping and product iteration. But with the current ease of access to 3D printing technologies and significant price reduction, it can preferably be used for low production volume as well. I plan to use 3D printed parts also for production in volumes of several hundred up to one thousand pieces per year. According to my investigation up to this volume, 3D printing will be cheaper compared to injection molding. I also plan to greatly simplify the process of creating the light diffuser for the illuminated logo. In larger volumes, I will very probably use dual-color 3D print (partially translucent filament for logo infill) instead of pouring the SLA epoxy as described in the build instructions.

    In terms of cost, I have evaluated the material cost for the current prototype and compared it with an estimated cost for a version with a custom PCB (calculated for 1000 pcs batch). As can be seen from the tables below, the cost has been reduced to approximately 35% of the initial value.

    In order to complete the cost of the product, the labor costs of the production process and the costs of transport and packaging must also be calculated. The final cost of the sensor is evaluated in the following table. If you're wondering how I derived these figures, they're based on a combination of my experience in electronics manufacturing, Prusa 3D Printing Price Calculator and a discussion with ChatGPT 😊

  • Test, before the road test

    Jan Říha10/08/2023 at 14:25 0 comments

    I've already shared a couple of videos demonstrating how I test the PionEar sensor and its machine-learning model performance in a car. I've captured some nice footage of it detecting ambulance sirens. Before I actually get in the car and spend time driving, I first run tests using a PC and YouTube. This initial testing provides valuable insights into whether it's worth testing in a real-world environment. Additionally, I want to showcase that the current machine-learning model is quite resilient to various traffic noises.

  • Housing material and model update

    Jan Říha10/01/2023 at 15:24 0 comments

    During the hot summer days, I realized that the standard PLA material might be problematic for use in car interiors. The temperatures can easily reach more than 70°C or even 80°C. For this reason, I decided to conduct a simple test in a real environment. One day, when the outside temperature reached 35°C, I left a couple of sensor housings on the car dashboard. The car was oriented in such a way that the dashboard was fully exposed to sunlight. In this test, I compared the currently used PLA materials with one of the PET-G samples that I had bought, as I liked its color as well. I was hoping that this material sample might be the right choice for this application.

    After approximately 2-3 hours of sunlight exposure, I checked the results. As expected, the PLA samples became very soft and lost their shape after I squeezed them in my hand. The PET-G material (Spectrum PET-G Stardust Blue) remained rigid, and I could not deform it after pressing it with my fingers. This test showed that PET-G filaments are definitely a better choice for such applications. I cannot exclude the possibility of repeating this test under more extreme conditions and perhaps choosing a different material again. But for now, it seems that PETG filaments are good enough for most cases.

    It is also worth mentioning that since the last time, I have updated the mechanical model of the housing. Now it is ready for mounting the solar panel. The whole box is now slightly bigger in order to fit the panel on the top of the housing. The build instructions have been updated accordingly.

  • Low power firmware

    Jan Říha08/28/2023 at 08:12 0 comments

    In this latest update, I'm excited to introduce a new firmware that prioritizes energy efficiency. It leverages the fact that the TinyML board has a neural network accelerator (NDP101) and host processor (ATSAMD21G18) as two separate devices. The NDP101 can process the input audio with extremely low power consumption. While NDP101 is processing the audio, the host MCU can be in sleep mode until the moment the NDP101 recognizes the desired sound pattern. If this happens, the NDP101 wakes up the host MCU via a dedicated interrupt pin. As the host MCU takes the largest portion of energy, this approach will dramatically reduce total power consumption.

    Before the low-power mode was implemented the total current draw was about 15mA. It's now been impressively reduced to under 2mA ! This current is low enough to power the whole device with a small solar panel (keep the battery charged). FW is available on GitHub – Please follow the readme file for further instructions. You can also watch the demonstration video on YT. 

  • Road tests

    Jan Říha08/23/2023 at 20:35 0 comments

    After updating the ML model, I conducted more comprehensive road tests. I captured some compelling footage that demonstrates how the sensor performs under real-world conditions. Particularly, the first scenario is a major concern for deaf drivers.

  • ML model update

    Jan Říha08/23/2023 at 17:37 0 comments

    Today, I want to provide an update regarding the ML model which I have updated. First of all, I will describe an issue with the old model. During the testing I noticed, that detection sensitivity is very poor when I drive a car at moderate/high speed. After analyzing this problem I have found, that this is caused by additional low-frequency noise generated by the car (engine, tire rolling, etc.), that masks the sound of sirens. 

    First I wanted to implement any kind of high-frequency/band-pass filter to analyze and evaluate just the frequency band where a siren typically emits a sound. But I found, that there is no easy way to make it with TinyML board or Edge Impulse studio.

    After this, I decided to create a new ML model that will "learn", that low-frequency noise in a car cabin together with siren sound should result in positive class detection. So I turned my current prototype into a sound recorder and recorded several hours of driving sound. After this, I mixed this with already existing siren sounds.

    I also used a new dataset for siren sounds: sireNNet. This Dataset seems to have better quality as the recordings are cleaner and do not contain any other (often very strange) type of sirens, that were present in the previous dataset. Even though this dataset is much smaller, the observed result in the real driving conditions was much better.

    For a better understanding of how I created the current dataset, please see the block diagram “Dataset_8_2023_diagram.pdf” which I share together with other files. I believe that this might be useful also for other people when trying to build any ML model that should run in a real environment.

    In the block diagram, you'll see that I use varying amplification levels for certain classes. I've discovered that this greatly helps the model in achieving a higher level of abstraction when recognizing sounds. Without this adjustment, the model frequently identified different classes (sounds) based on their average sound level rather than their frequency content.

    My current dataset is again shared on the Edge Impulse Platform here:

    https://studio.edgeimpulse.com/studio/267982

  • Li-pol battery safety

    Jan Říha08/21/2023 at 19:57 0 comments

    Despite the longer inactivity on my project page I have been still actively working on the project. One of the issues that I've been trying to figure out is the Li-pol battery safety concern.

    It's well-known that temperatures inside a car can soar, especially during the summer. Naturally, this raises concerns about whether the battery inside a sensor could overheat, leading to potential fires or explosions.

    After delving into the subject, here's what I've found: Standard lithium-polymer (Li-Po) batteries typically have a temperature range of up to 60°C for discharging and up to 45°C for charging. It's not hard to imagine a car's interior exceeding these temperatures. However, even if the temperature goes above these limits, a Li-Po cell remains stable. It's only when you hit the 130-150°C range that the cell becomes thermally unstable, which could result in a thermal runaway where flaming gases are released. But, realistically, reaching such extreme temperatures inside a car is quite improbable. The manufacturer's temperature guidelines seem to be more about minimizing cell degradation than they are about safety.

    Nevertheless, in the interest of maximizing safety, I looked into high-temperature batteries. Although there aren't many options on the market, I did discover that Zoncell produces high-temperature cells. These have a temperature range that extends up to 80°C for discharging (and 45°C for charging). Some might point out that 45°C as a charging limit might not be sufficient, especially since I plan to incorporate solar charging. It's worth noting that the lower charging temperature limit is primarily because of self-heating caused by the charging current. This is especially true for charging currents close to 1C or higher, where the heating effect is significant. However, based on my recent tests, solar panels can only deliver a maximum of 30mA, which is less than 0.1C. Such a low charging current won't result in substantial self-heating, so I'm confident that the true temperature limit is much closer to the 80°C discharge limit.

    So I ordered a few samples of Zoncell LP602530T (datasheet enclosed in shared files) and will consider their further use in the PionEar sensor.

  • Solar charging test

    Jan Říha05/25/2023 at 18:47 0 comments

    In the next iteration, I would like to integrate a solar panel into the PionEar sensor to avoid the need for charging. I did some research for suitable low-power solar charger modules with MPTT functionality. I discovered AMELION from @Jasper Sikken. In combination with IXYS SLMD960H09L solar panel, it seemed to be a good starting point. AMELION module is a great solution for low-power applications. It can still provide some current even in the indoor environment. I could measure the following charging currents:

    Indoor: 20 to 50 uA

    Outdoor (cloudy/shadow): 2 to 5 mA

    Outdoor sunlight: 25 to 30 mA

    These numbers seem to be promising. I think that behind the car's front window, there should be plenty of light to keep the battery charged - unless you park in the garage.

  • Test in the car

    Jan Říha05/23/2023 at 21:10 0 comments

    Usually, as a first test for audio event detection, I use YouTube as a source for various audible events. But the real-life test is irreplaceable. In the case of PionEar sensor, it was a bit tricky to catch the ambulance with sirens on and have a camera ready. Last time I spent 2 hours driving around the nearest hospital and wasn't able to meet any car with sirens :-) Today, on my way to work I was lucky and made a shot.

View all 10 project logs

  • 1
    TinyML and LED board wiring

    First of all, prepare the assembled LED expansion board and new TinyML board from Syntiant. You will need to make a small change on the TinyML board  - connect Vbat test point (battery voltage) to an unconnected (free) connector pin. This will serve later as a supply pin for the LED expansion board. Follow the wiring diagram as in the picture below.

  • 2
    Solar panel + charger and battery assembly

    Prepare another separate assembly where you connect the solar panel, AEMLION solar charger module, power switch, and Li-pol battery together. Follow the wiring diagram and picture for reference below.

  • 3
    3D print instructions

    Few instructions for the 3D printing of the provided housing. Preferably use PETG material for printing the housing and car holder because of its higher temperature resistance, which is advantageous for use in the interior of the car. For reference I have used: Spectrum PET-G Stardust Blue. Print both parts oriented by face on the SMOOTH print sheet (do not use textured sheets).

View all 11 instructions

Enjoy this project?

Share

Discussions

Marcela Krejzová wrote 01/16/2024 at 05:51 point

Jsem učitelka autoškoly pro neslyšící v Praze a ráda bych se svými neslyšícími klienty vyzkoušela vaši novinku. Učím v centru u Vinohradské nemocnice, kde sanitky několikrát denně potkáváme. Bylo by možné se nějak spojit a spolupracovat? Email mac.crazy@seznam.cz

Děkuji, Krejzová 

  Are you sure? yes | no

craig wrote 05/25/2023 at 03:51 point

I had been thinking about this sort of project for people in general and more particularly that it would warn if the emergency vehicle was approaching.

I would avoid using a battery powered device as they degrade and can become dangerous in a hot vehicle. May be less of a problem in the UK, but a major issue elsewhere. I don't think there would be a problem with the device being powered via the 12 Volt system plug.

  Are you sure? yes | no

Jan Říha wrote 05/25/2023 at 19:16 point

Hi, thanks for your comment. Ideally, I would like to deliver a device without any plugs and cables. Even though I understand that in the car you have a practically infinite source of energy :-) You're right about the Li-on/Li-pol limitation. For the concept, it is the easiest solution and I can live with it for some time. In case of wider use, I would consider using eg. Lithium ion capacitors:

https://www.digikey.cz/en/blog/lithium-ion-capacitors-can-help-you-provide-high-quality-power

Although it has lower energy density, it can still be a good choice when solar charging will provide a positive energy balance for most users.

  Are you sure? yes | no

nramkarran wrote 05/24/2023 at 16:39 point

This looks amazing! My main problem though is direction. I just can never pick up the directionality of the siren. My hearing issues are age related, so not total hearing loss.

  Are you sure? yes | no

craig wrote 05/25/2023 at 03:48 point

I don't think this an age related issue. It is a function of the sound isolation in modern vehicles, which makes it difficult to hear sirens and then to determine their direction. Direction is more difficult to determine as the sound can bound off building and leak into the vehicle from a location that isn't in the direction of the emergency vehicle.

A project like this example potentially can determine if the emergency vehicle is approaching or leaving via the Doppler effect on the pitch.

  Are you sure? yes | no

TheGrim wrote 05/23/2023 at 12:29 point

It's a cool concept. I like what I see. Cheers.

  Are you sure? yes | no

Jan Říha wrote 05/23/2023 at 21:26 point

Thank you!

  Are you sure? yes | no

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates