Audio beat detection - reliable options?
Simon Merrett wrote 08/24/2019 at 21:42 • 3 pointsI'm working on a charity project to provide haptic feedback to a deaf person in sync with the beat of a music play list (songs played one after the other, probably mixed).
I'm looking for a reliable beat detection routine/algorithm from a mic + preamp (with agc) input. The mic can be right next to the speaker if necessary as we can use an rf link to the user's haptic actuator.
I'm open to hardware from teensy, ESP32s and SAMD51s and down. If we have to use a smartphone or raspberry pi, we will. But I've tried several real-time beat detection approaches (fft/fht, dsp low pass and bandpass filters, autocorrelation) and nothing seems to work reliably for every beat or even most beats. Filters that focus on drums fall over when the rhythm is in other instruments, etc.
The problem I now have is that with no clear leader amongst the approaches, I don't know which one to persevere with. Does anyone have a go-to real-time beat detection system for a range of genres? I'm not talking classical music, there will always be clear rhythm, but it might not be drums.
The other problem I have is that if we miss a beat we want the system to put one in. We can keep track of the beat with a moving average filter but would a kalman be better for estimating the timing of the next beat?
Thanks
Discussions
Become a Hackaday.io Member
Create an account to leave a comment. Already have an account? Log In.
In the bad old days of Disco Light systems, I worked on several systems that just took a line in from the mixer and used a low pass filter, rectified it and fed that to a comparator to trigger the light system sequencer. It never worked very well on something as predictable as Disco.
Are you sure? yes | no
I've used this library in Pure Data before for beat detection. https://aubio.org/
Not an off the shelf solution, but it is open source and in C so you could take a look at how they did it at least.
Are you sure? yes | no
@Daren Schwenke thanks, it is relatively clear as to what is going on in the source and I wouldn't have thought to look there!
Are you sure? yes | no
Simon Dixon's (and perhaps other contributors') Beatroot might contain some good nuggets but I'm really hoping for someone who has implemented something similar to pop up and testify as to what works in realtime on more portable hardware platforms.
https://www.eecs.qmul.ac.uk/~simond/pub/2007/jnmr07.pdf
Are you sure? yes | no
I think you don't actually need beat detection. You want haptic feedback. So maybe just use lowpass filter then rectification, then straight to vibration motor? Then you don't have to sync anything and deaf user will feel all beats and even how long or energetic they are. If you have motor, you need to rectify signal, if you use small coil and magnet, you could just send lowpass filtered signal and user would feel vibrations directly. It's much simplier than beat detection and conveys more information than on/off.
Are you sure? yes | no
If I understand you correctly, you're suggesting something similar to the filtering I have already tried. It is OK at buzzing with the music but when I put earplugs in, it seems highly arrhythmic for songs without a VERY strong bass beat. It is the implementation here: https://web.archive.org/web/20170617105835/http://damian.pecke.tt:80/beat-detection-on-the-arduino
He was specifically trying to filter out everything except the strong bass beat at live electronic music concerts, so his approach is excellent for that application.
Thanks for the suggestion. Please come back if I have misunderstood.
Are you sure? yes | no
Yep, it's exactly what I've suggested. Thanks for responding.
Are you sure? yes | no
@Simon Merrett
Sorry, my software does not use microphone sampled audio.
There is no beat synchronization from my software. It just reads information from a MusicXML file.
If you really want to go with the option of using a microphone and real-time beat detection, then, your best bet will likely be streaming audio and using machine learning to identify a beat from drums, guitar, and other musical instruments.
https://medium.com/@awjuliani/recognizing-sounds-a-deep-learning-case-study-1bc37444d44d
https://www.quora.com/Music-How-can-machine-learning-be-used-to-analyze-music
There might be a trick you can do where the microphone is used to recognize a song and then the program in the device will just produce preset haptic feedback for the specific song recognized.
Are you sure? yes | no
Your suggestion is fascinating: to use the mic in and XXX (loads of signal analysis) to get a match to a normative spec for the music. I would also imagine that you could then continue to use the processed mic in signal -- weakly accurate that it might be -- to create a 'beat locked loop' with the normative music spec. So you get the clean rhythm info from the spec, and the realtime synchronization from the environment.
Are you sure? yes | no
Thanks! That sounds like a good idea.
Are you sure? yes | no
Thanks - like Shazam to songbpm.com and then a rough sync to the current track? I'll keep it under consideration but it sounds like a lot of dependencies on a fast connection to the internet and several APKs that I wouldn't know where to start with.
Are you sure? yes | no
Yes. Something along those lines.
This may help. https://github.com/spotify/echoprint-server
Are you sure? yes | no
I am willing to help you on this project if you decide to include me in it.
I work with projects involving electronics and software related to audio.
You can use my free open-source application.
https://github.com/adct-the-experimenter/MusicXML-to-Hex
In my opinion, the goal of providing haptic feedback to a deaf person in sync with the beat of a music playlist can be achieved without real-time beat detection.
You can more easily just use MusicXML files or decode audio information and convert it to information for haptic feedback without real-time detection.
Are you sure? yes | no
@Pablo Antonio Camacho Jr. That's a very kind offer, thanks. I took a quick look at your repository and have some questions. I don't think a source file for the music will be available to the user/device, so a microphone will be the only input. Do you think we can use microphone sampled audio with your software? Also, how do you synchronise the beat from your software with the track that is playing?
I should explain that the deaf person will be coordinating body movements to the music (in this case an exercise class) along with hearing people who will be moving to the music.
Are you sure? yes | no