Close
0%
0%

TapTDOA

Turn almost any flat surface into a sensor using cheap piezos and time difference of arrival.

Public Chat
Similar projects worth following
TapTDOA is a project to explore the possibility of making a generic tap interface controller.

Many of us live and work in places with large flat surfaces all around us. What if you could turn them into sensors? Turn a wall into a light switch or dimmer. Add a secret knock lock to a door.

Piezo elements make excellent contact microphones and small and cheap. By attaching them to the surface of a rigid material, sound waves traveling through the material can be detected, and with multiple sensors the waves will arrive at different times. The waves can then be filtered and correlated to find precise time offsets, and Time Difference of Arrival (TDOA) methods can be used to find a position given known sensor locations and the speed of sound through the material.

For practical applications, once the sensors are attached, training can be done to discover some of the material properties like the speed of sound through the material at room temperature.

Prior art / inspiration

Some prior art and inspiration comes from these excellent projects:

Hardware

The STM32f303RE was chosen for it's 4 very fast ADCs and included op amps with PGA. This reduces board complexity somewhat, and allows for more control programatically. This chip also comes on a very affordable NUCLEO dev board w/ included programmer/debugger!

Giant bag of mostly peripherals

Extensive use of DMA and peripheral features allows this chip to capture and store data in circular buffers without using any CPU, as well as using ADC channel watchdogs for threshold 'event' detection. The CPU only gets involved when one of the channels exceeds set thresholds.

            trigger   +------+ dma   +---------------+
            +-------> | adc1 +------>+  circ buffer  |
            |         +------+       +---------------+
            |         +------+ dma   +---------------+
            +-------> | adc2 +------>+  circ buffer  |
  +------+  |         +------+       +---------------+
  | tim3 +--+         +------+ dma   +---------------+
  +------+  +-------> | adc3 +------>+  circ buffer  |
            |         +------+       +---------------+
            |         +------+ dma   +---------------+
            +-------> | adc4 +------>+  circ buffer  |
                      +--+---+       +---------------+
                         |
                         |adc 1-4 watchdogs
                         +------------------->  captureEvent()

Once an event is detected, the capture continues for a bit less than the total buffer size. This allows both leading and following data to be captured for further analysis.

Piezo Mount.svg

SVG file for laser cutting Piezo mount

svg+xml - 10.53 kB - 08/24/2018 at 14:14

Download

  • Giant Calculator!

    Tom08/28/2018 at 02:20 0 comments

    We have kicked around many ideas for this from sports applications to a voting wall and of course combining it with video projection for fun game of whac-a-mole.    But today we are going a bit simpler and turning our lunch table into a calculator.

  • Sometimes It's Simple, Sometimes Not

    Ben Hencke08/24/2018 at 15:02 0 comments

    Look at this spectrogram. Just look at it.

    I never thought background noise could look so beautiful. Also blue fireballs of signal!

    It looks like there are two things going on here. There seems to be a bit of acoustic dispersion - the higher frequencies arrive earlier, and the low frequencies shift in time a bit. There seems to be a jump in the shift at the very low end through, and I still think those are possibly transverse vs longitudinal waves. 

    But if anything is certain, I need to filter to a smaller vertical slice to get a consistent time delta.

    Previous alignments were for equidistant sensors, so things aligned more easily. The various frequency bands hadn't phase-shifted apart. 

    So I tuned the FIR filter a bit, and I'm grabbing stuff right around 20Khz. 

    The top graphs are FFT of a before and after. The bottom are the waveforms before and after cross correlation alignment.

    As I move from GNU Octave to the STM32 where resources are tighter, this is lot to crunch across a huge sample. I'd like to keep the number crunching to a minimum, and I don't need/want any alignment of the echoes and other things that happen after the leading pulse.

    Using peak detection works remarkably well on the filtered signal, but not perfect. Getting a smaller window around the peak and cross correlating that seems to work well. 

    I tried filtering the full-wave version of the signal through a low-pass, but peak detection on just one half seemed to have similar results, and I plan to use cross correlation anyway as it is going to phase align the waveform phase better than anything else will.

    In most cases the peak-based alignment had the phase within 1 sample of what cross correlation found. You can see it fixed channel 4 (purple) above that was 1 period off.

    The mind blowing part is when you go back and look at the original signal (normalized, but otherwise unfiltered) and compare.

    When looking at the above, keep in mind that the FIR filter has shifted things on the bottom graph by about 400 samples.

    Now to port this into C using the ARM math libraries... 

  • Mounting the Piezos

    Tom08/24/2018 at 13:59 0 comments

    There are all sorts of ways you could attach the piezos but its really important that they be held consistently.    Just taping them works ok but the tape slowly gives up a little and that can have big effects on the signal.    This is what I have found works the best.

    Its a simple laser cut housing,  I used .25" acrylic but anything will work.  The housing is 1/2" deep and holds a 1" piece of  polyurethane that was also cut on the lase to the right diameter.  On the back side I have a simple screw terminal.

    This is held onto the surface with 3M Command removable strips.   The setup keeps the piezo firmly to the surface and still allows it to vibrate.

  • Cross (Correlating) the Streams

    Ben Hencke08/18/2018 at 21:04 0 comments

    With a bit of help from GNU Octave and an idea of which frequency bands are interesting, I can now play around with cross correlating the signals. 

    Here is the cross correlation output (xcorr) showing time offsets from negative on the left to positive on the right. 

    The interesting thing here is the positive peak. This indicates the delay where the signals most closely align. 

    To make things easier, I can make a function to find this delay:

    function ret = findDelay(a,b)
      [R,lag] = xcorr(a,b);
      [v,i] = max(R);
      ret = i - length(a);
    endfunction

    To get an idea of what that looks like when applied back to the signals, I can plot the unaligned and aligned signal by shifting the second by the delay.

    Looks pretty good!

    Now theoretically this will work great, but to calculate all of this on the STM32 will be a lot of number crunching. Each channel needs to be filtered, which is currently 6k samples per channel, and a FIR filter of sufficient size needs to be created. Octave has me covered there too, and this article has everything I need to know about designing these filters. It will be an accuracy/CPU time trade-off. 

    Next I need to cross-correlate the signals, though I don't really need or want the full result, I only need the peak and the delay of the peak. Still, that is a lot of vector dot products to calculate. Maybe I can cheat and check at a lower resolution, find a rough match, then fine tune it. 

  • Take It up an Octave

    Ben Hencke08/07/2018 at 16:33 0 comments

    With sensors mounted to an acrylic sheet, I get something like this:

    It looks like there is some noise, and a few frequencies going on in here. This is after a high-pass filter on the analog side around 1.6KHz. 

    One of the things I'm keeping an eye out for are different kinds of waves. There's transverse waves where the material wobbles up and down, the kind you'd think about with sound. With a sensor attached directly, it's possible to pick up longitudinal or compression waves that travel through the material. 

    Like earthquakes. The "P wave" travels faster and can give early warning of an incoming quake. If the signal has longitudinal waves, part of the signal will travel faster. Those should also have a higher frequency. Unless they are picked apart, I imagine they will mess with correlation. 

    So I wanted to pick apart the signals by filtering, but haven't done much digital filtering. After googling a lot, I found some FIR filter designers, but I wanted a way to quickly experiment and get a feel for what these are doing. 

    Enter GNU Octave.

    I found this absolute gem that has pretty much everything I need:

    https://www.allaboutcircuits.com/technical-articles/design-of-fir-filters-design-octave-matlab/

    With a bit of fiddling around, I can see different frequencies and pick them apart. This is way better/faster than anything I've done before.

    The blue line is the signal through a low pass, and the red line is after a bandpass for higher frequencies but excluding high frequency noise. It looks like there are 2 signals in there, and the higher frequency signal does look like it travels faster than the low frequency signal. Does this mean I found longitudinal/compression waves?

    I'll have to see what I get after correlating the signal components. Ch1 and 4 should be about the same, and ch2 and 3 should be about the same as these where intentionally arranged in 2 equidistant pairs from the tap source. 

  • Tiny Oops

    Ben Hencke08/06/2018 at 14:01 0 comments

    I didn't find any reference schematic for USB in the datasheet or reference manual, so I just kind of wired it up and hoped for the best. D+ to DP D- to DM, seemed logical enough. There's a Cube checkbox for phy, so I figured the chip must have whatever it needs internally. 

    Plugged it in, and nothing... Not even a blip on the console.

    You'll have to forgive my optimism, the only other project I've done with USB directly (instead of through UART to USB) was on one of those "Blue Pill" boards, and it just worked. 

    So I googled around and found a schematic for it, sure enough...

    Turns out I need an external pull-up resistor on D+

  • ​Thinking about cross correlation...

    Ben Hencke08/05/2018 at 23:44 0 comments

    Wow, the arm_math CMSIS stuff is really nice. Tons of stuff there, and the source is available, and even optimized to use SIMD instructions when available.

    I think I'm going to go with a vector dot product, and slide that down a smaller window based on where I think the opening waveform is. I don't have the memory or need for the full correlation waveform, just peak correlation detection, and I think I can use a smaller window around where the ADC WD tripped, then scan down the other channels looking for peaks in correlation where the start of the waveform matched. 

  • Beautiful Data With... Icicles?

    Ben Hencke08/05/2018 at 17:06 0 comments

    The piezo goes through a very simple high-pass filter into the op amp w/ PGA set to 16x, then on to the ADC. 

    (piezo is connected to the terminal on the right)

    This should give me about 1.5Khz cutoff to filter out the DC and any bass. I don't yet know exactly what kind of frequencies I'm dealing with yet. What does a 'tap' look like? The connectors (J4, J9) let me probe and experiment with values if necessary. 

    So with everything hooked up, lets see what we get:

    Beautiful! But... WTH are icicles doing in the data? Also my circular buffer math is off, signal starts about 1/16th of the way in and wraps around.

    It's on all channels at the same time! 

    I double checked power supplies. Maybe something on the chip (USB perhaps?) was activating on some interval causing load spikes on the power supply? I thought I added enough decoupling caps by following the datasheet!

    Maybe some kind of ADC spitback even though it's fed into the op amps? Did I create some kind of resonant circuit with the piezo?

    I tried disabling everything else the chip was doing. I tried changing the PGA gain. 

    Scoped everything (power, piezo inputs, pre op amp, post amp, ref voltages). I didn't find any causes, my poor scope (owon DS7102V) is either blind to the icicles or the probe suppresses them. 

    Mystery Fix

    I experimented a bit. Adding a 10k resistor in series with the piezo fixed the weird icicles, but I don't yet know why. 

    Noise is now about 2-3 LSB and some of that might be background audio getting picked up, or stray EMI on the piezo leads. Will have to check that out at some point, but the above looks plenty fine for tap detection and correlation!

    Maybe Related


    The PGA has a fixed GND reference, so I can't just float 1/2 of 3.3v on the op amp. Instead I use the DAC to output a reference voltage that once multiplied by the PGA is about 1.65v. By changing the DAC and PGA at the same time, I can keep close to ideal range and still have programmable gain control.

    So theoretically glitches on the DAC or extra buffer could be the cause, but scoping this didn't show noise.

  • Ode to HAL

    Ben Hencke08/05/2018 at 16:45 0 comments

    Oh HAL,
    We're not a match.
    I tried to be your pal
    but you make my head scratch.
    Your code is a puzzle
    and the interface is jagged.
    While CPU cycles you guzzle
    my interrupt is run ragged!
    But the hardware is elegant,
    you time wasting contraption.
    I've discovered registers most relevant
    now I'm done with this abstraction

    Hello ARM World

    ST has made getting in to ARM pretty easy. I had some false starts with NXP, but these STM32 NUCLEO dev boards with ST-Link debuggers and free high quality tools are really nice.

    These STM32F303RE are pretty complex chips with a lot going on, a lot of registers and clocks and all kinds of rules. Here's the clock tool in STM32CubeMX, a chip configurator and code generator:

    (it doesn't all fit on my screen)

    Using the tool definitely saved a ton of time. It has a lot of sanity checks, and generates code in either HAL (Hardware Abstraction Layer) or LL (Low Level). HAL code has a ton of extra sanity checks and promises portability across different chips should the need arise. LL on the other hand is pretty bare bones and gives light wrapper functions around poking registers.

    This can be paired with your favorite IDE, or if you want something that is free and takes less time to set up you can use SW4STM32 which based on Eclipse. I used Eclipse in a previous job quite extensively, so I'm comfortable enough with this setup. Its all backed by the all powerful GCC, and integrates well with the ST-Link debugger via Open OCD. The only trick is that you have to fiddle with the Cube settings and import the generated project in just the right way. 

    Getting Some Data

    So the basic capture architecture is this:

                trigger   +------+ dma   +---------------+
                +-------> | adc1 +------>+  circ buffer  |
                |         +------+       +---------------+
                |         +------+ dma   +---------------+
                +-------> | adc2 +------>+  circ buffer  |
      +------+  |         +------+       +---------------+
      | tim3 +--+         +------+ dma   +---------------+
      +------+  +-------> | adc3 +------>+  circ buffer  |
                |         +------+       +---------------+
                |         +------+ dma   +---------------+
                +-------> | adc4 +------>+  circ buffer  |
                          +--+---+       +---------------+
                             |
                             |adc 1-4 watchdogs
                             +------------------->  captureEvent()

    The timer tim3 triggers all 4 ADCs simultaneously, which then use DMA to write into a circular buffer. Meanwhile ADC watchdogs keep an eye on measured ADC values and interrupt when a value is outside of a predefined range. 

    That all happens without any CPU, its all just wiring peripherals and DMA together until the watchdog triggers. 

    I want to capture some leading data because I won't know when exactly the signal starts. It is likely going to precede the trigger as a weaker signal. Put another way, I want to stop capturing data into the circular buffers after (BUFFER_SIZE - X) more samples are taken, where X is the amount of leading data to capture.

    So I thought, I'll just add an interrupt handler to tim3 and decrement a counter. Tim3 is already triggering for each sample. The CPU runs at 72MHz, and the ADCs can run up to about 5Msps. Initially I'm going for 1Msps. That only leaves about 72 cycles between samples, not enough to really do much, but surely enough to decrement a counter, right?

    Enter HAL. HAL provides an abstraction layer, but also tons of checks. But it's not written in C++ templates or even a bunch of #ifdefs, its mostly just readable C code. And abstraction. So the interrupt fires, and the interrupt vector looks like this:

    The TIM3_IRQHander is of course provided by the generated HAL code. This calls HAL_TIM_IRQHandler, passing in a pointer to the tim3 hal data structure. But of course HAL_TIM_IRQHandler is generic for any timer, not just tim3, and not even just your flavor of tim3. So it runs through every possible reason any timer could ever interrupt. To be fair its only about 140 lines of code, but here is where my 'user' function gets called from:

    It's buried deep enough...

    Read more »

View all 9 project logs

  • 1
    Piezo Prep

    Soldering to the piezos:  The Piezos are a brass disc with a crystal on one side,  the brass is the '-' and the white part, the crystal is the '+'.  You can buy them with the wires already soldered in place or you can do it yourself.   

    You should use a thin flexible wire.  Soldering is pretty strait forward but caution should be used when soldering to the crystal,  I used pretty low heat, 250deg C.   Hotter temps can damage it.  For the brass part turn your heat up because it is such a heat sink,  I use 350deg C for the brass.

  • 2
    Cut and Glue Your Parts

    Laser cut 4 sets of Piezo Mount.SVG out of any non conductive ~.25" material.  

    These get stacked up and glued together,  I used acrylic and Rez-N-Bond.

  • 3
    Put it all together

    Next feed the leads thru the holes and solder on the screw terminal.

    Super glue the terminal black down.

    Cut a 1.5" circle from 1" thick polyurethane foam and cut a slit in it for the wires.  

    Place the foam in the housing and the piezo on the foam.

    Add the 3M Command Strips.

    Make 4

View all 3 instructions

Enjoy this project?

Share

Discussions

David wrote 10/29/2021 at 21:39 point

Hi Ben,

Great project here, congratulation and thanks for sharing. 

I am working as well on a similar set up for a interactiv board game but facing also lack of precision in the detection with use of a threshold.  Could you tell a bit more about the filtering and cross correlation methods that you used here?

  Are you sure? yes | no

Elliot Williams wrote 08/27/2018 at 14:04 point

Hi Ben!  Great project.

I tried, and failed, to do this about ten years ago, so I'm totally in your corner hoping for victory.  Basically, you've already discovered most of the stumbling blocks that I did -- diffusion in the medium makes for smearing of the signal, which in turn makes filtering hard / necessary. My experience was also that peak detection didn't cut it.  

Worse, it's probably dependent on the material that you're sensing through. I found different diffusions between a solid cherry coffee table and the chipboard tables at the hackspace, for instance, that lead me to give up...

So I really like your current approach -- hard bandpass filter and cross-correlate to get the time difference. Even if you don't know the speed of 20 kHz through maple, you can hope that it's roughly constant in different directions within the same table.  

If this comment is short on helpful details, I hope it makes up for it with warm thoughts and encouragement!  :)  

PS: Coming to Supercon this year?

  Are you sure? yes | no

Ben Hencke wrote 08/27/2018 at 14:41 point

Thanks for the wishes :)

For practical applications, I'm thinking once the sensors are attached, there's a few things that can be done to discover some of the material properties like the speed of sound through the material at room temperature. For a tinkerer, this might mean configuring/tuning parameters, or in some kind of more polished device a setup/learning step. (I should add this to the description!)

Yes, see you at Supercon!

  Are you sure? yes | no

Mick wrote 08/23/2018 at 23:06 point

Could be useful for building an electronic shooting target. 

  Are you sure? yes | no

Daniel wrote 08/22/2018 at 17:21 point

One of the older projects uses an Arduino plus a couple of op-amps to amplify, rectify, and trigger-on-threshold its 4 piezos, two for each axis (X-Y). That way the microcontroller is only measuring pulses. Don't know how to compare with a full DSP solution.

  Are you sure? yes | no

Ben Hencke wrote 08/23/2018 at 04:39 point

The problem I ran in to with a threshold detection system was that often one "pulse" gets missed (or early), then you have to wait for at least half the waveform period for the next pulse, which is eons compared so the speed of the signal through a solid. since some sensors are further away than others, the chance that closer sensors trigger on an earlier wave is fairly high. 

  Are you sure? yes | no

ohnoitsaninja wrote 08/21/2018 at 18:49 point

I recently did something similar, starting out with a grid of piezos attached to the underside of my desk.

I quickly moved from piezo to MPU6050, they are cheap(under 80 cents for a chip+breakout from china) and quite superior to piezos for this application, I use just the Z axis and 3 mpu6050s.

I use a LPF on them and then an envelope follower, effectively measuring the amplitude. I feed the amplitudes into a neural network and train it by tapping in a circle around the sensors. 

It works with enough resolution to reliably trigger 8-16 samples when using the device as a midi drum instrument. 


  Are you sure? yes | no

besenyeim wrote 08/22/2018 at 09:46 point

Interesting. Is it publicly documented somewhere? I wonder how placing items on the desk affects the signal. Or leaning on it stresses the desk a little, maybe changes the time of sound propagation. Does the controller have to adapt?

  Are you sure? yes | no

ohnoitsaninja wrote 08/22/2018 at 17:06 point

No it's not documented and leaning on the desk doesn't do anything. I suppose you could create a condition putting things on the desk that would cause it to misidentify knocks but it doesn't seem to have much of an impact. I imagine if you were more dependent on specific frequencies instead of just looking at signal amplitudes you'd have more issues with this. 

  Are you sure? yes | no

rafununu wrote 08/09/2018 at 10:22 point

I find this very interesting.  I think 3 sensors (an equilateral triangle) will be enough to decode any position. Let's find applications, maybe in the electronic music domain ?

  Are you sure? yes | no

Ben Hencke wrote 08/11/2018 at 13:09 point

Yeah! maybe this can drive a drum machine or something for the upcoming music segment!

  Are you sure? yes | no

kelu124 wrote 08/06/2018 at 17:40 point

Superb project! Will keep an eye on it =)

  Are you sure? yes | no

Ben Hencke wrote 08/11/2018 at 13:09 point

Thanks :) comments, questions welcome!

  Are you sure? yes | no

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates