Close
0%
0%

Bluetooth RPM Letterboard

Capacitive touch, field programable whiteboard giving distinct multi-sensory fingerprints to letters in RPM therapy for non-verbal autistics

Public Chat
Similar projects worth following
To be used during RPM communication / therapy sessions by non-verbal autistic individuals and their facilitators to enhance the sensory fingerprint of letters and symbols.

A mobile & programmable whiteboard with capacitive touch sensing acting as a bluetooth keyboard to use the Text-To-Speech capabilities of an iOS or Android device. It also provides on board visual (LED) and audio (piezo) feedback of touched letters and symbols as they are drawn on the whiteboard. Three keyboard layouts are possible: large ABC, medium ABC and QWERTY.

Why, What and How

Contrary to what they may seem on the outside, non-verbal autistic individuals are intellectually intact.  They may know exactly what they want to say, but may be unable to say it. They may know how to play a game, but sit motionless, or simply rock back and forth. As much as someone with autism might want to, it may seem impossible to turn a thought in his head into speech from his lips, or to convince his/her hand to pick up a pencil—in other words, bridging the gap between intention and action can be an impossible task.

Some research shows that providing gentle physical cues, or “proprioceptive feedback,” can help individuals with autism complete motor movements. A tap on the shoulder or elbow is often enough to help someone translate an intent to move into physical motion.

Soma Mukhopadhyay created a method called RPM to open a window into their minds where the teacher or facilitator learns how to access the open learning channels (auditory, visual, tactile and kinesthetic) of the student and adapts the sessions accordingly.  This allows them to communicate, slowly and painstakingly, by pointing at a letterboard to spell their thoughts as well as to expose them to information and interaction that they would otherwise never have.

Eventually, a few students can move to something like an iPad (LA Times: In the ‘silent prison’ of autism, Ido Kedar speaks out). The transition is not easy nor straightforward. There are a lot of sensory variables involved. Practice, patience and consistency are only a small portion of what's required.

To assist in that transition, I built this letterboard that acts as a Bluetooth keyboard, sending its output to an iOS or Android tablet which provides spoken feedback of each character and word through its Text-To-Speech capabilities, also providing visual feedback on the letterboard thru LEDs on each letter and a piezo for clicking sounds (and some Easter Eggs we'll cover later). 

The hope is that, by adding visual and audio feedback to its shape and form, each letter acquires a distinct sensory fingerprint that may create stronger bonds to bridge the gap between intention and action.

It would have been orders of magnitude easier to create a simple App on iOS/Android (and there are many commercially available apps for assistive communication), but some autistic individuals perceive the light emitted by a screen as sensory noise, and others associate the screen of a tablet to game time (i.e. YouTube, games, etc.), disrupting their efforts at purposeful communication.

Elements in the Letterboard

The letterboard consists of a 22x14 matrix of “Touch Electrodes” (TE). Each “TE” in turn, consists of a 2x3 matrix of copper pads. The pads are small enough so that a finger will touch at least 2 of the copper pads, providing the row and column that allow for its usage as a 2D touchpad (albeit with a 22x14 “pixel” resolution. Since this is only for letters, this is fine).

Top right corner of the front side of the PCB showing 5 columns x 3 rows of TE's. The (22x14) x (2x3) = 1848 copper pads are 4mm square

Each row and column of the TE’s connect to one channel of the three 12-Key Capacitive Touch Sensor Breakouts (Adafruit MPR121) for a total of 36 channels (22 columns + 14 rows = 36 channels). These 3 breakout boards use I2C to connect to the MCU and each has a distinct I2C address. The breakout boards are added to the PCB using SMD header pins (datasheet)

The MCU is an ESP32 (Adafruit ESP32 Feather). Because the MCU’s USB port will also serve as the letterboard’s battery charger port, it is added to the PCB using through-hole pins (to create a solid union that can withstand the multiple connect/disconnects of a USB charger).

The letterboard also has a 10x6 LED matrix interspersed in the TE matrix. The LEDs are controlled by two 74HC595 Shift Registers...

Read more »

RPM GEN3 ABC.zip

All files for enhancements and corrections as listed in Log#2

x-zip-compressed - 13.76 kB - 05/12/2023 at 23:38

Download

RPM Gen 3.kicad_pcb

KiCad 6 PCB

kicad_pcb - 2.37 MB - 05/08/2023 at 23:06

Download

RPM Gen 3.kicad_sch

KiCad 6 Schematic

kicad_sch - 798.08 kB - 05/08/2023 at 23:07

Download

RC3.ino

Main program, contains the setup() and loop() functions

ino - 4.30 kB - 05/08/2023 at 22:55

Download

Declarations.h

All variables, constants & #includes, all grouped by function

plain - 5.15 kB - 05/08/2023 at 22:55

Download

View all 10 files

  • 1 × Adafruit ESP32 Feather V2
  • 3 × Adafruit 12-Key Capacitive Touch Sensor Breakout
  • 2 × onsemi 74HC595 Shift Register
  • 2 × 0805 0.01uF SMD Capacitor
  • 60 × Reverse Mount SMD LED 1204 PCB Type

View all 9 components

  • Log #2: Improvements after field tests

    Pedro Martin05/12/2023 at 21:33 0 comments

    After some field tests, I have made some changes. 

    1. The Post-It Whiteboard & System Capacitance

    I discovered that the Post-It Whiteboard cover over the copper pads (i.e., over the Touch Electrodes) had a couple of disadvantages:

    • Its thickness allows for less than desired capacitance from the touch electrodes, requiring a very high value in charge and time to the capacitive sensors which in turn generates more capacitance noise. 
    • The idea of just erasing a keyboard layout and drawing a new one sounded (and still does) very good: lots of flexibility depending on each user’s need. The problem is that a touch keyboard means touching erasable ink, which means letters survive very little before requiring redrawing. 

    Because of these two effects, I decided to, for now at least, go for a single (i.e., fixed) keyboard layout printed in self-adhesive vinyl (which has less thickness and thus allows for better capacitance performance).  

    It is self-evident that a 2D matrix of copper pads is unnecessary if there is a fixed number of symbols of the touch keyboard (i.e., I could use instead a large copper square per letter instead of a matrix of copper pads). But for now, I´ll continue using the existing matrix until I come up with a way to allow for multiple layouts. 

    The fixed layout I chose was Large ABC, so the Function Switches also changed: the Phrase ON/OFF Text-to-Speech switch is no longer applicable. Instead, I’m using that switch to turn ON/OFF the Easter Egg songs (see below).

    2. Ground Bounce

    Because of the ground bounce effect on the Function Switches (mentioned in the previous log entry), using interrupts meant having to trap false positives in code. So, in this iteration, I’m using digital reads instead. The main program loops fast enough to make it unnoticeable. 

    3. Non-blocking Code in Tunes Playback & Easter Eggs

    Playing of short tunes for startup and layout configuration is a direct use of Robson Couto's program, which uses a delay() statement to play each note. This forced me to play only short excerpts of the songs. I also wanted to add Easter Eggs: when the user types the keyword of one of the songs, playback begins. The songs are (keyword in parenthesis):

    • Super Mario (mario)
    • Darth Vader (galaxia)
    • Pink Panther (pantera)
    • Happy Birthday (cumple)
    • Ode to Joy (alegria)
    • The Lion Sleeps Tonight (leon)
    • Brahms Lullaby (dormir)

    For these two reasons I changed Robson Couto's code to be non-blocking (identifying during the whole program when a note is being played as well as if a tune is in progress). 

    Now at startup a random short song is played (Super Mario, Darth Vader or Pink Panther) and 7 songs are ready to be played if discovered! Playback can be halted by typing "zz" at any time. 

    4. Persistent Capacitive Noise

    Even with the new self-adhesive vinyl cover and the new config values for the capacitive sensors, there is still noise that can generate false touch positives.

    I added a routine to trap this noise (or most of it). There are (rare) cases when these false positive capacitive sensor touch registers enter some kind of loop, in which case after trapping this condition based in a determined number of iterations, I now reset the three cap sensors breakouts. 

    The origin of this noise remains a mystery to me. Ground bounce? EMI? Voodoo?

    5. Bluetooth Reconnect in iOS

    iOS has some limitations, perhaps by design, when it comes to Bluetooth reconnect: if the user turns off the BLE radio in the letterboard (by turning off all Function Switches) after it had been connected (paired) to iOS, it won’t reconnect again when turned on.

    To fix this, I added an ESP.restart() statement when this happens to force a reconnect, identifying the reason for reboot to avoid regular startup songs and other regular startup functions. 

    6. Cap Sensors Calibration

    I changed the cap sensors configuration values after several calibration...

    Read more »

  • Log #1: Initial results after first release

    Pedro Martin05/08/2023 at 23:53 0 comments

    This is the third iteration at building a usable tool for RPM communication. They have evolved in functionality and complexity:

    And as in every version, there have been lessons to be learned:

    • The SPDT switches are affected by ground bounce. When using interrupts, the Interrupt Service Routine (ISR) is triggered continuously when more than 2 pins are LOW. The issue is currently managed in code and could be further mitigated (albeit not eliminated) by not using interrupts. Must read more on decoupling and possibly test using Wokwi.
    • The PCB can benefit from using 3 distinct types of ground (for analog, digital, and power circuits) with their respective cap separators.
    • The MCU's antenna is placed over the ground plane which can badly detune the antenna and impair the WiFi / Bluetooth performance. Beginner's luck: I placed the MCU over headers, creating a 2mm air gap that mitigated (but not eliminated) the issue.
    • Increase Copper Weight of Outer Layers to possibly increase capacitance of the touch pads (albeit nearly double the cost if 2oz copper weight), Look at Trace Resistance Calculator from AllAboutCircuits,
    • Use a connector instead of soldering the battery cables directly to holes in the PCB: with a little manipulation, it breaks off.
    • The smallest SMT were the caps @ 0805. Consider using bigger components since we don't really have am issue with available real-estate on the PCB.
    • ESP32 Usable Flash Memory: the ESP32 development boards from non-Espressif manufacturers usually can't use the full on-board memory because their partition schemes on Espressif's Arduino ESP32 core are limited. A quick (manual) fix is to put a partition.cvs in the sketch: it will be picked up instead of the one in the menu. Adafruit has "...a PR to Espressif to fix the partition scheme for 8MB. It may take a while for them to review and merge...".

View all 2 project logs

  • 1
    PCB - Design & Build

    KiCad & PCB

    Being my first PCB ever, I started from scratch with KiCad Like a Pro 3rd edition in Udemy which gave me a basic understanding both of Kicad 6 and of PCB design.  The course author also has a playlist on his YouTube channel with examples and key terms.

    Some additional sources:

    Capacitive Touch Sensing

    At the same time, I researched about capacitive touch sensing, starting with main concepts, common challenges and basic guidance, with focus on touch surfaces (as opposed to touch buttons) in self-capacitance sensors.  From Microhip's Applications Notes AN2934 page 13 (Capacitive Touch Sensor Design Guide). 

    After this basic knowledge, more specifics are found in NXP's AN3863: Designing Touch Sensing Electrodes, specifically:

    • Item 3.1 Traces, page 4 for clearance, width & routing
    • Item 3.2 Electrode Pattern Design, page 6
    • Item 3.3 Electrode Placement, page 7 
    • Item 3.4 Correct use of Planes, page 8 
    • Item 3.6 Virtual Ground and Considerations in Battery Operated Applications, pg 11
    • Item 4.5 Electrode Patterns: Touchpad, page 25

    The following image from KiCad shows the front of the PCB with its 22 x 14 TEs, each with 2x3 copper pads.  All pads are 4mm square, with 0.8mm separation if within a TE and 1.6mm between TE's.

    Letterboard's Touch Matrix

    The letterboard´s touch matrix operates with a single-finger touch, so it's designed to have a minimum of 1 and a maximum of four Touch Electrodes (TE) touched at any one time and always at least one row and one column. Any other case is filtered out in code (i.e. when multitouch or touched at the edges)

    There are 3 possible touch cases (pads are identified in the KiCad PCB file with Cxx and Rxx for Column and Row number):

    1. Intersection of 4 TE's (in red)
    2. intersection of 2 TE's (in orange)
    3. Intersection of a single TE's row & column pads.

    This layout required custom PCB symbols. I started learning it from atomic14's KiCad Tutorial: Custom Symbol, Footprint and 3D Model and then from my own discussions in the KiCad forum to use SMD pads as touch sensors with vias

    The letterboard has two kinds of TE's in the matrix: labeled EVEN and ODD for the number of the copper pad that's connected on the top layer and the copper pad that's connected using Via's in the bottom layer. 

    To keep distances to a minimum, odd columns connect to the top MPR121 and even columns to the bottom MPR121 with the rows connecting to the middle MPR121. This can be seen on the Touch Matrix portion of the Schematic:

    Letterboard's LED Matrix

    I got the basics from The Engineering Mindset's - LED Circuit Design (i.e. forward current & voltage, voltage drop, resistors, etc.) and from DroneBot Workshop's : Shift Registers with Arduino.

    Then, to control a LED matrix (10x6 in my case), I consulted Mike Cook's LED Matrix Workshopprotostack's: Introduction to 74HC595 Shift Registers, and, Jegqamas': Control 8x8 LED matrix with two Shift Registers.

    Armed with this knowledge, I chose NXP's 74HC595 SMD shift registers (datasheet) and Inolux's reverse mount SMD 1204 Amber LED (datasheet).  The MAX7219 could be an option (see mdraber's: Controlling 8x8 Matrix with MAX7219), but it's price point made me go with the 74HC595's. 

    PCB Manufacture

    I designed some small tests and sent them to JLCPCB (Manufacturing Capabilities) and then send the final files which triggered a discussion with their tech support people: Hot Air Solder Leveling (HASL) finish has a lower cost but would not be recommended for small copper pours.  Instead, I ordered Electroless Nickel Immersion Gold (ENIG) finish. See picture for examples of potential bridging issues:

    A comparison of HASL vs ENIG will show advantages and disadvantages of both. For a table comparing most finishing options, see here.

    Finally, I had to learn to do SMD soldering, for which I started with Electronoobs' SMD Soldering Tutorial and EEVblog's SMD Stencil Reflow Soldering Tutorial. A very therapeutic experience I must say. 

  • 2
    3D Printed Enclosure - Design & Build

    Once I had a fully built and functional SMD soldered PCB came the turn to learn Fusion360, for which I started with wermy's Crash Course, then took Designing for 3D Printing with Fusion 360 in Udemy, then practiced with Product Design Online's various videos. Needless to say, this is like painting or writing: It will require practice for the rest of my life. :)

    I sent for manufacture in JLCPCB a resin version and a filament version. Interesting to see the differences in texture and weight.  For future 3D printing outsources, Craftcloud is a great place to use.

    Resin resulted is a better finish, but a much higher weight and less precise dimensions. The filament enclosure was printed as separate parts and then glued together.

    To cover the PCB copper pads (i.e., the "Touch Electrodes") and create the visual user interface, I tested various options and settled on Post-it Dry Erase Whiteboard Film Surface, which allows to draw different keyboard layouts in the letterboard, and being Post-It, can be safely removed in case something comes up.  See below the bare PCB with exposed copper pads and covered with Post-It Whiteboard.

  • 3
    Software Considerations

    Software

    I started this project using Visual Studio Code for editing and the Arduino IDE 1.8 for compilation & upload (by declaring "Outside Editor" in the Arduino IDE's preferences). Now the Arduino IDE 2 makes this unnecessary (and it does have many useful features).

    MPR121

    Having gone through Adafruit’s learning guide (a good starting point) for their Capacitive Sensor Breakout Boards, I reached the limit of what can be done with the Set Thresholds public method of the MPR121 library

    I created a program for calibration of individual electrode sensitivity using baseline filtering, touch filtering & debounce, and the MPR121's autoconfigure feature, all during program execution, thus avoiding the need to recompile & reload with each scenario being tested.

    Its objective is finding the optimal calibration parameters to be later implemented in the appropriate production program. I posted the summary of the MPR121 capabilities, the NXP application notes, the changes made to the MPR121 library in a tutorial on the Arduino Forum. The code is on Github.

    Interrupts

    Reading the SPDT switches could be done in the main loop(), but I wanted to use Interrupts (to learn about them and to avoid using loop() cpu time, although now I see it is overkill, but that's what makes it fun!). 

    I went through Function Pointers in Gammon's Forum, but my comprehension ability hit a wall. Passing arguments to Interrupt Routines was the solution (the forum discussion includes advanced topics like compile-time recursion and lambda functions both of which I did not use). Also, useful to consider as consultation material, the Espressif Docs on GPIO: Interrupts.

    BLE Keyboard

    The only ESP32-BLE Keyboard library I've found is T-vK's. To minimize memory usage, it can use NimBLE-Mode albeit with some mods to compile in Arduino. Some tinkering is necessary to compensate the speed at which iOS can receive and Text-To-Speech characters and words. 

    Also, the library handles only standard US keyboard, to special characters (like "ñ") have to be sent using the equivalent keyboard character (like ";"),

    Debug Output

    A nifty method I found for managing debug output is creating a DEBUG variable and redirecting the Serial.print() commands to the bleKeyboard (for wireless serial outout), to themselves (for standard serial monitor) or to null (for production versions).

    Permanent Data Storage

    To store the selected configuration parameters, I use Espressif's Preferences library (unique to arduino-esp32, the replacement for the Arduino EEPROM library). Works best for storing many small values, rather than a few large values (in that case, better to use LittleFS as file system library instead). Don't miss Random Nerd Tutorials' take on the subject.

    LED Matrix

    This being a 10x6 matrix (columns 1 thru 10, rows 1 thru 6) managed by two 74HC595 shift registers (see schematic), specific data bytes had to be assembled for the different letterboard configurations (small ABC, large ABC and QWERTY) using one or two LEDs per letter for the shift out operations. 

    These two guides helped in understanding Serial to Parallel Shifting-Out with a 74HC595 and the advanced IO function shiftOut().  

    The data bytes are assembled in Declarations.h as seen here:

    Letterboard Configuration Mode

    When the user sets all of the SPDT 5 switches to OFF (i.e., no letter, no word, no phrase output and no LED nor Buzzer feedback), the system enters config mode. 

    At this point, all touch detection and management routines are focused on the three possible letterboard configurations: touching anything on the rightmost third of the board (column > 14), sets layout #3, touching anything on the middle of the board (columns > 7), sets layout #2 and touching anything on the first third of the board sets layout #1, saving these changes in permanent memory via Preferences.

    Program Structure

    All code is developed using functions (subroutines) with descriptive names and thoroughly commented to assist in understanding the code.  This allows for a short and simple main loop():

    There are 5 files (I name them .h to bypass the alphabetical order assigned by the Arduino IDE if they were all named .ino):

    • RC3.ino (i.e. Release Candidate 3) is the main program and contains the setup() and loop() functions.
    • Declarations.h has the variables, constants & #includes, all grouped by function.
    • MPR121.h is the modified & simplified version of the standard MPR121 library contained now in a single file (i.e., no.cpp file)
    • fCore.h contains the core functionality routines (programming mode, read & output touched data, manage config switches & interrupts)
    • fSupport.h contains the supporting functionality routines (buzz, LEDs, symbol tables, battery).
    • tunes.h contains the melodies and routines used in the config mode and at system startup to identify which mode is selected. The tunes functions were created by Robson Couto

View all 4 instructions

Enjoy this project?

Share

Discussions

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates