Speak To Me: A Button Box

Description

With my dad being 91 and losing short term memory, living in an assisted living facility about 5 hours away via ferry and road travel, I wanted something that would bring him back into the present.

I still use a DakCalendar app on a big TV to show date/time and slowly scroll through his week at a glance with a very big font, but with his macular degeneration he can't seeing complete letters and words.

He started going down to the front desk to ask when the next meal was and what day it was - and that could happen any time day or night. But it was his way of grounding himself.

This project has been built and tweaked over the last year as I try to deal with ideas, updates and new operating system releases on a Raspberry Pi.

The goal is for an ever present "button box" where a button press will tell him what day it is, what time of day it is and when his next meal is (or in the case of an early am button press - it also suggests he "should probably go back

Details

The Challenge

Find or build a low-cost self-help device to help an aging parent remove the frustrations from living alone with restricted abilities. There should be no need for great eyesight or short term memory, both skills that deteriorated with my Dad's normal aging. He wants to know what day and time it is, when his next meal is and when the care workers are arriving next.

The Solution

A button box with multiple buttons that can be pressed at any time of day or night that will tell him the date and time, when his next meal is and when he has any upcoming events. It will be tied to a Google Calendar so that when his online calendar is updated, the new events will play on the next button press.

We prerecord generic messages about care worker arrival times and use cron jobs to auto play the messages at the related times of day. The date/time messages will use Google's Text to Speech API.

Using a Raspberry Pi keeps the footprint and learning process to a minimum. Using Google APIs keeps the processing requirements down. This version can be run on on a Raspberry Pi Zero W.

I am also working on a standalone single button version only used for date/time and cron job alarms. Because it doesn't require a calendar, I'm hoping it can be be built as a portable box with UPS, using a local TTS product like PIPERor MIMIC3. This will require a Raspberry Pi 4B with 8GB Ram and 64 bit O/S.

The Speak-To-Me Button Box

This project uses a Google Calendar and Google Text-to-Speech to announce appointments and events on a scheduled or ad-hoc basis.

It is written with Python 3.9 scripts using the RPi.GPIO library

Examples of Usage:

When care workers are arriving
When it's time for next meal
Upcoming appointments or events coming soon
Current date/time and how long until next meal
Reading chapters from downloaded audio book with button to stop and then restart where it left off
Playing music files with button to stop and then restart where it left off
Downloading podcast on a schedule and playing on demand

Tools:

cron jobs for playing of prerecorded phrases
cron jobs to download podcasts on a schedule
rc.local and bash script to start button script on reboots
Python script stores meal times and meal description preset phrases

Wiring the buttons

This was a learning process for me. The buttons required 12V power and had to be wired to GPIOs for input to detect when they were pressed.

A final working circuit is shown below using the big buttons, resistors and a DC plug on a breadboard.

Here's the Falstad circuit diagram I used to make sure I got the 12V down to 3V before GPIO

When the button closes, it uses a voltage divider circuit with resistors to get the 12V down to 3V before to send a notice to the GPIO that its connected button has been pushed.

I made a video of an early version that reads aloud calendar events, next meal times, and play/stop/resume button for chapters of a book.

Future Changes:

Decided I'm going to need automated announcements for todays events - and possibly tomorrow. His confusion seems to be mounting when reading calendar on TV.

Some method of auto adding crontab announcements four hours and one hour before events might work. I'll have to find a way to auto-edit the crontab as well, removing announcements after event has passed.

One more ask from him is finding a way to hookup the audio book readings to his bluetooth headphones he uses for TV movies right now. I've had issues with Raspberry Pi and bluetooth in the past so that might be a big research project with much testing.

Files

crontab.txt

Example local crontab file that plays messages at specific times of day

plain - 1.96 kB - 05/24/2023 at 22:02

Download

play_prerecorded_phrase.py

Python script called by a local cron job to play your prerecorded alarm messages

plain - 640.00 bytes - 05/24/2023 at 22:00

Download

save_phrases.py

Example file you can use to create pre-recorded phrases into MP3 audio files for use in cron job alert messages

plain - 2.30 kB - 05/24/2023 at 21:59

Download

Release-History.txt

Summary of work done to date

plain - 6.91 kB - 05/24/2023 at 21:34

Download

speaktome-published.py

Speak To Me source code for python script

plain - 37.00 kB - 05/24/2023 at 21:33

Download

Components

1 × Audio Speakers 3.5mm auxiliary jack

1 × LED Button(s) Big colored LED buttons using 12V source

1 × Raspberry Pi Tested on Raspberry Pi Zero W and Raspberry Pi 4B

1 × Enclosure Box Picked up hinged one from Michael's Craft Store

1 × 12V 3A Power Supply for LED buttons 2.1 mm plug

Project Logs

Collapse

History of Changes

Don Mitchinson • 05/24/2023 at 21:35 • 0 comments

# Speak To Me #
# ***********
# TESTED ON: Python 3.9 on Raspbian Bullseye 64 bit (May 5 2023 release)
#
# Version 1.39 (May 20, 2023)
# Fix AM/PM reading for times between 4 and 6 PM
#
# Version 1.38 (May 14, 2023)
# Change bouncetime to 100ms
# Add 0.1 seconds sleep before checking input
#
# Version 1.37 (May 14, 2023)
# CHange bouncetime=100
# Debug statements for initial log file open
# pause slightly after reading log file close
# add sys.stdout.flush() after closing log file for read
# add TIME_PRESSED to check for elapsed time and ignore quick press
# set default start time to 0
# add delay after say_phrase before calling subprocess
#
# ??? use DEVNULL for stdin does it defaukt to subprocess.PIPE???
# ??? or use universal_newlines=True if it's a problem with ^M is lines
# change log file open to log = open(name, "w", 1) # line-buffered
# log.flush() in the main processes has no effect on the file buffers in the child processes
#
# Version 1.36 (May 13, 2023)
# Check for internet connected inside say_phrase(sayAloud...
# Play a prerecorded phrase when internet is down and don't exit program
# Pull everything out of function and put into GPIO callback button and it works
# add in folderName and description variables to GPIO if clause
#
# Version 1.35 (May 12, 2023)
# Test code using Red button to call playAudioFromFolder("news", "CBC WORLD NEWS")
#
# Version 1.34 (May 12, 2023)
# Set home_dir to script path
# Change name of Client Secret File constant to CALENDAR_OAUTH_FILE
# Test Green button to see if it will play from last spot stopped
# Attempted to use SIGKILL to stop sox process when program killed but was not saving Aborted message in log file
# Use SIGTERM when button pressed to retain ABORTED message in SOX Play log file
#
# Version 1.33 (May 10, 2023)
# Check test GREEN button using trim parameter with stop/start and different times
# Set SOX_CMD to full play path
# Remove PLAYING_AUDIO constant - use SOX_ID instead
#
# Version 1.32 (May 9, 2023)
# Add process ID to Audio Folder play function
# Merge today and tomorrow's events into one button
# Use another button for Audio Folder chapters
# Add global SOX_ID to Audio Folder function
#
# Version 1.31 (May 9, 2023)
# Test kill of subprocess when 4th button clicked
# Stop playback if button press causes Sox process to be killed
#
# Version 1.30 (May 9, 2023) - edited from v 1.27
# Add cbc news to 4th button
#
# Version 1.27 (April 20, 2023)
# Get start end dates collection before any announce events call

# Version 1.26 (April 6, 2023)
# Use shlex.split for popen args
# add delay after popen
# use creationflags=subprocess.DETACHED_PROCESS | subprocess.CREATE_NEW_PROCESS_GROUP (windows)
# start_new_session=True
#
# Version 1.25 (April 5, 2023)
# Use sox -d instead of play
# Change play command to use -d for sox device and - ta alsa to hide encode error
# hard code home directory instaed of os.getcwd to work with rc.local
# Tested use with rc.local and shell script
# Change say_phrase back to using play command
#
# Version 1.24 (April 3, 2023)
# Test version with cancelling GPIO inputs while button pressed
#
# Version 1.23 (March 29, 2023)
# Change kill process back to SIGTERM
# Split play command into array for all items separated by white space
# Change subprocess.Popen call to specify executable parameter
#
# Version 1.22 (March 26, 2023)
# Try to find out why process can't be killed
# Split command into arguments and try...

Build Instructions

Collapse

Preliminary Setup

Basic Hardware

Raspberry Pi with WIFI capability (tested on Raspberry Pi 4 and Zero W)
3.5mm Audio Speakers - can be run on USB speakers but trickier to setup
Breadboard
Resistors
12V power supply and breadboard power plug if using large LED buttons

Operating System

Latest 64 bit Raspberry Pi OS Lite (May 3rd 2023)
It will work with 32 bit Lite as well - watch for one path change in next step
Install using Raspberry Pi Imaging tool
Setup with local WIFI
Check that SSH is turned on using sudo raspi-config | Interfaces
Install script dependencies (see below)

Google Setup

Create a ".credentials" folder under your home folder on the Pi
Requires a Google account with basic Gmail address
Setup Calendar API (described in a following step)
Setup Google Text to Speech API (described in a following step)

Install Script dependencies

Update path to avoid pip installation warning messages

touch $HOME/.bash_profile
echo -e 'export PATH="$HOME/.local/bin:$PATH"' >> $HOME/.bash_profile
export PATH="$PATH: $HOME/.local/bin"

For 64 bit install, add libatomic.so library to path

echo -e 'export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libatomic.so.1.2.0' >> $HOME/.bash_profile
export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libatomic.so.1.2.0

For 32 bit install

echo -e 'export LD_PRELOAD=/usr/lib/arm-linux-gnueabihf/libatomic.so.1.2.0' >> $HOME/.bash_profile
export LD_PRELOAD=/usr/lib/arm-linux-gnueabihf/libatomic.so.1.2.0

Install Required Tools

sudo apt-get install python3-pip
/usr/bin/python3 -m pip install --upgrade pip
pip install setuptools -U
pip install --user --upgrade google-cloud-texttospeech
See what version of grpcio is installed in the success message like

"Successfully installed cachetools-5.3.0 google-api-core-2.11.0 google-auth-2.17.1 google-cloud-texttospeech-2.14.1 googleapis-common-protos-1.59.0 grpcio-1.53.0 grpcio-status-1.53.0 proto-plus-1.22.2 protobuf-4.22.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 rsa-4.9"

pip install grpcio-tools==#.##.# (where #.##.# from version above, eg 1.53.0)
pip install --upgrade google-api-python-client google-auth-httplib2 google-auth-oauthlib
pip install python-dateutil
sudo apt-get install sox libsox-fmt-mp3

Enable Google Calendar API

First we need to activate the Google Calendar API for a Google cloud project of our choice. Using my experience with this along with some Google Developer API docs and an online example...

Open Google Developers Console
From the top Project menu select Create project

If you haven’t created a developer account, you’ll be asked to

Agree to their terms of service
On top of page, click on top right button after “Start your Free Trial with $300 in credit…”

Enter Project Name – and if you’re in a Google Workspace enter your organization
Click Enable APIs and Services

Find and Select Google Calendar API
Click Enable
When the Google Calendar API is enabled and selected, click Create Credentials
Select User Type is External, choose User Data and click Next

Enter an App name is you want, no logo, User support Email, and Developer Support Email, then click Save and Continue.
Ignore scopes for now

Under OAuth Client ID set type to Desktop App, enter app name and click Create
This will generate your ClientID . Then click “Download JSON” and store the credentials file on your local PC. Move this file to a new “.credentials” subfolder on your Raspberry Pi and rename it “desktop-app.json”

Under Credentials | Create Credentials | API key

Edit the API Key and Restrict Access to the Google Calendar API

Discussions

Don Mitchinson wrote 07/20/2023 at 16:27

Woohoo! Just made finalist in the Assistive Tech challenge. Thanks for your support Hackaday and sponsors. Really looking forward to fleshing this out

Are you sure? yes | no

Don Mitchinson wrote 07/08/2023 at 15:03

Decided I need to work on a crontab option that automatically adds/edits/removes calendar events to the daily announcements.

He'll still have option to push button to get the events for the day and next day, but an automated announcement several hours down to one hour before will likely be a good addition for him. Confusion is something that is coming up more often now.

Are you sure? yes | no

Don Mitchinson wrote 05/24/2023 at 21:52

Currently testing a smaller more portable one function button box that could be used by the bedside to just alert him for time and next meal. It could also used be used as wake-up alarm for upcoming pills, dressing/undressing schedules (using crontab).

For next edition I'm planning on exploring other promising options to use a local TTS option without Google.

An offline calendar option will be tougher to incorporate unless I can build some type of local web server where events can be added to a locally installed calendar.

Are you sure? yes | no

Speak To Me: A Button Box

Description

Details

The Challenge

The Solution

The Speak-To-Me Button Box