Close
0%
0%

communication through gesture

A trigger is sent to a communications device when a hand moves above an adjustable threshold of acceleration

Similar projects worth following
This project enables people with physical disability to communicate through gesture using BBC microbits. One microbit is worn on the wrist (the transmitter) and a second is connected to a laptop or communications device (the receiver).

When a gesture is made, the motion is detected by the microbit on the wrist. This microbit uses its radio to signal the microbit connected to the laptop.

The microbit attached to the laptop then sends a trigger signal through the USB cable connected to the laptop. This signal is detected by a program running on the laptop which generates a simulated keystroke. This keystroke controls other software running on the laptop, such as communication software to create speech.

Please see a video of hand shake being tested at Beaumont College, Lancaster, UK here:

hand shake testing video

Project details can also be found on my website at:

hand shake project site

Setup instructions are on my website at:

hand shake set up instructions

The latest hand shake system uses the BBC micro:bit. One microbit is worn on the wrist and a second is connected to a laptop or communications device. When a gesture is made, the motion is detected by the microbit on the wrist. This microbit uses its radio to signal the microbit connected to the laptop. This microbit then triggers an event on the laptop through the USB cable connected to the laptop. The trigger can also be used as switch to control communications software, such as Smartbox’s Grid software. Thanks to Sensory Software for giving me a licence for this software to allow me to test the hardware with it.

Licence

This project and all of the associated files are covered by the GNU General Public License version 3 (https://opensource.org/licenses/GPL-3.0)

  • 2 × BBC micro:bit board About £12 each
  • 1 × wrist holder for one microbit board Make it yourself or buy one from e.g. Polulu
  • 1 × Freetronics Leostick (not required for most applications) This board appears as a physical keyboard to the PC, so can send 'real' keystrokes.

  • Testing opportunity with UCPLA

    matt oppenheim08/30/2020 at 17:48 0 comments

    I am sending a hand shake kit to Aragna Ker at UCPLA for trial. This is a great opportunity to get some real-world feedback.

    This gave me the kick to test out a couple of off-the-shelf wrist holder solutions for the microbit. Much as I like my own home-made holder, using a read-made sport armband for an iPod means that the microbit and battery holder will have a plastic cover, which makes the kit easy to clean, which is important in today's Covid-19 world.


    Thanks to Hackaday for making this possible.

  • Functional block diagram

    matt oppenheim08/22/2020 at 16:09 0 comments

    A block diagram of the system setup is shown below. The base.exe program runs on the communication device and monitors the USB port that the receiver microbit is connected to. When a trigger is received from the receiver microbit, a software keystroke is broadcast. This software keystroke is intended to control communication software or any other target software running. The default keystroke is F1, but this can be changed at run time.

    The receiver microbit can remotely change the threshold of shake detection on the transmitter microbit using the two buttons on the receiver microbit. The transmitter microbit is worn by the end user. The threshold of shake detection can also be adjusted on the transmitter microbit using the two buttons on this microbit. When a shake is detected by the transmitter microbit, a signal is sent by radio to the receiver microbit.

  • Benchmark

    matt oppenheim08/13/2020 at 09:55 0 comments

    This log answers the Benchmark requirement for the Hackaday 2020 prize entry rules:

    Benchmark-How well is the project impact and viability demonstrated? Are estimated costs realistic? How well does the project improve upon other currently available solutions?

    Benchmark

    Preliminary testing can be seen with the target user group on a short video at:

    https://youtu.be/gRrBVgBbUFQ

    The testing indicates the impact and viability of the project.

    The estimated costs are realistic as the components are repurposed off the shelf components.

    This project fills a gap in what is available as assistive technology. This gap was identified by the Technologists who work at Beaumont College with the target user group.

  • Production

    matt oppenheim08/13/2020 at 09:51 0 comments

    This log answers the Production requirement for the Hackaday 2020 prize entry rules:

    Production-Is the project realistically reproducible (taking into consideration necessarymaterials, skills, and production processes)? Are the manufacturing processes detailed? Are those processes realistic for scalability?

    Production

    The design comprises of two microbits and one wrist holder plus associated code. There are over 1.5 million microbits already produced. Microbits cost around $15 each. So obtaining these does not require any production.

    There are wrist holders commercially available, though I prefer my home made one. It would be straight forwards to ask a fabric shop to do a production run of wrist holders based on my home made one. Otherwise commercially available wrist holders can be used.

    The code is on my GitHub site, so can easily be replicated.

  • Design

    matt oppenheim08/13/2020 at 09:46 0 comments

    This log answers the Design requirement for the Hackaday 2020 prize entry rules:

    Design-Is there a depth of design detail available (like a system design, CAD models, project test methods, etc.)? Is there base-level planning for the functionality (e.g.,functional block diagram, list of specifications and descriptions of how they will be met, etc.)? How user-friendly is the design?

    Design

    Each block of the design is explained in the set up instructions on my website at:

    https://www.seismicmatt.com/handshake-set-up-instructions/

    The design is extremely user friendly as it repurposes off the shelf technology. The only construction necessary is to make a wrist holder for the microbit. How to do this is explained on my website at:

    https://www.seismicmatt.com/introduction/bbc-microbit-wrist-holder/

    Wrist holders are commercially available, but I found my home made one to be more secure and comfortable.

  • Concept

    matt oppenheim08/13/2020 at 09:41 0 comments

    This log answers the Concept requirement for the Hackaday 2020 prize entry rules:

    Concept-Is the project creative, original, functional, and pushing boundaries? Does the project effectively address the selected challenge?

    Creative

    This project is creative in that I created a solution to a recognised problem. Initially I created custom made hardware, which I realised would be difficult for others to replicate. There is also the safety issue associated using 'home-made' hardware with a vulnerable user group. I managed to re-purpose an off the shelf microcontroller board, which is aimed at high school education. This simplicity is the creativity.

    Original

    The need for this project was identified by the Lead Technologist at Beaumont College, Lancaster. This is a specialist educational centre for students, many of whom have cerebral palsy. The professional at Beaumont were unable to find off the shelf technology that fills the gap that this project intends to.

    Functional

    Preliminary testing was done with the target user group, both with my home made prototypes and later with the microbits. The technology works.

    Pushing boundaries

    This project pushes the boundaries of the technology that can be used to enable some people who currently cannot communicate to access communication technology.

    Enabling somebody who cannot communicate to communicate will empower that individual to push his or her boundaries.

    Does the project effectively address the selected challenge?

    This project is a solution for some people for the selected challenge. It is effective in that it works and is easy and safe to replicate.

  • Running in Administrator mode

    matt oppenheim08/04/2020 at 16:16 0 comments

    Sensory Software's Grid 3 is a popular communication software package, running in Windows. Naturally, I would like handShake to be able to operate this software through the software keystrokes that handShake generates.  To get Grid 3 to respond to a software keystroke, I have to 'elevate' handShake to run as an Administrator. I do have a second solution. I can use a Freetronics Leostick USB dongle as a pretend keyboard and have this generate keystrokes that appear as coming from a physical keyboard. I did this for a while, but this adds a layer of complexity and expense to the project. The simplest solution is to run handShake as an Administrator when using Grid 3, or other software that requires software keystrokes to come from an elevated source.

    I tested out adding the functionality for handShake to detect when Grid3 was running, then automatically try to elevate to run in Administrator mode. I got this running. Then removed the functionality. Why? Security. Software running as Administrator can damage your system if incorrectly or maliciously written.

    Now handShake detects if Grid 3 is running and advises that this requires the software to be restarted as an Administrator, but does not automatically restart in Administrator mode.

    Running handShake as Administrator is a choice that the user makes and implements if he or she deems necessary.  As all of the code is on the project GitHub site, the code can be reviewed by the user or whoever administers the communication device that handShake runs on. The code can be reviewed to check that it is safe to run as an Administrator.

    One of the advantages of open source projects is that they are open to scrutiny to find security flaws.

  • Testing

    matt oppenheim07/01/2020 at 17:07 0 comments

    I spent a short time working in mechanical engineering, for a company that makes quick release anchors for oil rigs. Every part was tested, as you can imagine. But not only the finished assembly. Even fragments of the castings used to construct the main component were taken for destructive testing to prove that the metal had the correct characteristics.


    Why should it be any different in software engineering? Software testing is an established part of producing reliable code. I have been slack at implementing this. So I took an online course through Pluralsight, kindly made available for free during the recent lockdown, on writing tests for software. The course is aimed at Python, which I use for scripting the software implemented on the PC.

    So I am slowly getting to grips with writing tests. I find writing stand alone doctests a useful tool. I test the code from the command line, then copy and paste the tests into a separate file. It is not quite as simple as this, but you get the idea. I am slowly getting to grips with mocks and pytest. Using the doctests is fast and easier, so I am more likely to actually write some tests.

  • What do I want out of posting this project

    matt oppenheim06/30/2020 at 15:18 0 comments

    There are so many excellent projects submitted to the Hackaday competitions, showing a level of skill that I don't have - why would I enter this? I hope that somebody who can use the project shown here, or who knows somebody that could potentially benefit from the project, reads this site and tries to implement hand shake. Please get in touch if there is anything that I did not explain well.

    If you do try this out, please let me know. I keep an unpaid position at my local university for when I am not working at sea. This gives me the resources and support to do this work. Any feedback helps me write publications, which I need to keep the position. The data will be anonymised.

  • Packaging python executables

    matt oppenheim06/30/2020 at 15:10 0 comments

    There are several libraries for making a python script executable. Some work for some folk some of the time. I am not sure if they all work for all folk all of the time. I used the cx_freeze library successfully with one of my projects. It looks to still be active. Currently I use pyinstaller.

    One issue I found using these packaging managers is that the size of the final executable can seem unreasonably large. Of course, a Python environment needs to be wrapped up in the executable. To minimise the size of this,  work in a virtual environment with only the extra modules installed that you need for your script to run. I got this tip from the website here in the post dated

View all 11 project logs

  • 1
    hand shake setup instructions

    Please check to see if there are more recent set up instructions on my web site here:

    https://www.seismicmatt.com/handshake-set-up-instructions/

    I am slowly porting these instructions to the project GitHub page here:

    https://hardwaremonkey.github.io/microbit_hand_shake/

    This is generated using mkdocs with markdown and image files. I recently learned how to do this  from the series of short videos here:

    https://calmcode.io/mkdocs/material-mkdocs.html

    All the markdown and images needed to generate the instructions will be on the gh-pages branch of the GitHub repository. Then the instructions will be open source.


    Please find a copy of the setup instructions from my website below.

  • 2
    Instructions update 8 August 2020

    Safety

    The BBC micro:bit was designed to be safe to use in education with 11-12 year-olds, so has safety designed in.

    The transmitter board is powered by two AAA batteries in a case. Normal safety practice should be followed with the batteries – keep them dry.

    Parts

    2 x BBC micro:bit boards

    1 x AAA battery pack for a micro:bit board

    1 x wrist holder – construction instructions below

    1 x micro USB cable to connect the receiver micro:bit to the communications device.

    I am testing an iPod touch 4G Sport Armband made by Tune Belt and an iPod classic holder made by igadgitz. Both of these work as a wrist holder for a microbit and battery pack.

     I made my own, picture below, before finding the two off-the-shelf solutions. I still think the one I made looks better. But you can save yourself a few hours by buying a ready-made solution.

    Build details for the holder that I made are at the end of these instructions.

    microb:bit and AAA battery holder on a home-made wrist holder

    How to set up the system-summary

    The transmitter micro:bit goes on your wrist, the receiver micro:bit is connected to your PC or laptop computer using a micro USB cable.

    Each of the micro:bits needs different code loaded onto it. The hex files can be drag and dropped onto each micro:bit through a file explorer to program them.

    The transmitter is shaken. This shake sends a signal to the receiver. This causes the receiver to send a signal to the computer through the USB port.

    A piece of software running on the computer detects this message and sends a keystroke out. This keystroke can be used to control communication software running on the computer.

  • 3
    How to connect the micro:bit to a Windows machine and load code

    Hopefully, you can just plug the micro:bits into the Windows machine and they will be automagically detected. A folder called MICROBIT will appear in your File Explorer. Then you can drag and drop the .hex file containing the code for the micro:bit onto this folder and the board will load the code in about 10s, merrily flashing a little LED. Details below on what hex file goes where.

    If you need more detailed instructions on how to connect a micro:bit to a Windows machine and how to load code onto it, please look at my blog post: https://www.seismicmatt.com/2020/04/27/connecting-the-bbc-microbit-to-windows-and-adding-code-to-it/

View all 11 instructions

Enjoy this project?

Share

Discussions

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates