Close
0%
0%

SWAN open source AAC

A communicator for non-verbal disabled people.
An acronym for SWitch to AlphaNumeric.
An auditory scanned alphabet, message creator.

Similar projects worth following
Development of an augmentative and alternative communication (AAC) device for people who can't talk or see. An android phone running the SWAN app says the alphabet, the user selects letters to form a sentence. Selection can be done using any AAC switch, via SWAN's electromagnetic interface. Selection is also possible using only the phone's internal sensors, including sip & puff capability. The message can be read or can be sent as a text message. SWAN is completely open source, the app will be free.

SWAN will enable someone to communicate, who, through disease, injury or approaching death, can't talk, see or write. If they can hear, and are able to make a movement, they can use SWAN to write a word or message. Instead of being trapped, unable to say they're thirsty, or that they love you. The incommunicado.

A video introducing SWAN is viewable via the link below.

SWAN enables severely disabled folk to communicate their needs and thoughts to their friends, family and carers.

It consists of a smartphone, an electromagnetic interface module, a switch and a pair of headphones.

SWAN will enable someone to communicate, who, through disease, injury or approaching death,can't talk, see or write. If they can hear, and are able to make a movement, they can use SWAN to write a word or message. Instead of being trapped, unable to say they're thirsty, or that they love you. The incommunicado.

SWAN is an assistive communication device for severely disabled non-vocal folk; individuals who have a neurological or neuromuscular disorder that result in loss of speech, sight, writing and typing ability. e.g. -

  • Motor neurone disease
  • Cerebal haemorrhage or stroke
  • Cerebral palsy
  • Multiple sclerosis
  • Traumatic head injury
  • Brain cancer
  • An accident victim, temporarily unable to see, talk or write because of injuries, could use SWAN to give details about the accident.

My interest in developing SWAN goes back around twenty years, when my friend's mother was dying of Motor Neuron Disease. His accounts of the increasing difficulty of communication with her were harrowing. Some years later, I became very aware of the need for a communication device when my father and sister each had brief lucid moments during their last days of suffering cancer. I'll never forget the joy I felt, when my questions where answered by a squeeze, or a slight movement, at a stage, so close to death, where it was easy to assume they were comatose.

These responses, however, indicated that there was a possibility that they had something to say, but couldn't due to the disease progression and medication. There is a need for a communication device which is simple and easy to learn how to use, as time may be very limited.

It should be free, and obtainable in minutes. That's SWAN. A smart phone (which can be obtained very cheaply second hand, or indeed borrowed), running a free "app", speaks the alphabet at a variable rate (selected on set-up by the carer). The user selects letters, either using the smartphone or an external switch, to construct a message.

The auditory scanning method of creating a message is very time consuming, but the alternative is no communication at all.... I'm trying to speed up the scanning process , see here.

The user can edit and listen to the message being constructed. An auditory menu of commands allows the user to do several things when the message is finished-

Display the message on the smartphone screen
Send the message as an SMS text
Save the message

The user could build up a store of messages, or just words, for future use.

If the user is unable to use the touch screen of the smartphone for selection, they may be able to use the orientation switch function of SWAN. Here the smartphone is placed on a piece of foam (a pan-scrubber), and the internal accelerometers provide rate of change of orientation data, to detect the user gently pressing on the phone.

Those without the dexterity to use the phone's internal sensors for letter selection can use any commercial switch specifically designed for use by the disabled. AAC switches are designed to be used by those with very limited dexterity. They plug into the SWAN electromagnetic interface - a small circuit board which influences the phone's geomagnetic sensor. More on this can be found here.

SWAN is

- a communicator for non-verbal disabled people.

- an acronym for SWitch to AlphaNumeric.

- an auditory scanned alphabet, message creator.

The SWAN electromagnetic interface, being an extremely simple, inexpensive and non-intrusive method of connecting the real world with a smartphone, has uses apart from a communicator, which are detailed here.

The full documentation of the SWAN project is at http://aacswan.blogspot.co.uk/

First SWAN - the microcontroller prototype

First experimentation using App Inventor - recreating Ørsted's 1820 discovery

Electromagnetic Interface - External Switch...

Read more »

  • 1 × android smartphone HTC Wildfire
  • 1 × a wound coil, or PCB copper trace coil 30 turns, 46 SWG, ! cm dia
  • 1 × current limiting resistor 150 ohms
  • 1 × li-ion battery >70mAH
  • 1 × charge management IC MCP73831 or MAX1555 (with associated caps)

View all 10 components

  • Experimental PCB coil - Using an excellent PCB milling service

    Roger Curry03/16/2015 at 21:24 0 comments

    I'd designed the PCB for just the coil part of the electromagnetic interface some time ago (see here). I thought I'd have a go at a 1 inch square test PCB. Keeping the layout simple, a square coil it was, and with the connection pads my trace design came out a little oblong, but the concept would be provable. It's got trace all the way to the centre so that I can experimentally determine the amount of unwanted turns at the axis.

    One inch coil

    42 turns of 0.01 inch trace

    The vertical lines are screen faults on my "weathered" laptop


    Getting a PCB made, from the Eagle CAD file I'd created, went on the back-burner. This delay was caused by my acquisition of an Arduino and its subsequent learning curve. I need the Arduino in order to control and obtain data from various aspects of SWAN development, a tool rather than part of the project.



    Unfortunately I've been completely sidetracked by the Arduino, my spare time having been spent using it for martial arts data gathering experimentation (a project which followed the completely useless "electronic candle," see here. Still, you have to make stuff to learn).



    The crunch came when I was about to buy some hardware for the new project. Realising SWAN was about to languish for a while (yet again!), I shelved punch monitoring. But I'd gained a modicum of Arduino experience.



    Back to SWAN. I'd come across an outfit on ebay (eminence2001) which is set up to produce prototype PCBs using a CNC miller to remove copper, leaving only the traces. I emailed the owner Marek, with the Eagle file attached and a link to this blog. He was kind enough to produce my board very quickly for the princely sum of £5. Here it is, it's a gem -

    Remember, this double-sided board is only an inch square. The track width is a minuscule 10 mil (0.25mm, Marek usually works to a minimum 0.6mm). The job was very precise, and I'd heartily recommend Marek's PCB milling services.



    I've done some quick testing of the PCB. By probing at various points on the coil at 40mA, I determined that the board could actually be single sided, and that the amount of unwanted turns in the middle was rather more than I had expected... In short, it works.



    Next I'll hook up the Arduino, and do some proper testing. With the results, the next coil PCB design can be created using Eagle, this time with an "proper" octagonal or spiral coil layout, and additional circuitry. I'll certainly be using Marek's PCB milling service to make the prototype.



    He trades as eminence2001 on ebay. His current listing is here



    email marek.kuczynski@ntlworld.com

    "PCB Printed Circuit Board Prototype Production/Milling/Manufacture/Manufacturing"

  • Auditory scanning

    Roger Curry08/20/2014 at 10:19 0 comments

    Everyone able to construct a word, knows the sequence of the alphabet. During auditory scanning, this is favourable, since the user is aware when the desired letter will be announced. Unfortunately the most commonly used letters are not in "alphabet order". This could be very annoying to a user of SWAN with excellent mental acuity.

    letter frequencies, sorted alphabetically

    In order to accommodate these folk, it's planned to have an option to scan through the alphabet in statistical order.

    letter frequencies, sorted by frequency

    Also, I'm experimenting with grouping of letters, to enable even faster selection. Text prediction is also possible, although beyond my software capabilities.

  • Simplicity vs. Swiss Army Knife

    Roger Curry08/20/2014 at 10:17 0 comments

    To open the brown ale bottle, there's a choice. Use the simple bottle-opener, or select the bottle opener from the plethora of choices of the Swiss Army knife, and then use a design compromised tool.

    When I began considering the use of an android phone as the platform for SWAN, I became quite engrossed with the possibilities that could be offered to the user. Libraries of commonly used words and sentences. Text prediction. Twitter. The radio. Speaking clock and alarm. Mp3 player. Bluetooth control of appliances... etc.

    This is known as the dreaded feature creep. SWAN soon became a Swiss army knife of menus, functions.... and confusion and complexity. Longer development time and steeper user learning curve result.

    Any deviation from having SWAN as simple as possible, could mean that someone, somewhere, just can't use it.

    And so SWAN is without bells and whistles. No fancy graphics, no unnecessary functions. It just does what it says on the lid.

    1) Enables a person to construct a written message.
    2) The message can then be displayed on the phone screen or sent by SMS text.

    This stage in development is my target. At this point SWAN will be available for use. Here though, the beauty of open source comes in. Third party individuals or software houses can easily use the work done by myself, to expand the capabilities of SWAN to include all sorts of functions, using the same switch interface and auditory scanning.

    Whatever Swiss Army knife evolves from this remains to be seen, but the original, simple SWAN will remain as the first option for someone with an immediate need to communicate, such as an accident victim unable to verbalise.

  • Other uses for the SWAN electromagnetic interface

    Roger Curry08/20/2014 at 10:14 0 comments

    The ability to very simply and cheaply add a switch input to an inexpensive android phone, offers many possibilities for use in areas other than AAC.

    A burglar alarm - a passive infra red motion sensor module such as this......

    ..... costing £0.99 from china, connected with the SWAN electromagnetic interface, would enable a highly sophisticated alarm or room monitoring system to be developed. An app could easily be written in Tasker which could, for instance, send a text message to another mobile phone upon detecting a person in the monitored area. The person receiving the text could then text back with instructions to the android alarm system app -

    on - bluetooth an audio file of an alarm sound to a bluetooth speaker.

    msg - same with an audio message, e.g. "Leave the premises immediately!"

    off - switch audio off

    log - send no more text messages, just log times of detection

    auto - sound alarm automatically when intruder is detected

    Since the PIR module can output pulses >1 sec, the speed issues I'm having with SWAN using MIT App Inventor and Tasker do not arise.

    Industrial uses - for instance, a switch at the end of a production line could be triggered as each manufactured item is finished. Connected to an inexpensive mobile phone, data such as line rate, down time, shift total etc. could be sent by text to the production manager at set intervals. The phone could be protected, and made robust enough for industry, by a suitable IP65 enclosure.

  • It Doesn't Work!

    Roger Curry08/20/2014 at 10:12 0 comments

    I've written the SWAN software using MIT App Inventor. At present, it scans through the alphabet and selection is done by touching the screen, or depressing the smartphone whilst on a foam base (here I'm using the accelerometer to detect rate of change of orientation). Implementing the electromagnetic interface, so that letter selection is done by a commercial AAC switch, has become problematic because of speed issues.

    Unfortunately App Inventor just can't do it, and neither can Tasker. For some reason Tasker only registers changes of magnetic field over second in duration, and with all the azimuth calculation App Inventor does, it's not surprising that it's way too slow.

    I know that the android OS is fast enough, through Joe Desbonnet's work, and also free sensor apps such as "Magnetic Field Detector" show detection of short pulses well.

    Other routes to interface switches were then considered. One solution is to have the phone's charger plugged in during use. The power lead is plugged into an external female micro USB socket. The +5V from the socket is taken to the normally closed switch, then back to a short lead terminated by a male micro USB plug, which is plugged into the phone. In this way, the supply can be interrupted, and this can be detected by the android OS. Unfortunately App Inventor doesn't yet support power monitoring, and although Tasker (the other app development software I'm experimenting with) can do this, the idea is a no-goer because most commercial AAC switches can't handle the possibly high charging current and may not have a normally closed mode. And of course, the flexibility in having SWAN independently powered would be lost.

    Another method considered was to use a small piece of aluminium foil taped to the touch screen over a "button". A wire from this to the switch and then to ground could simulate a button press by direct influence on the capacitive screen. After a bit of experimentation, the idea was deemed too dodgy and doomed. Interesting though, and perhaps with the inclusion of a FET it could be made to work reliably. Here's a Hackaday post where the method is demonstrated.

    Also considered for use as a switch interface were an ultrasonic transducer next to the phone's microphone, the microphone input jack socket, and the ambient light sensor. All these require more hardware than the simple coil of wire. The electromagnetic interface is the way to go.

    So, where to go from here? I'm designing the PCB for the electromagnetic interface. I'm continuing development of the functionality, with both App Inventor & Tasker, but the project really needs a software"plug-in" of some sort, to handle the speed issues. If I can't get any help in this direction, my completed software algorithm could be used by a third party software house as a template to produce some professional code. This would mean I'd have to do a modest "Kickstarter", in order to pay for it. I'll get some quotes, once my algorithm is finalised. 

    If I had the time, I'd learn the Android APK development environment, and the whole thing could be done in a professional manner. But it's been way too time consuming just to get to the stage I'm at - I've a modicum of expertise in App Inventor, and just learning Tasker.

  • Electromagnetic Interface - External Switch to Android Phone

    Roger Curry08/20/2014 at 10:06 0 comments

    The selection of letters from SWAN auditory scanning can be accomplished using the phone's own sensor inputs (i.e. the touch screen, accelerometer or proximity sensor). However, using them may be impossible because of insufficient feedback, or a user's lack of dexterity. It's highly desirable therefore, that an external switch, specifically designed for use by the disabled, can be used with SWAN via a 3.5mm jack socket.

    These are called AAC switches, examples can be seen here -

    Augmentative and alternative communication switches

    more can be found here


    To interface these switches to the android smartphone, electromagnetic coupling will be used (details here). A microswitch, for example, can be connected to a coil of wire and a battery via a resistor to limit the current to around 60mA. The coil is placed on the phone, when the switch is closed a magnetic field is generated by the coil, and is detected by the geomagnetic sensor on the phone.
    Proof of concept -

    9 V battery, microswitch, 150 ohm resistor, coil of 30 turns of 46 SWG wire, cable ties.

    Instead of the microswitch, any commercial AAC switch could be incorporated into this interface via a 3.5mm jack socket.
    The simplicity of this means that those of a hacking nature should be able to make such an interface with little effort, from materials at hand, or from a scrapyard or dump. I plan to produce a video showing how quick and easy such a build is, with only parts gleaned from a scrap car.

    However, a properly designed interface is desirable, for those able to afford it. To this end I've been learning how to use Eagle PCB software. Years ago, I earned a living using acetate sheets, transfers, black tape of various widths and a scalpel. Now, using CAD is wonderful, compared to the "olden days" of tape up. Here's where I'm at with the design of a double sided PCB to (hopefully, if it works - i.e. produces sufficient magnetic field) replace the coil of wire...

    One inch coil, 42 turns of 0.01 inch trace. The vertical lines are screen faults on my "weathered" laptop

    The PCB will be populated with -

    a wound coil (if the copper trace coil doesn't work)
    current limiting resistor
    >70mAH li-ion battery
    MCP73831 or MAX1555 charge management IC (with associated caps)
    charge LED
    micro USB socket
    test push button
    3.5mm mono jack socket

    In order that the magnetic interface can be used for purposes other than AAC, pads and traces will be on the board for optional components (a transistor & resistors) and connections (to a PIR device for example, see here).

  • First experimentation using App Inventor - recreating Ørsted's 1820 discovery

    Roger Curry08/20/2014 at 10:01 0 comments

    When I first started experimenting (as an android virgin) with a donated HTC Wildfire in February, I constructed a series of apps which graphed the output of each sensor in the phone. Such is the engineer, and these provided valuable steps in learning how to use App Inventor. At this point, using an android phone as the platform for SWAN (a long dormant project, see here) hadn't occurred to me.

    The sensor which immediately held my interest was the geomagnetic sensor, found in most smartphones. App Inventor converts the output from the HTC's internal AK8973 3-axis geomagnetism sensor to an azimuth reading (0 - 360 degrees), for utility as a compass. I was disappointed with this, and had hoped that the device's raw data would be available. Anyway, the graph of azimuth vs time proved most interesting.

    The trace shows intriguing hunting behaviour when close to active electronics (laptop etc.) and also I was surprised by the sensitivity. A two pence coin (post 1992, bronze to steel) moved near the phone distorts the Earth's magnetic field and the trace moves considerably. At this point I realised I could interface with the phone; I swiftly cut a spare charger lead up, to provide a couple of feet of wire, stripped both ends and, with a 18650 Li-Ion battery ripped from a scrap Vaio power pack, I recreated Hans Christian Ørsted's momentous experiment of 1820. He would have recognised only the wire.

    The trace on the screen moved, just as Ørsted's compass needle turned, when current flowed through the wire. So, I reasoned, if you have a small coil of wire fixed onto the phone, over the point where the geomagnetic sensor IC is, then software in the phone can respond whenever current flows though the coil. With only the coil, a resistor to limit current, a battery and a switch, we have a very simple method of wireless on/off communication with the phone.

    I thought I was onto something new (not with electromagnetism, being nearly two centuries too late!), but a little research on the internet soon turned up some excellent work by Joe Desbonnet. He details his experiments on his blog here.
    Joe uses 30 turns of enamelled copper wire (about a meter) to form a coil with an internal diameter of 1cm. He's energising the coil using an arduino (5V) through a 120 ohm resistor to limit the current to 40 mA. He uses a free magnetic sensor app to determine where the sensor is within the phone, then determines the maximum speed he can send data to the phone from the arduino.

    This video demonstrates a means of low cost (less than 20c) and low bandwidth (less than 10bps) communications between an Arduino and an Android cellphone. See this blog post for details:

    http://jdesbonnet.blogspot.com/2011/0...

    The SWAN project, shelved in the 2001, came back to life at that moment - I realised that instead of an LCD, a PCB and a handful of chips, I could use readily available second hand smartphones as the platform for SWAN. At the moment they're available on Ebay at the £30 mark - much cheaper than it would cost to make the dedicated electronics! With a switch interface sorted out, the functionality - an app - could be built using App Inventor or Tasker.

  • First SWAN - the microcontroller prototype

    Roger Curry08/20/2014 at 09:56 0 comments

    My first attempt to produce a SWAN device was in 2001, using a microcontroller. My sister bought me PICBASIC software, I made a PIC programmer, a friend got me a handful of PIC 16F84 microcontroller chips, and I began to develop SWAN.

    The prototype used a modified computer mouse as an ergonomic switch - the mouse button switches being simply connected to the cord, no use being made of the internal electronics. A PIC 16F84 microcontroller was interfaced with an ISD 1016A "chipcorder" IC. This device, an addressable non-volatile audio record/playback chip, had my voice saying the alphabet recorded on it. The microcontroller then sequenced through the alphabet, and the user could hear this through the headphones. Letters were selected using the mouse buttons, and the created text was shown on an HD 44780 liquid crystal alphanumeric display, having two lines of 16 characters.

    Here's a bit of documentation from 2001 -

    The project stalled and was shelved. I'd contacted some people in the field of communications for the disabled, unfortunately they didn't like the fact that I wanted SWAN to be "freeware" - they wouldn't commit to further development and manufacture without sole ownership of the intellectual property rights. I got pissed off, especially since I was giving the design to them for nothing. End of SWAN...... Until 13 years later, I was given a smartphone, and the concept became do-able with minimal hardware cost, and a free "app".

    The above single page description of SWAN is about all that remains of the project. The prototype went when my old shed biodegraded. The PICBASIC source code, compiled microcontroller hex code, the sound .wav file, schematic, bill of materials, costings, half-finished eagle PCB layout and other documentation all went the journey when my old 486's hard drive self destructed, many moons ago.....

View all 8 project logs

Enjoy this project?

Share

Discussions

Anil Kumar wrote 09/11/2014 at 18:56 point
Good luck Roger, this is what tech is all about. Good luck and hope you win the prize and get this in market for everyone... :)

  Are you sure? yes | no

Roger Curry wrote 09/12/2014 at 20:18 point
Thanks Anil,
glad you like the project!
Cheers,
Rog

  Are you sure? yes | no

Roger Curry wrote 09/10/2014 at 11:16 point
Hi mrobinson801,
Please contact me by email. We can discuss how SWAN can help the man you're caring for. I can send you an app that may be of value to him, if he's willing to test some features. My emai can be found on this page of my blog -
http://aacswan.blogspot.co.uk/2014/08/contact-details.html
Kind Regards,
Rog

  Are you sure? yes | no

mrobinson801 wrote 08/20/2014 at 10:50 point
A very worthy project! As a full time caretaker for a man with severe cerebral palsy, I would be very interested in trying this out for situations away from home (he does have a head switch operated synthetic speech computer at home...ca. 1999!). I also like the implementation of statistical ordering for letters; my own method of unaided communication involves slowly stepping through the alphabet (starting with/prioritizing vowels is either helpful or confusing) and watching for a head nod to accept each letter...a very tedious and error prone endeavor. I will gladly report back with feedback once I try this out

  Are you sure? yes | no

Roger Curry wrote 08/20/2014 at 11:41 point
Cheers mrobinson801,
This feedback is exactly what I wanted from joining Hackaday Projects. I will be in contact in the next few weeks, about you beta-testing SWAN. Unfortunately I'm going to be away from internet access from sat 23rd, for a week or so, gold panning and looking for pyromorphite in Scotland. When I get back, we'll see if the project has got through to the next round, but be assured that development of SWAN will continue, regardless of the outcome of this competition.
Thanks very much for your interest,
Regards, Rog

  Are you sure? yes | no

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates