BCI-based assisted living and robot avatar system

This is a brain-computer interface system to allow people suffering from locked-in syndrome to experience a higher quality of life

Similar projects worth following
People with locked-in syndrome - which can result from stroke, multiple sclerosis, traumatic brain injury, poisoning , ALS (Amyotrophic lateral sclerosis) - are conscious and can think and reason, but are unable to speak or move. This proposed system will use EEG to enable a brain-computer interface, thus allowing those affected to communicate via text-to-speech, access internet and entertainment streams, control their home, control wheelchairs, and control a robot avatar.

Introduction and aims

The aim here is to replace the traditional mouse/keyboard interface system with something a little more direct, since locked-in people can't use their limbs, or even move their eyes significantly in some cases (thus ruling out eye-tracking). We will use an EEG device, initially the OpenBCI, to acquire signals from the surface of the skull. These signals will be subjected to amplification, ADC, software-based signal analysis and pre-processing, in order to extract two special signals which are widely used in the field of brain-computer interfacing: p300 and SSVEP.

These signals will be used to control cursors (SSVEP), select characters for text input (SSVEP and p300)., and select from image-based menus (p300). You can see examples of these in other people's projects by looking at this project log. These input functions, in place of traditional mouse/keyboard, will allow the user to:

  • communication with text-to-speech (both locally and remotely via robot-avatar)
  • control web browser
  • read books
  • control video and audio playback, and selection
  • control motor functions of a wheelchair
  • control a robot avatar
  • control a smart-home (e.g. lights on/off, temperature control)

All of this has been accomplished within academia, but our aim is to accomplish this with a low-cost extensible open-source system (<£1500) that people affected can actually afford to make or buy!

Locked-in Syndrome

Locked-in syndrome is caused by damage to specific portions of the lower brain and brainstem.. "Locked-in syndrome was first defined in 1966 as quadriplegia, lower cranial nerve paralysis, and mutism with preservation of consciousness, vertical gaze, and upper eyelid movement.1 It was redefined in 1986 as quadriplegia and anarthria with preservation of consciousness".

Neuroscience involved: EEG, ERPs, p300 and SSVEP

What is an ERP?

Event-related potentials are voltage fluctuations in the ongoing EEG (electroencephalogram) which are time-locked to an event (e.g. onset of a stimulus). The ERP manifests on the scalp as a waveform comprising a series of positive and negative peaks varying in amplitude, and polarity as the waveform manifests itself over time. It’s usual for ERP researchers to assume a peak represents the underlying ERP component – although this is not actually the case as it’s the components between peaks that reflect neural processes. The scalp voltage fluctuations obtained by EEG reflect summation of postsynaptic potentials (PSPs) which are occurring in cortical pyramidal cells within the brain. The PSPs themselves result from changes in electrical potential due to ion channels on the postsynaptic cell membrane opening or closing – thus allowing ions to flow in/out of the cell. If a PSP occurs at one end of a cortical pyramidal neuron, the neuron will become an electrical dipole – one end positive and one negative. If the PSPs occur in multiple neurons which all have their dipoles in the same direction – the dipoles will sum to form a large dipole. This current dipole will be large enough to detect on the scalp surface! Thousands of neurons are required to produce such a summed dipole, called an equivalent current dipole. This is most probable to occur in the cerebral cortex where groups of pyramidal cells are lined up together perpendicular to the cortical surface.

The distribution of positive and negative voltages manifested on the scalp for a given equivalent current dipole will be determined by the location and orientation of the dipole in the brain. So it’s important to note that each dipole will produce both positive and negative voltages on the scalp. (Oxford Handbook of ERPs, Luck & Kappenman, 2013)

What is p300?

P300 is an event-related potential (ERP) that is elicited within the framework of the oddball paradigm. In the oddball paradigm, a participant is presented with sequence of events that can be classified into one of two categories. Events in one of the categories are rarely presented to the participant, whilst events in...

Read more »

View all 11 components

  • Development timeline for the BCI-based assisted living and robot avatar system

    Neil K. Sheridan04/24/2016 at 19:25 0 comments

  • Robot-Avatar Design

    Neil K. Sheridan04/20/2016 at 19:15 0 comments


    The robot-avatar will have data sent/received using TCP/IP. The data we will be sending/receiving will comprise: commands for motor control (forward, left, right, back, control nerf gun), streaming video & audio from control computer to robot-avatar, and streaming video & audio from robot-avatar to control computer. Text-to-speech will be done by the communication software module on control computer and sent as audio to the robot-avatar.

    Outline of main parts required:

    • Raspberry Pi Model 3
    • Arduino
    • H-Bridge Motor Shield for Arduino
    • Robot platform with wheels and DC motors
    • Raspberry Pi Camera (or another HD camera ->USB with microphone e.g. GoPro)
    • Ultrasonic distance sensors
    • Stereo Amplifier and speakers
    • 24" LED HDMI monitor

    Overview schematic for the Robot-Avatar

  • Ideas and primary components for low-cost computer

    Neil K. Sheridan04/08/2016 at 21:08 0 comments

    Motherboard: 970A-DS3P AMD 970A (Socket AM3+) DDR3 [AMD 970 Chipset, x4 DDR3, x2 PCI-E x16, x3 PCI-E x1, x2 PCI, x6 SATA 6GB/s, x1 Gigabit LAN, x2 USB 3.0, x12 USB 2.0] £53

    CPU: AMD Piledriver FX-4 Quad Core 4300 3.80GHz £60

    RAM: Kingston HyperX Savage Red 16GB (2x8GB) DDR3 PC3-19200C11 2400MHz £78

    GFX: MSI Radeon R7 240 "Low Profile" 2048MB GDDR3 PCI-Express Graphics Card £45

    HDD Solid State : PM951 256GB M.2 PCI-e 3.0 x 4 NVMe £80

    Monitor: G2260VWQ6 22" 1920x1080 TN FREESYNC 1ms Gaming Widescreen £100

    Various: £100

    Total cost: £516

  • Research Materials

    Neil K. Sheridan04/07/2016 at 17:53 0 comments

    1. Electrode placement for p300 and SSVEP

    2. Comparative Study of SSVEP- and P300-Based Models for the Telepresence Control of Humanoid Robots

    3. P300 brain computer interface: current challenges and emerging trends

    4. A Low-Cost EEG System-Based Hybrid Brain-Computer Interface for Humanoid Robot Navigation and Recognition

    5. A Review of P300, SSVEP, and Hybrid P300/SSVEP BrainComputer Interface Systems [pdf]

  • Examples of SSVEP and p300 BCIs

    Neil K. Sheridan04/07/2016 at 17:21 0 comments

    As it's rather difficult and long-winded to describe the functioning of SSVEP and p300-based BCIs using written language, I've included these videos of other people's projects as examples [skip to fourth to see a robot!]:

    In this first video, an SSVEP-based speller is shown. As you can see, each letter flickers at a different frequency..

    In this second video, an SSVEP-based cursor is shown. As you can see, each direction choice flickers at a different frequency..

    In this third video, a p300-speller is shown..

    In this fourth video, somewhat more excitingly, Nao is controlled by SSVEP. The user attends to the flickering lights on the frame which correspond to which of the four commands they desire to send to Nao (e.g. walk forward)

  • Rough outline of modules for software control and software->actuator control

    Neil K. Sheridan04/06/2016 at 20:17 0 comments

    * Communication Module

    p300 or SSVEP speller

    * Wheelchair Module

    Can be a flickering light box for direction (SSVEP) and stop; p300 menu for fave locations (photo of kitchen, front door, lounge, find friend* etc]

    *friend can have RFID tag so we can find them in the house easily

    * Assistive Living Environment Module

    Rooms broadcast to BCIALERAS (brain computer interface assisted living entertainment and robotic avatar system) so we know where user is

    -> Adjust lighting / turn on off including lamps (photo menu of lights)

    -> Text-to-voice using SSVEP or p300 speller to answer door

    -> Adjust temperature either home-wide or local room (e.g. AC / fans)

    -> Fave food p300 menus to communicate with friends which food/drink would like easily! These could be sent to smart phone of friend so they can be purchased or brought as required.

    * Robotic Avatar Module

    When confined to one room and only able to move slowly in wheelchair (if at all when in bed) it could be frustrating to hear noises etc. and wonder what they are. Or perhaps may wish to explore the garden and look at plants, ponds etc. Or perhaps watch TV with friend whilst they are in sitting room and user is in bed. This is accomplished using a robot avatar. Simply a tracked robot with a 1m vertical extension featuring video cameras (including zoomable) and voice synth (for SSVEP/P300 speller to speech) and microphone. It could additionally feature a screen broadcasting video of the user.

    - Robot can be controlled directly, or move semi-autonomously to fave locations.

    - An automated rescue feature for robot to reach safety in case of getting stuck in corners and frustrating the user.

    The robot could feasibly leave the house and be accompanied by friend on visits to shops, parks, galleries, museums car rides. Communicating via 4G to user stuck at home.

    * Entertainment Module

    Top Menu with links [Reading / Video / Audio / Internet ]

    -> Reading -> Control of Kindle App using KDK [turn page/close/select book]

    Net-> Web Browsing -> Cursor for Hyperlinks + pop up keyboard for URLs/fields

    Net-> Twitter -> + pop up keyboard

    Net -> Video -> Control using Amazon Video API and YouTube

    Net -> Audio -> local library, iTunes?, streaming services

    ? keyboard option of p300 speller?

    ? p300 speller to choose books

    Net -> Email control

View all 6 project logs

Enjoy this project?



Jean Pierre Le Rouzic wrote 09/11/2023 at 19:35 point

Wonderful project!
Did you make some progress since, for example connecting the BCI to a wheelchair or a robotic arm?

  Are you sure? yes | no

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates