Close
0%
0%

NeuroLS System

Plug & Play Hardware + Software Interface for Real-Time DMX and MIDI control through brain signals

Public Chat
Similar projects worth following
We aim to provide a plug & play interactive and generative hardware + software interface that will turn brain signals into light and sound, allowing hobbyists and artists to create their own audiovisual pieces through brain signals, in real time.

State-of-the-art, scientific, neural signal interpretation involves thorough and time-consuming processes typically starting with the apparently-simple collection of data, but also involving complex preprocessing (filtering) and processing (feature extraction + classification) techniques which are not trivial at all.

In the scope of scientific projects, such manual and involved techniques are appropriate and necessary; but in terms of interactive design applications, it is possible to develop, apply and take advantage of faster, more automated interpretation techniques -provided they may not be the most scientifically sound- to produce audiovisual works that result from real-time communications between the brain and the computer.

In 2015, we began developing Neuroscientific Research focused on Art in collaboration with the University of Houston, Tecnológico de Monterrey and the Contemporary Art Museum of Monterrey (MARCO), resulting in scientific articles published in journals such as "Frontiers in Neuroscience" (2017; Zurich, Switzerland).

Currently, we work not only on scientific development in this area, but also in integrating it with interactive design. 

Together with L14, a group specialized in developing audio technology and interaction design we focus on applying state-of-the-art human-machine interfaces in other interesting areas.

Our goal is to use new creative platforms in order to increase the reach and the impact of our efforts, and of the technology with which we work.

Concept: Build an innovative interface for humans to talk to machines or machines to talk to humans.

With the NeuroLS System, we aim to provide a plug & play interactive and generative hardware + software interface that will turn brain signals into light and sound, in a way in which will allow hobbyists and artists around the world to create their own audiovisual pieces through brain signals, in real time.

For this first version, we focus on building the system to control DMX universes. In which the user can select a range of colors for each  MUSE HEADBAND sensor; to be controlled by each of the brain Absolute Band Powers (Alpha, Beta & Gamma). Thus the DMX universe can be mapped out and controlled by the brain signals.

The challenge

The state of the art of neural signal interpretation involves a thorough and time-consuming process which typically starts with the apparently-simple collection of data, but also involves a preprocessing stage including signal filtering and requires complex processing techniques which, in science and research, is no trivial task at all.

Thereby the integration of hardware pieces and software in order to control the systems with the brain signals in real time was complex challenge to solve.

Approach to solve the problem

In the process of developing scientific projects, such manual and involved techniques are appropriate and even necessary, but in terms of interactive design applications, it is possible to develop, apply and take advantage of faster and more automated interpretation techniques -provided they may not be the most scientifically sound- to produce audiovisual works that result from real-time communications between the brain and the computer.

How it works

A Muse headband collects Raw EEG data from the user and sends the information via Bluetooth to a computer.  A software from Muse Tools, processes the brain signals to Absolute Band Powers (Alpha, Beta, & Gamma) for each sensor. NeuroLS Software (Pure Data patch) registers this data to arrays to determine the predominant brainwave for each sensor and allows the user to customize the output of the signal to a DMX-MIDI system.

The user can also select a range of colors for each sensor of the MUSE HEADBAND to be controlled by each of the brain Absolute Band Powers (Alpha, Beta & Gamma). So the DMX universe can be mapped out and controlled by the brain signals.

Our extra mile: Workshops -- More Developers + More Projects

The goal of our studio is to use the scientific tools that we develop to make awesome things; and we know sharing with more people will result in even more awesomeness. This is why, in the beginning of last year, we started offering workshops, for both adults and young students, through which we share -in a practical learning process- our technical and scientific knowledge with the local and national community, and we provide them with tools to make their own creative projects.

We started with the goal of changing the world, we realized the tools were in science, and now we aim to create the extraordinary through...

Read more »

LS-System-v5.pd

Added Muse OSC support. Determines the predominant brainwave. Added GUI for is_good value.

pd - 83.49 kB - 08/23/2018 at 17:11

Download

LS-System-v4.pd

Added individual waveform and time values for each brainwave.

pd - 81.55 kB - 08/23/2018 at 17:11

Download

LS-System-v3.pd

Third version of the NeuroLS System. Added ramp controls for each sensor: waveform (direction), and time.

pd - 69.72 kB - 08/15/2018 at 23:02

Download

LS-System-v2.pd

Second version of the NeuroLS System. Store 2 colors for each for type of brain wave: alpha, beta and gamma. Improved GUI. Added a predominant brain wave switch for simulation, this parameter will later be mapped to the EEG headband.

pd - 60.83 kB - 08/15/2018 at 20:33

Download

LS-System-v1.pd

First version of the NeuroLS System. A simple ramp between two colors. Registers Alpha and Beta absolute for each sensor in an array.

pd - 11.44 kB - 08/15/2018 at 20:33

Download

  • Neon SciFi: Neuro

    Memo Santos08/27/2018 at 00:40 0 comments

    We have made an agreement with Museo MARCO to develop and present an event in this venue (estimated quorum: 600 people) where we will implement this system and a couple of others which were requested (also involving Neuro-Lighting and Neuro-Sound) in November 2018. Join us!

  • Definition of a venue for public testing

    Memo Santos08/26/2018 at 23:42 0 comments

    We contacted Baby-Robot Studio and Dakota Studio Bar because they are friends of us who are involved in the local artistic scene and we thought they might help. We offered them to prove the NeuroLS System experience and decide if it could benefit their performances. Baby-Robot Studio refused but Dakota Studio Bar accepted provided we did a proper documentation of the experience. 

  • Fig. 6 Headset + NeuroLS + LED lights integration

    Memo Santos08/26/2018 at 22:51 0 comments

    Figure 6. Headset + NeuroLS + LED lights integration. A connection was made between our NeuroLS system with LED lights; programed to vary a sequence depending on the predominant wave signal read by a MUSE band headset.

  • Fig. 5 Construction of 3D-printed pieces.

    Memo Santos08/16/2018 at 21:22 0 comments

    Figure 5. Construction of 3D-printed pieces. a) Lamp built by a 3D-printer from Innovation gym CoWork with PLA filament. b) Material required for the construction of  the LED lamp: LED strip lights; Arduino Mega; Jumpers; Protoboard; monster driver. c) Electronic assembly of the devices. 

  • Headset Connection with PureData software

    Memo Santos08/16/2018 at 21:20 0 comments

    A connection between MUSE headset and PureData software was stablished, after understanding MUSE Tools. The Pd Patch was programmed to plot and sonify raw FFT neural data. (Fig. I) 

    Fig. I

    A subsequent Pd patch was program to relate the alpha, beta & gamma waves to certain colors of our election. (Fig. II)

    Fig. II

  • Fig. 4 Stroboscopic LED lamp Render.

    Memo Santos08/16/2018 at 14:32 0 comments

    Figure 4. Stroboscopic LED lamp Render. Visualization made in solid works to be printed with PLA filament. After NeuroLS Software interface converts the brain signal into audiovisual representation format, the LED lamp receive a DMX  signal to reproduce a stroboscopic protocol. 

  • NeuroLS System Version 3

    Miguel Moreno08/15/2018 at 23:01 0 comments

    The current objective of the NeuroLS System Version 3 is to change the oscillation between two colors depending on the predominant brain wave of each sensor.

    Luckily the Muse EEG headband has Muse Tools, a software that receives the Muse Raw EEG Data, and processes it to Raw FFT, Absolute Band Powers, Relative Band Powers, etc (http://developer.choosemuse.com/tools/available-data).

    In this case we receive the three Absolute Band Powers (Alpha, Beta, & Gamma) for each sensor (four in total).

    Deltha and Theta brain waves were omitted since the nature of the headset doesn't allow the read this wave length properly.

    After registering this data to arrays we can then determine the predominant brainwave for each sensor. 

    Fig. I

    Once determined, the predominant brainwave is "lit up" by a red rectangle under the brainwave switch, located on the upper left corner (Fig. II). This switch allows to see and change the two stored colors of each brainwave.

    The user can then change the two stored colors through a set of three number boxes (RGB), one for each color. These number boxes, located on the bottom (Fig. II), range from 0 to 255. In addition, there is instant feedback showing the color over the number boxes.

    Fig. II

    Lastly, the waveform switch and time number box, located on the top middle (Fig. II), change the way these two colors oscillate. The waveform switch has five different waveforms:

    • UP: upwards ramp
    • DW: downwards ramp
    • SN: sinewave oscillation
    • TR: triangle wave (upwards and downwards ramp)
    • SQ: square wave (instantly change between both colors).

    The time number box allows to change the oscillation time in milliseconds.

    All this also has an instant feedback, located in the upper right corner (Fig. II) where the oscillation between colors is visible.

  • Testing headset connectivity

    Memo Santos08/15/2018 at 20:19 0 comments

    We were able to use some of our research lab’s material for free. We got access to 2 Muse headsets which were not being used. We had to make sure they were working properly. We had never used the actual SDK kit because we normally get raw data and do the whole processing on Matlab. We are getting familiar with the Muse proprietary tools.

  • Project Introduction: NeuroLS System

    Memo Santos08/03/2018 at 21:43 0 comments

    Special thanks to INNOVaction Gym, our oldest co-work collaborator in Monterrey, México.

    Visit them @ www.fb.com/InnovactionGym/

  • Fig. 3 Pure Data schematic

    Memo Santos08/03/2018 at 21:22 0 comments

    Figure 3. Pure Data schematic. A) Listening for incoming messages from UDP network. B) Parse OSC packets into PD messages. C) Write absolute band powers of Alpha and Beta brainwaves into an array for audiovisual representation.

View all 13 project logs

Enjoy this project?

Share

Discussions

Curt White wrote 08/27/2018 at 03:40 point

Cool project! Would you mind posting your code?

  Are you sure? yes | no

Memo Santos wrote 08/28/2018 at 22:05 point

Thanks Curt! Check the files section you could find our programs there, but make sure you have Pure Data installed and a preferably Muse Headband. https://puredata.info

  Are you sure? yes | no

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates