Close
0%
0%

interFACE.place - Smile Powered Synthesizer

A face controlled musical instrument that uses only your web camera

zenzen
Public Chat
Similar projects worth following
The goal of this project is to create new ways to control technology that require no physical touch, or any specialised equipment, and that are entirely inclusive, accessible and adaptable.

In this first experiment, the technology is used to control an expressive musical instrument simply using the movements of your face, tracked by your device's camera.

This is both a creative tool but also proof of the power of the concept, showing that an expressive and simple interface can control more than household appliances.

Using motion capture in real-time allows us to track a person's own physical expressions and movements so as to control and manipulate hardware, software and data resulting in an augmented reality, machine-learning powered, holographic screen compatible, general MIDI controller and synthesizer that is used as a website, requires no instructions and can be extended to control almost anything with a little bit of custom hardware!

This is a project that uses your web camera and cutting edge motion-tracking technology to instantly bestow your movements with new capabilities and superpowers.

Protoype : interface.place

Current state of the art motion capture devices are complicated, expensive and bespoke pieces of equipment - I intend to remove the need for any specialised equipment entirely and remove the cost so as to completely democratise accessible interface technology. This software aims to bring more equality to skills and leverages technology to empower and include the most diverse and varied audience - giving confidence to the most underrepresented and unprivileged.

As the technology is very new, this product is the first of it's kind and the scope of it's ability falls far beyond a musical instrument. This is as much a research project into perfecting natural instinct interfaces for the future of human-machine interaction. 

This product will allow as many people to make music as possible from the youngest children to the oldest adults - in many ways it will be the purest accessible musical instrument as there is no physical touch required which post COVID is of extra importance!

By connecting to a flashed Pico or Arduino with USB HID MIDI capabilities we can make the software control any electronics we like, from radio controlled cars to video games controllers!

  • 1 × webcam If you don't have a camera on your device, grab a USB one
  • 1 × modern computer / mobile with GPU GPU accelerated computers run faster
  • 1 × MIDI Instrument Optional MIDI device to play from software
  • 1 × PI Pico To allow the software to control other things than MIDI

  • 0.6.1 is finally here

    zen06/28/2021 at 18:58 0 comments

    So after trialling it out with some sensitive induvials, it turns out that certain features can appear quiet scary so to make it easier for kids to use, and for an extra layer of fun, you can now superimpose a range of different eyes, from Terminator style red eyes to Frank Sidebottom style googly eyes! 

    NB. Safari may now work, thanks for all your valuable feedback!

  • Hosting fails!

    zen06/22/2021 at 20:08 0 comments

    Well so much for a free ride.... Turns out CloudFlare have a 20,000 file limit for their hosting which means only one sound collection can be used so looks like interface.dance will always be throttled :(

    I am migrating interface.place to github which I am hoping fixes this issue!

  • New domains / new capabilities

    zen06/14/2021 at 08:54 0 comments

    Some new domains to host and change the nature of the project have been acquired and now the project will be permanently hosted at 

    interface.place

    the latest test build and beta version is available at

    interface.dance

    for kids / fun versions, they should be available at

    interface.lol

    for body version a new domain will be announced soon!

  • Hardware integration - RC Car

    zen06/13/2021 at 11:12 0 comments

    As this is an experiment into the future of human-machine interactions, it is very important that the concepts can be used outside of the context of musical instruments. 

    Using the available everywhere web-MIDI, a simple adapter can be created that can then be patched into any other electronics.

    I've been thinking about the best hardware to use as a demonstration and I think I've found it and will attempt to create a face controlled RC car!

    If anybody has recommendations for cheap hackable cars please comment :)

  • Version 0.6 is here

    zen06/13/2021 at 10:44 0 comments

    For the Hackday screen contest comes a new release!

    - New kid friendly mode

    - Different masks and colour schemes

    - Better sounding audio engine with configurable reverbs

    - Toggle buttons to disable audio if other control hardware is connected

    - Extra settings panel and new "advanced mode"

  • Launching version 0.5

    zen06/08/2021 at 12:46 0 comments

    Check it out and please relay your feedback!

    Version 0.5 has gone live and offers a new DUET mode and some extra settings for improving performance.

    WebMIDI is available - Pi Pico can read USB MIDI and so will be able to intercept the commands for use controlling things other than musical instruments.

View all 6 project logs

  • 1
    Checkout the source code

    Head on over to the github repository and download the entire folder or checkout using GIT.

  • 2
    Install dependencies

    Install NodeJS and Yarn if you have not already done so. Once installed, navigate to the downloaded directory and type 

    yarn 

  • 3
    Build the software

    At this point you can run 

    yarn start

    to launch a local webserver that hosts the application and any changes to the code will be reflected in the application.

View all 6 instructions

Enjoy this project?

Share

Discussions

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates