PhotoSynth - Instant Musical Superpowers!

Smile Powered Synthesizer

Similar projects worth following
The goal of this project is to create new ways to control technology that require no physical touch, or any specialised equipment, and that are entirely inclusive, accessible and adaptable.

In this first experiment, the technology is used to control an expressive musical instrument simply using the movements of your face, tracked by your device's camera.

This is both a creative tool but also proof of the power of the concept, showing that an expressive and simple interface can control more than household appliances.

By connecting a microcontroller via USB such as a PICO, arduino or ESP to any device and you now have face control over anything, from remote control cars to washing machines - this even works from mobile devices thanks for the universality of MIDI!

This is a project that uses instantly bestow your movements with new capabilities and musical superpowers!

Protoype :

Current state of the art motion capture devices are complicated, expensive and bespoke pieces of equipment - I intend to remove the need for any specialised equipment entirely and remove the cost so as to completely democratise accessible interface technology. This software aims to bring more equality to skills and leverages technology to empower and include the most diverse and varied audience - giving confidence to the most underrepresented and unprivileged.

This is as much a research project into perfecting natural instinct interfaces for the future of human-machine interaction as it is an awesome way to make music!  

This product will allow as many people to make music as possible from the youngest children to the oldest adults - in many ways it will be the purest accessible musical instrument as there is no physical touch required which post COVID is of extra importance!

By connecting to a flashed Pico or Arduino with USB HID MIDI capabilities we can make the software control any electronics we like, from radio controlled cars to video games controllers!

  • 1 × Webcam or Camera phone If you don't have a camera on your device, grab a USB one
  • 1 × Modern computer or mobile device GPU accelerated computers run faster
  • 1 × (Optional) MIDI Instrument Optional MIDI device to play from software
  • 1 × (Optional) PI Pico To allow the software to control other things than MIDI
  • 1 × (Optional) Holographic display Special hologram mode on Looking Glass Portrait device

  • MIDI Control

    zen05/29/2023 at 11:09 0 comments

    Making the software control external hardware from every device is possible thanks to WebMIDI! My original plans for use serial port still works but only really has access when used on a desktop system... However all these messages can also be sent over WebMIDI which just so happens to work across *all* devices including mobiles and tablets! The MIDI signal can be sent over a standard 2.5mm stereo jack and so with the right adapter (USB-C to MIDI2) you can use the Photosynth to control other musical equipment as well as using the signals to controls other hardware such as pointing devices, letter selectors, volume dials, smart home lights and assistants such as Siri and Alexa.

  • 0.6.1 is finally here

    zen06/28/2021 at 18:58 0 comments

    So after trialling it out with some sensitive induvials, it turns out that certain features can appear quiet scary so to make it easier for kids to use, and for an extra layer of fun, you can now superimpose a range of different eyes, from Terminator style red eyes to Frank Sidebottom style googly eyes! 

    NB. Safari may now work, thanks for all your valuable feedback!

  • Hosting fails!

    zen06/22/2021 at 20:08 0 comments

    Well so much for a free ride.... Turns out CloudFlare have a 20,000 file limit for their hosting which means only one sound collection can be used so looks like will always be throttled :(

    I am migrating to github which I am hoping fixes this issue!

  • New domains / new capabilities

    zen06/14/2021 at 08:54 0 comments

    Some new domains to host and change the nature of the project have been acquired and now the project will be permanently hosted at

    the latest test build and beta version is available at

    for kids / fun versions, they should be available at

    for body version a new domain will be announced soon!

  • Hardware integration - RC Car

    zen06/13/2021 at 11:12 0 comments

    As this is an experiment into the future of human-machine interactions, it is very important that the concepts can be used outside of the context of musical instruments. 

    Using the available everywhere web-MIDI, a simple adapter can be created that can then be patched into any other electronics.

    I've been thinking about the best hardware to use as a demonstration and I think I've found it and will attempt to create a face controlled RC car!

    If anybody has recommendations for cheap hackable cars please comment :)

  • Version 0.6 is here

    zen06/13/2021 at 10:44 0 comments

    For the Hackday screen contest comes a new release!

    - New kid friendly mode

    - Different masks and colour schemes

    - Better sounding audio engine with configurable reverbs

    - Toggle buttons to disable audio if other control hardware is connected

    - Extra settings panel and new "advanced mode"

  • Launching version 0.5

    zen06/08/2021 at 12:46 0 comments

    Check it out and please relay your feedback!

    Version 0.5 has gone live and offers a new DUET mode and some extra settings for improving performance.

    WebMIDI is available - Pi Pico can read USB MIDI and so will be able to intercept the commands for use controlling things other than musical instruments.

View all 7 project logs

  • 1
    Checkout the source code

    Head on over to the github repository and download the entire folder or checkout using GIT.

  • 2
    Install dependencies

    Install NodeJS and Yarn if you have not already done so. Once installed, navigate to the downloaded directory and type 


  • 3
    Build the software

    At this point you can run 

    yarn start

    to launch a local webserver that hosts the application and any changes to the code will be reflected in the application.

View all 6 instructions

Enjoy this project?



Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates