Close
0%
0%

Magpie MIDI - An Adaptive MIDI/Interface Harmonica

Creative expression tool for cerebral palsy patients

Similar projects worth following
Magpie MIDI is an affordable adaptive tool that enables cerebral palsy patients and others with muscle control disabilities to express themselves in new ways. Meant to be easily customizable to meet different needs of varying degrees of disabilities, every aspect of hardware and software is open-source. The device offers new means for cerebral palsy patients and alike to express their creativity in areas of computer games, music, and writing.

The video below summarizes the progress we have made so far. For more details, please read the descriptions below this video.


Introduction

Magpie MIDI is an adaptive computer accessory for people with cerebral palsy and other muscle control disabilities. These kinds of disorders often have symptoms like impaired movements and involuntary movements. Such symptoms make many of the everyday tasks most of us take for granted difficult and challenging. We believe that if there is one ability that could make these people happier, it’s creative expression. Being able to creatively express ourselves gives us the independence and the freedom we enjoy.

More specifically, we believe that the most accessible way for any of us to express ourselves freely is by using computers and specialized softwares like Photoshop and Logic. However, although adaptive computer accessories do exist, many are expensive, inflexible and are limited in what it can do. Our project aims to resolve all of these issues. It integrates four common computer accessories into one simple easy to use tool: USB keyboard, USB Mouse, Gaming controller, and a MIDI controller.

Be it music, digital arts, writing gaming or whatever else, having all four capabilities in one adaptive device expands the number of tasks individuals with cerebral palsy or other disabilities are able to do on their own.

Core Functions

Users interact with the tool by blowing or drawing their breath into the device’s air holes. Each of the air holes can be customized to execute the desired actions depending on the selected functionality. 

  • MIDI Mode: The Magpie MIDI imitates the functionality of a regular harmonica. At the moment, our device is able to play single notes from C3 to A4. However, by using all the pressure sensors available, upto 26 notes can be played at the same time. We’ve also added a MIDI potentiometer so that it can be used to control sound design parameters such as volume, gain, kill, pan and reverb.
  • Mouse Mode: To use the mouse, the user can simply move their head to move the cursor similar to using a joystick. Blowing into the device will left-click and drawing from the device will right-click.
  • Keyboard Mode: With the 13 air holes on the harmonica, the user can type out the full 26 alphabet letters without the need for additional buttons or controllers. For example, blowing into the first air hole for example will type out the letter ‘a’, and drawing from the first air hole will type out the letter ‘b’.
  • Gamepad Mode: This feature is still in progress, but we are working to make the device able to do everything that a typical console gaming controller is able to do.

Additional Details

  • Switching Between Modes: Users will be able to switch between different modes by what we will be calling ‘gestures’. Our idea currently is to make it so that when the user blows air from left to right of the harmonica unit, Magpie MIDI will switch its function. We plan on adding more ‘gestures’ for other miscellaneous tasks as well such as for typing special characters.
  • Software: We plan on creating a dedicated software for customizing what each of the airways in the device does. Many softwares require shortcuts such as pressing CTRL and W at the same time. Using the software, the users will be able to program this without writing a single line of code.
  • Haptic Feedback: We will be adding haptic feedback to our device. With haptic feedback driven by vibration motors similar to ones found inside phones, the users will find it easier to interact with our device.
  • Multiple Magpie MIDI: We want to support multiple Magpie MIDI to be used simultaneously. For example, many MIDI musicians like to have a MIDI keyboard and MIDI drum pad or MIDI knob placed next to each other.

Future Plans and Improvements

Currently, our device is still in the prototyping stage and is made from 3D printed components and CNC milled circuit boards. We are planning to add the additional features mentioned above...

Read more »

  • Hardware Update - Prototyping the Harmonica Unit

    Shu Takahashi09/01/2020 at 12:50 0 comments

    We have been working to prototype the harmonica unit for the past couple of weeks, but we have yet to share this on our log page. However, now that most of the work on the harmonica unit is done, we’re ready to share what we have accomplished so far.

    At this stage, the design requirements for the harmonica were very clear. It needed to hold an array of 13 air pressure sensors, hold the multiplexer breakout board, be in the form of a harmonica, and fit together with the base unit. This meant designing both the plastic components as well as a PCB board to hold all the electronics components together.

    I first designed the air holes and the PCB to go with it. I used the same dimensions we used when Pato and I experimented with the air pressure sensor a few days ago. Designing the PCB and the air pathway component to fit together was really challenging because it required a close coordination between my CAD program, Fusion 360 and my EDA program EAGLE. I initially tried to do everything in Fusion 360 with the new update that integrated EAGLE; however, I had problems with routing and exporting at a later stage, so I used two separate programs.

    Printing the air pathway component was straight forward. Although the back part of the component warped a little bit because I used ABS plastic. Fabricating the circuit board was a little bit more challenging. To begin with, this was my first time milling my own PCB using a CNC machine. Learning the process for operating the CNC machine was quite a bit of a learning curve. Also, I unintentionally designed the PCB from the bottom side. This resulted in a PCB layout that was mirrored horizontally. Luckily this problem was resolved by flipping the export image horizontally, but the whole process took about three days, which delayed this project by a few days.

    In the circuit board, I also included a transistor switching circuit to control the vibrational motor from the Arduino. The first circuit board had errors, but the second attempt was a success. I was able to place all the required electronic components in the harmonica unit into a one nice PCB.

    The circuit board and the air pathway plastic component fit together well on the first try which was a bit of a satisfying moment. In the air pathway component, I made a small tunnel to route the wires through so that it can access the Arduino Leonardo in the base unit.

    Now all there was to design for the harmonica unit was a nice enclosure to hide all the electronic components and the air pathway component. Designing the enclosure required a bit of thinking and quite a lot of time as well. First, I wasn’t sure how the enclosure should fit together. I initially thought of making the enclosure like a sleeve, but this was not possible because we were still at a prototyping stage and so the electronics needed to be easily accessible in case something went wrong. At the end, I designed a two part enclosure that covered the components from the top and the bottom. I designed the two enclosure components to be held together using a nut and a bolt. I put an effort into hiding the screws from the top side to make the enclosure look nice.

    Printing the bottom and the top enclosure took a very very long time. This is partly because I chose to use an SLA printer as the two enclosure components had some complicated geometries that could be damaged from typical FDM supports. But the wait was worth it. Although there were some warps in the final print, it still looked very similar to what I designed in CAD.

    Although the prints looked great, it still needed some post processing. I filed the sides that had the supports to get a smoother surface. The pockets I designed in the top enclosure to fit the nuts came out too small. Instead of trying to reach in and remove the excess material inside the pockets, I grinded the nuts so that it can fit inside.

    Everything was coming together nicely. But that was until I tried assembling the harmonica...

    Read more »

  • More Experiments with Electronics and Sensors

    Shu Takahashi09/01/2020 at 11:55 0 comments

    Choosing Electronic Components

    Having chosen the air pressure sensor, there were still many electronic components to select and test. Most notably, the microcontroller and the joystick sensor.

    For the microcontroller, I initially wanted to choose something with plenty of memory space and options for wireless connectivity such as the ESP32. However, as I was testing out the mouse library with my Arduino UNO, I realized that most boards I’ve used in my past projects do not support it. I found out that to use the native mouse and keyboard library, the microcontroller had to be 32u based. This narrowed down my options significantly. With the limited time I had left to order and wait for parts to arrive, my best option was to use the Arduino Leonardo. This board doesn’t have any wireless options built-in nor much memory space compared to boards like the ESP32, but it’s suffice for the first proof of concept prototype.

    Arduino Leonardo has 12 analog input pins. However, for this project I will need a total of 16 analog pins. 13 from the pressure sensors, two from the joystick unit and one from the potentiometer. The first intuitive solution I had in my head was to use a multiplexer. Without much thinking, I ordered a breakout board for a multiplexer IC CD74HC4067 from Texas Instruments. When the breakout board arrived and I thought about how the program would work, I realized that I would not be able to program an external interrupt. There may be a work around to address this oversight such as using a scheduled interrupt, but for now, my option will be limited to polling. This may limit the features and performance of the output programs (i.e. MIDI or USB Mouse), but it will be enough for the first proof of concept prototype.

    Choosing the joystick sensor was relatively straightforward. As Pato and I discussed in our CTQ, we wanted to make this device as accessible and affordable as possible. For this reason, we decided to use the joystick unit from a Xbox 360 controller. These joystick units can be bought in many countries around the world as a replacement part. Just for making the prototyping process easier, I also bought a joystick unit that came with a breakout board from a company called HiLetgo. This joystick unit also seemed to be very popular as I could find the identical product on Amazon in the US, Mexico and Japan. Coincidently, when I compared these two modules, the joystick was identical. So for now, I decided to use the Joystick module from HiLetgo because it already had a breakout board attached. In the later prototyping stages, I will be able to design a PCB for the Xbox joystick, so that it can also be used to build the device.

    Testing the Joystick Mouse

    To make sure that the joystick module and the Arduino Leonardo I ordered worked together, I ran the example code from the official Arduino documentation. The program worked as expected and I was able to control my computer mouse using the joystick module. It’s not too clear yet how this program will work in conjunction with everything else the processor needs to process, but hopefully I will figure it out in the coming days.

    Testing the Multiplexer

    The multiplexer board will be connected to the 13 air pressure sensors. It will essentially act as a rotary switch to choose which channel to connect the Arduino to. The Arduino with the multiplexer will be constantly switching between the 13 channels and writing the results to an array.

    This array can then be used by the program to take specific actions based on the sensor values. At this stage, my goal was to make sure that the multiplexer could be used to switch between the 13 channels and have the data recorded on to an integer array.

    Which channel to connect to is controlled using four digital pins that receive the channel number in binary. For example writing LOW, LOW, LOW, HIGH to the four pins will connect the Arduino’s analog pin to channel number one on the multiplexer. For each air pressure sensor,...

    Read more »

  • Choosing and Testing Air Pressure Sensor

    Pato Montalvo08/25/2020 at 02:51 0 comments

    The most fundamental component in our device is the air pressure sensors. All of the functionalities and actions revolve around the effective use of this technology, meaning that we need to choose a suitable sensor to have a good overall performance.

    While in team discussions, we went over what were the most important aspects that these sensors should have. Like all of our other desired parts, they should be accessible and affordable. 

    Regarding the technical specs, we started by brainstorming the general design, like Shu described in the previous post. We knew that each one of the harmonica’s air holes would have one individual sensor. This meant that the sensors will be exposed to atmospheric pressure constantly, in contrast with manometric sensors. That narrowed our options to using absolute pressure sensors that measure gauge pressure. 

    As seen on our BOM log post, we considered numerous options. Shu ordered a couple of them each and we designed a test sequence to determine which sensor is the most appropriate.

    Shu designed a simple air hole to measure different interactions with the sensors. As you can see in the picture below, we chose a square air hole to replicate that of an original harmonica and added a pressure relief exhaust orifice at the back of the air hole. The dimensions of these simulations were also evaluated using fluid dynamics principles (Bernoulli’s Principle).

    We considered the following formula for our design:

    Where:

    Δp = difference in pressure (desired on the inside vs avg atmospheric pressure on the outside)

    ρ = Air density

    Q = Hydraulic Head 

    A2= Area of the orifice

    A1= Area of the air hole inlet

    For these designs, we also researched the common lung capabilities of the average human. We decided to choose values that would be an average of general human capabilities, (eg. 6 m/s blow airspeed[1] / 0.8 psi exhalation pressure). With the blow speed and the cross-sectional area of the airhole, we calculated the head value and proceeded to iterate until we decided to set for a 2mm diameter orifice.

    As you can see in the picture below, my partner Shu conducted the tests for each of the sensors and its compatibility with our designs. 

    An important thing to note is that we are considering a blow and draw functionality in our device, just like in a regular harmonica. That is why we had to make sure that our potential sensor could detect both sip and puff.

    After the tests, we concluded that the MPS-2400 series sensors were the most suitable. We are confident we’ve made the right decision choosing this sensor, especially because of the process we used to select it. It will definitely bring the best out of this new adaptive tool.

    References:

    [1]Flow Through an Orifice
    https://mcnallyinstitute.com/13-html/13-12.htm

    [2]Human Exhaled Air Energy Harvesting with Specific Reference to PVDF Filmhttps://www.sciencedirect.com/science/article/pii/S2215098616300830

    [3]Maximum Static Inspiratory and Expiratory Pressures with Different Lung Volumes https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1501025/

  • Hardware Update - Prototyping the Base Unit

    Shu Takahashi08/21/2020 at 01:43 0 comments

    The first part we decided to start designing was the base unit. This part is responsible for providing the joystick movements as well as the MIDI knob (potentiometer) movements.

    We decided to work on this first because we were still discussing how the harmonica unit would work. Besides, working on this part was one of our top priorities as this must function flawlessly and hold the harmonica unit steadily. I had an idea of how the joystick module and the potentiometer were going to function together, so I made a quick sketch before creating any models in CAD.

    I began by creating simple test designs for the joystick handle part using Fusion 360. I had never 3D printed parts for a joystick before, so I wanted to make sure that I could design this component properly first. I took a picture of the joystick handle that came with the module and made measurements from there. I also added a ball joint and a cover to ensure that the joystick could still move freely.

    Before moving on to designing the base unit, I also decided to test fit the joystick module and the joystick knob handle to find the appropriate measurements. It’s always difficult to find the right dimensions for the parts to fit together snuggly and this time was no different. Luckily for this test component, only two re-prints were needed.

    Now that I had tested some of the detailed parts I was concerned with, I was confident that I could design the entire base unit. The first draft of the design took about two hours. For now, I decided to use small wood screws to hold the components together. I plan on considering using other methods for the assembly process later.

    All of the components printed without any major issues, except one. The component I had trouble with was the joystick handle. This component was both difficult to design and fabricate. This component had to be designed so that wires from the circuit board inside the harmonica unit could pass through the component while providing the snug fit to the joystick module. In terms of fabrication, I first tried 3D printing the component with an FDM printer, but as you can see in the photo, the outer part was too thin. So, I tried printing this component with an SLA printer. It worked but I had to re-print three times and make additional adjustments. As you can also see, in the final version, wires could pass through without affecting the snug fit to the joystick module.

    I also came across another issue. The rotary joint between the joystick module and the potentiometer wiggled significantly when the joystick was moved. I should have noticed this in CAD, but it’s always a challenge to think about how things are going to work in the real world versus what it looks like in CAD.

    To make the platform, which the potentiometer module sits on more stable, I added a circular guide around the potentiometer to provide additional support.

    It took a while to make adjustments, wait for the 3D printers, and to test fit the components; however, all this patience paid off at the end and I was satisfied with the result. The video below is a short demo of the base unit.

    All there is left to do for the base unit is to add the enclosure for holding the electronics components that will be located in the base unit. The walls of the base unit still wiggles a little bit, but I hope to resolve this issue in the next prototyping stage.


  • Bill Of Materials

    Pato Montalvo08/21/2020 at 01:25 0 comments

    To move forward, we decided to organize our proposed components in an excel sheet. The purpose is to evaluate and compare each one of the parts against its alternatives to meet the specified ends. We created a document dubbed “Bill of Materials Draft” where we listed components based on our desired functionality. We included different alternative components for each functionality. For each part, we gathered information like a link to its datasheet, a list of suppliers with different prices, its picture, and a brief description. 

    Before we had this, discussion on specific sensors was not efficient. This document has definitely helped a lot in deciding the main components. As of now, we have most of the major components on our BOM, and we expect to add more components as we make more progress. 

  • CTQs and Defining Our Project

    Pato Montalvo08/21/2020 at 01:10 0 comments

    While discussing the scope of the project, we had different ideas about the functionalities the device would have. We had been considering multiple features until that point, but nothing was really set in stone.

    That is why we decided to evaluate the potential the device could have. Through research and discussion, we attempted to define its capabilities. 

    We settled on the following functionalities:

    • MIDI Instrument: A mouthpiece simulating a harmonica will be built to control a Musical Instrument Digital Interface (MIDI). The device will have 13 airways with built-in air sensors, each of which can be configured to execute different commands. As a musical instrument, it would be able to play numerous octaves and notes through the interface.
    • Gaming Controller / Mouse: The device will also work like a computer mouse and a gaming controller. It will support numerous additional outputs including left and right-click to game function buttons that are on typical gaming pads.
    • Keyboard: The13 air sensors will also have a “key” functionality allowing users to “type” using the harmonica.

    Our next step was starting a critical to quality analysis (CTQ) where we defined the critical aspects of each functionality. Initially, we did this separately, with Shu and I making our own analysis. Then, we evaluated each other's work and created a definitive CTQ analysis by aggregating the individual ones. The diagram below is our final CTQ showing the requirements for each functionality.

    The full diagram can be viewed from the link below:

    https://docs.google.com/document/d/e/2PACX-1vToisXb_tbWGin7M6cg-1xUX57Q9Fi7K6FQsVNLn1ia1_xWr_dVCbtfjrOn8ZflgTRp2OaFlYzunCt-/pub

    Creating this diagram was an important milestone. It opened the doors to the exploration of the components that would be needed to make all of these functionalities possible. 

    I know that getting an idea of our full CTQ is difficult from the diagram alone. For some of the requirements, we had long discussions to clarify what each of us meant. I’ve listed the details we discussed for some of the key requirements that are worth mentioning below.

    “Must also be able to function as MIDI pad and knob controller”

    We initially only had a MIDI keyboard in mind, but we realized that by adding a potentiometer to the device, it could also function as a MIDI knob controller and MIDI pad like in the image below.

    Figure adopted from the image cited below.

    (B&H Photo, 2020)

    “The controller must be as universal as possible and support many users with different needs”

    We know that cerebral palsy has a wide range of effects on motor controls, and this is different for everyone. Although we know that this device alone will not be able to support every user with cerebral palsy, we want to do our best to help as many people as possible. One way we’re planning to do this is to make everything, including hardware, electronics, and software open-source. By making everything open to the public, we believe that a community of users and developers could come together to share hardware and software customizations.

    As we design our product, we will also keep in mind about universality. For example, we’ve already decided to use 1/4-20 UNC threads, the most common type of thread used in consumer-grade cameras. This will allow various camera accessories such as flexible tripods or stands to be used in conjunction with our device.

    We believe that small design choices like this can add up to make our device as universal as possible. Additionally, we know that making this device truly universal is not possible, and perhaps this is the most challenging aspect in designing adaptive devices; however, we believe that creating a user and developer-driven community will give the best chance for this device to reach its full potential.

    “The device must give some sort of live feedback → i.e. haptic feedback”

    We added this to our CTQ after...

    Read more »

  • Coming up With Our Idea

    Shu Takahashi08/11/2020 at 07:39 0 comments

    We decided to give ourselves some time before our first video call meeting to thoroughly read the design briefs for each nonprofit challenge and think about ways to solve each issue. Then during the meeting, we discussed the benefits and drawbacks of each idea, debated on the feasibility of each idea given the time and resources we had available and contributed our thoughts. Below, we’ve summarized the discussion we had for the idea we decided to pursue as well as two other ideas we had.

    Mouth Controlled MIDI/Game Controller/Mouse / United Cerebral Palsy Los Angeles / Shu

    This device is an external computer accessory for patients with cerebral palsy and other disabled patients that struggle with fine motor controls. The device is controlled using the user’s mouth, head movement and breath. It will support multiple purposes for creative expressions and recreations. The primary functions of the device include being used as a computer mouse, gaming controller, keyboard and a MIDI controller (to play music using DAW software). Each of these functions can be switched by using predetermined breath gestures. For example, puffing into the harmonica from left to right would switch the device from the mouse function to a keyboard function. Also, this system could be expanded by adding additional devices on the sides to meet different needs of the users. For example, a musician using this system could have the main device in the front acting as the MIDI keyboard and the device on the side acting as a MIDI knob device. Such functions of what each of the puffs does will be fully customizable through an easy to use software that maps out this device visually.

    I came up with this idea after watching a TED talk called Disability Technology by Jeff Paradee. He introduced a sip and puff device that was specially designed for playing video games. This device has joystick that is controlled using the mouth to act as a computer mouse and an airway that allowed users to click on elements in a GUI environment. I instantly thought that this was one form of creative expression and wondered if this concept of sip and puff could be applied to other means of artistic expressions.

    (Quadstick, 2020)

    After some thoughts, my mind settled on creative writing, digital arts, music composition and of course gaming. The idea was to add more airways to add additional functionalities to the device. From observing that a MIDI device and a keyboard is just an array of buttons, I drew an analogy between this idea and a harmonica. The main source of inspiration for drawing this analogy was from seeing musicians play the harmonica hands-free while playing the guitar.

    (Rowland Scherman, National Archives and Records Administration, 2020)

    We saw some potential problems that could get in our way while prototyping this device. The first potential issue was how we were going to detect the sip and puff from the users. We initially thought about using an array of microphones to detect the breaths; however, after more research, we found that an air pressure sensor was more appropriate for this purpose. We also thought of other means to detect the change in airflow, and we’re still in progress of deciding what kind of sensor to use. Another potential issue was how a gaming joystick and the potentiometer was going to work together mechanically. We are still thinking of ways to overcome this problem.

    Despite the potential issues we saw in our way, we believed that this idea was the most suitable project given the time and resources available. This project was challenging enough for us, but we could still see the light at the end of the tunnel. It was a project we felt we could utilize both of our skills both of us had developed over the years.

    Smart Text to Speech Glass / United Cerebral Palsy Los Angeles / Shu

    This is an adaptive tool designed to facilitate communication for cerebral palsy patients and other patients with similar needs. I saw the need for such a device...

    Read more »

  • Introduction from Pato

    Pato Montalvo08/11/2020 at 01:51 0 comments

    Hello everyone, Pato here! I’m so excited to be starting this project with my good friend Shu.

    I graduated from Electro-mechanical engineering last year, and since then, I’ve been looking for experiences to complement my professional development. As Shu mentioned, I have a great passion for electrical engineering, but I also love prototype design. 

    When Shu told me about this project and invited me to work together as a team, I couldn’t say no.  I know that he is a very talented maker, and I’ve learned a lot from him. I knew this was an opportunity to keep honing skills I’m interested in, and most importantly, to create something of value and transcendence. 

    I’m confident that whatever comes out of this collaboration will be a product with a meaningful impact. Let’s get to work!

  • Introduction from Shu

    Shu Takahashi08/11/2020 at 01:49 0 comments

    Hi and welcome to our first log on this project! My name is Shu and in this log, I will be introducing myself followed by my partner, Pato in the next log.

    In mid-July, I was stuck at home for a couple of months after graduating high school waiting for my first year at university to begin. I had been trying to work on some digital fabrication projects, but frankly, I felt that I could be spending my time on some other projects that would actually help people. On top of that, I wanted to work on a project with somebody else because I had only been listening to my own thoughts all break and it was driving me insane.

    This is about the time I read about this prize on the Hackaday website. I instantly liked the way the prize had organized real world problems into tangible project challenges. I thought that if I could participate in one of the challenges, I could be sure that I’m contributing my time and effort to something meaningful.

    To combat my annoying internal thoughts and also to hold myself accountable to somebody else, I decided to invite my friend Pato. I met Pato during my internship last year. He has substantial expertise in electrical engineering and I was sure that his experience could be a great help. I also wanted an opportunity to work on a project together since he left Japan from an internship that felt like such a short time.

    Fortunately, he agreed to work on this prize with me. We decided to meetup over a video call to discuss our ideas for the nonprofit challenges. We will follow up with another log to recall our discussion where we decided on which challenge to tackle. But before that, Pato will introduce himself briefly in the next log. Thank you for reading and we look forward to keeping you updated.

View all 9 project logs

Enjoy this project?

Share

Discussions

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates