Magpie MIDI - An Adaptive MIDI/Interface Harmonica

Project Logs

Collapse

BOM Update
Pato Montalvo • 10/05/2020 at 15:16 • 0 comments

As part of our update, we upgraded our BOM for it to contain more information about our device.

Level Hierarchy

First, I included a level hierarchy. This makes it easier for the reader to understand how the different components come together. Level 1 is the main device, and as the level go higher, you can see the components that are used in each sub-assembly.

As an example, on how to uses this nomenclature, let's pick component BUPCB2B Base Unit Controller PCB. As you can see, this component is listed as LVL 4. If you go up, you'll see that the first LVL 3 sub-assembly is the BUMUV2 Base Main Unit. That means that BUPCB2B is part of BUMUV2.

Component Index

We also assigned a Tag to all of our components to help us understand them faster. Tags begin with either E, I, or X. "E" means that this component comes from an external supplier. "I" indicates that this is a component that we designed and produced. Finally, "X" is used for sub-assemblies.
Component Universality

One of the main goals of this project is to make this device accessible. To accomplish that goal, all of our external supplier components were checked to verify their accessibility in Central and North America (MEX/USA/CANADA) and Japan. This will help guarantee that people living in these zones will be able to find the necessary components produce their own MAGPIE MIDI.
Long Big Update
Shu Takahashi • 10/05/2020 at 13:54 • 0 comments
Hello everyone, it’s been a while since we last updated the community on the progress we’ve made. First of all, we are incredibly honoured to be one of the finalists for the Hackaday Prize 2020. We are extremely excited to keep on pursuing this project. I know it’s been a long time, but trust us, we’ve been working hard to make more improvements so we can get this device to people that need it fast.

Admittedly, we have found it more difficult to make progress because of the time constraints. I started my first year at the university about five weeks ago in Canada (but I’m in Japan) which has taken away the time I had to work on this project during the summer holidays. Pato changed his position at his company and this made it more difficult for both of us to coordinate together, not to mention the significant time difference we have between ourselves.

Nonetheless, we have both tried to contribute as much as we could to this project. Although we’ve worked on this project throughout the last month, it’s been more difficult to write a log for each of the small chunks of the tasks we finished. But enough about ourselves, here is a long update of everything we’ve accomplished in the last month. I’ve written this log with the headings listed below, and hopefully this will make it easier to read this update.
- Hardware Updates
- Software Updates
- General Updates
Hardware Updates

We’ve made significant changes to the hardware of Magpie MIDI. Magpie MIDI 2.0 aims to be more fabricatable at large scales. Although the end result is in no means perfect, it was overall a good experiment to test the design ideas we had after creating the first prototype.

New PCB designs

We mentioned that we were going to move on from CNC milled circuit boards to manufacturable PCBs, and this is exactly what we have done. We’ve redesigned the PCBs both for the harmonica unit and the base unit.

The harmonica unit’s PCB now uses the SMD version of the multiplexer, CD74HC4067 as well as other SMD components except for the air pressure sensors. This made the PCB much more compact and opened the door for new potential designs. In addition, we’ve also added a male USB type-C connector to the PCB for reasons we will explain in the following section.

The base unit now has two PCBs. One is used to connect the USB type-C connector from the harmonica unit to the main PCB. The other is the main PCB which has the Arduino board. We’ve decided to switch from Arduino Leonardo to Arduino Micro so that the base unit is more compact, and the USB cable is able to come out from the sides, not from the bottom of the base unit. Although this has improved the base unit overall, in the future, we want to get rid of the Arduino and make our own 32u based boards to better customize it to our needs.

New harmonica unit design

Because of the redesigned PCB, we were able to make significant improvements to the harmonica unit. In the first prototype, it was extremely difficult to assemble the harmonica unit as it required multiple components to be held together at the same time. In the new harmonica unit design, we were able to simplify the design so that it’s easier to fabricate and assemble.

As I mentioned earlier, the harmonica unit’s PCB now has a male USB type-C connector. This connector is used to connect the electronics in the harmonica unit to the electronics in the base unit. In the first prototype, this connection was made using thin wires that ran through the joystick unit. This made the assembly much more difficult and made the base and harmonica unit inseparable. By using a USB type-C connector, the two units can now be taken apart for easier assembly and better transportability.

The outer casing is now in the shape of a single piece sleeve instead...
Read more »
Hardware Update - Prototyping the Harmonica Unit
Shu Takahashi • 09/01/2020 at 12:50 • 0 comments

We have been working to prototype the harmonica unit for the past couple of weeks, but we have yet to share this on our log page. However, now that most of the work on the harmonica unit is done, we’re ready to share what we have accomplished so far.

At this stage, the design requirements for the harmonica were very clear. It needed to hold an array of 13 air pressure sensors, hold the multiplexer breakout board, be in the form of a harmonica, and fit together with the base unit. This meant designing both the plastic components as well as a PCB board to hold all the electronics components together.

I first designed the air holes and the PCB to go with it. I used the same dimensions we used when Pato and I experimented with the air pressure sensor a few days ago. Designing the PCB and the air pathway component to fit together was really challenging because it required a close coordination between my CAD program, Fusion 360 and my EDA program EAGLE. I initially tried to do everything in Fusion 360 with the new update that integrated EAGLE; however, I had problems with routing and exporting at a later stage, so I used two separate programs.

Printing the air pathway component was straight forward. Although the back part of the component warped a little bit because I used ABS plastic. Fabricating the circuit board was a little bit more challenging. To begin with, this was my first time milling my own PCB using a CNC machine. Learning the process for operating the CNC machine was quite a bit of a learning curve. Also, I unintentionally designed the PCB from the bottom side. This resulted in a PCB layout that was mirrored horizontally. Luckily this problem was resolved by flipping the export image horizontally, but the whole process took about three days, which delayed this project by a few days.

In the circuit board, I also included a transistor switching circuit to control the vibrational motor from the Arduino. The first circuit board had errors, but the second attempt was a success. I was able to place all the required electronic components in the harmonica unit into a one nice PCB.

The circuit board and the air pathway plastic component fit together well on the first try which was a bit of a satisfying moment. In the air pathway component, I made a small tunnel to route the wires through so that it can access the Arduino Leonardo in the base unit.

Now all there was to design for the harmonica unit was a nice enclosure to hide all the electronic components and the air pathway component. Designing the enclosure required a bit of thinking and quite a lot of time as well. First, I wasn’t sure how the enclosure should fit together. I initially thought of making the enclosure like a sleeve, but this was not possible because we were still at a prototyping stage and so the electronics needed to be easily accessible in case something went wrong. At the end, I designed a two part enclosure that covered the components from the top and the bottom. I designed the two enclosure components to be held together using a nut and a bolt. I put an effort into hiding the screws from the top side to make the enclosure look nice.

Printing the bottom and the top enclosure took a very very long time. This is partly because I chose to use an SLA printer as the two enclosure components had some complicated geometries that could be damaged from typical FDM supports. But the wait was worth it. Although there were some warps in the final print, it still looked very similar to what I designed in CAD.

Although the prints looked great, it still needed some post processing. I filed the sides that had the supports to get a smoother surface. The pockets I designed in the top enclosure to fit the nuts came out too small. Instead of trying to reach in and remove the excess material inside the pockets, I grinded the nuts so that it can fit inside.

Everything was coming together nicely. But that was until I tried assembling the harmonica...
Read more »
More Experiments with Electronics and Sensors
Shu Takahashi • 09/01/2020 at 11:55 • 0 comments

Choosing Electronic Components

Having chosen the air pressure sensor, there were still many electronic components to select and test. Most notably, the microcontroller and the joystick sensor.

For the microcontroller, I initially wanted to choose something with plenty of memory space and options for wireless connectivity such as the ESP32. However, as I was testing out the mouse library with my Arduino UNO, I realized that most boards I’ve used in my past projects do not support it. I found out that to use the native mouse and keyboard library, the microcontroller had to be 32u based. This narrowed down my options significantly. With the limited time I had left to order and wait for parts to arrive, my best option was to use the Arduino Leonardo. This board doesn’t have any wireless options built-in nor much memory space compared to boards like the ESP32, but it’s suffice for the first proof of concept prototype.

Arduino Leonardo has 12 analog input pins. However, for this project I will need a total of 16 analog pins. 13 from the pressure sensors, two from the joystick unit and one from the potentiometer. The first intuitive solution I had in my head was to use a multiplexer. Without much thinking, I ordered a breakout board for a multiplexer IC CD74HC4067 from Texas Instruments. When the breakout board arrived and I thought about how the program would work, I realized that I would not be able to program an external interrupt. There may be a work around to address this oversight such as using a scheduled interrupt, but for now, my option will be limited to polling. This may limit the features and performance of the output programs (i.e. MIDI or USB Mouse), but it will be enough for the first proof of concept prototype.

Choosing the joystick sensor was relatively straightforward. As Pato and I discussed in our CTQ, we wanted to make this device as accessible and affordable as possible. For this reason, we decided to use the joystick unit from a Xbox 360 controller. These joystick units can be bought in many countries around the world as a replacement part. Just for making the prototyping process easier, I also bought a joystick unit that came with a breakout board from a company called HiLetgo. This joystick unit also seemed to be very popular as I could find the identical product on Amazon in the US, Mexico and Japan. Coincidently, when I compared these two modules, the joystick was identical. So for now, I decided to use the Joystick module from HiLetgo because it already had a breakout board attached. In the later prototyping stages, I will be able to design a PCB for the Xbox joystick, so that it can also be used to build the device.

Testing the Joystick Mouse

To make sure that the joystick module and the Arduino Leonardo I ordered worked together, I ran the example code from the official Arduino documentation. The program worked as expected and I was able to control my computer mouse using the joystick module. It’s not too clear yet how this program will work in conjunction with everything else the processor needs to process, but hopefully I will figure it out in the coming days.

Testing the Multiplexer

The multiplexer board will be connected to the 13 air pressure sensors. It will essentially act as a rotary switch to choose which channel to connect the Arduino to. The Arduino with the multiplexer will be constantly switching between the 13 channels and writing the results to an array.
This array can then be used by the program to take specific actions based on the sensor values. At this stage, my goal was to make sure that the multiplexer could be used to switch between the 13 channels and have the data recorded on to an integer array.

Which channel to connect to is controlled using four digital pins that receive the channel number in binary. For example writing LOW, LOW, LOW, HIGH to the four pins will connect the Arduino’s analog pin to channel number one on the multiplexer. For each air pressure sensor,...
Read more »
Choosing and Testing Air Pressure Sensor
Pato Montalvo • 08/25/2020 at 02:51 • 0 comments

The most fundamental component in our device is the air pressure sensors. All of the functionalities and actions revolve around the effective use of this technology, meaning that we need to choose a suitable sensor to have a good overall performance.

While in team discussions, we went over what were the most important aspects that these sensors should have. Like all of our other desired parts, they should be accessible and affordable.

Regarding the technical specs, we started by brainstorming the general design, like Shu described in the previous post. We knew that each one of the harmonica’s air holes would have one individual sensor. This meant that the sensors will be exposed to atmospheric pressure constantly, in contrast with manometric sensors. That narrowed our options to using absolute pressure sensors that measure gauge pressure.

As seen on our BOM log post, we considered numerous options. Shu ordered a couple of them each and we designed a test sequence to determine which sensor is the most appropriate.

Shu designed a simple air hole to measure different interactions with the sensors. As you can see in the picture below, we chose a square air hole to replicate that of an original harmonica and added a pressure relief exhaust orifice at the back of the air hole. The dimensions of these simulations were also evaluated using fluid dynamics principles (Bernoulli’s Principle).

We considered the following formula for our design:

Where:

Δp = difference in pressure (desired on the inside vs avg atmospheric pressure on the outside)

ρ = Air density

Q = Hydraulic Head

A2= Area of the orifice

A1= Area of the air hole inlet

For these designs, we also researched the common lung capabilities of the average human. We decided to choose values that would be an average of general human capabilities, (eg. 6 m/s blow airspeed[1] / 0.8 psi exhalation pressure). With the blow speed and the cross-sectional area of the airhole, we calculated the head value and proceeded to iterate until we decided to set for a 2mm diameter orifice.

As you can see in the picture below, my partner Shu conducted the tests for each of the sensors and its compatibility with our designs.

An important thing to note is that we are considering a blow and draw functionality in our device, just like in a regular harmonica. That is why we had to make sure that our potential sensor could detect both sip and puff.

After the tests, we concluded that the MPS-2400 series sensors were the most suitable. We are confident we’ve made the right decision choosing this sensor, especially because of the process we used to select it. It will definitely bring the best out of this new adaptive tool.

References:

[1]Flow Through an Orifice
https://mcnallyinstitute.com/13-html/13-12.htm

[2]Human Exhaled Air Energy Harvesting with Specific Reference to PVDF Filmhttps://www.sciencedirect.com/science/article/pii/S2215098616300830

[3]Maximum Static Inspiratory and Expiratory Pressures with Different Lung Volumes https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1501025/
Hardware Update - Prototyping the Base Unit
Shu Takahashi • 08/21/2020 at 01:43 • 0 comments

The first part we decided to start designing was the base unit. This part is responsible for providing the joystick movements as well as the MIDI knob (potentiometer) movements.
We decided to work on this first because we were still discussing how the harmonica unit would work. Besides, working on this part was one of our top priorities as this must function flawlessly and hold the harmonica unit steadily. I had an idea of how the joystick module and the potentiometer were going to function together, so I made a quick sketch before creating any models in CAD.

I began by creating simple test designs for the joystick handle part using Fusion 360. I had never 3D printed parts for a joystick before, so I wanted to make sure that I could design this component properly first. I took a picture of the joystick handle that came with the module and made measurements from there. I also added a ball joint and a cover to ensure that the joystick could still move freely.

Before moving on to designing the base unit, I also decided to test fit the joystick module and the joystick knob handle to find the appropriate measurements. It’s always difficult to find the right dimensions for the parts to fit together snuggly and this time was no different. Luckily for this test component, only two re-prints were needed.

Now that I had tested some of the detailed parts I was concerned with, I was confident that I could design the entire base unit. The first draft of the design took about two hours. For now, I decided to use small wood screws to hold the components together. I plan on considering using other methods for the assembly process later.
All of the components printed without any major issues, except one. The component I had trouble with was the joystick handle. This component was both difficult to design and fabricate. This component had to be designed so that wires from the circuit board inside the harmonica unit could pass through the component while providing the snug fit to the joystick module. In terms of fabrication, I first tried 3D printing the component with an FDM printer, but as you can see in the photo, the outer part was too thin. So, I tried printing this component with an SLA printer. It worked but I had to re-print three times and make additional adjustments. As you can also see, in the final version, wires could pass through without affecting the snug fit to the joystick module.
I also came across another issue. The rotary joint between the joystick module and the potentiometer wiggled significantly when the joystick was moved. I should have noticed this in CAD, but it’s always a challenge to think about how things are going to work in the real world versus what it looks like in CAD.
To make the platform, which the potentiometer module sits on more stable, I added a circular guide around the potentiometer to provide additional support.
It took a while to make adjustments, wait for the 3D printers, and to test fit the components; however, all this patience paid off at the end and I was satisfied with the result. The video below is a short demo of the base unit.

All there is left to do for the base unit is to add the enclosure for holding the electronics components that will be located in the base unit. The walls of the base unit still wiggles a little bit, but I hope to resolve this issue in the next prototyping stage.
Bill Of Materials
Pato Montalvo • 08/21/2020 at 01:25 • 0 comments

To move forward, we decided to organize our proposed components in an excel sheet. The purpose is to evaluate and compare each one of the parts against its alternatives to meet the specified ends. We created a document dubbed “Bill of Materials Draft” where we listed components based on our desired functionality. We included different alternative components for each functionality. For each part, we gathered information like a link to its datasheet, a list of suppliers with different prices, its picture, and a brief description.

Before we had this, discussion on specific sensors was not efficient. This document has definitely helped a lot in deciding the main components. As of now, we have most of the major components on our BOM, and we expect to add more components as we make more progress.
CTQs and Defining Our Project
Pato Montalvo • 08/21/2020 at 01:10 • 0 comments
While discussing the scope of the project, we had different ideas about the functionalities the device would have. We had been considering multiple features until that point, but nothing was really set in stone.

That is why we decided to evaluate the potential the device could have. Through research and discussion, we attempted to define its capabilities.

We settled on the following functionalities:
- MIDI Instrument: A mouthpiece simulating a harmonica will be built to control a Musical Instrument Digital Interface (MIDI). The device will have 13 airways with built-in air sensors, each of which can be configured to execute different commands. As a musical instrument, it would be able to play numerous octaves and notes through the interface.
- Gaming Controller / Mouse: The device will also work like a computer mouse and a gaming controller. It will support numerous additional outputs including left and right-click to game function buttons that are on typical gaming pads.
- Keyboard: The13 air sensors will also have a “key” functionality allowing users to “type” using the harmonica.
Our next step was starting a critical to quality analysis (CTQ) where we defined the critical aspects of each functionality. Initially, we did this separately, with Shu and I making our own analysis. Then, we evaluated each other's work and created a definitive CTQ analysis by aggregating the individual ones. The diagram below is our final CTQ showing the requirements for each functionality.
The full diagram can be viewed from the link below:

https://docs.google.com/document/d/e/2PACX-1vToisXb_tbWGin7M6cg-1xUX57Q9Fi7K6FQsVNLn1ia1_xWr_dVCbtfjrOn8ZflgTRp2OaFlYzunCt-/pub

Creating this diagram was an important milestone. It opened the doors to the exploration of the components that would be needed to make all of these functionalities possible.

I know that getting an idea of our full CTQ is difficult from the diagram alone. For some of the requirements, we had long discussions to clarify what each of us meant. I’ve listed the details we discussed for some of the key requirements that are worth mentioning below.

“Must also be able to function as MIDI pad and knob controller”

We initially only had a MIDI keyboard in mind, but we realized that by adding a potentiometer to the device, it could also function as a MIDI knob controller and MIDI pad like in the image below.

Figure adopted from the image cited below.

(B&H Photo, 2020)

“The controller must be as universal as possible and support many users with different needs”

We know that cerebral palsy has a wide range of effects on motor controls, and this is different for everyone. Although we know that this device alone will not be able to support every user with cerebral palsy, we want to do our best to help as many people as possible. One way we’re planning to do this is to make everything, including hardware, electronics, and software open-source. By making everything open to the public, we believe that a community of users and developers could come together to share hardware and software customizations.

As we design our product, we will also keep in mind about universality. For example, we’ve already decided to use 1/4-20 UNC threads, the most common type of thread used in consumer-grade cameras. This will allow various camera accessories such as flexible tripods or stands to be used in conjunction with our device.

We believe that small design choices like this can add up to make our device as universal as possible. Additionally, we know that making this device truly universal is not possible, and perhaps this is the most challenging aspect in designing adaptive devices; however, we believe that creating a user and developer-driven community will give the best chance for this device to reach its full potential.

“The device must give some sort of live feedback → i.e. haptic feedback”

We added this to our CTQ after...
Read more »
Coming up With Our Idea
Shu Takahashi • 08/11/2020 at 07:39 • 0 comments

We decided to give ourselves some time before our first video call meeting to thoroughly read the design briefs for each nonprofit challenge and think about ways to solve each issue. Then during the meeting, we discussed the benefits and drawbacks of each idea, debated on the feasibility of each idea given the time and resources we had available and contributed our thoughts. Below, we’ve summarized the discussion we had for the idea we decided to pursue as well as two other ideas we had.

Mouth Controlled MIDI/Game Controller/Mouse / United Cerebral Palsy Los Angeles / Shu

This device is an external computer accessory for patients with cerebral palsy and other disabled patients that struggle with fine motor controls. The device is controlled using the user’s mouth, head movement and breath. It will support multiple purposes for creative expressions and recreations. The primary functions of the device include being used as a computer mouse, gaming controller, keyboard and a MIDI controller (to play music using DAW software). Each of these functions can be switched by using predetermined breath gestures. For example, puffing into the harmonica from left to right would switch the device from the mouse function to a keyboard function. Also, this system could be expanded by adding additional devices on the sides to meet different needs of the users. For example, a musician using this system could have the main device in the front acting as the MIDI keyboard and the device on the side acting as a MIDI knob device. Such functions of what each of the puffs does will be fully customizable through an easy to use software that maps out this device visually.

I came up with this idea after watching a TED talk called Disability Technology by Jeff Paradee. He introduced a sip and puff device that was specially designed for playing video games. This device has joystick that is controlled using the mouth to act as a computer mouse and an airway that allowed users to click on elements in a GUI environment. I instantly thought that this was one form of creative expression and wondered if this concept of sip and puff could be applied to other means of artistic expressions.

(Quadstick, 2020)

After some thoughts, my mind settled on creative writing, digital arts, music composition and of course gaming. The idea was to add more airways to add additional functionalities to the device. From observing that a MIDI device and a keyboard is just an array of buttons, I drew an analogy between this idea and a harmonica. The main source of inspiration for drawing this analogy was from seeing musicians play the harmonica hands-free while playing the guitar.

(Rowland Scherman, National Archives and Records Administration, 2020)

We saw some potential problems that could get in our way while prototyping this device. The first potential issue was how we were going to detect the sip and puff from the users. We initially thought about using an array of microphones to detect the breaths; however, after more research, we found that an air pressure sensor was more appropriate for this purpose. We also thought of other means to detect the change in airflow, and we’re still in progress of deciding what kind of sensor to use. Another potential issue was how a gaming joystick and the potentiometer was going to work together mechanically. We are still thinking of ways to overcome this problem.

Despite the potential issues we saw in our way, we believed that this idea was the most suitable project given the time and resources available. This project was challenging enough for us, but we could still see the light at the end of the tunnel. It was a project we felt we could utilize both of our skills both of us had developed over the years.

Smart Text to Speech Glass / United Cerebral Palsy Los Angeles / Shu

This is an adaptive tool designed to facilitate communication for cerebral palsy patients and other patients with similar needs. I saw the need for such a device...
Read more »
Introduction from Pato
Pato Montalvo • 08/11/2020 at 01:51 • 0 comments

Hello everyone, Pato here! I’m so excited to be starting this project with my good friend Shu.

I graduated from Electro-mechanical engineering last year, and since then, I’ve been looking for experiences to complement my professional development. As Shu mentioned, I have a great passion for electrical engineering, but I also love prototype design.

When Shu told me about this project and invited me to work together as a team, I couldn’t say no. I know that he is a very talented maker, and I’ve learned a lot from him. I knew this was an opportunity to keep honing skills I’m interested in, and most importantly, to create something of value and transcendence.

I’m confident that whatever comes out of this collaboration will be a product with a meaningful impact. Let’s get to work!