Bot-thoven: A Robot Musician

Description

After performing together at a retirement home, we were met with a variety of music requests from the residents. Realizing the need to make live music performances accessible to anyone without the need of a human performer, we decided to construct a robot which will use computer vision to read sheet music and perform it on an instrument with musical expression. We plan on improving existing models of instrument-playing robots by creating a mallet system that can strike a xylophone key at different forces and lengths to produce this "musical" effect.

Details

Bot-thoven: A Musical Robot Performer

Problem/Issue

From Honda’s theremin playing robot (Tsujino, Okuno, & Mizumoto, 2014) to the robot used in Nigel Stanford’s music video “Automatica,” a numerous number of instrument-playing robots have been created in the past. While instrument playing robots demonstrate impressive accuracy in hitting correct pitches or performing in perfect rhythm, these robots lack performing with musical expression that one would find through a human performer (Kemper & Barton, 2018). Human performers use techniques such as dynamics - difference in volume levels - and articulation - difference in the clarity of sound - in order to convey various musical emotions to the listener. By working with David Cha in the Computer Systems Lab, we propose to develop a robot which can read sheet music and perform with musical techniques like that of a human performer.

We have chosen the xylophone to serve as the instrument for our project. Research shows that the strength at which a xylophone key is struck conveys different emotional responses for a listener (Chau & Horner, 2016). The six emotional categories which showed the strongest effects due to differing strengths was calm, mysterious, comic, angry, scary, and sad, with softer strokes especially conveying comical emotions while harder strokes conveyed mysterious and angry feelings.

Research shows two possible ways to vary the strength at which a xylophone key is struck on a robotic instrument. One is through time varying torque as used by drum-playing robots when achieving a diverse drum stroke pallet (Gopinath & Weinberg, 2016). Regarding the xylophone mallet as a rod and the pivot point where the mallet is controlled by the servo as a fulcrum creates a time varying torque resulting in a diverse range of applicable forces for an instrument playing robot. A second option is through changing a servo’s rotation degrees per second (Oh & Park, 2013). By changing the rotation degrees per second on a servo, a mallet-servo system can strike a key at different velocities, resulting in an instrument playing robot to perform at various dynamics.

Creating a musical instrument-playing robot will allow anyone to listen to a personalized live performance at any time or location. The robot will be able to perform any given repertoire with musical expression like a human performer as long as the sheet music is printed for the program to read. While our project currently focuses on a single instrument - a xylophone, our work on how to have robotic instruments play with dynamics and musical techniques has the potential to be applied for a variety of instruments. The expansion of our research to various instruments will give listeners even more freedom in personalization. In addition, music has been shown to help dementia and Alzheimer’s patients in helping regain memory loss. Our technology has the potential to be used in the field of music therapy - where Bot-thoven can be set up in nursing homes to help residents use the power of music for positive recovery.

Objectives

Jason’s first objective is to create a working servo-mallet system which is able to strike the xylophone key at various dynamics. By achieving this goal, our project would have achieved the aspect of performing on an instrument with musical expression like a human performer. After completing one successful servo-mallet system, Jason will be able to quickly duplicate the model to create the necessary amount needed for the entire xylophone.

David’s first objective is to create a working program which is able to take the music input data and create an output conveying details on the appropriate pitch, length, and musical expression (ex: dynamic level). By achieving this goal, David will be able to send data for Jason to use when telling the servo-mallet system at which strength to strike the key.

After completing our project, we plan on setting...

Files

stand.stl CAD STL File for Servo-Mallet System Stand. Use for 3D Printing. Standard Tesselated Geometry - 156.53 kB - 11/14/2019 at 14:25		Download
First Servo with Xylophone Test.mov Shows robot playing a two-note passage with a mix of single notes and chords (two notes at once). The passage is played four times, with the articulation changing from a lot of ringing to a muted sound, the tempo getting slower, and the dynamics becoming softer. quicktime - 25.64 MB - 12/12/2019 at 14:16		Download
First Speed Test.mp4 Video of servo-mallet system moving at different speeds. MPEG-4 Video - 817.63 kB - 11/14/2019 at 14:27		Download
First Test on Xylophone.mp4 Video of testing servo-mallet system on Xylophone w/o stand. MPEG-4 Video - 627.22 kB - 11/14/2019 at 14:26		Download

Components

1 × Xylophone ammoon 30 Note Glockenspiel Xylophone Wooden Frame Percussion Musical Instrument
30 × Xylophone Mallet JETEHO Drum Mallets, Bell Mallets Percussion Sticks for Kids Play Log Drums, Tongue Drums and Keyboard Percussion, 10.2"x0.8"
1 × Arduino UNO Microcontroller; Arduino Uno R3 Development Board
30 × Standard Servos HS-422 HITEC Servo Motor
2 × Servo Driver Adafruit PCA9685 16-Channel Servo Driver

View all 6 components

Project Logs

Collapse logs

1/13 - 1/19
Jason P. • 01/16/2020 at 14:53 • 0 comments

Plan

I will need to attach the servo-mallet systems I have created to the stands I have 3D printed and connect it to the servo driver/Arduino to test if the servo-mallet systems function as intended.

Procedures

I attached the servo-mallet systems to the stands and then ran my code to test each individual one to make sure they can strike the xylophone key as intended.

Progress

I initially took the wooden frame my partner built and assembled it with the xylophone (shown below).

After attaching my servo-mallet systems to the stands, I placed each of the systems in their proper location. Each of the systems were able to strike a xylophone key at varying dynamics while running my code, so I now have 15 functioning servo-mallet systems prepared for use.

Problems

For next week, I will need to continue building the final set of servo-mallet systems so that I can test my code with all the keys on the xylophone.
1/6 - 1/12
Jason P. • 01/16/2020 at 14:52 • 0 comments

Plan

I will need to build more servo-mallet systems so that I can test my code with more keys (and therefore play more interesting musical passages).

Procedures

I used the 3D Printer to print additional copies of the stand.

Progress

I was able to print additional copies of the stand for the servo-mallet system using the 3D printer. An image is shown above depicting the additional stands which were printed.

In addition, I was able to create additional servo-mallet systems so that I will be able to test my robot with passages consisting of more notes. An image is shown above depicting the additional servo-mallet systems created. I achieved my progress report goal which I set to accomplish by the end of January of “creating at least 15 servo-mallet systems.”

Problems

For next week, I will need to attach the servo-mallet systems I have created to the stands I have printed and connect it to the servo driver/Arduino to test if the servo-mallet systems function as intended.
12/16 - 12/22
Jason P. • 12/19/2019 at 14:57 • 0 comments

Plan

I will need to take the musical elements I’ve been able to add to my code and test it on new musical passages.

Procedures

I used the code I had written last week to hit a xylophone key with different musical expressions and tested it on a new musical passage.

Progress

I was able to add an additional servo-mallet system to run with my servo driver. This allowed me to experiment with three-note tunes like Mary Had a Little Lamb.

A video can be found here where my robot is playing Mary Had a Little Lamb with a mix of single notes and chords (two notes at once). The passage is played three times, with the articulation changing from a lot of ringing to a muted sound, the tempo (speed) getting slower, and the dynamics becoming softer.

Problems

I will now need to build more servo-mallet systems so that I can test my code with more keys (and therefore play more interesting musical passages), as well as try using a rubber band to fasten the mallet on to the servo horn instead of a zip tie.
12/9 - 12/15
Jason P. • 12/12/2019 at 14:21 • 0 comments
Plan

I will need to take what I have learned from familiarizing myself with the PCA9685 library and write code to have the servos appropriately strike a xylophone key.

Procedures

I used the example “Servo” code from the Adafruit Library as a baseline to add upon it and write my own code. I had to find the correct PWM values for my servos to correctly strike the keys and update the code to play a tune at varying dynamics/articulations.

Progress

After familiarizing myself with the PCA9685 library last week, I was able to write a code to control multiple servos (2 as of now) using a single Arduino pin. After being able to strike a xylophone key with my servo-mallet system, I added improvements to my code to achieve the following:
- Playing a tune consisting of two notes
- Playing a key with different articulation (lots of ringing vs. stiff/muted)
- Playing a key at different tempo (fast vs. slow)
- Playing a key with different dynamics (loud vs. soft)
- Playing multiple notes at the same time (chords)
A video can be found here where a robot is playing a two-note passage with a mix of single notes and chords (two notes at once). The passage is played four times, with the articulation changing from a lot of ringing to a muted sound, the tempo getting slower, and the dynamics becoming softer.

In terms of my budget, I have used $114.70 so far. The table above shows the details of my purchases I have made for my project as of now. Once I order an additional PCA9685 in the Spring, my total budget should sum up to $129.65.

Problems

I will now need to build more servo-mallet systems so that I can test my code with more keys (and therefore play more interesting musical passages). While testing, I also realized that the zip ties may not be the best option to attach the mallets on the servo, so I will need to try using other methods such as a rubber band.
12/2 - 12/8
Jason P. • 12/12/2019 at 14:20 • 0 comments

Plan

This week, I will construct more servo-mallet systems to use for testing the control of multiple servo movements with my PCA9685 PWM/Servo Driver from Adafruit.

Procedures

I took the stands which finished 3D printing from last week and attached the servo and mallet onto it to build three additional complete servo-mallet systems. I also had to solder all the pins onto the Servo Driver which arrived from Adafruit.

Progress

Above is an image of the Servo Driver from Adafruit which arrived this week. The image shows the driver after I completed soldering on all of the pins.

After completing putting together my new additional servo-mallet systems (shown above), I attached the servos to my Servo Driver shield. I spent Thursday taking the time to understand the PCA9685 library and running the example “Servo” code onto my board. Running the example code allowed me to verify that the Servo Driver works as intended in controlling multiple servo movements at the same time through just two analog pins.

Problems

I will need to take what I have learned from familiarizing myself with the PCA9685 library and update the example code to have the servos move along my desired goal of appropriately striking a xylophone key.
11/18 - 11/24
Jason P. • 12/12/2019 at 14:20 • 0 comments

Plan

This week, I will continue printing more copies of the prototype of the servo-mallet system.

Procedures

I will continue 3D printing out more copies of the stand prototype and building more servo-mallet systems. This will allow me to test my code for varying servo speeds (for dynamics when hitting the xylophone key) and see if it will work when the microcontroller has to control several servos at once.

Progress

Above is an image from the CURA 3D printing software detailing the layout of my current print. The print is currently in progress (image below) and will be finished by Thursday evening.

I have also placed my order for the 16-Channel 12-bit PWM/Servo Driver from Adafruit this week, which will allow me to test if the shield is compatible with my microcontroller.

Problems

Once the shield from Adafruit arrives, I will need to take all the servo-mallet systems that I made this week and test the shield to make sure it functions as intended: allowing a single microcontroller to control multiple servos at once.
11/11 - 11/17
Jason P. • 12/12/2019 at 14:17 • 0 comments

Plan

This week, I will print the prototype for the servo-mallet system I will be using.

Procedures

Using the initial CAD I designed, I will use a 3D Printer to print out a stand to create my first prototype for the servo-mallet system.
Progress

Below is the first physical prototype I created for the servo-mallet system.

To address the servo size problem I brought up in the past several weeks, David and I disassembled the xylophone into two separate parts to free up space around the xylophone frame. The wooden blocks show where the servo-mallet systems will be placed (all around the xylophone) to ensure we can fit all the servos.

Technical Specification

The microcontroller which I am using is the Arduino Leonardo. The technical specification, obtained from the Arduino website it shown below.

All the parts I will be using for my project function and connect to the board as intended. A video of my HITEC Servo functioning when connected to the Leonardo microcontroller is available to view here.

Problems

I will need to take the problems I found with my first prototype and create an improved design for the second prototype.
11/4 - 11/10
Jason P. • 11/14/2019 at 14:21 • 0 comments

Plan

This week, I will test the prototype for the servo-mallet system I will be using.

Procedures

Using the servo-mallet system I built last week, I will continue testing out the prototype to find areas to fix and improve on.

Progress

Below is an image of the servo-mallet system I have built as a first prototype. Currently, David and I are testing out the servo to move at different speeds, in order to achieve hitting the xylophone key with varying dynamics (loudness).

In terms of fitting all 30 servos across the xylophone, I have decided to test the method of staggering mallet length. This week, I have built copies of the servo-mallet prototype. With these additional mallets, I will be able to physically test this method next week to see if it will work. If not, I will look into other options such as building a new, wider frame for the xylophone keys or using a smaller servo as a replacement.

Problems

I will need to test the staggering mallet length method to see if I will be able to resolve my problem regarding how my servos are currently too big to fit 30 of these systems across all of the xylophone keys.
10/28 - 11/3
Jason P. • 11/14/2019 at 14:20 • 0 comments

Plan

This week, I will create a prototype for the servo-mallet system I will be using.

Procedures

Using the materials which arrived from Amazon last week, I built my first servo-mallet prototype to test it out on a xylophone.

Progress

I first tested my code on varying the mallet speed by changing the servo’s rotation degrees per second. A video can be found here.

I followed up by testing out the system onto the actual xylophone. A video can be found here.

Problems

After testing out the servo-mallet system on a xylophone, I realized that the size of the servos is currently too big to fit 30 of these systems and align it with all of the xylophone keys. For the upcoming weeks, I will need to think of a way to fit all 30 servo-mallet systems across my xylophone, whether it is using new servos, extending the xylophone frame, or modifying my mallet system.
10/21 - 10/27
Jason P. • 11/14/2019 at 14:20 • 0 comments

Plan

This week, I will create CAD for the servo-mallet system prototypes I will be making.

Procedures

I used Fusion360 and the parts which arrived from Amazon to come up with an initial CAD design for the servo-mallet system.

Progress

The mallets, zip ties, and xylophone arrived through Amazon as pictured above.

I finished an initial CAD design showing how I will use the mallet, a servo, and zip tie to create a servo-mallet system to strike a xylophone key.

Problems

I will need to begin testing out the prototype by 3D printing the body (in green). By testing out the initial design, I will be able to make adjustments and changes as necessary to improve the servo-mallet system.