Bot-thoven: A Robot Musician

Description

A Senior Research Project by Jason Pak and David Cha.

A musical instrument-playing robot will allow anyone to listen to a personalized live performance at any time or location. We plan on improving existing models of instrument-playing robots by creating a servo motor based mallet system that can strike a xylophone key at different forces and lengths to allow it to perform with musical expression like a human performer.

Video Demos:
Bot-thoven Performing Beethoven and Other Famous Pieces (Final Update)
https://youtu.be/9Ag7zkTR_XE

Performing a Scale in 3 Different Styles (March Update)
https://drive.google.com/file/d/1WGXOwnS6DCYbZWAmw7SPrEdkh6lR6mPg/view?usp=sharing

Details

Bot-thoven: A Musical Robot Performer

Problem/Issue

From Honda’s theremin playing robot (Tsujino, Okuno, & Mizumoto, 2014) to the robot used in Nigel Stanford’s music video “Automatica,” a numerous number of instrument-playing robots have been created in the past. While instrument playing robots demonstrate impressive accuracy in hitting correct pitches or performing in perfect rhythm, these robots lack performing with musical expression that one would find through a human performer (Kemper & Barton, 2018). Human performers use techniques such as dynamics - difference in volume levels - and articulation - difference in the clarity of sound - in order to convey various musical emotions to the listener. By working with a partner in the Computer Systems Lab, we propose to develop a robot which can read sheet music and perform with musical techniques like that of a human performer.

We have chosen the xylophone to serve as the instrument for our project. Research shows that the strength at which a xylophone key is struck conveys different emotional responses for a listener (Chau & Horner, 2016). The six emotional categories which showed the strongest effects due to differing strengths was calm, mysterious, comic, angry, scary, and sad, with softer strokes especially conveying comical emotions while harder strokes conveyed mysterious and angry feelings.

Research shows two possible ways to vary the strength at which a xylophone key is struck on a robotic instrument. One is through time varying torque as used by drum-playing robots when achieving a diverse drum stroke pallet (Gopinath & Weinberg, 2016). Regarding the xylophone mallet as a rod and the pivot point where the mallet is controlled by the servo as a fulcrum creates a time varying torque resulting in a diverse range of applicable forces for an instrument playing robot. A second option is through changing a servo’s rotation degrees per second (Oh & Park, 2013). By changing the rotation degrees per second on a servo, a mallet-servo system can strike a key at different velocities, resulting in an instrument playing robot to perform at various dynamics.

Creating a musical instrument-playing robot will allow anyone to listen to a personalized live performance at any time or location. The robot will be able to perform any given repertoire with musical expression like a human performer as long as the sheet music is printed for the program to read. While our project currently focuses on a single instrument - a xylophone, our work on how to have robotic instruments play with dynamics and musical techniques has the potential to be applied for a variety of instruments. The expansion of our research to various instruments will give listeners even more freedom in personalization. In addition, music has been shown to help dementia and Alzheimer’s patients in helping regain memory loss. Our technology has the potential to be used in the field of music therapy - where Bot-thoven can be set up in nursing homes to help residents use the power of music for positive recovery.

Objectives

The first objective is to create a working servo-mallet system which is able to strike the xylophone key at various dynamics. By achieving this goal, our project would have achieved the aspect of performing on an instrument with musical expression like a human performer. After completing one successful servo-mallet system, we will be able to quickly duplicate the model to create the necessary amount needed for the entire xylophone.

Another objective is to create a working program which is able to take the music input data and create an output conveying details on the appropriate pitch, length, and musical expression (ex: dynamic level). By achieving this goal, we will be able to send data for the Arduino board to use when telling the servo-mallet system at which strength to strike the key.

After completing our project, we plan on setting...

Files

scales.ino Code which allows the robot to play a scale in the full range of the xylophone. ino - 7.92 kB - 05/17/2020 at 06:42		Download
stand.stl CAD STL File for Servo-Mallet System Stand. Use for 3D Printing. Standard Tesselated Geometry - 156.53 kB - 11/14/2019 at 14:25		Download

Components

1 × Xylophone ammoon 30 Note Glockenspiel Xylophone Wooden Frame Percussion Musical Instrument
30 × Xylophone Mallet JETEHO Drum Mallets, Bell Mallets Percussion Sticks for Kids Play Log Drums, Tongue Drums and Keyboard Percussion, 10.2"x0.8"
1 × Arduino UNO Microcontroller; Arduino Uno R3 Development Board
30 × Standard Servos HS-422 HITEC Servo Motor
2 × Servo Driver Adafruit PCA9685 16-Channel Servo Driver

View all 6 components

Project Logs

Collapse logs

Final Log Entry
Jason P. • 05/21/2020 at 19:41 • 0 comments

Bot-thoven performs Beethoven and Other Famous Piano Works (Final Demo)
3/9 - 3/15
Jason P. • 04/12/2020 at 02:22 • 0 comments

Plan

I will begin working with David to combine his MIDI file converting code with my robot.

Procedures

I will connect my xylophone-playing robot and run David’s MIDI file converting code through Arduino and Processing software.

Progress

We were able to upload the code onto the Arduino board and check if it would work. Currently, the code is not functioning perfectly as intended because it is unable to take a MIDI file data and have the robot play it.

Problems

For next week, I will continue working with David to make his MIDI file converting code compatible with my robot.
3/2 - 3/8
Jason P. • 04/12/2020 at 02:21 • 0 comments

Plan

I will need to finish connecting the rest of my servo-mallet systems to a servo driver with my second order for the PCA9685.

Procedures

I will connect the servo-mallet systems with the servo driver and run my Arduino code to verify that the system can strike the xylophone key at varying dynamics.

Progress

The second servo driver has arrived and I was able to finish connecting the remaining 15 of my servo mallet systems to the servo driver for testing. I was able to run my Arduino code successfully through the servos, allowing me now to have the ability to perform a 30-note range passage with musical expression.

Here is a link to Bot-thoven playing across the xylophone (scale) in 3 different musical styles.

Problems

For next week, I will need to begin working with David to combine his MIDI file converting code with my robot.
2/24 - 3/1
Jason P. • 04/12/2020 at 02:21 • 0 comments

Plan

I will need to connect my servo-mallet systems to the PCA9685 servo driver.

Procedures

I will connect the servo-mallet systems with the servo driver and run my Arduino code to verify that the system can strike the xylophone key at varying dynamics.

Progress

I have finished taking 15 out of 30 of my servo mallet systems and attaching it to the servo driver for testing. I was able to run my Arduino code successfully through the servos, allowing me now to have the ability to perform a 15-note range passage with musical expression.

Problems

For the upcoming weeks, I will need to continue connecting the rest of my servo-mallet systems to a servo driver whenever my second order for the PCA9685 arrives.

Problems Faced During Research

One problem I faced while pursuing my research was the use of zip-ties as a fastener between the servo head and xylophone mallet. While I initially believed that using zip-ties would be secure enough to attach the mallet to the servo motor, the zip-ties kept slipping off. To address this problem, I used rubber bands as an extra fastener for my second prototype of the servo-mallet system, which resulted in the mallet to stop slipping off of the servo horn and become much more secure.

Another problem I faced while pursuing my research was integrating my work in the Robotics Lab with that of my partner David’s in the Computer Systems Lab. While we initially wanted to use computer vision to convert an image of sheet music into data for my robot to perform, we realized that this method may be too difficult to achieve within our timeline. Instead, we decided to take a more simple approach by using a MIDI file and converting it into data for my robot to perform.

A final obstacle I faced was the long delivery time for my parts to arrive. My order for the second PCA9685 servo driver, a part crucial to finish the next step in my project, has still not arrived yet. To address this problem, multiple orders of the servo driver have been placed in hopes that the part will arrive very soon.
2/10 - 2/16
Jason P. • 02/18/2020 at 14:17 • 0 comments

Plan

I will need to finish building the remaining 15 servo-mallet systems.

Procedures

I will attach the mallets onto all of the servos and use a 3D printer to print the stands to hold up the servo-mallet system.

Progress

I have finished printing the final 15 stands needed for my servo-mallet system. Using zip-ties and rubber bands, I also completed attaching the mallets to the servo horn.

In addition, as shown above, the new components I had ordered arrived. This includes the power supply, the power adapter, and the servo extension cables.

Problems

For next week, I will need to continue building the final set of servo-mallet systems by attaching the servo to the stands I have finished 3D printing.
2/3 - 2/9
Jason P. • 02/18/2020 at 14:17 • 0 comments

Plan

I will need to begin building the remaining 15 servo-mallet systems.

Procedures

I will screw the long servo horns on to all of the servos and use a 3D printer to print the stands to hold up the servo-mallet system.

Progress

I have finished attaching the long servo horns to the 15 servos. The stands are currently printing right now through the 3D printer and should be finished printing by Tuesday.

Problems

For next week, I will need to continue building the final set of servo-mallet systems by attaching the mallet using a zip tie.

Practical Need or Problem Trying To Be Solved

Creating a musical instrument-playing robot will allow anyone to listen to a personalized live performance at any time or location. The robot will be able to perform any given repertoire with musical expression like a human performer as long as a MIDI file of the music is prepared for the program to read. While our project currently focuses on a single instrument – a xylophone, our work on how to have robotic instruments play with dynamics and musical techniques has the potential to be applied for a variety of instruments. The expansion of our research to other instruments will give listeners even more freedom in personalization. In addition, music has been shown to help dementia and Alzheimer’s patients in helping regain memory loss. Our technology has the potential to be used in the field of music therapy – where Bot-thoven can be set up in nursing homes to help residents use the power of music for positive recovery.

Another reason why this project is important is because it will help develop fine motor control of a robot. Bot-thoven requires precise servo movement when controlling a mallet to strike the xylophone with varying musical emotions. The project’s main goal of “playing an instrument expressly” requires meticulous motor control, expanding Bot-thoven’s application to fields beyond music. Since motor control is an important aspect of any robotics project, the methods used to develop fine servo motor control through Bot-thoven will serve as important research and information for all roboticists.
1/13 - 1/19
Jason P. • 01/16/2020 at 14:53 • 0 comments

Plan

I will need to attach the servo-mallet systems I have created to the stands I have 3D printed and connect it to the servo driver/Arduino to test if the servo-mallet systems function as intended.

Procedures

I attached the servo-mallet systems to the stands and then ran my code to test each individual one to make sure they can strike the xylophone key as intended.

Progress

I initially took the wooden frame my partner built and assembled it with the xylophone (shown below).

After attaching my servo-mallet systems to the stands, I placed each of the systems in their proper location. Each of the systems were able to strike a xylophone key at varying dynamics while running my code, so I now have 15 functioning servo-mallet systems prepared for use.

Problems

For next week, I will need to continue building the final set of servo-mallet systems so that I can test my code with all the keys on the xylophone.
1/6 - 1/12
Jason P. • 01/16/2020 at 14:52 • 0 comments

Plan

I will need to build more servo-mallet systems so that I can test my code with more keys (and therefore play more interesting musical passages).

Procedures

I used the 3D Printer to print additional copies of the stand.

Progress

I was able to print additional copies of the stand for the servo-mallet system using the 3D printer. An image is shown above depicting the additional stands which were printed.

In addition, I was able to create additional servo-mallet systems so that I will be able to test my robot with passages consisting of more notes. An image is shown above depicting the additional servo-mallet systems created. I achieved my progress report goal which I set to accomplish by the end of January of “creating at least 15 servo-mallet systems.”

Problems

For next week, I will need to attach the servo-mallet systems I have created to the stands I have printed and connect it to the servo driver/Arduino to test if the servo-mallet systems function as intended.
12/16 - 12/22
Jason P. • 12/19/2019 at 14:57 • 0 comments

Plan

I will need to take the musical elements I’ve been able to add to my code and test it on new musical passages.

Procedures

I used the code I had written last week to hit a xylophone key with different musical expressions and tested it on a new musical passage.

Progress

I was able to add an additional servo-mallet system to run with my servo driver. This allowed me to experiment with three-note tunes like Mary Had a Little Lamb.

A video can be found here where my robot is playing Mary Had a Little Lamb with a mix of single notes and chords (two notes at once). The passage is played three times, with the articulation changing from a lot of ringing to a muted sound, the tempo (speed) getting slower, and the dynamics becoming softer.

Problems

I will now need to build more servo-mallet systems so that I can test my code with more keys (and therefore play more interesting musical passages), as well as try using a rubber band to fasten the mallet on to the servo horn instead of a zip tie.
12/9 - 12/15
Jason P. • 12/12/2019 at 14:21 • 0 comments
Plan

I will need to take what I have learned from familiarizing myself with the PCA9685 library and write code to have the servos appropriately strike a xylophone key.

Procedures

I used the example “Servo” code from the Adafruit Library as a baseline to add upon it and write my own code. I had to find the correct PWM values for my servos to correctly strike the keys and update the code to play a tune at varying dynamics/articulations.

Progress

After familiarizing myself with the PCA9685 library last week, I was able to write a code to control multiple servos (2 as of now) using a single Arduino pin. After being able to strike a xylophone key with my servo-mallet system, I added improvements to my code to achieve the following:
- Playing a tune consisting of two notes
- Playing a key with different articulation (lots of ringing vs. stiff/muted)
- Playing a key at different tempo (fast vs. slow)
- Playing a key with different dynamics (loud vs. soft)
- Playing multiple notes at the same time (chords)
A video can be found here where a robot is playing a two-note passage with a mix of single notes and chords (two notes at once). The passage is played four times, with the articulation changing from a lot of ringing to a muted sound, the tempo getting slower, and the dynamics becoming softer.

In terms of my budget, I have used $114.70 so far. The table above shows the details of my purchases I have made for my project as of now. Once I order an additional PCA9685 in the Spring, my total budget should sum up to $129.65.

Problems

I will now need to build more servo-mallet systems so that I can test my code with more keys (and therefore play more interesting musical passages). While testing, I also realized that the zip ties may not be the best option to attach the mallets on the servo, so I will need to try using other methods such as a rubber band.