-
Audio Synthesis Engine Design
08/31/2018 at 07:22 • 0 comments(Original post date: 28/01/16)
I’ve spent the past 10 days in sunny Anaheim, California at The NAMM Show 2016 exhibiting with Modal Electronics, so I haven’t had much time to work on my project. However it’s given me a chance to think a lot about my synthesis engine design, so I thought I’d use this weeks (well, a late last weeks) update to give a brief overview of audio synthesis types, the essential components, as well as outlining my current design ideas and how they have changed over the project so far.
Before I continue, it is probably worth stating that I am by no means either a synthesiser enthusiast or an expert at developing audio synthesis engines, and part of me doing this project is to advance my knowledge and experience in both of these areas. I have got a good intermediate understanding of audio synthesis, but if you want to know about synthesis types and components at an advanced level I’d rather leave that to other sources than for me attempt to explain everything in detail here. Also I am going to talk about audio synthesis from a very general point of view, and not discuss the differences between analogue and digital synthesis; however it is worth mentioning that as I'll be implementing the audio synthesis engine on the BeagleBone Black it is going to be completely digital.
Synthesis Types
There are many different types of synthesis, which all create their own distinctive sounds in particular ways.
The main types of synthesis are as follows:
- Additive synthesis - Adding together simple waveforms (usually sine waves), called partials, to create more complex waveforms. Not very common in modern synthesisers due to the complexity of designing a user interface that allows each partial to be controlled in an intuitive way. A good example of an additive synthesis software synthesiser is FL Studio's Harmor.
- Subtractive synthesis - Filtering harmonically-rich waveforms. Probably the most common type of synthesis within modern commercial synthesisers, and is relatively simple to implement and process. An example of a classic subtractive synthesis synthesiser is the Moog Minimoog.
- Frequency Modulation (FM) synthesis - Modulating the frequency of waveforms with other waveforms in the audio range. At its simplest level it creates a very particular distinctive sound, however it is very complex to implement a versatile FM engine. A good example of a class FM synthesiser is the Yamaha DX7; the first commercially successful digital synthesiser.
- Wavetable synthesis - Similar to subtractive synthesis, but instead of using equations to generate the sound waves, it stores small samples of a single cycle of a waveform and plays back the stored sample over and over again. It was very popular in early digital synthesisers due to it’s advantages of taking up less memory and processing power. An example of this is the PPG wave 2.
- Granular synthesis - Manipulating very short samples of sound called grains, played back in unconventional ways. Great for creating soundscapes, textures, and effects. A good example of granular synthesis is the Collidoscope.
- Physical-Modelling synthesis - The process of using equations and algorithms to simulate real instruments or physical sources of sound. This type generally uses a very different set of parameters and controls compared to the other synthesis types. My personal favourite physical-modelling software synth is Apple's Sculpture.
Essential Building Blocks of Audio Synthesis
Below is a list of the four important building blocks needed for any decent sound synthesis engine of any synthesis type:
- Oscillators - used to create the raw sound source/waves/tones, and can come in many different wave shapes (e.g. sine, saw, square, noise)
- Filters - used to shape the timbre of the sound created by oscillators, with the most popular filter types being low-pass, high-pass, and band-pass. Important for subtractive synthesis.
- Envelopes - used to modulate parameters of the sound (most commonly the volume) in the time domain. Most envelopes are ADSR envelopes which provide control over 4 distinct time related parts of a sound - attack, decay, sustain, and release.
- Low Frequency Oscillators (LFOs) - used as source for creating rhythmic modulation, such as tremolo (by modulating the sounds volume), or vibrato (by modulating a sounds pitch).
On top of that you’ll find synthesisers will come with other components and controls such as effects (e.g. delay, reverb, chorus, flanger), sequencers, other modulation sources (e.g. key velocity, aftertouch, wheels/pedals/joysticks), but are not essential to the core of a synthesis engine.
Number of Voices
Other than synthesis type and essentially components, another main thing to consider in audio synthesis design is how many notes the synthesis engine can play simultaneously. There are most commonly two possible ‘modes’ here - polyphonic (multiple notes) or monophonic (a single note).
In a polyphonic system you then need to consider the number of voices, also called the polyphony value. A greater polyphony value allows bigger chords and textures to be played, however it increases the needed processing power. There are also different levels of polyphony - in a true polyphonic synthesis each voice/note will have it’s own oscillators, filter, envelopes, and LFOs; however some polyphonic systems are simplified so that, while multiple notes can be played simultaneously, every voice shares a single element such as a filter and/or LFO, which is more commonly known as a paraphonic synth. Polyphonic synthesisers usually have an option to put them into monophonic mode if desired.
Example flow charts of of monophonic, polyphonic, and paraphonic audio synthesis engines.
In these examples each voices has two oscillators.
Development of my Synthesis Engine Design
So far throughout this project my design ideas for the synthesis engine have changed quite a bit, and I’m expecting it to keep changing right up until the end. There are a number factors that have caused this, which will ultimately determine what my final synthesis engine design will be:
- The capabilities and power of the target hardware, the BeagleBone Black
- The capabilities of the synthesis library I’m using, most probably Maximilian
- The space on the toy piano enclosure for synth parameter controls
- The number of inputs on an Arduino Pro Mini that I can used for controlling these controls
- My knowledge and experience in implementing synthesis engines
- Time!
Here are the main design changes that have happened so far, with reasons why:
- Originally I was planning on having a very complex synthesis engine with many modulation destinations, digital effects, and some quite advanced parameters; totalling to about 60 parameters/controls. However after experimenting with Maximilian on the BBB I found that I’m not going to have enough processing power to get it all working without audio glitches. Also, I’m not sure I’m going to have enough time or space to wire and attach 60 controls to the toy piano enclosure.
- I wanted to have 12 or 16 note polyphony, but testing Maximilian on the BBB seems to show that in order to have such high polyphony I would need to heavily simplify other parts of the synthesis engine, such as making it paraphonic instead of polyphonic.
- Originally I was just going to have a basic filter which could either be low-pass, band-pass, or high-pass. However after experimenting with Maximilian I found that its SVF (state-variable filter) offers more control over the sound and uses less processing power.
My Current Synthesis Engine Design
I have decided to base my synthesis engine on subtractive synthesis for a couple of reasons:
- It can create a relatively varied set of sounds without being too complex to implement or use
- It’s the synthesis type I have had most experience with, from the synths I help develop at Modal Electronics
Minimum set of Parameters
Here are the minimum set of components and parameters that I want to implement into my vintage toy synthesiser:
Parameter Category Parameter Value Range Panel Control Description Oscillators Sine wave level 0-127 Potentiometer Sets the level of a sine oscillator Triangle wave level 0-127 Potentiometer Sets the level of a triangle oscillator Sawtooth wave level 0-127 Potentiometer Sets the level of a sawtooth oscillator Pulse wave level 0-127 Potentiometer Sets the level of a pulse oscillator Pulse amount 0-127 Potentiometer, centre-detented Sets the pulse wave shape Sub level 0-127 Potentiometer Square wave 12 semitones down State-Variable Filter Frequency cutoff 0-127 Potentiometer Sets the cutoff/centre frequency of the filter Resonance 0-127 Potentiometer Sets the resonance of the filter Low-pass mix 0-127 Potentiometer Sets the level of the low-pass mix High-pass mix 0-127 Potentiometer Sets the level of the high-pass mix Band-pass mix 0-127 Potentiometer Sets the level of the band-pass mix Notch mix 0-127 Potentiometer Sets the level of the notch mix Amplitude Envelope Attack 0-127 Potentiometer Sets the time it takes for the amplitude to reach each its max value when a note is triggered Decay 0-127 Potentiometer Sets the time it takes for the amplitude to go from the max value to the sustain value Sustain 0-127 Potentiometer Sets the amplitude level that the sound/note stays at for it's duration until the note/key is released Release 0-127 Potentiometer Sets the time it takes for the amplitude to go from the sustain level to 0 after a note/key is released Amount 0-127 Potentiometer Sets the amount of envelope modulation on the amplitude. Can also act as a volume/gain control. Filter Envelope Attack 0-127 Potentiometer Sets the time it takes for the filter cutoff to reach each its max value when a note is triggered Decay 0-127 Potentiometer Sets the time it takes for the filter cutoff to go from the max value to the sustain value Sustain 0-127 Potentiometer Sets the filter cutoff value that the sound/note stays at for it's duration until the note/key is released Release 0-127 Potentiometer Sets the time it takes for the filter cutoff to go from the sustain level to 0 after a note/key is released LFO Wave shape Sine, triangle, sawtooth, square, random Potentiometer, detented Sets the shape of the LFO Rate 0-127 Potentiometer Sets how slow/fast the LFO goes Depth -64-+63 Potentiometer, centre-detented Sets the depth of the LFO (Possibly not needed if each LFO mod destination has it's own depth controls) Keys/Voices Octave -2-+2 Potentiometer, centre-detented Sets the octave that the keyboard keys can play Scale Chromatic, Major, Minor, others Potentiometer, detented Sets the musical scale that the keyboards keys can play Voice mode Poly, mono Toggle switch Sets whether the synth is polyphonic or monophonic mode Transpose -6-+6 Potentiometer, centre-detented Sets the semitone offset of the keyboard Modulation Depths Velocity to amplitude -64-+63 Potentiometer, centre-detented Sets the amount of applied keyboard velocity modulation for note amplitude LFO to amplitude -64-+63 Potentiometer, centre-detented Sets the amount of LFO modulation for note amplitude LFO to cutoff -64-+63 Potentiometer, centre-detented Sets the amount of LFO modulation for filter cutoff LFO to resonance -64-+63 Potentiometer, centre-detented Sets the amount of LFO modulation for filter resonance Aftertouch to cutoff -64-+63 Potentiometer, centre-detented Sets the amount of applied keyboard aftertouch modulation for filter cutoff Aftertouch to LFO depth -64-+63 Potentiometer, centre-detented Sets the amount of applied keyboard aftertouch modulation for LFO depth Effects Distortion amount 0-127 Potentiometer Sets the amount of distortion applied to the overall sound Global Vintage amount 0-127 Potentiometer Unique to this synthesiser. Sets the amount of random voice/oscillator detuning; random filter cutoff value offsets; and random crackles/pops applied to the overall sound, essentially emulating an old/broken analogue synthesiser. Volume 0-127 Potentiometer Sets the system volume of the BBB. Not a 'patch' parameter like the rest of them. I want it to be true polyphonic synthesiser, so that I can use the poly aftertouch I have implemented into the keyboard as well as implement filter envelopes, both which need to be per note/voice, with a polyphony value of at least 4 voices. Therefore there would be one instance of the oscillators, filter, envelopes, and LFO for each voice; the rest of the components/parameters would be global to the overall/mixed sound.
A flow chart of my synthesis engine design
Extended Parameters
If I'm able to, based on the 6 factors above, there are a number of other parameters I would like to build into my synthesiser:
Parameter Category Parameter Value Range Panel Control Description Oscillators Oscillator coarse tune -24-+24 Potentiometer, centre-detented Sets the coarse tune (in semitones) for each oscillator. There would be one instance of this parameter for each wave type. Oscillator fine tune -64-+63 Potentiometer, centre-detented Sets the fine tune (in cents) for each oscillator. There would be one instance of this parameter for each wave type. Noise level 0-127 Potentiometer Sets the level of the noise sound source. Double Mode off/on Toggle switch If an oscillator has a 'note' value not set to 0, a second instance of that oscillator is created which plays at the root note, allowing two-note chords to be created. Osc phase offset 0-127 Potentiometer Sets the degree of offsets of phase for the set of oscillators. Filter Envelope Amount/Depth -64-+63 Potentiometer, centre-detented Sets the depth of the envelope modulation on the filter cutoff. LFO Single shot mode off/on Toggle switch Sets whether the LFO only cycles once, acting more like an envelope Delay 0-127 Potentiometer Sets how long it takes for the LFO to start once a note is triggered LFO2 - - - A second LFO will all the same controls as the existing LFO1 Keys/Voices Glide 0-127 Potentiometer Sets the amount of glissando between notes Voice size 1-4 Potentiometer, detented Sets how many voices are played with each note Voice spread 0-127 Potentiometer If there is a voice size of above 1, this sets how detuned each voice in the stack is Modulation Depths Velocity to cutoff, resonance, LFO depth -64-+63 Potentiometer, centre-detented Sets the amount of applied keyboard velocity modulation for each destination. There would be one parameter for each destination. LFO to filter mix -64-+63 Potentiometer, centre-detented Sets the amount of applied LFO modulation for each of the filter mix parameters. There would be one parameter for each of the filter mix parameters. Aftertouch to resonance, LFO rate -64-+63 Potentiometer, centre-detented Sets the amount of applied keyboard aftertouch modulation for each destination. There would be one parameter for each destination. LFO2 mod depths -64-+63 Potentiometer, centre-detented Various modulation destination depths for LFO2 Filter envelope depths -64-+63 Potentiometer, centre-detented Allow the filter envelope to also modulate other parameters, with individual depths for each destination. If implementing this then the proposed filter envelope depth control could actually just be a 'envelope->filter depth' control here, with the envelope just being called 'mod envelope' rather than specific to the filter. Effects Chorus level 0-127 Potentiometer Sets the amount of chorus applied to the overall sound. Reverb level 0-127 Potentiometer Sets the amount of reverb applied to the overall sound. Delay level 0-127 Potentiometer Sets the amount of delay applied to the overall sound. Delay time 0-127 Potentiometer Sets the delay time of the delay effect Distortion type pre-filter, post-filter Toggle Switch Sets where in the signal chain that the distortion is applied. Global Vintage pitch amount 0-127 Potentiometer Unique to this synthesiser. Sets the amount of random voice detuning, essentially emulating an old/broken analogue synthesiser. This would replace the single 'vintage amount' parameter. Vintage cutoff amount 0-127 Potentiometer Unique to this synthesiser. Sets the amount of random filter cutoff offsets, essentially emulating an old/broken analogue synthesiser. This would replace the single 'vintage amount' parameter. Vintage crackle amount 0-127 Potentiometer Unique to this synthesiser. Sets the amount of random crackles/pops applied to the overall sound, essentially emulating an old/broken analogue synthesiser. Patch save and recall - Push buttons A set of buttons that are used to save and load 'patches' - a set of parameter values -
MIDI I/O - Part 2 (Processing MIDI messages)
08/31/2018 at 07:11 • 0 comments(Original post date: 24/01/16)
In my last log I talked about the implementation of the electronics needed for adding a MIDI interface to my vintage toy synthesiser. As a suitable follow-on, within this post I thought I'd talk in-depth about MIDI message processing; specifically about five factors of the MIDI message format that make processing MIDI messages more complicated than it appears, or at least in regards to allowing full compatibility with all MIDI gear. As I'm not using any MIDI library (which would typically be used to handling the processing of MIDI messages) to develop the software for this project, I needed to write this code from scratch; C code which I have shared at the bottom of this blogpost.
Basics to MIDI Processing
MIDI is a form of serial communication, therefore bytes are transmitted one-byte at a time rather than in chunks or packets. Therefore to process MIDI messages coming from a serial port, each byte needs to be read and processed individually.
The first byte of any MIDI message, the 'status' byte, will always have a value of 128 or above, with the following 'data' bytes having a value of less than 128, therefore the first part to correctly processing MIDI messages is to check the value of each byte against this value - if the byte is 128 or above you know you have just received the start of a new MIDI message, or if the byte is 127 or below you that this byte is part of the previous MIDI message.
However the second part to correctly processing MIDI messages is to know the length of each MIDI message based on the status byte value, so that you know when you've received a full MIDI message which can now be used. Different MIDI messages have different lengths, and as the status byte represents the message type, this will indicate how many data bytes we should expect to receive after the status byte. Once we've received the correct amount of data bytes after a status byte, we know we've received a full MIDI message which can now be used within the software/system.
An example of using the above two rules in a MIDI processing algorithm:
- From a serial port you read a byte of value 176. Because it is a value greater than 127 you know it is a status byte, and the start of a new message, so you store this byte at the start of a message buffer.
- You look up what the status byte represents - it is CC message on MIDI channel 0. As it is a CC, you expect to receive two data bytes next to complete this message, and you flag that so far you have only received one out of three bytes.
- You read a second byte of value 1. As you've flagged that we've currently received just one byte of a CC message, this must be the first data byte, or the CC number. You store this value in the second index of the message buffer, and flag that so far you have received two out of three bytes.
- You read a third byte of value 23. As you've flagged that we've currently received two bytes of a CC message, this must be the second data byte, or the CC value. You store this value in the third index of the message buffer. As you have now received all three bytes of a CC message, you flag that this message have been fully received and can be used to trigger an event within your application.
The above rules mean that invalid message can be caught and discarded rather than the system attempting to use them. For example, if a new status byte is received before all the data bytes of the last message are received, you know to discard the previous message in the message buffer and start storing and processing a new message. Or if too many data bytes are received after a status byte, as you would have already processed the MIDI message and are waiting to receive a status byte to start processing a new message, any extra data bytes would just be ignored.
It is worth mentioning that there is one type of MIDI message that can't be processed in this way - System Exclusive (or SysEx) messages. However these are easier to process - they always start with status byte value 240, and end with a byte of value 247, with a variable number of data bytes in-between.
Advanced MIDI Processing
As mentioned at the start, there are some factors to the MIDI message format that mean processing MIDI messages isn't always as simple as just checking for a status byte and a message length. Here are five factors that could be unknown to some MIDI developers, with examples of how these factors can be processed in my example code at the bottom.
1. Note-on Note-off Messages
This factor is fairly well know to MIDI developers, but can be easily forgotten from time to time. Note-on and Note-off messages have their own set of status byte values, however 'note-off' messages can also be sent using the note-on message format but with a velocity value (2nd data byte) of zero. Many older MIDI controllers and synthesisers use this feature of MIDI in conjunction with the next factor I'll talk about, running status.
Therefore to correctly processing note-on messages, if it has a third byte of value 0, this message must actually be converted to or used as a note-off message (but with the same channel and note number).
2. Running Status
Running Status is a feature of MIDI that allows messages to be sent without the status byte, if the status byte is the same as that of the previous message. This feature was used in a lot of older MIDI equipment which had limited processing power available, as it allows less bytes to be sent. This explains why some MIDI keyboards send note-off messages as note-on messages, as it allows any number of keys to be pressed and released but without needing to send a new status byte on each event. Only Voice Category messages (e.g. note, CC, aftertouch) can be sent using running status.
To correctly process running status messages, after receiving a full voice category MIDI message, instead of setting the byte counter back to 0 at this point you set it to a value of 1. This will put your processing algorithm into a state where it thinks it has already received a status byte and is waiting for the messages data bytes. If a status byte is received instead at this point it just restarts processing a new message.
3. 14-Bit CCs
The majority of MIDI messages have 7-bit value bytes, due to data bytes not being able to have a value greater than 127. This isn't always enough resolution for control, therefore it is possibly to send 14-bit CC values by sending a pair of CCs instead. CC controller 0-31 can be used to send 14 bit values by following the CC with a second CC with a controller value of 32-63, where the combined 14-bit value is split between the value byte (3rd byte) of each message. For example, if you want to send a mod wheel message with a 14 bit resolution value, split the value into two 7 bit values (known as coarse and fine values), send the coarse value using a CC message with controller number (2nd byte) set to 0, and send the fine value using a CC message with a controller number set to 32.
Processing 14 bit CCs needs to be done one way or another, even if you only ever want to use 7 bit CCs. Therefore to correctly process CC messages you must always store the controller number of the previous CC you received, and if the new CC number is between 32 and 63 where the previous CC number is equal to the new CC number minus 32, flag that you have received a 14 bit CC value. It's then up to you whether you want to process this as a 14 bit CC or not. In my project/code I don't want to process 14 bit CCs so I just ignore the second CC (else my application thinks I've received two separate CCs and could do odd things based on this), however if you want to use 14 bit values it would be at this point that you combine the coarse and fine values to create a 14 bit number.
For more info on 14 bit MIDI CCs, including how they are encoded and decoded, see here.
4. RPNs and NRPNs
Another way that 14 bit resolution values can be sent using MIDI is using Registered Parameter Numbers (RPNs) and Non-Registered Parameter Numbers (NRPNs), which are sent as a succession of 3 or 4 specific CCs. RPN/NRPNs not only allow greater resolution of values, but also allow for a greater number of controller/parameter numbers. First an RPN/NRPN parameter number is sent using a pair of CCs (101 and 100 for RPNs, 99 and 98 for NRPNs), and then the parameter value is sent using either one or two further CCs (6 - course value, and 38 - fine value) depending on whether it is a 7-bit or 14-bit number. These CC numbers are specified for sending RPNs and NRPNs, and ideally should not be used for any other purpose in order to have full compatibility with other MIDI hardware and software.
To process RPNs/NRPNs an RPN/NRPN-specific message buffer needs to be used. If CC 101 or 99 is fully received you need to store the value of the CC and wait for the next parameter CC (CC 100 or 98), and once you have that combine the two received parameter numbers into a single parameter number, much like how 14 bit CC coarse and fine values are combined. Once you get the following CC 6 message you now have the full NRPN message to use as desired, though if the RPN/NRPN is followed by CC 38 you then need to adjust the value or this NRPN to include this fine value.
RPN/NRPN processing is something I have yet to add to my MIDI processing code. You can read more about RPNs and NRPNs here.
5. Intwined MIDI Messages
Most MIDI equipment won't start sending a new MIDI message until it has finished sending the previous one, however this isn't the case for all MIDI gear. You may find that some MIDI devices will send Realtime Category messages (e.g clock, active sensing) intwined within Voice Category messages, which is valid and needs to be processed correctly.
For example, the following is an example of a series of MIDI messages, with the bytes received in the order you'd expect:
This example shows four CC messages (status byte in red, data bytes in orange) with several Clock Timing messages (one byte MIDI messages) (in black) in between.
However here is an example of the same set of messages but with the bytes sent in a different order:
All the same messages are sent, but the clock messages are sent in the middle of CC messages. Unfortunately this is a valid stream of MIDI messages, and without proper processing only the last CC message in the previous example would have been processed correctly.
When I was working on an algorithm to process intwined MIDI messages correctly, this webpage proved to be very helpful. Even though it is mainly talking about running status, it offers an answer on how to handle intwined messages:
A recommended approach for a receiving device is to maintain its "running status buffer" as so:
- Buffer stores the status when a Voice Category Status (ie, 0x80 to 0xEF) is received.
- Nothing is done to the buffer when a RealTime Category message is received.
As can be seen in my example code below, only when a voice category message is received do I update my running status variable; if a clock message is received I just store this in my message buffer and flag that I've received a clock message. If the clock message was received within a voice message and I then start receiving the rest of the message, as the running status variable is still equal to the last received voice message I will keep processing this message.
Example MIDI Processing Code
This is the C code I'm currently using to process MIDI messages coming from the MIDI serial port on my BBB. It is a function that is called each time I receive a byte, which the byte is passed into, and returns a non-zero number when a full message has been received indicating the message type. I'm not suggesting the is the best or definitive way of processing MIDI messages, and it is currently only processing MIDI messages I care about (note, CC, aftertouch, pitch bend, program change, clock, and SysEx), however it is a good working example of how it can be done.
#define MIDI_NOTEOFF 0x80 #define MIDI_NOTEON 0x90 #define MIDI_PAT 0xA0 #define MIDI_CC 0xB0 #define MIDI_PROGRAM_CHANGE 0xC0 #define MIDI_CAT 0xD0 #define MIDI_PITCH_BEND 0xE0 #define MIDI_NOTEOFF_MAX 0x8F #define MIDI_NOTEON_MAX 0x9F #define MIDI_PAT_MAX 0xAF #define MIDI_CC_MAX 0xBF #define MIDI_PROGRAM_CHANGE_MAX 0xCF #define MIDI_CAT_MAX 0xDF #define MIDI_PITCH_BEND_MAX 0xEF #define MIDI_CLOCK 0xF8 #define MIDI_CLOCK_START 0xFA #define MIDI_CLOCK_CONTINUE 0xFB #define MIDI_CLOCK_STOP 0xFC #define MIDI_SYSEX_START 0xF0 #define MIDI_SYSEX_END 0xF7 uint8_t ProcInputByte (uint8_t input_byte, uint8_t message_buffer[], uint8_t *byte_counter, uint8_t *running_status_value, uint8_t *prev_cc_num) { /* A recommended approach for a receiving device is to maintain its "running status buffer" as so: Buffer is cleared (ie, set to 0) at power up. Buffer stores the status when a Voice Category Status (ie, 0x80 to 0xEF) is received. Buffer is cleared when a System Common Category Status (ie, 0xF0 to 0xF7) is received - need to implement this fully!? Nothing is done to the buffer when a RealTime Category message (ie, 0xF8 to 0xFF, which includes clock messages) is received. Any data bytes are ignored when the buffer is 0. (http://www.blitter.com/~russtopia/MIDI/~jglatt/tech/midispec/run.htm) */ //static uint8_t running_status_value = 0; //static uint8_t prev_cc_num = 127; //don't init this to 0, incase the first CC we get is 32, causing it to be ignored! uint8_t result = 0; //===================================================================== //If we've received the start of a new MIDI message (a status byte)... if (input_byte >= MIDI_NOTEOFF) { //If it's a Voice Category message if (input_byte >= MIDI_NOTEOFF && input_byte <= MIDI_PITCH_BEND_MAX) { message_buffer[0] = input_byte; *byte_counter = 1; result = 0; *running_status_value = message_buffer[0]; } //If it's a clock message else if (input_byte >= MIDI_CLOCK && input_byte <= MIDI_CLOCK_STOP) { //Don't do anything with MidiInCount or *running_status_value here, //so that running status works correctly. message_buffer[0] = input_byte; result = input_byte; } //If it's the start of a sysex message else if (input_byte == MIDI_SYSEX_START) { message_buffer[0] = input_byte; *byte_counter = 1; } //If it's the end of a sysex else if (input_byte == MIDI_SYSEX_END) { message_buffer[*byte_counter] = input_byte; *byte_counter = 0; result = MIDI_SYSEX_START; } // If any other status byte, don't do anything } //if (input_byte >= MIDI_NOTEOFF) //===================================================================== //If we're received a data byte of a non-sysex MIDI message... //FIXME: do I actually need to check *byte_counter here? else if (input_byte < MIDI_NOTEOFF && message_buffer[0] != MIDI_SYSEX_START && *byte_counter != 0) { switch (*byte_counter) { case 1: { //Process the second byte... //Check *running_status_value here instead of message_buffer[0], as it could be possible //that we are receiving running status messages entwined with clock messages, where //message_buffer[0] will actually be equal to MIDI_CLOCK. //TODO: process NRPNs, correctly (e.g. process 9 byte NRPN, and then as 12 bytes if the following CC is 0x26) //if it's a channel aftertouch message if (*running_status_value >= MIDI_CAT && *running_status_value <= MIDI_CAT_MAX) { message_buffer[1] = input_byte; result = MIDI_CAT; //set the correct status value message_buffer[0] = *running_status_value; //wait for next data byte if running status *byte_counter = 1; } //if it's a program change message else if (*running_status_value >= MIDI_PROGRAM_CHANGE && *running_status_value <= MIDI_PROGRAM_CHANGE_MAX) { message_buffer[1] = input_byte; result = MIDI_PROGRAM_CHANGE; //set the correct status value message_buffer[0] = *running_status_value; //wait for next data byte if running status *byte_counter = 1; } //else it's a 3+ byte MIDI message else { message_buffer[1] = input_byte; *byte_counter = 2; //set the correct status value message_buffer[0] = *running_status_value; } break; } case 2: { //Process the third byte... result = 0; //TODO: process NRPNs, correctly //if it's not zero it's a note on if (input_byte && (*running_status_value >= MIDI_NOTEON && *running_status_value <= MIDI_NOTEON_MAX)) { //3rd byte is velocity message_buffer[2] = input_byte; result = MIDI_NOTEON; //set the correct status value message_buffer[0] = *running_status_value; } //if it's a note off else if ((*running_status_value >= MIDI_NOTEOFF && *running_status_value <= MIDI_NOTEOFF_MAX) || (!input_byte && (*running_status_value >= MIDI_NOTEON && *running_status_value <= MIDI_NOTEON_MAX))) { //3rd byte should be zero message_buffer[2] = 0; result = MIDI_NOTEOFF; //set the correct status value message_buffer[0] = *running_status_value; } //if it's a CC else if (*running_status_value >= MIDI_CC && *running_status_value <= MIDI_CC_MAX) { //if we have got a 32-63 CC (0-31 LSB/fine CC), //and the last CC we received was the MSB/coarse CC pair if (message_buffer[1] >= 32 && message_buffer[1] <= 63 && (*prev_cc_num == (message_buffer[1] - 32))) { //Don't do anything. Right now if this is the case we just want to ignore it. //However in the future we may want to process coarse/fine CC pairs to //control parameters at a higher resolution. printf ("[VB] Received CC num %d directly after CC num %d, so ignoring it\r\n", message_buffer[1], *prev_cc_num); } else { message_buffer[2] = input_byte; result = MIDI_CC; //set the correct status value message_buffer[0] = *running_status_value; } //else //store this CC num as the previously received CC *prev_cc_num = message_buffer[1]; } //if it's a poly aftertouch message else if (*running_status_value >= MIDI_PAT && *running_status_value <= MIDI_PAT_MAX) { message_buffer[2] = input_byte; result = MIDI_PAT; //set the correct status value message_buffer[0] = *running_status_value; } //if it's a pitch bend message else if (*running_status_value >= MIDI_PITCH_BEND && *running_status_value <= MIDI_PITCH_BEND_MAX) { message_buffer[2] = input_byte; result = MIDI_PITCH_BEND; //set the correct status value message_buffer[0] = *running_status_value; } // wait for next data byte (if running status) *byte_counter = 1; break; } default: { break; } } //switch (*byte_counter) } //else if (input_byte < MIDI_NOTEOFF && message_buffer[0] != MIDI_SYSEX_START && *byte_counter != 0) //if we're currently receiving a sysex message else if (message_buffer[0] == MIDI_SYSEX_START) { //add data to the sysex buffer message_buffer[*byte_counter] = input_byte; *byte_counter++; } return result; }
-
MIDI I/O - Part 1 (Electronics)
08/31/2018 at 07:05 • 0 comments(Original post date: 20/01/16)
MIDI is an essential part of any serious piece of electronic music equipment. In a nutshell, MIDI is "a technical standard that describes a protocol, digital interface and connectors and allows a wide variety of electronic musical instruments, computers and other related devices to connect and communicate with one another". For example, it allows a consumer musical keyboard from one company to trigger notes or control audio within a piece of software developed by a completely different company, pretty much out-of-the-box.
You've probably seen from one of my previous logs that I use the MIDI messaging format to send note messages from the keyboard mechanism to the BeagleBone Black, however to make my toy piano synth fully MIDI-compatible I need to add some kind of MIDI input and output connections to the synth. In this blogpost I'm going to talk about the hardware and electronics I've used to allow MIDI messages to be sent to and from the BeagleBone Black, integrating a fully functional MIDI interface into my vintage toy synthesiser.
MIDI Hardware Transport Options
The original and most common hardware transport option for MIDI I/O is using a pair of five-pin DIN connectors, and I have planned on using this hardware transport option from the outset of this project because of the following reasons:
- They are the most common MIDI connectors found in commercial synthesisers
- They can quite simply be connected to one of the UARTs on the BeagleBone Black
- I've used MIDI-DIN connectors in past projects
A standard MIDI-DIN connector (top) and a pair of standard MIDI-DIN cables (bottom)
Throughout the project I have considered alternative or addition connections for sending and receiving MIDI messages, however for this project there weren't enough good reasons to use them. Other hardware transport options I had consider include the following:
- USB MIDI - MIDI can be sent over USB, and in regards to computer music it is becoming the most common hardware interface for MIDI. Most modern MIDI controllers include USB-MIDI, sometimes instead of MIDI-DIN connectors, as it allows MIDI hardware to be plugged straight into computers, however it is still not that common on commercial synthesisers. I believe that the mini USB port on the BBB can be used as a USB client port, making it a USB slave device (such as a commercial MIDI controller), however I'm not 100% sure of this as I haven't seen any examples, and I haven't had any experience with programming USB comms in Linux.
- Ethernet MIDI - MIDI can be transmitted over a network protocol such as Ethernet, and as the BBB includes a network port this would be an option for this project. However MIDI over Ethernet isn't as supported as the other options, plus I've not had as much experience with network comms compared to standard serial comms.
- Bluetooth MIDI - MIDI over Bluetooth is starting to become quite common on modern MIDI controllers that are designed to be portable and need to be wireless. However, even though it's fairly small, my vintage toy piano is a bit too bulky to be considered a 'portable' instrument, plus I'd need to attach a Bluetooth transmitter/receiver to the BBB for this option, which is a technology I've had no experience with, so it didn't make sense as a MIDI transport option for this project.
The Circuit
The circuit for the MIDI interface within my synth is made up of two separate circuits - a MIDI-in circuit and a MIDI-out circuit, which each connects a MIDI-DIN connector to a TX or RX serial port on the BBB. There are plenty of examples of these circuits and how they are connected to boards such as the BBB, and below I'm going to highlight the specific guides I used, as well as any of my own additions or changes.
MIDI-In
The guide I used for building the MIDI-in circuit was the Libre Music Production Arduino and MIDI in guide, which is fully transferable for the BBB.
The main components needed for the circuit are:
- 1 x female MIDI DIN connector
- 3 x 220 Ohm resistor
- 1 x 1N4148 diode
- 1 x 10kOhm resistor
- 1 x 6N138 optocoupler
As explained in the guide, an optocoupler is very important here as it allows the two electronic circuits (the BBB and the connected MIDI gear) to be electronically isolated from each other, which prevents the occurrence of ground loops and protects equipment from voltage spikes. Here is a breadboard diagram of the circuit from the guide that I used:
A couple of corrections and things to mention about this diagram:
- The anode of the diode should actually be connected to pin 3 of the optocoupler, not pin 4
- The viewport of the MIDI connector is from the back
- Obviously in my project this circuit is being attached to a BBB instead of an Arduino. I mention how this circuit is specifically connected to the BBB below.
MIDI-Out
The guide I used for building the MIDI-out circuit was the official Arduino MIDI guide, which again is fully transferable for the BBB.
The MIDI-out circuit is a lot simpler than the MIDI-in circuit, and it only requires the following components:
- 1 x female MIDI DIN connector
- 2 x 220 ohm resistors
Here is a diagram of the circuit from the guide that I used:
A couple of things to mention about this diagram:
- The viewport of the MIDI connector is from the front
- Again, in my project this circuit is being attached to a BBB instead of an Arduino.
The Combined Circuit
Here is a photo of the above two circuits combined onto a single piece of strip board for my project:
In the above photo, the wires at the top are going to the two MIDI-DIN connectors, and the wires on the left are going to the BBB.
Combining the two circuits into a single one has allowed me share the power and ground lines between the two, meaning I only need to use a single pair of wires from the BBB for power and ground.
You'll also notice that I've used a set of screw terminals for connecting the MIDI-DIN connectors to the circuit. I've done this so that, once everything is attached to the toy piano enclosure, I can remove this particular circuit from the piano if needed without having to remove the connectors too, or vice-versa.
Connecting to the BeagleBone Black
As mentioned above MIDI is a serial communication method, meaning that the above circuit can be simply attached to the BBB via a pair of UART pins. I'll be using UART2 for MIDI, therefore I've attached the circuit to the BBB using the following pins:
- Orange wire to BBB P9_21 pin (UART2 TXD), for sending MIDI messages from the BBB to an external device
- Green wire to BBB P9_22 pin (UART2 RXD), for receiving MIDI messages from an external device to the BBB
- Black wire to BBB P9_01 pin (a DGND pin), for allowing the MIDI circuit to be powered by the BBB
- Red wire to BBB P9_03 pin (a VDD_3V3 pin), for powering the MIDI circuit using the BBB
As per connecting the keyboard mechanism, this circuit needs to be powered by a 3.3V pin instead of 5V, as the BBB serial ports run at 3.3V.
Connections to UART 2 on the BBB
Setting the Required Serial Baud Rate in Linux
While this blogpost covers the electronics of the MIDI I/O connection within my toy piano synth, I just thought I'd briefly talk about how I successfully got MIDI messages being send from and to software running on the BBB, as this took me a little while due to the serial baud rate needed for MIDI.
MIDI communicates using a serial baud rate of 31250, which is not a standard or common baud rate. The code I've shown in a previous blogpost for setting up serial comms in Linux wouldn't work here as 31250 is not a recognised rate when using the most common method of setting up serial comms (or at least what I consider to be the common method!). After a lot of Googling I found this thread in which a very helpful man called Peter Hurley provided some example code on how to use the BOTHER method of setting a custom baud rate. Using this example code I have now replaced my serial setup code with the following in order to get work MIDI comms:
int SetupSerialPort (const char path[], int speed, bool should_be_blocking) { int fd; struct termios2 tio; // open device for read/write fd = open (path, O_RDWR); //if can't open file if (fd < 0) { //show error and exit perror (path); return (-1); } if (ioctl (fd, TCGETS2, &tio) < 0) perror("TCGETS2 ioctl"); tio.c_cflag &= ~CBAUD; tio.c_cflag |= BOTHER; tio.c_ispeed = speed; tio.c_ospeed = speed; if (ioctl( fd, TCSETS2, &tio) < 0) perror("TCGETS2 ioctl"); printf("[VB] %s speed set to %d baud\r\n", path, speed); return fd; }
-
Getting started with BBB audio and the Sound Synthesis Engine
08/30/2018 at 21:25 • 0 comments(Original post date: 16/01/16)
This week for the project I've been attempting to get started on the sound synthesis engine for my synth, as well as setting up an audio output on my BeagleBone Black. There have been quite a few frustrating evenings in doing this, and while I now finally have synthesised sound coming out of my BBB, it looks like I may need to rethink what library I'm going to use to make my synthesis engine, or redesign and simplify my original idea for the synthesis engine.
USB audio on the BeagleBone Black
Initially I was planning on building my own DAC for the audio output on my BBB, however after looking into the audio IO options on the board I decided that taking advantage of the USB audio support would be the best option.
I tried out a couple of USB audio adapters to use with my BBB, and finally settled on an EC Technology adapter. At first I bought this budget adapter, however the sound quality was really terrible, whereas the EC Technology one sounds just as good (if not a little better) than the built-in soundcard in my MacBook.
A quick visual review of USB audio adapters I tried out
To set up USB audio as the default sound device on my BBB I used this guide. ALSA was already installed on my BBB, and contrary to the guide all I had to do was disable HDMI to make USB audio the system default. This was done by adding the following line to my /boot/uEnv.txt file:
optargs=capemgr.disable_partno=BB-BONELT-HDMI,BB-BONELT-HDMIN
After doing that and rebooting my BBB, any audio I played on my BBB was coming out of the USB audio adapter. However if you're attempting to do that same I recommend reading the whole guide incase you need to do the extra steps.
The Synthesis Library
Compiling Maximilian
As stated in the proposal of this project, I've been planning on using the C++ audio synthesis/DSP library Maximilian to develop the synthesis engine. While I have never used it before, I was drawn to this particular library for a few reasons:
- It is supported on Linux, which is obviously very important for this project
- It uses C++, which is my preferred language
- It looks easy to use and comes with a lot of code examples
- It seems to still be support and regularly updated
As stated in my last log, I've been using a cross-compiler on OS X for compiling my BBB programs on so far, however when it came to attempting to cross-compile Maximilian example code everything became a lot more difficult. As Maximilian (and I'm guessing all Linux-based audio libraries and applications) requires the OSS or ALSA library to be compiled, one of these frameworks libraries needed to be installed and configured in such a way so that my cross-compiler could use it. After many attempts at doing this, getting fairly close using the advanced sysroot installation guide for my cross-compiler, I wasn't able to successfully compile the Maximilian example project.
So I went onto plan B - compiling the code on the BBB itself. This process involves editing the code on my MacBook, followed by scp'ing it onto the BBB, and then using the standard GCC compiler on the board. My first attempt at this didn't work, and came up with the following compilation errors for a number of variables:
maximilian.h:412:18: error: ISO C++ forbids initialization of member 'x' [-fpermissive]
maximilian.h:412:18: error: making 'x' static [-fpermissive]
maximilian.h:412:18: error: ISO C++ forbids in-class initialization of non-const static member 'x'I'm not completely sure why the compiler thinks these variables are static, and everything compiles fine on both OS X and Intel Linux, however I fixed this by simply removing the initialisation of these variables in the hope that doing so won't break the library in anyway. After doing this the Maximilian program compiled, and I was able to run it and get example sound coming out of the USB audio.
Using Maximilian
The first few Maximilian examples I tried worked fine on the BBB, however it wasn't until I tried the polysynth example that I discovered something I hadn't yet considered - Maximilian may not be optimised for single-board computers such as the BBB, or least the more complex examples. When running the unedited polysynth example with no GCC optimisation the audio was very glitchy and the program was using 99% of the CPU - a program that runs without issue on my MacBook. Even when adding the -O3 optimisation flag (full optimisation for performance) to the compile command the audio would still glitch, and after finding this guide on getting and setting BBB CPU speed I found my BBB was already running at it's max 1000MHz frequency.
This led me on to test the limits of what I could do with Maximilian before I start getting audio artefacts and a ridiculous CPU usage. The polysynth example is a good program to test with as it includes a fairly large number of synth components (18 oscillators, 6 filters, and 6 envelopes) which provides 6 note polyphony (with each note containing 2 oscillators and an LFO), and is a good simple example of the synthesis engine I was hoping to develop for this project. This is what my tests found, all with full performance optimisation:
- I can get up to 4 note polyphony of the polysynth example before getting glitches
- Removing all filters drops the CPU usage down to 30-40%, and allows me to increase polyphony up to at least 16 without getting glitches
Going forward, this gives me the following options:
- Use Maximilian but with a redesigned, simplified sound engine (e.g. a low polyphony number, a single global filter instead of an individual filter for each note).
- Attempt to optimise Maximilian for the BBB, somehow...
- Use another library or framework for creating my synthesis engine
At this point in time I still want to experiment a bit more with Maximilian to see if I can get it to work for me without having to simply my synthesis engine design too much, however there is another similar looking library, STK, that I may test to see if it offers better performance. Either way, it is good to finally hear synthesised sound coming out of my BeagleBone Black!
-
Getting Started with the BeagleBone Black
08/30/2018 at 21:16 • 1 comment(Original post date: 10/01/16)
After completing the majority of the key mechanism for my vintage toy synthesiser, which I covered in my last log, I thought it was about time I cracked open the BeagleBone Black board and attempted to connect the key mech to it. Setting up the BBB for my preferred development language and environment, as well as getting the Arduino-to-BBB comms working, was a bit more complex than I thought it would be, nevertheless I have now got the BBB receiving key interaction data from the keyboard.
This log covers the following main things:
- Setting up the BBB to be tethered to a computer
- Installing a BBB-compatible ARM cross-compiler on OS X
- A method for developing C/C++ based software for the BBB, from writing code to testing compiled binaries
- Enabling all UART/serial ports on the BBB, and writing software that reads from a connected serial device
My Preferred Development Languages and Environment
Professionally, and as a hobbyist, I mainly develop software using the C and C++ languages, which is what I plan to use when developing the BBB software for the vintage toy synthesiser.
When it comes to developing software for Linux-based single-board computers such as the BBB, my preferred way of doing it is using a cross-compiler that allows me to develop and compile the software on my main computer running OS X, and then using something such as Secure Copy (scp) to transfers the binaries onto the target hardware. The majority of this blog post talks about the tools and methods used to get this environment set up.
Tethering the BBB to a Computer
As per the official BBB Getting Started guide, the most common way to use and and develop on the BBB is to connect it to a computer and use the network-over-USB access. This is done using the following simple steps:
- Connect the BBB to your computer via USB
- Wait for a new mass storage device to appear
- On the mass storage device, open START.htm
- Follow the provided instructions to install the needed drivers
Once that has been done, possibly followed by a needed computer restart, you can now access your BBB through the 192.168.7.2 IP address. My preferred access method is to use Secure Shell (ssh) through a command line interface (CLI), using the command:
ssh -l root 192.168.7.2
BBB-Compatible ARM Cross-Compiler for OS X
It is entirely possible to develop software for the BBB directly on the board by accessing it over a network. However there are a couple of pitfalls here:
- You're stuck using CLI programs which are not everyones preferred method of interacting with a computer, especially when it comes to text editing
- When compiling your software you're limited to the power of the BBB which is probably not as great as your personal computer, making the process a lot slower
- You'll be developing using Linux, which may not be your preferred OS to use
The way around this is to develop the software on your personal computer, where you have a GUI and greater CPU/RAM specs, and then transfer it over to the BBB afterwards. The main obstacle in doing that though is the fact that the processor type and OS of your personal computer (most probably Windows or OS X running on an Intel or AMD processor) is probably different from that of the BBB (Linux running on an ARM processor), so you need to use a compiler/toolchain that will run on one type of system (the host) but build software to be run on another type of system (the target). This is known as a cross-compiler.
The toolchain needed for cross compiling for the BBB is arm-linux-gnueabihf; which compared to the more-commonly used arm-linux-gnueabi has hardware FPU (Floating Point Unit) support which is needed for compiling for the BBB target. The arm-linux-gnueabihf cross-compiler toolchain is officially available for both Linux and Windows as a GCC-based compiler released by Linaro. For OS X there is an unofficial (but working) version of the Linaro toolchain available from here - this is the cross-compiler that I have installed and started using.
My Preferred Development Method
Now that I have my cross-compiler installed, I can start developing software for the BBB using my preferred environment and tools.
This is my preferred development method, from writing code to testing the compiled binaries:
- I write my code on OS X using a programming text editor - my personal favourites are Xcode and Sublime Text
- I compile my code using a Terminal window with the one of the following command:
For C code:
/usr/local/linaro/arm-linux-gnueabihf/bin/arm-linux-gnueabihf-gcc [source file] -o [compiled binary name]
For C++ code:
/usr/local/linaro/arm-linux-gnueabihf/bin/arm-linux-gnueabihf-g++ [source file] -o [compiled binary name] - With the same Terminal window I copy the compiled binaries to the BBB using scp:
scp [binary file] root@192.168.7.2:[destination directory]
- Using a second Terminal Window which has a running ssh session logged into the BBB (see the Tethering...section above) I test running the binaries using the following command:
./[binary file]
Eventually I will want my BBB software to start on boot, but I'll talk about that in a later blog post.
Connecting the Key Mech Arduino via Serial
Apart from the obvious Hello World program, the first application I have developed for the BBB is a simple program that reads serial data coming from the pianos key mech Arduino, displaying read bytes to the console.
The BBB has six on-board serial ports - one that is coupled to the boards serial console, and five UART ports that can be found on the boards expansion headers. By default only the serial console port is enabled, so to use any of the other UARTs you must allow them to be enabled at boot. I did this by following the "Section 1" steps on this tutorial. Note that on my BBB the uEnv.txt file was in the /boot/ directory, not /boot/uboot/ as the tutorial suggests.
Once this had been done, I connected the Arduino Pro Mini to the BBB using the following connections:
- Arduino TX pin to BBB P9_26 pin (UART1 RX), for sending serial data from Arduino to BBB
- Arduino GND pin to BBB P9_01 pin (a DGND pin), for allowing the Arduino to be powered by the BBB
- Arduino RAW pin to BBB P9_03 pin (a VDD_3V3 pin), for powering the Arduino using the BBB
The key mechanisms Arduino Pro Mini connected to the BeagleBone Black via the UART1 port
Lastly I developed a small piece of code that opens the UART1 device file (/dev/ttyO0) and displays any byte it reads from it. You can see this code below:
#include <stdio.h> #include <stdint.h> #include <stdlib.h> #include <termios.h> #include <fcntl.h> #include <unistd.h> #include <string.h> #include <stdbool.h> #include <errno.h> #define KEYBOARD_SERIAL_PATH "/dev/ttyO1" int main (void) { printf ("Running test_key_mech_input (v2)...\n"); int keyboard_fd; uint8_t keyboard_input_buf[1] = {0}; //========================================================== //Set up serial connection printf ("Setting up key mech serial connection...\n"); struct termios tty_attributes; // open UART1 device file for read/write keyboard_fd = open (KEYBOARD_SERIAL_PATH, O_RDWR); //if can't open file if (keyboard_fd < 0) { //show error and exit perror (KEYBOARD_SERIAL_PATH); return (-1); } tcgetattr (keyboard_fd, &tty_attributes); cfmakeraw (&tty_attributes); tty_attributes.c_cc[VMIN]=1; tty_attributes.c_cc[VTIME]=0; // setup bauds (key mech Arduino uses 38400) cfsetispeed (&tty_attributes, B38400); cfsetospeed (&tty_attributes, B38400); // apply changes now tcsetattr (keyboard_fd, TCSANOW, &tty_attributes); // set it to blocking fcntl (keyboard_fd, F_SETFL, 0); //========================================================== //Enter main loop, and just read any data that comes in over the serial port printf ("Starting reading data from key mech...\n"); while (true) { //attempt to read a byte from the serial device file int ret = read (keyboard_fd, keyboard_input_buf, 1); //if read something if (ret != -1) { //display the read byte printf ("Byte read from keyboard: %d\n", keyboard_input_buf[0]); } //if (ret) } ///while (true) return 0; }
Next Steps
Now that I have got the BBB up and running the next step is to start the development of the sound synthesis engine. This will involve developing some software that creates a simple controllable tone, as well as configuring the BBB to output the audio via one of its audio outputs.
-
Development of the Key Mechanism (Part 2 - Production)
08/30/2018 at 21:02 • 0 comments(Original post date - 04/01/16)
If you read my last log you would have seen that I started my project by prototyping ideas for the digitisation of the pianos key mechanism. Well, after spending the best part of a week attached to my soldering iron and multimeter I've finally completed most of the electronics and software for the final implementation of the key mech. The implementation for detecting key presses and releases has changed considerably since my initial design, and I've still got a bit of software tweaking to do, however I've now got a fairly reliable working key mechanism which is fully polyphonic, velocity-sensitive, and polyphonic pressure-sensitive.
Hardware
Components used:
- Velostat
- 10k resistor (x 18)
- 4051 multiplexier (x 2)
- Arduino Pro Mini (3.3V version)
Sensors
As per my prototyping, the sensor I have used to detect a press for each key is Velostat - a flexible conductive material that reduces resistance as pressure is applied to it. It is very similar to off-the-shelf pressure or bend sensors, but it has a number of advantages which have been useful for me:
- It's a lot cheaper
- It's more flexible
- It can be cut to any shape or size
- There are no set connection terminals
Using a pressure sensitive material was useful as it has allowed me to implement velocity sensitivity and polyphonic pressure sensitive is a fairly simple way.
Due to using Velostat, the circuit used for each key of the piano is essentially the same as that for an FSR sensor connected to a microcontroller, where one side of the sensor is attached to power, and the other to a pull-down resistor to ground as well as to an analogue input:
An FSR sensor attached to an Arduino. Source: https://learn.adafruit.com/force-sensitive-resistor-fsr/using-an-fsr
A small piece of Velostat has been attached to the underside of each key, and when a key is pressed it causes the Velostat to touch two contact points which closes a circuit (in the same way that a push button/switch works) and sends a pressure value to a dedicated analogue input for the pressed key. Since the prototyping stage I have rewired the keyboard so that the wires are soldered away from the area that the Velostat joins the circuit, allowing the contact area to be more even and allow greater touch sensitivity of the keys.
The Velostat attached to foam on the underside of each key, which connects two contact points going to power and ground/input when the key is pressed
Microcontroller
Even though it will be the BeagleBone Black microcontroller that uses the values from the keyboard to trigger and modify sound, I decided not to connect the sensors directly to the BBB but to use a second mirocontroller in between that communicates with the BBB via one if its serial interfaces. I did this for the following reasons:
- Splitting tasks - The main job for the BBB in this project is to run a sound synthesis engine which is going to be time critical, so I don't want it to be doing any extra tasks that could slow it down. Also the scanning of the pianos 18 keys needs to be done as fast as possible so that the keys trigger sound as soon as they are pressed, so using a dedicated microcontroller for this task would be preferable.
- More Modular - Connecting a microcontroller to the BBB rather than connecting 18 sensors directly requires a lot less connections and wires to the BBB, which makes it easier to remove the key mech or BBB from the piano if desired.
I chose to use an Arduino Pro Mini microcontroller for the key scanner as it is a small, cheap microcontroller that I personally have had a lot of experience with. I'm using the 3.3V version (rather than the 5V version) as I believe that the serial ports on the BBB run at 3.3V which would not be able to communicate with a 5V microcontroller without using a logic level shifter. As the Pro Mini only has 8 analogue pins I've had to use two 4051 multiplexers to connect all 18 sensors to the Arduino; the first 8 keys are connected to 1 multiplexer, the last 8 keys are connected to a second multiplexer, with the middle 2 keys connected directly to the Arduino. This has meant I'm only using a total of 4 analogue inputs, as well as 6 digital outputs to control the multiplexers. Finally the serial transmit (TX) pin is used to send serial messages to the BBB.
The Complete Circuit
The completed circuit for the key mech has been developed using stripboards which have been screwed to the base of the piano, as well a copper tape used as the power line to each key as well as for the contacts for the 'switch' below each key. Below is a breadboard diagram of the circuit I have built, followed by some photos of the circuit.
Breadboard diagram of the completed circuit, where the Velostat sensors have been replaced by FSRs
Software
As mentioned above, all the processing of key interaction is handled using an Arduino microcontroller, so the only software required for the key mech is a single Arduino sketch that needs to handle four main things - processing key presses/releases, processing key velocity, processing key pressure, and sending these three attributes to the BBB as serial-based messages.
I have created a GitHub repository to host all my code and schematics/diagrams for this project. To see the current state of the key scanner code click here.
Processing Key Presses/Releases
Key presses are detected by reading an initial analogue input value of above 0, which happens when force is applied to one of the Velostat sensors which causes the keys circuit to be closed, whilst the key is currently flagged as 'off'. This will then trigger a note-on message for the pressed key as soon as we have generated a key velocity value (see below). Key releases are detected by reading an analogue value of 0 whilst the key is current flagged as 'on', which happens when one of the Velostat sensors is removed and breaks/opens the keys circuit. This will then trigger a note-off message for the released key.
Processing Key Velocity
Key velocity is the attribute of how soft or hard a key is pressed, and is most commonly mapped to the volume/gain of the triggered sound though can usually also be mapped to other sound parameters. My implementation of velocity sensitivity in the toy piano synth involves starting a timer when we get an initial key press, and then after 10ms it uses the highest sensor analogue value that was received to work out a velocity value.
For example, if a key is pressed lightly only a small force is applied to the Velostat sensor, causing the analogue input value reached within 10ms to be fairly small, giving us a small velocity value. On the other hand, if a key is pressed harder a greater force is applied to the Velostat sensor, causing the analogue input value reached to be larger, giving is a greater velocity value.
Once the key velocity has been generated it then sends a note-on message for the pressed key. I am using a time value of 10ms here as it has been found that the minimum latency the ear can detect is 11ms, so providing the key press triggers a sound before this time there will be no apparently delay.
Processing Key Pressure
Some synthesisers and MIDI keyboards have an attribute called 'Aftertouch' - the amount of pressure/force that is applied to a key after is has been initially pressed, which can then be mapped to modulate a number of sound parameters. There are two types of Aftertouch - channel aftertouch (also known as Channel Pressure) and polyphonic aftertouch (also known as Poly Pressure), where channel aftertouch modulates every note of the same channel or keyboard at the same time, and polyphonic aftertouch modulates each note individually. Polyphonic aftertouch offers much better expression and modulation possibilities, however unfortunately it is quite rare among consumer keyboards and instruments as it is both expensive and more complex to implement.
As each key in my toy piano synth has its own pressure sensor, in this case I am able to implement polyphonic pressure sensitivity in quite a simple way. Currently it is implemented so that once a key has been pressed and a note-on message with a velocity value has been sent, any further changes to the force applied to the key while being held down that are greater than the initial force get converted to pressure messages for that key. However this implementation only works for softer key presses, as a hard press won't leave any scope for applying any more force that can be converted to an analogue signal, so I need to rethink how I can allow pressure sensitivity for all velocity values.
Serial Message Format
Being an electronic musical instrument, the most obvious message format to use here is MIDI. There are three types of MIDI messages that the Arduino will send to the BBB over serial, which are all three-byte messages that start with a status byte (a value greater than 127), which contains the message type and MIDI channel values; followed by 2 data bytes (which each have a range of 0 to 127):
- Note-on message - [144 + MIDI channel] [note/key number] [velocity value]
- Note-off message - [128 + MIDI channel] [note/key number] [0]
- Polyphonic aftertouch message - [160 + MIDI channel] [note/key number] [pressure value]
Example Video
Here is a demonstration video of the key mechanism in action. For this demo I have connected the Arduino Pro Mini to an Arduino Mega (through serial) which has been modified with HIDUINO so that it can be recognised as a USB-MIDI controller, sending MIDI messages to Logic Pro. Sending the messages from the keyboard to external MIDI devices will eventually be one of the tasks for the BBB.
Changes From Initial Design
The main part of the key mech implementation that has changed since the initial design is the removal of the set of switches on the back of each key, which were going to be used to generate velocity values by timing how long it takes for each key to be fully pressed; a method used by most consumer keyboard instruments for implementing velocity sensitivity. However after realising I could use the Velostat pressure values to work out velocity in a similar kind of way I decided not to add this set of switches, as it would double the size of the circuit and the amount of components. Also it would mean the Arduino would have to scan double the amount of inputs, possibly slowing down the key scanning task.
Therefore, contrary to my beliefs when devising the original design for this key mech and writing my last blog post, using sensors similar to FSRs has proven to be the easiest and most effective solution, rather than using a dual set of switches like consumer keyboards use.
Next Steps
Most of the key mechanism is now complete, and there are only a few things left to do on this part of the project:
- Connect the Arduino Pro Mini to the BBB to provide serial transmission, as well as to power the Arduino from the BBB.
- Some of the keys are a lot less sensitive than others, most probably due to the switch contact area not being even, which needs fixing.
- A couple of the Velostat sensors can't reach high pressure values, causing the keys to have decreased velocity and pressure sensitivity, which also needs fixing. I'm not yet entirely sure why this is, but it could be down to the position or size of the sensor.
- Each Velostat sensor produces slightly different results, causing the velocity and pressure sensitivity of each key to be slightly different. This could be addressed and calibrated in the software by defining maximum pressure values that each key/sensor can send.
- Improve the polyphonic aftertouch algorithm so that pressure sensitivity works better when keys are pressed more forcefully.
-
Development of the Key Mechanism (Part 1- Prototyping)
08/30/2018 at 20:41 • 0 comments(Original post date - 28/12/15)
The first part of this project I have decided to undertake is the development of the key mechanism - allowing the existing toy piano keys to generate note messages in the digital domain. I have decided to start with this particular task as, from a technical point of view, it could arguably be the most challenging part of the project, with next to nothing in regard to existing examples of it being done. I've done a lot of trial and error in past projects to find a reliable solution for doing this, however I'm now fairly confident that I'm on a path towards a working implementation.
A cross-section of the existing toy piano key mechanism, annotating the way it was designed to work.
Past implementations and ideas
This isn't the first project I've undertaken where I've needed to digitalise a toy piano key mech. Back in May 2015 at the MIDI HACK hackathon in Berlin I turned a vintage toy piano into a simple MIDI controller; the original idea for this synth project. For that particular project I decided to use piezo sensors that would be struck by the keys in order to create MIDI note on and off messages. I decided to use piezos as they're cheap sensors that have been a tried and tested solution among musical instrument projects, allowing great sensitivity as well as a mechanism for generating velocity values. My intention was to have a row of piezos stuck to the underside of the top of the piano, which would be struck by the keys hammers to generate note-on messages along with a velocity value created by the hardness of the piezo strike, coupled with a second row of piezos underneath the back of the keys for generating note-off messages when the keys were released. However the piezos ended up being pretty unreliable for me due to the handmade construction of the piano - they wouldn't always get struck by the hammers, sometimes hammers would strike or trigger multiple sensors, and the velocity values wouldn't always create consistent results.
The piezo implementation for my existing MIDI toy piano project.
This circuit was stuck to the underside of the top of the piano to allow the keys hammers to strike the piezos.
Another option I thought about for this project was to use Force Sensitive Resistors (FSR's) instead of piezos. These could have been placed underneath the front of the keys and generated note-on messages from the force applied to the keys when being pressed along with a velocity value, with note-off messages generated when the forced is released. FSR's would have also allowed me to implement MIDI polyphonic aftertouch into the piano, allowing each key to control a continuous parameter value by applying various amounts of pressure whilst the key is held down. However due to the increased expense of FSR's over piezos I didn't see it as a viable solution for the project. Also from past experiences I found that the touch sensitivity of FSR's isn't as great as I'd want it.
Current implementation idea
When approaching this synth project I realised a much simpler solution that could be implement - turning each key into a pair of simple switches, making them work in essentially the same way that most velocity-sensitive synth/MIDI keyboards work. This implementation is as follows:
- When a key is initially pressed it opens a switch/circuit that starts a timer
- When the key is fully pressed it closes a second switch/circuit that stops the timer. The time value is used to generate a velocity value, and a note-on message is sent.
- When the key is released it closes the first switch/circuit, generating a note-off message.
This is potentially a better solution than using existing sensors as it is a tried and tested method for electronic musical keyboards. As I need the touch sensitivity to be very sensitive I found I can't use any mechanical push switches/buttons here; even some membrane buttons I tried weren't sensitive enough. The best solution seems to be to just make the keys open and close circuits, acting in the same way that buttons/switches do.
Here's a basic sketch of the circuit for each key:
Where I'm at so far
So far I have created the switch/circuit on each key that will generate the note on messages (the switch at the front of the key). To do this I have done the following:
- drilled some holes below the keys to lay a wire for each key
- stuck two pieces of copper tape underneath the front of each key to act as the contact points for the switch
- connected a wire from one side of the contact point to an input on a microcontroller, one for each key
- attached the second side of the contact point of each key to a single strip of copper tape, acting as the power line
- attached some foam and Velostat to the underneath of the front of each key that closes the circuit when the key is pressed
Originally I was using copper tape on the underside of each key to close the circuit when a key is pressed, however I found that some keys needed to be heavily pressed for them to work, which was not desirable. To improve on this I went about trying to replicate the way that silicone keypads work, however after attempting get hold of some conductive pills (or the materials needed to make them) I discovered Velostat - a conductive and pressure sensitive flexible material that was recommended as an alternative to conductive silicone or rubber. I'm not 100% sure why (though I'm sure somebody reading this with more electronics knowledge and experience could tell me!), but using Velostat instead of copper tape works so much better, offering greater sensitivity to the keys. An added bonus to using Velostat is its pressure sensitivity, theoretically allowing me to implement polyphonic aftertouch into the piano. A small layer of foam has been inserted between the keys and the Velostat to give the keys a more tactile feel, especially when applying pressure. The foam has also decreased to the distance that the key needs to move in order to trigger a note message, making the keys less clunky and easier to play.
Here is a video of eight of the keys in action, wired up to an Arduino Mega board which is sending MIDI messages to Ableton Live software. There is no velocity or pressure sensitivity implemented yet, however notes of different lengths can be played, and it is fully polyphonic.
Next steps
I've still got a way to go before the key mechanism will be fully operational and connected to the BBB. Here are the main things I've got left to do:
- Improve the key sensitivity - some keys still aren't as sensitive as I'd like, probably because I've soldered the switch contact points directly below the key, meaning the contact area is uneven (a bit of a rookie mistake!). Ideally the contact area needs to be flat, so these solder points need to be moved.
- Add the circuitry for the back set of switches so that I can implement velocity sensitivity.
- Connect the front set of Velostat switches to a set of analogue inputs on an Arduino Pro Mini through some multiplexers.
- Connect the back set of switches to a set of digital inputs on the Arduino through some shift registers.
- Develop Arduino code for generating velocity-sensitive note messages and pressure messages
- Sending MIDI messages from the Arduino to the BBB via one if its serial interfaces