Okay! So much to report here all of a sudden. Yesterday I had both a discouraging and victorious day regarding this assignment.
I spent a large portion of the morning troubleshooting on my computer only to find that somehow I managed to delete Logic Pro (how does that even happen..). So a lot of time vanished as I was trying to fix this, which only really left me with 1 hour to try things out before going to work in the afternoon. At which point I discovered another weird problem. Anywho, point of the story was that not much happened on the computer, but my brain has been hard at work!
But! After work in the evening, I made a lot of progress in a relatively short amount of time with very little/possibly no troubleshooting at all. In fact I was actually able to set up most of my installation. So as you will have read, I will be using Genesis (a recent composition of mine) as the experimental platform to demonstrate the skills that I have taught myself, and the creative potential of this interactive installation. I’ve decided to use Mainstage as the host for the installation. If you are not familiar with Mainstage, basically it is a live version of Logic, designed to be used for performing.
Here’s a photo of all the controllers (including the Wii Balance Board and my iPad) that I have access to and considering using. I have a feeling I won’t be able to use all of them but I will let you know how many I can use without cluttering the interactivity.
The first thing I achieved was muting a large number of tracks in the final mix of Genesis (in Logic) to make a backing track in which I can layer interactive instruments on. I then opened the Playback plugin in Mainstage and inserted the newly bounced audio file.
The next step was to open each instrument (that I had muted in the final mix) in new channel strips within Mainstage. The interactivity will come from mapping the controllers that I have to the various instruments and their effects.
In my composition Genesis there’s a couple of sounds that I made using The Mangle, which is a granular synth plugin. I sampled my mum washing up one night (thankfully I didn’t damage my mic!) and opened the audio sample within The Mangle. Taking little (editable) grain sized chunks of the gurgles and sloshes of washing up, The Mangle can play them back at any pitch, speed, reversed, panned etc. The cross hairs you can see is where the grain is playing back from in the audio file. I also randomised the section of where the grains are playing back from whilst also randomising the volume and reversability of the grain play back. This creates a pretty unpredictable sound, especially when the source sound is water.
It’s pretty limitless. From there I mapped the the rate of the grain playback as well as the pitch of the playback to the Mod Wheel Midi CC – which is 1.
There are Midi CC standard messages which most softwares conform to – here’s a link to a handy site that says what each one typically does.
The next picture shows the Midi mapping from within The Mangle.
This is all pretty cool if you have a keyboard that is already sending through Midi CC 1 from the Mod Wheel. But I am only halfway there as I want to control this with a Wii Remote. This is where OSCulator comes into play. When you open up OSCulator for the first time it looks like this. Pretty blank.
Having automatically found the wii remote using the sync button, OSCulator has a number of ‘events’ that each button or motion control can be used for. If you’re using OSCulator for the first time, make sure you press every button on your remote so that the program knows about them. It will automatically know about the motion controls however, as they are always sending through data. In the picture below, you can see that I have two wii remote devices connected. One is in fact the Wii Balance Board and the other is a standard Wii Remote connected to a Guitar Hero Guitar attachment.
The reason behind the Wii Balance Board is that I want participants of this installation to have a designated place to experience the music from. Looking at the picture below, you can see up the top that I have the ‘SUM’ message mapped to the Midi Note D1. This means whenever someone steps on the board, it will sense their weight and play a D1 on an instrument. I then went ahead and mapped the top left and right regions of the board to two other notes so when the person shifts their weight on the board different notes/pitches will play. However, these pitches aren’t really going to be heard as the sound it will be triggering is going to be water sloshes.
I got really excited when I started playing with the actual Wii Remote events as I mapped the motion ‘Pitch’ event to the Midi CC 1 Value. What this means is that the Wii Remote effectively just became the Mod Wheel of a keyboard and instantly starting sending value messages to The Mangle grain pitch and rate playback. The graph below shows the values (up to 127) of the Pitch event being changed by the Wii Remote motion within OSCulator (angling the remote up and down).
All this together creates a morphing, ambient sloshing and gurgling of water that can only be controlled by standing on the Wii Balance Board and angling the Wii Remote up and down – which means it can be played by anyone. Totally cool. At this point I bounded around the house in victory.
But I wasn’t finished yet! I also had plans for the Pad Sound that I had also created. Sampling my own voice, I was able to use The Mangle to create a very smooth Synth Pad sound by editing the Attack, Sustain and Release of the grain playback. As I used only 4 chords for the Pad Sound, I decided I wanted to map each chord to a different button the Guitar Hero Guitar.
Using the Chord Trigger Midi FX in Mainstage, I was able to map a different key to play one of the four chords. It looks like this (below). Hit the learn button, choose the note you want to play the chord with on the blue keyboard (top). Then choose the notes that you want in your chord on the orange keyboard below. Don’t forget to hit Learn button again before moving on to the next chord.
The reason why I chose notes so high on the blue keyboard was because I haven’t yet found a way to make OSCulator send sounds to a different Midi input. This meant that as I played the pad sound, the water gurgles were also being triggered by the Guitar Hero Notes. I went back to The Mangle and adjusted which keys are allowed to play the gurgles so that I didn’t get any over laps. This is my quick fix, but if you happen to know a solution please comment below as it will just get more messy to program as I add more sounds in.
So now I have both the Wii Remote, the Balance Board and the Guitar Hero Guitar all working together! The really cool part is that all of these are playable by a single person at once as the Wii Remote sits inside the Guitar Hero Guitar. So to adjust the ‘Pitch’ event in OSCulator all you have to do is pretend to be a rock star with the ‘Guitar’ and angle it up and down. Combine that with standing on the Board you can control the sloshes and gurgles whilst playing the Pad chords with the coloured buttons in your left hand.
I want as much of this installation as possible to be ‘computer free’ and so I came up with the idea of mapping a single button on the Wii Remote to begin playing the backing track. That way the entire performance can be controlled without touching the computer. So I mapped the Home button on the Wii Remote to the Playback plugin Play/Stop button in the assignments/mapping menu in Mainstage. Below is a picture. See how on the left there’s a column with ‘OSCulator Out’? Under channel 1, I have made the Midi CC number 21 . Following on to the right you can see it’s mapped to the Play/Stop Switchable button.
In OSCulator, the event I chose was a Midi CC Toggle event and I also had to choose CC 21. However you have to be careful when trying to map it within Mainstage as it will automatically think that you’re trying to map motion control as they are always sending messages, remember? So you have to enter it in manually within Mainstage to only look for Midi CC 21 and not anything else. To do that click on the ‘Ctrl 21 (21) and edit it in the little menu that pops up.
I’m really excited to share this! The next things that I plan to do is add in some of the other effects that I have in Genesis such as the Thunder (door slams), Tongue Clicks, and Dishwasher Hits. These could prove to be difficult as I’ll have to choose notes that aren’t being used by any other instrument unless I find a way to send through another Midi input from OSCulator into Mainstage.
I also plan to make a poster or something similar to accompany the installation showing how it works so people aren’t so lost when they have a go.
I think that’s everything for now, I’ll let you know how the rest is going shortly!
Peace – Jonno