Dynamic Live Flash animation via MIDI and Game Controllers

From 1734 when Louis-Bertrand Castel, a Jesuit monk, created the Clavecin Oculaire, a 'color harpsichord,' until today, artists have created instruments for performing color compositions. In the late 19th century, Alexander Rimington created the first 'color organ' for use in public performance. In the 1920s and 30s, Thomas Wilfred traveled the United States and Europe performing 'lumia' recitals with his 'clavilux.' In this tradition, I developed a computer system that allowed for the performance of user-controlled Flash animation.

The user interface is a Mad Catz Playstation 2 game controller and the UC-33e Evolution MIDI controller by m-Audio. The computer is an Apple iMac G5 with 1.8 GHz processor and 1.5 GB of RAM. The MIDI and game controller signal data is sent to Flash (.swf) files being played through the stand-alone Flash Player, utilizing ActionScript 2.0 code in Flash 8 via MAX/MSP 4.6 software. An LCD projector and sound system are the final destination outputs.

Several movie clips with different shapes and motions were created in the Flash file. The choice of the current movie clip is made by selecting a button on the game controller. The parameters of the Flash movie clips are altered by the MIDI data values which change according to the movement of faders on the MIDI device: altering x/y translation and scale, switching between greyscale and color modes, individual RGB color channel values, and level of transparency. The parameters are altered via relationships established in the Flash ActionScript code. As new movie clips are added to the screen and build up, older ones are deleted. Up to one hundred movie clips are on the screen at any given moment which creates a trail of motion.

I also created automated 'visual music' compositions in which the the attributes of a Flash animation dynamically reflect the changing values in a sound piece. These 'synaesthesia' pieces are influenced by the work of artists such as John Whitney, Oskar Fischinger, and Mary Ellen Bute who pioneered the representation of sound in animated visual form. Automated MIDI / Flash pieces can be created by incorporating Ableton Live software for exporting the MIDI note and velocity values for a MIDI instrument track directly to MAX/MSP where the values can be sent to Flash via the Flashserver.

15 minute performance with sound - excerpt from my MFA thesis exhibition from November 4, 2005, Orleans Hall, Savannah GA

To create a MIDI controlled Flash animation:

you need the following software:

Flash 8
MAX/MSP
Flashserver external for MAX/MSP by Olaf Matthes
*allows for communication between MAX/MSP and Flash
Ableton Live (for automated MIDI pieces)

hardware:

G5 Mac with 1.5 GB of RAM (suggested)
Playstation 2 Controller (SmartJoy adapter to connect via USB)
UC-33e Evolution MIDI Controller
LCD projector and screen (optional)

Step by Step Instructions:

Using MAX/MSP, create a patch which allows for the reading of input from both the Playstation controller and the MIDI controller.

Use the Flashserver exeternal to create a connection to a Flash file. In Flash, author ActionScript code to accept the incoming data from the MAX/MSP Flashserver and create variables to allow for changes in the parameters (dimensions, translation, and RGB color values) of movie clips.

Publish the Flash file and run the resulting .swf file in the standalone Flash player. Set the security settings to allow the .swf file to accept data from the Flashserver.

For an automated MIDI / Flash animation: Set up Ableton Live software to playback MIDI tracks and export the note and/or velocity values to MAX/MSP
in place of creating MIDI values from a MIDI controller.

Demo Files:

MAX/MSP 4.6 File
(install the Flashserver component to make this work)
Flash 8 File (you will need this .as file to make the Flash file work)
Place all files in the same folder - and run the published .swf file from the same folder.


Links:

Flash 8
http://www.adobe.com
MAX/MSP software
http://www.cycling74.com
Flashserver component for MAX/MSP
http://www.nullmedium.de/dev/flashserver
Ableton Live
http://www.ableton.com
M-Audio products
http://www.m-audio.com
Center for Visual Music
http://centerforvisualmusic.org
The IOTA Center
http://iotacenter.org
Thomas Wilfred / Lumia
http://www.lumia-wilfred.org
The Savannah College of Art and Design
http://www.scad.edu

 

Fred Leighton, MFA 2006, studied the history of Visual Music and developed these techniques for live control of Flash animation as part of his graduate studies in Interactive Design and Game Development at The Savannah College of Art and Design, Savannah, GA

Contact: fred@artsdigital.com