thesis
immersive music visual instrument installation
How can I make an immersive MusicVisual experience using a touch free interface without a device or VR headset?
Description
The experience involves multiple projectors to display visuals on the walls, ceiling and floor of the space. Depth Data from a Kinect type sensor is mapped with music and visual elements. It will be used as a music visual instrument/installation for performance. The audience is for people who enjoy music and exploring how their movement affects the music and visuals. It will enable musicians to perform visuals with their music. It enables visual artist to have a musical accompaniment to their work. It assists dancers who wish to have music and visuals compliment their movements and bodies. The key features include a venue space, where sound and projections submerge the room. A performer steps into the space and performs the music and visuals by their movement through the space.
Research Approach
I plan to research venue spaces for thesis and beyond. I have seen ideas that inspire me in examples of the holo-deck on Star Trek, the holographic interface in the films Minority Report and Matrix Reloaded, and the V Motion Project. I will need to find a place where I can exhibit the functionality of what I produce. I need to research what is required for multiple projections and integrating the system with a touch free sensor. I need to talk to people who have done room scale installations and research the methods to accomplish these tasks.
Personal Statement
I am a composer, songwriter, visual artist, director, performer and chef. I care about dreams, inspiration, the wonderful and fantasy. My experience as an auteur filmmaker and music composer have shaped my art to focus on the compelling nature of music score and mesmerizing effect of visual synchronicity. I am intrigued by dreams and this area of our lives that is seemingly inaccessible to our waking experience. I am inspired to articulate the idea of navigating dreams, to sail celestial waters. To express this idea through the medium of a touch free music visual instrument installation performance.
Bill of materials
- SPACE
Booked for some time around Week 9, 10, 11
- Computer
- MaxMSP/Jitter System, Processing, OSC, Visuals & Music
- Kinect 1 or Kinect 2
- Projectors (3 + )
- Matrox TripleHead2Go for Multiple Displays
- USB AtoB
- Thunderbolt to Displayport Dongle
- DPI male to male Patch Cord
- Displayport Dongle (3x)
- DPI to HDMI (3x)
- Mounts, Stands, Tripods for Projectors
- Screens?
Schedule
Thesis Response
I want to make an immersive music visual instrument installation using a touch free interface without a device or VR headset. I have tried a VR headset and felt disoriented and extremely uncomfortable. I most definitely do not want to work in that medium. I do not like to be unaware of my surroundings and feel that would be the wrong platform direction to take my artwork into. I am inspired by how technology can assist and enable us to do wonderful things with our artwork, not inspired by how technology is assimilating our ideas.
It is important that the participant not touch things like the conductor does not play a specific instrument in an orchestra, rather directs the performance. My experience with piano performance is that it is not very physically moving, the piano is stationary and the performer is seated at the keyboard. There is little showmanship or stage presence. I have felt the need to perform my music in another way. I am repulsed by the idea of being a dj to perform my music because it does not do the justice to my passion practice of composing and producing my musical expression. Most of the work I put into my music occurs at a piano, keyboard and computer. I compose for a range of sounds, melodies, percussions, rhythm and harmonies. It is a lot like the novelty of a one-man band and a reality with how technology has made advancements with music production and composition. These tools and instruments are not the way I see my artwork articulated for performance. As the author of the music, I feel I must conduct the performance, to direct the sounds via a touch free interface.
The limitations with the technologies are mapping the depth data from Kinect to my musical samples. It is a prototype system that is inhibited by the workflow. For example, I must compose my music samples in a Digital Audio Workstation, Logic Pro X, and import them to the music visual system, Max MSP/Jitter. The depth data comes from the Kinect and sent to Max where the music samples are triggered. It would be nice, (yet too large in scope) to design and develop a platform for easily integrating music samples with my music visual system or configure a way to compose with the system. I write my music using a piano keyboard and use it to produce a song. I would use the Immersive Music Visual Instrument Installation to perform the song.
I am considering having the visuals being an ensemble player to the music composition and performance. I have more of a sense of urgency to develop a way to perform my music with stage presence, movement and showmanship. I feel that enjoying music in headphones is a fraction of what the real music experience can be. The idea of having an immersive experience in an installation that is the instrument packages the idea of modern music composition and performance with visuals that compliment the sound and without the pressure of venues, booking, stages, minimums, and costs that come with live acts.
My goal is to build a platform that works so I can perform a single piece. It is unclear whether work has been done in this area. Certainly there are works that are similar or resemble the idea in one way or another. I am building upon skills that I have developed at ITP and am inspired to continue working and iterating on these ideas. The nexus would be the composer being the conductor, the performer.
USER SCENARIO
LIST OF ENTITIES THAT HAVE DONE SIMILAR WORKS
United Visual Artists Collective
https://uva.co.uk/
Light Surgeons
http://www.lightsurgeons.com/
AV&C Systems
http://www.av-controls.com/
Rockwell Group
https://www.rockwellgroup.com/projects
V Motion Project
Visuals test
system prototype test
Scheduled space for performance
In my quest for a space to show my project I was fortunate to schedule an evening to perform at NYU Steinhardt Education Building, Recital Room 303, 35 W. 4th Street for Wednesday, March 14th @ 7pm. I was referred by Kimmel Center Operations to Luis Mercado who handles the scheduling of those rooms and he informed me that this space is completely booked after spring break. He was able to schedule this time for me and I gratefully obliged. My prototype is ready and I will be able to perform something and it will give me an opportunity to get some quality documentation of my project in preparation for presentations. I’ve asked Jenny Lim to help me with camera work. Still looking for an assistant to help carry equipment and help with production logistics.
performance promotion
I sent out a promotional card to invite the community. An email was sent to the student list of ITP and I created a Facebook event invitation. I don’t expect a big turnout as it is during spring break and many are out of town. I still find this opportunity to be rather awesome and to meet it with a radiant performance.
Rehearsal
Tuesday, March 13, 2018 6pm Conference Room
PERFORMANCE PREPARATION
Because of scheduling availability at NYU Steinhardt’s piano recital room, the performance took place on Wednesday, March 14th, 2018, during spring break. On March 2nd, I spoke on the phone with Luis Mercado, who does scheduling at Steinhardt, for booking the room. He informed me that there was no availability after Spring Break, and the room is for Professors and students in their department. So I agreed to book the room for that Wednesday of Spring Break. We then exchanged email to confirm and go over the procedures for using the room.
Hi Brandon,
You are confirmed in room 303 on 3/14, 7 – 10pm. The room is available as-is — meaning piano, 74 chairs, and whiteboard will be in the space. I will create a document for Public Safety and deliver the day of so your guests have access to the space. If you can restore the room to the way you found it we should be good to go! Let me know if there is any food involved in this production as we’ll need to discuss that further.
All best,
Luis
-The space was confirmed on March 5th after he spoke with his supervisor.
I immediately went into triple overtime on developing the system, mapping sounds, calibrating pixel thresholds and preparing equipment for the rehearsals and performance.
Processing Sketch:
MAX Patch:
I composed the piece in the key of A Major (Omit E) with Logic Pro X:
Implemented instruments such as: Orchestra Strings, Synth Pads, Ambient Soundscapes, Oud, Guitar, Distorted Synth Lead, Bass and Rhythmic Beat.
MUSIC SAMPLES COMPOSED:
VISUALS
The Visuals are a generative composite with a blend of video, particle systems, p5.js sketch, Adobe After Effects / Photoshop, Final Cut Pro X and JITTER.
Preparations included setup of three tripods for three projectors, and another tripod for the documentation camera, Canon 5D Mk III.
The process of checking out the equipment was extensive. Some tripods were not functioning properly, they would slouch and not stay fastened. There were two projectors that needed recalibration or keystone adjustment. Another two projectors needed mounting for the Manfrotto Arm Head Mount because that is the only one compatible with the wood mount piece for the bottom of the projectors and corresponding tripods without the proper mount. Thanks to Marlon and Nitish in the ER for assistance with mounting and adjustments!
I was able to check out the Roland Speaker Monitors to get louder audio for the performance. Each speaker required its own power cable and I used 1/4″ to RCA Connector Adapters for a Y 1/8″ connection to the audio output of my laptop.
For transporting the equipment I was able to fit the speakers, projectors, cables, power cords, extensions cords, and arms in the large Pelican Case. Michael Kripchak helped with pushing the case to 35 West 4th Street Room 303.
Marlon then gave me an official NYU Memo document to carry around with me while I am moving the equipment from the ITP floor to the performance space. To: TSOA Security Re: Removal of Gear
Day of REHEARSAL & Equipment Check:
Arrived at 35 West 4th Street Room 303 @ 6:20pm…
SETTING UP:
The show was a success and it was an inspiring evening. I was able to perform for the duration of 7-8pm. Because Spring Break Hours have the Equipment Room close early at 8, I had to make special arrangements with floor staff, Chetan Hasabnis to return the equipment when we finished.
I would have loved to stayed there for at least 3 hours to flush out all the documentation coverage and perform more music and visuals. Considering all the circumstances and being respectful to everyone’s time and the help I received, I was thankful to all and for everything running smoothly.
I was able to have others perform the instrument as well - Dominic Barrett, Michael Kripchak & Christina Elizabeth Hall. I will be getting their feedback and interviewing them about the performance & system.
Meticulous preparation for the show and planning contributed to a successful evening. I will be interviewing the attendees for their feedback and response to the performance.