What Are Interactive Music Videos?

A special topics course at RIT called "Applications in Virtual Reality: Interactive Music Videos" was an interesting blend of art and technology. 

In case you've never seen experienced once before, an interactive music video basically is any digital experience that involves a song and input from a user. These terms are pretty ambiguous which means that interactive music videos can be vastly different and unique. Do a quick Google search to find some, or look at these popular examples:

Arcade Fire (Warning: a little freaky. Move different body parts in the video): http://www.beonlineb.com/

Ellie Goulding (Control the path of a flying object in a virtual world): http://lights.elliegoulding.com/ 

Cold War kids (Select band members to switch instruments and effect the song): 

http://www.coldwarkids.com/iveseenenough/

What Did I Do?

For the project, we worked in small teams of artists and computer scientists. Together we had to contact a local band or musician to work with, and then negotiate possible interactive music video ideas for them. The project was as much of a test in business as it was technology.

My group worked with Giant Panda Guerilla Dub Squad, a reggae band from Rochester. After speaking with the band we decided we would create an interactive experience with an audio and video mixer that users could manipulate to change the sound and visuals while the band performed a song.  

From a technical standpoint my role in the project was the audio manager. I am interested in audio engineering so it was exciting to pair audio with programming.

As a proof of concept for the audio mixer I put an interactive HTML5 project together, using Javascript and some open source libraries that I found on GitHub. Eventually my mixer was combined with the rest of the project that my teammates worked on, which includes video, 3D, and photo elements.

Here's the final product:  

http://giantpanda.cias.rit.edu/

(You may need to refresh the page several times in order to get it to load all of the media correctly. If the loading % gets stuck, refresh your page. Google Chrome only) 

Problems and Improvements

You might have noticed that the final product was a little buggy. 

The video files in the project alone add up to a few Gigabytes of data that needs to be streamed to the user's browser, in addition to the audio, which means that the experience can't begin until all the media is preloaded. And the preloading of media sometimes stalls and and ceases entirely before it is finished. Not an optimal experience.

If we had a better understanding of web programming, in regards to media streaming and downloading, we may have been able to avoid this. We also set our project scope too large given that the technical students on our team didn't have a strong web background. If we did it again, I would press the group to replace the video segments with still images to avoid the media throttling issue. Also more research into media loading via web would have been invaluable.

Final Thoughts

Overall the project was rewarding and I gained a ton of experience in a variety of roles that I enjoy:

• Business: negotiating requirements with the band members
• Art: Created art assets and worked closely with videographers and photographers. I even was able to check out a high-end camera from the RIT Photo cage and record the band live at a concert.
• Music/Audio: Used Audacity to process files, and new HTML5 Audio functions to dynamically alter and add effects to them through the web. 
• Web Programming: Became familiar with Javascript and got refreshers with HTML and CSS.
• Team Work: Worked on a team of programmers and artists. Had to mitigate conflict between team members. 

If I could do it again, I would encourage the team to scale down our project scope, but I still would advocate for the combination of 3D visuals and audio. Projects like this that combine art and technology are rare and rewarding.