During the MANIFEST:IO festival in Berlin, I will be presenting the project as an artist and discussing my process. We talk about the potential of the interactive music video form and Dolby Atmos, but also about the hurdles that exist with this technology today.
To experience more immersive and realistic sound we were limited to 5.1 or 7.1 surround-sound-systems which are difficult to consume due to the necessary hardware.
3D sound, however, is based on a new file format and does not work with fixed audio tracks that are premixed for the speaker channels, but combines sounds with metadata. This makes it possible to place sounds in three-dimensional space and then render them on any speaker layout in real time, so that the sound seems to come from the correct direction in any scenario. So we don’t store a fixed mix in a file, but all sounds individually with the corresponding intention about their behavior.
I saw the potential and wanted to explore it. How can a music video work with 3D sound? Doesn’t it just make sense, thinking of seminal formats like non-linear storytelling or interactive videos, to use object-based sound?
The project experiments with the interplay of visual effects and three-dimensional sound. With the help of object-based sound production, an immersive sound experience was created. In the Dolby Atmos sound mix, instruments, voices and sound effects were placed spatially and react to visual stimuli. The user is thus drawn deeper into the events of the song, creating a tightly knit audiovisual experience with an emotional story.
This new form of music video allowed us to increase the immersion by:
- Room of the sounds
- Interplay between visual impulses and auditory signals
- Interactive component that involves the viewer
Limitations due to current technical hurdles are present. If there were an interactive player that could also decode object-based sound, features such as head-tracking would be possible (objects would be „anchored“), or users could change the properties of individual sound objects through interaction.
But I would definitely say that the open nature and flexibility of this format, which both secures the musical intent of a song and increases immersion in the audiovisual content, will vastly improve the user experience.
I also think that the process has shown that artists from different disciplines should work more closely together from the beginning.
A music video that is not only based on the finished song, but can be created together with it, brings a completely different experience.
