During the COVID-19 pandemic, the HLRS visualization team was hard at work finding ways to use virtual reality (VR) and augmented reality (AR) technologies to bring people closer together in digital space.
As partners in the KoLab-BW project, HLRS staffers have been working with six partner organizations to develop software that, in conjunction with commercially available VR headsets, allows researchers to meet in virtual space despite being in different parts of a city, country, or the world at large.
Jul 12, 2022
HLRS Projects
Media & Art
Virtual Reality
CAVE
Visualization
See all news
Recently, the team has applied this concept to a fast-paced, dynamic arena for collaboration—making music together. In recent months, HLRS Visualization Department Head Dr. Uwe Wössner has partnered with Dr. Jeremy J. Ham, a musico-spatial designer, architect, and independent researcher based in Australia, to design a “Networked Musicospatial Virtual Environment,” or NMSVE.
“I’m interested in what I call the ‘musico-spatial crossover,’ which basically is bringing together music and spatial design,” Ham said. “I’m also focused on developing 3D spatial notation of music, which is an emergent field of study. Most of all, though, I am interested as a musician in improvisation as a methodology. In that respect, what the HLRS visualization team designed and what we worked on was a way of capturing performance and highlighting how different music can be spatialized differently.”
In partnering together, Ham and Wössner created an approach using the HLRS-developed COVISE software environment to notate and share visualizations of improvised music collaborations in real time across the world.
Dr. Jeremy Ham performing as part of Ars Electronica Garden Aotearoa.
Ham and Wössner recently showcased their NMSVE concept as part of the New Zealand-based offshoot of ARS Electronica, the internationally renowned digital art festival, which was held in Wellington, New Zealand as ARS Electronica Garden Aotearoa. The researchers named the project “Synaespatia,” a play on “synaesthesia,” a rare phenomenon in which people experience one sense through another—think “tasting” music, “hearing” color, or the like. “Synaespatia, for me, is the sensing of music and sound in the spatial domain,” Ham said. For the exhibition, Ham, in collaboration with Stuttgart-based musician Timo Frey and Wössner, showcased the team’s NMSVE with live, collaborative performances taking place simultaneously in Wellington and Stuttgart. Using COVISE, the musicians are not only able to hear one another from 18,000 kilometers away, but also look around while wearing a VR headset to see dynamic and distinct music notation floating around them to represent what is being played on keyboards and electronic drum kits at any given moment.
“In KoLab-BW, we support collaboration in virtual environments between different VR hardware such as CAVEs, Powerwalls, head-mounted displays, and augmented reality,” Wössner said. “During the ARS Electronica event I was using the CAVE and Jeremy was using a head-mounted display and also a big LED wall. This event showed us that these technologies could be successfully integrated into a live music setting, even when the musicians are far apart.”
Ham hopes that innovations such as these could offer engaging, immersive environments for enhancing music education. Furthermore, he sees opportunities for using NMSVEs to bring music to those who may be hearing impaired or otherwise have limited accessibility to music.
-Eric Gedenk
