MULTIPARTIES: Multi-Party Augmented Reality Telepresence System
Brandenburg Labs, plazz, Consensive and TU Ilmenau bring people closer together using Augmented Reality
Brandenburg Labs GmbH, plazz AG, Consensive GmbH, and Technische Universität Ilmenau are launching the joint research project “MULTIPARTIES – Multi-Party Augmented Reality Telepresence System.” The focus is on the development of a 3D communication system that enables realistic online meetings between several people over distances. The two-and-a-half-year joint project is being funded by the German government as part of “KMU-innovativ: Interaktive Technologien für Gesundheit und Lebensqualität”.
Online meetings still differ a lot from real meetings. A circumstance many people experienced during the COVID-19 pandemic. Important interpersonal aspects of communication, such as body language or the ability to whisper to the other person, are lost. Online meetings also lead to a higher cognitive load than real meetings. MULTIPARTIES aims to overcome these limitations and enable spatially separated people to have more natural conversations online.
Using augmented reality (AR), i.e., the computer-aided enhancement of reality perception and spatial audio technologies, the people participating in a MULTIPARTIES meeting will be able to perceive, communicate and interact with each other as naturally as possible. For this purpose, realistic 3D avatars of people with expressive gestures and facial expressions, as well as spatially plausible audio, are seamlessly integrated into the real environment. This creates the impression of actually meeting the other conversation partners. Prof. Karlheinz Brandenburg, CEO of Brandenburg Labs GmbH, sees the future of online conferencing in so-called “AR telepresence systems”: “The goal of such systems is by no means to replace real interpersonal contact. Rather, we want to facilitate digital learning as well as work processes, promote togetherness, and create more diverse opportunities for social participation.”
The interdisciplinary consortium of three Thuringian SMEs and the Technische Universität Ilmenau brings with it many years of experience in their respective fields. The Brandenburg Labs GmbH, the project coordinator, is dedicated to spatial audio reproduction via headphones, which are integrated into the system. These enable a natural reproduction of sound, allowing participants in the virtual meeting to speak simultaneously, for example, or to “whisper in the ear” of other participants. Jürgen Mayer, CEO of the plazz AG in Erfurt, Germany, expects valuable insights for the event sector: “With our event and community platform Polario, we connect a large number of people in a natural way, in different environments and scenarios. With MULTIPARTIES, we enable users to create deeper connections and improve digital work environments. In-house and external events that are organized in a sustainable virtual or hybrid way, especially due to the COVID-19 pandemic, but also climate-friendly online marketing and sales meetings will benefit from MULTI-PARTIES.” The Weimar-based start-up Consensive GmbH brings its expertise in 3D representation and interaction of participants in mixed reality into the project. An important starting point in the project is, for the start-up, the use of “digital twins”. This involves digitally recreating the participants’ real-life environments and recombining them to create a shared space.
Three departments of the TU Ilmenau are also involved in the project, coordinated by Prof. Wolfgang Broll. The Department of Virtual Worlds and Digital Games, which he heads, is responsible for researching AR technologies and interaction techniques. “A major challenge lies in particular in the capture and reproduction of individual facial expressions and gestures in order to enable realistic and expressive avatars. In addition, we will explore new interaction techniques for controlling the application as well as for collaboration between participants,” Broll said.
The Audiovisual Technology department, headed by Prof. Alexander Raake, is addressing the questions of which factors contribute to the most natural communication possible and how this can be measured. The focus is on hearing, seeing, speaking, and communication through mimicry and gestures. Dr. Stephan Werner heads the Department of Electronic Media Technology, which is involved in the project with fundamental investigations into the recognition and assessment of relevant and irrelevant acoustic information, as well as the development of audio reproduction.
