Augmented Communication for XR

Our research on "Augmented Communication" aims to enhance remote communication in virtual and augmented reality through the integration of cutting-edge technologies such as machine learning, eye tracking, visual augmentation, and gesture recognition. Through this research, we have developed innovative solutions such as Visual Captions, which proactively suggests relevant visuals to aid in open-vocabulary conversations; ThingShare, a video-conferencing system that facilitates the sharing of physical objects; GazeChat, a remote communication system that utilizes gaze-awareness to represent users in 3D profile photos; and CollaboVR, a framework that enables multi-user communication in virtual reality through the design of interactive and reconfigurable layouts. Our goal is to further the state-of-the-art in real-time systems for augmented communication in VR and AR, ultimately making remote communication more universally accessible and effective.

Publications

teaser image of Visual Captions: Augmenting Verbal Communication With On-the-fly Visuals

Visual Captions: Augmenting Verbal Communication With On-the-fly VisualsOpen Source, Real-time, Live!

Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI), 2023.
Keywords: augmented communication, large language models, video-mediated communication, online meeting, collaborative work, augmented reality, XR interaction


teaser image of Experiencing Visual Captions: Augmented Communication With Real-time Visuals Using Large Language Models

Experiencing Visual Captions: Augmented Communication With Real-time Visuals Using Large Language Models

Adjunct Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST), 2023.
Keywords: augmented communication, large language models, video-mediated communication, online meeting, collaborative work, dataset, textto-visual, AI agent, augmented reality


teaser image of ThingShare: Ad-Hoc Digital Copies of Physical Objects for Sharing Things in Video Meetings

ThingShare: Ad-Hoc Digital Copies of Physical Objects for Sharing Things in Video Meetings

Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI), 2023.
Keywords: video-mediated communication, object-centered meetings, online meeting, collaborative work, augmented communication, XR interaction


teaser image of Modeling and Improving Text Stability in Live Captions

Modeling and Improving Text Stability in Live Captions

Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems (CHI EA), 2023.
Keywords: live captions; real-time transcription; visual instability; flickering metric; speech-to-text; text stability; tokenized alignment; augmented communication

teaser image of GazeChat: Enhancing Virtual Conferences With Gaze-aware 3D Photos

GazeChat: Enhancing Virtual Conferences With Gaze-aware 3D Photos

Proceedings of the 34th Annual ACM Symposium on User Interface Software and Technology (UIST), 2021.
Keywords: eye contact, gaze awareness, video conferencing, video-mediated communication, gaze interaction, augmented communication, augmented conversation, eye tracking, XR interaction
teaser image of CollaboVR: A Reconfigurable Framework for Multi-user to Communicate in Virtual Reality

CollaboVR: A Reconfigurable Framework for Multi-user to Communicate in Virtual Reality

Zhenyi He, Ruofei Du, and Ken Perlin
2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2020.
Keywords: chalktalk, virtual reality, collaborative work, layout, telepresence, communication, XR interaction, augmented communication

Videos

ThingShare: Ad-Hoc Digital Copies of Physical Objects for Sharing Things in Video Meetings


Visual Captions: Augmenting Verbal Communication With On-the-fly Visuals


CollaboVR: A Reconfigurable Framework for Multi-user to Communicate in Virtual Reality


Talks

Visual Captions: Augmenting Verbal Communication with On-the-fly Visuals Teaser Image.

Visual Captions: Augmenting Verbal Communication with On-the-fly Visuals

Ruofei Du

CHI 2023, Hamburg, Germany.


Interactive Perception & Graphics for a Universally Accessible Metaverse Teaser Image.

Interactive Perception & Graphics for a Universally Accessible Metaverse

Ruofei Du

Invited Talk at UCLA by Prof. Yang Zhang , Remote Talk.


Interactive Graphics for a Universally Accessible Metaverse Teaser Image.

Interactive Graphics for a Universally Accessible Metaverse

Ruofei Du

Invited Talk at ECL Seminar Series by Dr. Alaeddin Nassani , Remote Talk.


Interactive Graphics for a Universally Accessible Metaverse Teaser Image.

Interactive Graphics for a Universally Accessible Metaverse

Ruofei Du

Invited Talk at Empathic Computing Lab , Remote Talk.


Interactive Graphics for a Universally Accessible Metaverse Teaser Image.

Interactive Graphics for a Universally Accessible Metaverse

Ruofei Du

Invited Talk at UMD by Prof. Amitabh Varshney , College Park, Maryland.


Computational Interaction for a Universally Accessible Metaverse Teaser Image.

Computational Interaction for a Universally Accessible Metaverse

Ruofei Du

Invited Talk at GAMES Seminar , Remote Talk.


Fusing Physical and Virtual Worlds into An Interactive Metaverse Teaser Image.

Fusing Physical and Virtual Worlds into An Interactive Metaverse

Ruofei Du

Invited Talk at UCLA by Prof. Yang Zhang , Remote Talk.


Polymerizing Physical and Virtual Worlds into  An Interactive Metaverse Teaser Image.

Polymerizing Physical and Virtual Worlds into An Interactive Metaverse

Ruofei Du

Invited Talk by Prof. Arthur Theil at Birmingham City University , Remote Talk.


Blending Physical and Virtual Worlds into  An Interactive Metaverse Teaser Image.

Blending Physical and Virtual Worlds into An Interactive Metaverse

Ruofei Du

Invited Talk at Wayne State University , Remote Talk.


Fusing Physical and Virtual Worlds into 
Interactive Mixed Reality Teaser Image.

Fusing Physical and Virtual Worlds into Interactive Mixed Reality

Ruofei Du

Invited Talk at George Mason University , Remote Talk.


Cited By

  • I Cannot See Students Focusing on My Presentation; Are They Following Me? Continuous Monitoring of Student Engagement Through "Stungage". arXiv.2204.08193.Snigdha Das, Sandip Chakraborty, and Bivas Mitra. source | cite | search
  • ARENA: The Augmented Reality Edge Networking Architecture. 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).Nuno Pereira, Anthony Rowe, Michael W Farb, Ivan Liang, Edward Lu, and Eric Riebling. source | cite | search
  • What Was Hybrid? a Systematic Review of Hybrid Collaboration and Meetings Research. https://arxiv.org/pdf/2111.06172.pdf.Thomas Neumayr, Banu Saatci, Sean Rintel, Clemens Klokmose, and Mirjam Augstein. source | cite | search
  • A Dynamically Weighted Multi-Objective Optimization Approach to Positional Interactions in Remote-Local Augmented/Mixed Reality. 2021 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR).Akshith Ullal, Cadence Watkins, and Nilanjan Sarkar. source | cite | search
  • Usability Testing of Virtual Reality Applications\textemdashThe Pilot Study. Sensors.Dorota Kamińska, Grzegorz Zwoliński, and Anna Laska-Le{\'{s}}niewicz. source | cite | search
  • ARcall: Real-Time AR Communication Using Smartphones and Smartglasses. https://arxiv.org/abs/2203.04358.Hemant Bhaskar Surale, Yu Jiang, ThamBrian A., and SmithRajan Vaish. source | cite | search
  • Ubiq: A System to Build Flexible Social Virtual Reality Experiences. Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology.Sebastian Friston, Ben J Congdon, David Swapp, Lisa Izzouzi, Klara Brandstätter, Daniel Archer, Otto Olkkonen, Felix Johannes Thiel, and Anthony Steed. source | cite | search
  • How Will VR Enter University Classrooms? Multi-stakeholders Investigation of VR in Higher Education. CHI Conference on Human Factors in Computing Systems.Qiao Jin, Yu Liu, Svetlana Yarosh, Bo Han, and Feng Qian. source | cite | search
  • Mixed Reality Collaboration for Complementary Working Styles. Special Interest Group on Computer Graphics and Interactive Techniques Conference Immersive Pavilion.Keru Wang, Zhu Wang, Karl Rosenberg, Zhenyi He, Dong Woo Yoo, Un Joo Christopher, and Ken Perlin. source | cite | search
  • Local Free-View Neural 3D Head Synthesis for Virtual Group Meetings. 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW).Sebastian Rings and Frank Steinicke. source | cite | search
  • Augmented Chironomia for Presenting Data to Remote Audiences. arXiv.2208.04451.Brian D. Hall, Lyn Bartram, and Matthew Brehmer. source | cite | search
  • A State of the Art and Scoping Review of Embodied Information Behavior in Shared, Co-present Extended Reality Experiences. Electronic Imaging.Kathryn Hays, Arturo Barrera, Lydia Ogbadu-Oladapo, Olumuyiwa Oyedare, Julia Payne, Mohotarema Rashid, Jennifer Stanley, Lisa Stocker, Christopher Lueg, Michael Twidale, and Ruth West. source | cite | search
  • Smart Factory Using Virtual Reality and Online Multi-User: Towards a Metaverse for Experimental Frameworks. Applied Sciences.Luis Omar Alpala, Dar{\'{\i}}o J. Quiroga-Parra, Juan Carlos Torres, and Diego H. Peluffo-Ord{\'{o}}{\~{n}}ez. source | cite | search
  • Visual Transitions Around Tabletops in Mixed Reality: Study on a Visual Acquisition Task Between Vertical Virtual Displays and Horizontal Tabletops. Proceedings of the ACM on Human-Computer Interaction.Gary Perelman, Emmanuel Dubois, Alice Probst, and Marcos Serrano. source | cite | search
  • Investigating Document Layout and Placement Strategies for Collaborative Sensemaking in Augmented Reality. CHI EA '21: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems.Weizhou Luo, Anke Lehmann, Yushan Yang, and Raimund Dachselt. source | cite | search
  • Colocation for SLAM-Tracked VR Headsets With Hand Tracking. Computers.Dennis Reimer, Iana Podkosova, Daniel Scherzer, and Hannes Kaufmann. source | cite | search
  • A Survey on Synchronous Augmented, Virtual and Mixed Reality Remote Collaboration Systems. ACM Computing Surveys.Alexander Schäfer, Gerd Reis, and Didier Stricker. source | cite | search
  • OpenMic: Utilizing Proxemic Metaphors for Conversational Floor Transitions in Multiparty Video Meetings. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems.Erzhen Hu, Jens Emil Sloth Gr{\o}nb{\ae}k, Austin Houck, and Seongkook Heo. source | cite | search
  • ChameleonControl: Teleoperating Real Human Surrogates Through Mixed Reality Gestural Guidance for Remote Hands-on Classrooms. arXiv.2302.11053.Mehrad Faridan, Bheesha Kumari, and Ryo Suzuki. source | cite | search
  • ViGather: Inclusive Virtual Conferencing With a Joint Experience Across Traditional Screen Devices and Mixed Reality Headsets. Proceedings of the ACM on Human-Computer Interaction.Huajian Qiu, Paul Streli, Tiffany Luong, Christoph Gebhardt, and Christian Holz. source | cite | search
  • Investigating Psychological Ownership in a Shared AR Space: Effects of Human and Object Reality and Object Controllability. arXiv.2308.13953.Dongyun Han, Donghoon Kim, Kangsoo Kim, and Isaac Cho. source | cite | search
  • A Reference Framework for Evaluating Virtual Conferences/submitted by Alexander Gindlhumer. JOHANNES KEPLER UNIVERSITY LINZ.Alexander Gindlhumer. source | cite | search
  • iBall: Augmenting Basketball Videos With Gaze-moderated Embedded Visualizations. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems.Zhutian Chen, Qisen Yang, Jiarui Shan, Tica Lin, Johanna Beyer, Haijun Xia, and Hanspeter Pfister. source | cite | search
  • A Reference Framework for Evaluating Virtual Conferences. Universitätsbibliothek Linz.Lisa-Marie Huber and Alexander Gindlhumer. source | cite | search
  • An Extended AI-Experience: Industry 5.0 in Creative Product Innovation. Sensors.Amy Grech, Jörn Mehnen, and Andrew Wodehouse. source | cite | search
  • DisETrac: Distributed Eye-Tracking for Online Collaboration. Proceedings of the 2023 Conference on Human Information Interaction and Retrieval.Bhanuka Mahanama, Mohan Sunkara, Vikas Ashok, and Sampath Jayarathna. source | cite | search
  • SceneFusion: Room-Scale Environmental Fusion for Efficient Traveling Between Separate Virtual Environments. IEEE Transactions on Visualization and Computer Graphics.Miao Wang, Yi-Jun Li, Jinchuan Shi, and Frank Steinicke. source | cite | search
  • How Users Cognitively Appraise And~emotionally Experience The~metaverse: Focusing on Social Virtual Reality. Information Technology & People.Ayoung Suh. source | cite | search
  • Meet Me in VR! Can VR Space Help Remote Teams Connect: A Seven-week Study With Horizon Workrooms. International Journal of Human-Computer Studies.Katarzyna Abramczuk, Zbigniew Bohdanowicz, Bartosz Muczy{\'{n}}ski, Kinga H. Skorupska, and Daniel Cnotkowski. source | cite | search
  • An End-to-End Review of Gaze Estimation and Its Interactive Applications on Handheld Mobile Devices. ACM Computing Surveys.Yaxiong Lei, Shijing He, Mohamed Khamis, and Juan Ye. source | cite | search
  • CallMap: A Multi-dialogue Participative Chatting Environment Based on~Participation Structure. Lecture Notes in Computer Science.Masanari Ichikawa and Yugo Takeuchi. source | cite | search
  • Pseudo-mutual Gazing Enhances Interbrain Synchrony During Remote Joint Attention Tasking. Brain and Behavior.Chun-Hsiang Chuang and Hao-Che Hsu. source | cite | search
  • Key Technologies for Networked Virtual Environments. Multimedia Tools and Applications.Juan Gonz{\'{a}}lez Salinas, Fernando Boronat Segu{\'{\i}}, Almanzor Sapena Piera, and Francisco Javier Pastor Castillo. source | cite | search
  • Facilitated Model-based Reasoning in Immersive Virtual Reality: Meaning-making and Embodied Interactions With Dynamic Processes. International Journal of Computer-Supported Collaborative Learning.Michelle Lui, Kit-Ying Angela Chong, Martha Mullally, and Rhonda McEwen. source | cite | search
  • CrossTalk: Intelligent Substrates for Language-Oriented Interaction in Video-Based Communication and Collaboration. Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology.Haijun Xia, Tony Wang, Aditya Gunturu, Peiling Jiang, William Duan, and Xiaoshuo Yao. source | cite | search
  • Feeling Present! From Physical to Virtual Cinematography Lighting Education With Metashadow. Proceedings of the 31st ACM International Conference on Multimedia.Zheng Wei, Xian Xu, Lik-Hang Lee, Wai Tong, Huamin Qu, and Pan Hui. source | cite | search
  • Effects of Avatar Transparency on Social Presence in Task-centric Mixed Reality Remote Collaboration. IEEE Transactions on Visualization and Computer Graphics.Boram Yoon, Jae-eun Shin, Hyung-il Kim, Seo Young Oh, Dooyoung Kim, and Woontack Woo. source | cite | search
  • Stay In Touch