Customizing Generated Signs and Voices of AI Avatars: Deaf-Centric Mixed-Reality Design for Deaf-Hearing Communication
Project Overview
The document explores the transformative role of generative AI in education, specifically through the application of mixed-reality (MR) technologies aimed at improving communication for Deaf and hard-of-hearing (DHH) individuals. Utilizing a participatory design approach, the study actively involves DHH students to identify their communication needs and preferences for AI avatars capable of interpreting sign language and spoken language in real-time. Key findings from the research highlight the significance of customization and emotional expression in AI tools, as well as the necessity for these technologies to respect the social norms of both DHH and hearing communities. The overarching goal is to enhance accessibility in educational settings and provide a cost-effective alternative to human interpreters, thereby addressing the myriad communication challenges faced by DHH students. Through these advancements, the integration of generative AI in education not only fosters inclusivity but also empowers DHH individuals to engage more fully in their learning environments.
Key Applications
Mixed-Reality (MR) technology with AI avatars for interpreting sign language and spoken language.
Context: Facilitating communication between Deaf and hard-of-hearing (DHH) individuals and hearing peers in educational settings.
Implementation: Engaged 15 DHH students in participatory design sessions to brainstorm and sketch ideas for AI avatar functionalities.
Outcomes: Participants expressed enthusiasm for the technology, highlighting preferences for customization in avatar appearance and functionality, as well as emotional expression to enhance communication.
Challenges: Concerns regarding the accuracy and social acceptability of AI-generated voices and the need for technology to navigate cultural norms.
Implementation Barriers
Technology limitation
Current AI-generated voices may not meet the social and emotional communication needs of DHH individuals, potentially leading to misunderstandings. Additionally, there is a need for technology to respect both DHH and hearing social norms during communication.
Proposed Solutions: Integrating customizable options for AI-generated voices and ensuring they align with social norms and individual preferences. Involving DHH community members in the design process to ensure the technology is culturally aware and appropriate.
Accessibility issues
Limited availability of qualified human interpreters can lead to communication barriers for DHH individuals.
Proposed Solutions: Providing an affordable and accessible AI-based interpreting service to supplement human interpreters.
Project Team
Si Chen
Researcher
Haocong Cheng
Researcher
Suzy Su
Researcher
Stephanie Patterson
Researcher
Raja Kushalnagar
Researcher
Qi Wang
Researcher
Yun Huang
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Si Chen, Haocong Cheng, Suzy Su, Stephanie Patterson, Raja Kushalnagar, Qi Wang, Yun Huang
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai