Generating Signed Language Instructions in Large-Scale Dialogue Systems
Project Overview
The document explores the implementation of a multimodal conversational AI system designed to generate American Sign Language (ASL) instructions, aimed at improving accessibility for Deaf and Hard-of-Hearing (DHH) individuals in educational contexts. By leveraging Large Language Models (LLMs) for gloss translations and a video retrieval system for delivering signed instructions, the authors highlight the importance of community engagement in the design process to ensure that the system accurately meets the needs of its users. The evaluation results showcase the system's effectiveness, achieving high levels of user satisfaction while successfully minimizing cognitive load for DHH users. This initiative underscores the potential of generative AI to enhance educational accessibility and inclusivity, demonstrating a meaningful application of technology to support diverse learning needs.
Key Applications
Multimodal dialogue system generating ASL instructions
Context: Accessible to Deaf and Hard-of-Hearing users via Amazon Alexa devices
Implementation: Developed within the Alexa Prize TaskBot Challenge; users input tasks via touchscreen, and the system generates ASL video instructions using LLMs.
Outcomes: High retrieval accuracy and user satisfaction ratings on par with non-signing variants; reduced cognitive load for users.
Challenges: Disjointed nature of signing videos; potential confusion due to segmented ASL presentations.
Implementation Barriers
Technical barrier
Limited access to camera footage for sign recognition due to privacy regulations.
Proposed Solutions: Focus on instruction generation instead of sign recognition; utilize touchscreen inputs for user interaction.
Cognitive barrier
Higher cognitive load for DHH users when interpreting text-based instructions compared to signed instructions.
Proposed Solutions: Design user interface to reduce cognitive load by integrating visual and signed instructions.
Project Team
Mert İnan
Researcher
Katherine Atwell
Researcher
Anthony Sicilia
Researcher
Lorna Quandt
Researcher
Malihe Alikhani
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Mert İnan, Katherine Atwell, Anthony Sicilia, Lorna Quandt, Malihe Alikhani
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai