ASL STEM Wiki: Dataset and Benchmark for Interpreting STEM Articles
Project Overview
The document discusses the application of generative AI in education, particularly focusing on the ASL STEM Wiki, a resource designed to enhance STEM education for Deaf and hard-of-hearing (DHH) students. By providing a parallel corpus of STEM-related Wikipedia articles in both English and American Sign Language (ASL), this initiative seeks to overcome the accessibility challenges faced by DHH learners. The dataset includes 254 articles interpreted into ASL, featuring over 300 hours of video created by certified interpreters. Key applications of this resource involve automatic sign suggestion, fingerspelling detection, and the exploration of sign variation, which collectively facilitate a richer and more inclusive learning experience. The findings indicate that the ASL STEM Wiki significantly addresses the lack of standardized ASL resources in STEM education, thereby promoting equitable access and understanding in these subjects for DHH students. Overall, the integration of generative AI in this educational context showcases its potential to transform learning environments, making them more accessible and tailored to diverse learners' needs.
Key Applications
ASL Interpretation Enhancement
Context: This application serves as an educational resource for Deaf and hard-of-hearing students in STEM disciplines. It includes automatic sign suggestion, detection and alignment of fingerspelling, and analysis of ASL translations influenced by English syntax to enhance understanding of technical vocabulary and concepts.
Implementation: AI models are employed to detect fingerspelling in ASL videos, suggest appropriate ASL signs based on context, and identify translationese effects in ASL interpretations. These models align fingerspelling instances with corresponding English words to facilitate better understanding.
Outcomes: Improves ASL interpretation quality, enhances comprehension of technical vocabulary in ASL, facilitates better understanding of STEM concepts, and promotes the use of standardized ASL terminology.
Challenges: Challenges include limited standardization of ASL signs for technical terms, high variability of fingerspelled signs, complexities in accurately identifying translationese, and differences in English-ASL syntax.
Implementation Barriers
Technical barrier
The lack of standardized ASL signs for many STEM concepts leads to inconsistencies in interpretation. Additionally, limited annotated data for fingerspelling detection and alignment hampers model training.
Proposed Solutions: Develop systems to detect and suggest appropriate ASL signs based on context and audience. Expand dataset size through additional data collection and annotations, and consider data augmentation techniques.
Educational barrier
Deaf students often rely on fingerspelling due to a lack of established ASL signs, which can hinder understanding.
Proposed Solutions: Training interpreters more effectively on ASL terminology and improving educational resources in ASL.
Project Team
Kayo Yin
Researcher
Chinmay Singh
Researcher
Fyodor O. Minakov
Researcher
Vanessa Milan
Researcher
Hal Daumé III
Researcher
Cyril Zhang
Researcher
Alex X. Lu
Researcher
Danielle Bragg
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Kayo Yin, Chinmay Singh, Fyodor O. Minakov, Vanessa Milan, Hal Daumé III, Cyril Zhang, Alex X. Lu, Danielle Bragg
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai