Data Augmentation with Paraphrase Generation and Entity Extraction for Multimodal Dialogue System
Project Overview
The document explores the implementation of generative AI in education, specifically through the development of a multimodal dialogue system that supports young learners, aged 5-8, in grasping fundamental math concepts via interactive, game-based learning. Emphasizing the enhancement of the Natural Language Understanding (NLU) component, the system employs data augmentation strategies, such as paraphrase generation, to improve responsiveness and engagement. By integrating physical elements and multimodal inputs, the approach aims to create a more immersive and effective educational experience. The findings suggest that such interactive systems can significantly bolster children's understanding of math, making learning both enjoyable and effective while showcasing the potential of AI technologies in transforming educational methodologies.
Key Applications
Multimodal Dialogue System for early childhood education (Kid Space project)
Context: Game-based learning for children aged 5-8, focusing on basic math concepts.
Implementation: The system uses spoken dialogue systems (SDS) that process user utterances and multimodal inputs (visual, audio, gestures) to facilitate interactions.
Outcomes: Improved NLU performance for intent recognition from user utterances; engagement and interactive learning experiences for children.
Challenges: Limited datasets for training NLU models; need for robust interaction handling in a multimodal environment.
Implementation Barriers
Technical Barrier
Limited datasets for training the Natural Language Understanding models. Challenges in ensuring accurate understanding of multimodal inputs (e.g., gestures, speech) for effective dialogue management.
Proposed Solutions: Data augmentation techniques such as paraphrase generation to create more training data. Iterative training and evaluation of NLU models; integrating various sensing technologies for better input recognition.
Project Team
Eda Okur
Researcher
Saurav Sahay
Researcher
Lama Nachman
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Eda Okur, Saurav Sahay, Lama Nachman
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai