Towards a Multimodal Document-grounded Conversational AI System for Education
Project Overview
The document explores the development and evaluation of MuDoC, a multimodal document-grounded conversational AI system designed to enhance educational experiences through the integration of text and visuals. By allowing users to verify AI-generated content, MuDoC aims to improve multimedia learning. A user study comparing MuDoC with a traditional text-only system, TexDoC, indicates that both systems effectively engage learners, but MuDoC is particularly successful in fostering greater trust and engagement due to its interactive features. Despite these positive outcomes, the study found no significant differences in problem-solving performance between the two systems. Overall, the findings suggest that while generative AI tools like MuDoC can enhance learner engagement and trust, their impact on actual problem-solving abilities may require further investigation.
Key Applications
MuDoC - a multimodal document-grounded conversational AI system
Context: Educational settings, specifically for graduate students studying AI-related subjects at Georgia Institute of Technology.
Implementation: A user study where participants solved problems using MuDoC compared to a text-only system (TexDoC). MuDoC provides interleaved text and images from educational documents.
Outcomes: Enhanced learner engagement and trust in AI-generated content; visuals helped with memory retention and understanding.
Challenges: No significant performance improvement in problem-solving tasks; some participants felt overwhelmed by the amount of information presented.
Implementation Barriers
Cognitive Load
Participants experienced information overload due to lengthy responses with multiple images, leading to potential confusion and reduced effectiveness in problem-solving.
Proposed Solutions: Future designs should focus on concise responses and allow learners to self-regulate cognitive load by providing options to minimize information clutter.
Project Team
Karan Taneja
Researcher
Anjali Singh
Researcher
Ashok K. Goel
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Karan Taneja, Anjali Singh, Ashok K. Goel
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai