Multimodal Emotion Recognition by Fusing Video Semantic in MOOC Learning Scenarios
Project Overview
The document explores the application of generative AI in education, particularly through a novel approach to multimodal emotion recognition in Massive Open Online Courses (MOOCs). It emphasizes the critical role that instructional video content plays in influencing learners' emotional states. By integrating video semantic information with physiological signals, such as eye movement and photoplethysmography (PPG), the study proposes a method that utilizes a large language model to generate descriptive content for videos. This innovative fusion enhances the accuracy of emotion recognition in learners, leading to significant improvements in understanding their emotional responses during online learning experiences. The findings suggest that such advancements can provide valuable insights for optimizing educational strategies, ultimately contributing to more effective and engaging learning environments.
Key Applications
Multimodal emotion recognition method integrating video semantic information and physiological signals
Context: MOOC learning scenarios targeting online learners
Implementation: Video descriptions generated using a pre-trained large language model are fused with eye movement and PPG signals using a cross-attention mechanism.
Outcomes: Improved emotion recognition performance with accuracy enhancements over 14%. Provides insights into the influence of video content on learners' emotions.
Challenges: Recognition of neutral emotions (Interest and Confusion) is difficult due to similar feature distributions.
Implementation Barriers
Data imbalance
High dropout rates in MOOCs due to emotional deficiencies, and challenges in accurately recognizing specific emotions like Interest and Confusion due to unbalanced sample sizes.
Proposed Solutions: Implemented ADASYN sampling method to balance the distribution of samples across different emotion categories.
Project Team
Yuan Zhang
Researcher
Xiaomei Tao
Researcher
Hanxu Ai
Researcher
Tao Chen
Researcher
Yanling Gan
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Yuan Zhang, Xiaomei Tao, Hanxu Ai, Tao Chen, Yanling Gan
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai