Elucidating STEM Concepts through Generative AI: A Multi-modal Exploration of Analogical Reasoning
Project Overview
This document explores the application of generative AI, specifically large language models, to revolutionize STEM education. The research introduces a multimodal system that simplifies complex concepts by transforming them into understandable metaphors, visualized as storyboards, and then animated videos. The system leverages generative AI to create analogies and visual representations, aiming to enhance student comprehension and engagement with STEM subjects. The study employs a randomized A/B/C test to evaluate the effectiveness of this AI-driven approach in improving learning outcomes.
Key Applications
AI-powered Visual Analogy Generation
Context: STEM education, specifically mathematics, physics, and programming concepts, for learners of all levels. Transforming analogies into engaging visual storyboards.
Implementation: Employs GPT-4 (or similar LLMs) to generate analogies for STEM concepts. These analogies are then transformed into visual storyboards and animated videos using a text-to-image generative model.
Outcomes: Aims to enhance learners’ understanding of STEM concepts, improve their learning engagement, and improve comprehension of complex STEM algorithms.
Challenges: Generating effective visual representations of complex concepts, ensuring the generated content accurately and comprehensively represents the concepts, and the generative model frequently falling short in representing each part comprehensively.
Implementation Barriers
Technical Limitations
The generative model struggles to accurately and comprehensively represent complex STEM concepts in visual form, including difficulties representing multiple components of a concept within a single image. There are also challenges in creating dynamic visual analogies, specifically in generating suitable transitions or motions to articulate the analogy dynamically.
Proposed Solutions: Further refinement of the prompts used to guide the image generation process and exploring different generative models. Further development of the video generation process to include better animation and transitions.
Project Team
Chen Cao
Researcher
Zijian Ding
Researcher
Gyeong-Geon Lee
Researcher
Jiajun Jiao
Researcher
Jionghao Lin
Researcher
Xiaoming Zhai
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Chen Cao, Zijian Ding, Gyeong-Geon Lee, Jiajun Jiao, Jionghao Lin, Xiaoming Zhai
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gemini-2.0-flash-lite