Towards Adaptive Feedback with AI: Comparing the Feedback Quality of LLMs and Teachers on Experimentation Protocols
Project Overview
The document explores the transformative role of generative AI, particularly large language models (LLMs), in enhancing education through personalized and adaptive feedback, especially in science learning. It highlights a comparative study assessing the quality of feedback provided by LLMs against that of human teachers and science education experts, revealing that while LLMs can deliver feedback of comparable quality overall, they struggle with contextual understanding and error identification. This gap suggests that while LLMs offer valuable support in educational settings, their effectiveness can be significantly improved through collaboration with human expertise. The findings underscore the potential of integrating AI technologies into educational practices, facilitating tailored learning experiences that can support educators and students alike, while emphasizing the importance of maintaining human oversight to address limitations in AI applications.
Key Applications
LLM feedback agent for student experimentation protocols
Context: Science education for middle school students (grades 6-8) conducting experiments.
Implementation: The LLM agent was developed to detect errors in student experimentation protocols and provide real-time adaptive feedback. Feedback was compared with that from human teachers and experts.
Outcomes: The LLM agent provided feedback of similar quality to that of human teachers and experts, with a mean rating of 3.784 across all dimensions. It effectively summarized goals and suggested improvements.
Challenges: The LLM struggled with contextual understanding and providing specific error identification, particularly in the Feed Back dimension.
Implementation Barriers
Technical Limitations
LLMs often lack nuanced contextual understanding and struggle to identify specific errors in student work.
Proposed Solutions: Combining LLM-generated feedback with human expertise to enhance educational practices.
Implementation Challenges
Limited validation of LLM feedback systems in real-world educational contexts.
Proposed Solutions: Conducting real-world studies to evaluate LLM effectiveness in classroom settings.
Project Team
Kathrin Seßler
Researcher
Arne Bewersdorff
Researcher
Claudia Nerdel
Researcher
Enkelejda Kasneci
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Kathrin Seßler, Arne Bewersdorff, Claudia Nerdel, Enkelejda Kasneci
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai