How Adding Metacognitive Requirements in Support of AI Feedback in Practice Exams Transforms Student Learning Behaviors
Project Overview
The document examines the implementation of a generative AI-enhanced practice exam tool in a large undergraduate biology course, utilizing OpenAI's GPT-4o to deliver personalized feedback tailored to students' confidence levels and rationales behind their answers. While the study found that this AI-generated feedback did not lead to significant improvements in exam performance compared to conventional methods, it highlighted the positive impact of integrating metacognitive strategies, which promoted increased student engagement, confidence, and enhanced study habits. The findings suggest that fostering structured reflection and self-assessment may yield greater benefits for student learning than the feedback alone, indicating that the combination of generative AI and metacognitive elements can effectively support educational outcomes by encouraging deeper learning behaviors and self-awareness among students.
Key Applications
AI-powered practice exam system
Context: Large undergraduate introductory biology course with 1002 students
Implementation: Integrated AI-generated feedback with metacognitive requirements, including confidence ratings and answer explanations, within practice exams.
Outcomes: Increased student engagement with textbooks (40% use rate), high satisfaction (M=4.1/5), and improved self-regulation and confidence on midterm topics.
Challenges: No statistically significant performance improvement observed; reliance on AI could diminish critical thinking skills.
Implementation Barriers
Technical
Need for reliable technical infrastructure and time for instructors to integrate AI systems into their courses.
Proposed Solutions: Ensuring robust technical support and training for faculty on using AI tools effectively.
Pedagogical
Potential misalignment between AI-generated feedback and instructors' expectations or grading criteria.
Proposed Solutions: Instructors should verify and adapt AI feedback to ensure pedagogical appropriateness.
Cognitive Load
Students may find metacognitive requirements (like explaining answers) mentally exhausting or unnecessary, particularly for confident answers.
Proposed Solutions: Optional metacognitive requirements for straightforward questions to reduce cognitive load.
Project Team
Mak Ahmad
Researcher
Prerna Ravi
Researcher
David Karger
Researcher
Marc Facciotti
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Mak Ahmad, Prerna Ravi, David Karger, Marc Facciotti
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai