Can Large Language Models Match Tutoring System Adaptivity? A Benchmarking Study
Project Overview
The document explores the application of Generative AI, specifically Large Language Models (LLMs), in educational settings, particularly in comparison to Intelligent Tutoring Systems (ITS). It proposes a framework to evaluate how LLMs generate instructional responses across different tutoring scenarios, focusing on their adaptivity and pedagogical effectiveness. The findings indicate that while Llama3-70B exhibited some contextual responsiveness, none of the evaluated LLMs demonstrated the same level of adaptivity as traditional ITS, which provide structured and context-aware guidance essential for effective learning. This suggests that while LLMs offer potential in educational support, they currently fall short of replicating the nuanced and responsive interactions provided by established ITS, highlighting the need for further development to enhance their effectiveness in personalized education.
Key Applications
LLM-based tutoring systems
Context: Hybrid tutoring environments where LLMs provide real-time instructional support to learners
Implementation: LLMs were prompted with tutoring scenarios extracted from an open-source Intelligent Tutoring System (ITS) to evaluate their adaptivity and pedagogical soundness.
Outcomes: The study revealed that LLMs could provide conversational support but struggled to replicate the adaptivity of ITS. Llama3-70B showed some significant adaptivity to student errors, while Llama3-8B received higher pedagogical quality ratings despite formatting issues.
Challenges: LLMs demonstrated limited adaptivity to context signals, struggled with instruction adherence, and produced overly direct feedback that deviated from effective tutoring practices.
Implementation Barriers
Technical barrier
LLMs lack the ability to respond adequately to key context signals, such as student errors and knowledge components, which are essential for effective tutoring.
Proposed Solutions: Future research should prioritize hybrid settings that integrate LLMs within ITS frameworks to improve instructional effectiveness.
Pedagogical barrier
LLMs tend to provide overly direct feedback that may contradict effective instructional principles.
Proposed Solutions: Improvements in the design of LLMs, such as tuning the parameters for better alignment with pedagogical strategies, are necessary.
Project Team
Conrad Borchers
Researcher
Tianze Shou
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Conrad Borchers, Tianze Shou
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai