Improving Assessment of Tutoring Practices using Retrieval-Augmented Generation
Project Overview
The document explores the application of Generative Pre-trained Transformers (GPT), specifically GPT-3.5 and GPT-4, in assessing the social-emotional competencies of novice math tutors during one-on-one tutoring sessions. It emphasizes the critical role of social-emotional learning in tutoring to enhance educational outcomes. The study investigates various prompting strategies, notably Retrieval-Augmented Generation (RAG), which demonstrated improved accuracy and cost-effectiveness in evaluating tutor performance. The findings indicate that these generative AI models can effectively automate tutor assessments, offering valuable insights that can inform personalized training interventions for tutors. Overall, the use of AI in education, particularly through generative models, presents significant opportunities to improve tutoring practices and support the development of essential competencies in educators.
Key Applications
Generative Pre-trained Transformers (GPT-3.5 and GPT-4)
Context: One-on-one tutoring sessions for middle school math students
Implementation: Utilized GPT models to evaluate tutoring transcripts based on social-emotional learning principles using various prompting strategies.
Outcomes: The RAG prompt demonstrated higher accuracy in assessments and lower financial costs compared to other prompting strategies.
Challenges: Challenges included the potential for 'hallucination' in generated outputs and the need for human expert involvement in assessments.
Implementation Barriers
Technical
The challenge of accurately assessing social-emotional learning competencies without human expert involvement.
Proposed Solutions: Utilizing large language models (LLMs) like GPT to automate assessments and improve cost-effectiveness.
Financial
High costs associated with using advanced GPT models for real-time assessments.
Proposed Solutions: Identifying cost-effective prompting strategies, such as RAG, to reduce financial burdens.
Project Team
Zifei FeiFei Han
Researcher
Jionghao Lin
Researcher
Ashish Gurung
Researcher
Danielle R. Thomas
Researcher
Eason Chen
Researcher
Conrad Borchers
Researcher
Shivang Gupta
Researcher
Kenneth R. Koedinger
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Zifei FeiFei Han, Jionghao Lin, Ashish Gurung, Danielle R. Thomas, Eason Chen, Conrad Borchers, Shivang Gupta, Kenneth R. Koedinger
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai