Do Tutors Learn from Equity Training and Can Generative AI Assess It?
Project Overview
This document explores the application of generative AI, particularly large language models (LLMs) such as GPT-4o and GPT-4-turbo, in evaluating the performance of tutors participating in equity-focused training. The study involved 81 undergraduate tutors engaged in scenario-based lessons designed to enhance their ability to support students facing inequities. Findings revealed that the tutors experienced slight improvements in their learning outcomes and reported greater confidence in utilizing equity-focused skills. Furthermore, the generative AI models proved effective in assessing tutor responses; however, challenges persist in ensuring that AI evaluations align closely with human assessments. Overall, the use of generative AI in this educational context highlights its potential to enhance training and evaluation processes while also emphasizing the need for further refinement in AI-human assessment alignment.
Key Applications
Using generative AI for assessing tutor performance in equity training
Context: Online training for undergraduate tutors assisting middle school students
Implementation: Integrating GPT-4o and GPT-4-turbo to evaluate open-ended tutor responses based on scenario-based learning
Outcomes: Tutors reported increased confidence in addressing student inequities, and marginally significant learning gains were observed
Challenges: Aligning AI assessments with human evaluations and handling subjective interpretations of open-ended responses
Implementation Barriers
Technical barrier
Challenges in ensuring that LLMs accurately assess nuanced and subjective tutor responses
Proposed Solutions: Refining prompt engineering techniques and developing more objective assessment metrics
Logistical barrier
Limited number of trained tutors available to implement equitable practices in educational environments
Proposed Solutions: Increasing training programs and leveraging generative AI tools to scale assessments and support
Project Team
Danielle R. Thomas
Researcher
Conrad Borchers
Researcher
Sanjit Kakarla
Researcher
Jionghao Lin
Researcher
Shambhavi Bhushan
Researcher
Boyuan Guo
Researcher
Erin Gatz
Researcher
Kenneth R. Koedinger
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Danielle R. Thomas, Conrad Borchers, Sanjit Kakarla, Jionghao Lin, Shambhavi Bhushan, Boyuan Guo, Erin Gatz, Kenneth R. Koedinger
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai