Comparing Few-Shot Prompting of GPT-4 LLMs with BERT Classifiers for Open-Response Assessment in Tutor Equity Training
Project Overview
The document investigates the application of generative AI in education, specifically focusing on the use of BERT and GPT-4 models for evaluating open-ended responses in equity training for tutors. It highlights a comparative study that reveals how fine-tuning the BERT model on a limited dataset can lead to superior performance over the more advanced and resource-heavy GPT-4 in nuanced educational scenarios. The findings underscore BERT’s potential for automated assessments in educational contexts, especially in areas where conventional evaluation methods struggle to provide adequate feedback. Overall, the document illustrates the promise of generative AI, particularly BERT, as a practical tool for enhancing assessment processes in education, offering insights into its effectiveness in fostering equity training among educators.
Key Applications
BERT and GPT-4 models for assessing open responses in equity training.
Context: Higher education students in a tutor training program aimed at middle school tutoring.
Implementation: BERT was fine-tuned on a dataset of human-annotated tutor responses, while GPT-4 models were prompted with few-shot examples.
Outcomes: BERT showed superior performance, with higher accuracy and resource efficiency compared to GPT models in grading nuanced open-ended responses.
Challenges: GPT models struggled with capturing nuanced patterns due to reliance on few-shot prompting, leading to variable performance.
Implementation Barriers
Technical Barrier
GPT models may not capture nuanced response patterns effectively due to reliance on few-shot prompt engineering.
Proposed Solutions: Consider fine-tuning models on task-specific datasets to improve performance on complex assessments.
Resource Limitation
GPT models are resource-intensive and face issues such as changing model versions and downtime.
Proposed Solutions: Utilize more resource-efficient models like BERT for educational applications.
Project Team
Sanjit Kakarla
Researcher
Conrad Borchers
Researcher
Danielle Thomas
Researcher
Shambhavi Bhushan
Researcher
Kenneth R. Koedinger
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Sanjit Kakarla, Conrad Borchers, Danielle Thomas, Shambhavi Bhushan, Kenneth R. Koedinger
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai