Cyborg Data: Merging Human with AI Generated Training Data
Project Overview
The document examines the role of Generative Large Language Models (GLMs) in the field of education, specifically focusing on their application in automated essay scoring (AES) through an innovative method called 'Cyborg Data.' This approach integrates human-generated and AI-generated training data, significantly reducing the reliance on large amounts of manually scored essays while still achieving scoring accuracy comparable to traditional methods. The findings indicate that models trained with limited original data, supplemented by synthetic data from a larger AI model, can yield effective results similar to those trained on extensive human-annotated datasets. Nonetheless, the research highlights important concerns regarding potential biases in the AI's scoring process, particularly affecting underrepresented demographic groups, underscoring the need for careful consideration of fairness and equity in AI applications within educational settings. Overall, the document illustrates both the promise of generative AI in enhancing educational assessments and the critical challenges that must be addressed to ensure fair outcomes for all students.
Key Applications
Cyborg Data method for Automated Essay Scoring
Context: Large-scale education assessments targeting middle and high school students (Grades 6-12)
Implementation: Using a Teacher model (large generative model) to generate synthetic scores for essays, training a smaller Student model on a combination of original and synthetic data.
Outcomes: Achieved comparable performance to models trained on the entire dataset using only 10% of the original hand-scored data, reducing costs of AES development significantly.
Challenges: Potential bias in scoring across demographics, lower average scores for certain groups compared to human annotators.
Implementation Barriers
Technical Barrier
The quality and fidelity of synthetic data generated by AI can lead to model performance issues, including bias in scoring.
Proposed Solutions: Implementing regression-based models to better calibrate scores and regular evaluations to assess and mitigate bias in the synthetic data.
Project Team
Kai North
Researcher
Christopher Ormerod
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Kai North, Christopher Ormerod
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai