Automated Text Scoring in the Age of Generative AI for the GPU-poor
Project Overview
The document explores the role of generative AI, particularly generative language models (GLMs), in education, with an emphasis on applications like automated essay scoring (AES) and automated short answer scoring (ASAS). It reveals that smaller, open-source GLMs can be more effective than large proprietary models, offering advantages in transparency, security, and efficiency. Research findings indicate that fine-tuning these smaller models can achieve satisfactory performance in both scoring and providing constructive feedback to students. However, the document also identifies challenges, including concerns regarding the quality of the generated feedback and the necessity for ongoing validation and collaboration with educators to ensure the models meet educational standards. Overall, the integration of generative AI in educational assessment presents promising opportunities while also requiring careful consideration of its implementation and impact on teaching and learning outcomes.
Key Applications
Automated Text Scoring (ATS)
Context: Educational assessment for students, specifically for essay and short answer scoring.
Implementation: Fine-tuning small, open-source GLMs for AES and ASAS using the Automated Student Assessment Prize (ASAP) dataset.
Outcomes: Achieved adequate performance in scoring essays and short answers, with some models providing feedback that justified scores based on rubrics.
Challenges: Quality of feedback can be inconsistent; models may produce hallucinations and require rigorous evaluation for validity.
Implementation Barriers
Technical limitations
Challenges with the quality and reliability of model-generated feedback, including instances of hallucination. There is a need for rigorous evaluation methodologies.
Proposed Solutions: Collaboration with educators to develop feedback systems and ensure they meet educational needs.
Social implications
Concerns over the ethical development and implementation of AI tools in education, emphasizing the need for collaboration with educational practitioners.
Proposed Solutions: Involve educators in the design and deployment of generative AI tools to ensure they meet educational needs.
Project Team
Christopher Michael Ormerod
Researcher
Alexander Kwako
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Christopher Michael Ormerod, Alexander Kwako
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai