Reducing the Cost: Cross-Prompt Pre-Finetuning for Short Answer Scoring
Project Overview
The document explores the application of generative AI in education, particularly focusing on Automated Short Answer Scoring (SAS) through a two-phase approach that leverages cross-prompt training to enhance efficiency in preparing training data for various prompts. This method involves an initial pre-finetuning of a model using established rubrics and answers, followed by a finetuning phase tailored to a specific new prompt. The findings indicate that this approach significantly improves scoring accuracy, especially in contexts where training data is scarce, thereby addressing critical challenges such as data accessibility and the necessity for diverse training inputs. Overall, the use of generative AI in this context demonstrates promising outcomes in automating assessment processes, potentially leading to more streamlined and accurate evaluation methods in educational settings.
Key Applications
Automated Short Answer Scoring (SAS)
Context: School education and e-learning environments, targeting instructors and educational institutions.
Implementation: A two-phase training approach where a model is pre-finetuned on existing annotated prompts and then finetuned on a new prompt using key phrases.
Outcomes: Improved scoring accuracy and reduced training data requirements, particularly in settings with limited data.
Challenges: Data accessibility for cross-prompt data and the need for generalizability across different prompts.
Implementation Barriers
Data accessibility
Limited access to cross-prompt data can hinder the ability to effectively train a SAS model for new prompts.
Proposed Solutions: The proposed two-phase approach allows the model to be trained on existing rubrics and answers without needing access to cross-prompt data during the finetuning phase.
Generalizability
It's uncertain whether a model can learn useful scoring criteria from cross-prompt data for scoring new prompts.
Proposed Solutions: Designing the model to learn general properties of scoring tasks can enhance its ability to generalize from cross-prompt training.
Project Team
Hiroaki Funayama
Researcher
Yuya Asazuma
Researcher
Yuichiroh Matsubayashi
Researcher
Tomoya Mizumoto
Researcher
Kentaro Inui
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Hiroaki Funayama, Yuya Asazuma, Yuichiroh Matsubayashi, Tomoya Mizumoto, Kentaro Inui
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai