Matching Exemplar as Next Sentence Prediction (MeNSP): Zero-shot Prompt Learning for Automatic Scoring in Science Education
Project Overview
The document explores the application of generative AI in education, specifically focusing on the development of a zero-shot automatic scoring system for student-written responses in science using the Matching Exemplars as Next Sentence Prediction (MeNSP) technique. This innovative method utilizes pre-trained language models (PLMs) to evaluate student responses without relying on extensive training datasets, thereby reducing the time and costs typically associated with model training. The findings indicate that MeNSP can achieve scoring performance comparable to that of human raters, with the potential for further improvement by incorporating a limited number of labeled samples. By enhancing assessment practices, particularly for complex written responses in science education, this approach represents a significant advancement in the use of generative AI to support educational evaluation and feedback processes.
Key Applications
Matching Exemplars as Next Sentence Prediction (MeNSP)
Context: Science education for students in grades 5 to 8
Implementation: Developed a zero-shot approach to score student responses using PLMs without fine-tuning.
Outcomes: Achieved machine-human scoring agreements with Kappa values ranging from 0.30 to 0.57, and F1 scores from 0.54 to 0.81. The method significantly reduced the cost of model training.
Challenges: Challenges included finding appropriate scoring rubrics and ensuring the PLMs understood the scoring procedure.
Implementation Barriers
Data Collection
Collecting sufficient labeled student responses for training NLP models is time-consuming and costly.
Proposed Solutions: Employing zero-shot and few-shot learning approaches to minimize the need for large datasets.
Model Performance
Ensuring that the PLMs can accurately understand and perform the scoring task without extensive training.
Proposed Solutions: Using prompt learning techniques to align scoring tasks with pre-trained models.
Project Team
Xuansheng Wu
Researcher
Xinyu He
Researcher
Tianming Liu
Researcher
Ninghao Liu
Researcher
Xiaoming Zhai
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Xuansheng Wu, Xinyu He, Tianming Liu, Ninghao Liu, Xiaoming Zhai
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai