Automatic Short Math Answer Grading via In-context Meta-learning
Project Overview
The document explores the innovative application of generative AI, particularly through the MathBERT model, to enhance automatic short answer grading (ASAG) in mathematics education. It identifies significant limitations of conventional automated grading systems, which often struggle with novel questions and lack adaptability. To address these issues, the authors introduce an in-context meta-learning framework that enables the grading model to generalize better to previously unseen questions, resulting in higher scoring accuracy compared to existing methods. This advancement not only streamlines the grading process but also supports educators by providing more reliable assessments of student responses, ultimately fostering a more effective learning environment in mathematics. The findings underscore the potential of generative AI to transform educational assessment practices, making them more responsive and accurate as they adapt to a dynamic array of student inquiries.
Key Applications
MathBERT for Automatic Short Answer Grading
Context: Used in educational settings for grading student responses to open-ended math questions, targeting students from various educational levels.
Implementation: The model is fine-tuned on actual student responses using an in-context learning approach, where it receives examples of responses and their scores to provide context for grading.
Outcomes: Significantly improved scoring accuracy, especially for new questions not seen during training, with up to 50% improvement in some metrics.
Challenges: Challenges include the need for a model that can handle the combination of natural language and mathematical expressions, and the issue of model storage due to the large size of pre-trained models.
Implementation Barriers
Technical and Generalization Barrier
Existing models are not well-adapted to educational subject domains and often require separate models for each question, leading to storage issues. Additionally, models often struggle to generalize to new, unseen questions, which limits their applicability in real-world educational contexts.
Proposed Solutions: Using a meta-learning approach that allows for a single model to generalize across multiple questions and incorporate context effectively, along with implementing in-context learning to improve the model's ability to adapt to new questions using few examples.
Model Performance Barrier
MathBERT did not outperform the traditional BERT model, indicating potential limitations in processing mathematical language.
Proposed Solutions: Future exploration of more effective models for mathematical language and enhancing the contextual information provided to the model.
Project Team
Mengxue Zhang
Researcher
Sami Baral
Researcher
Neil Heffernan
Researcher
Andrew Lan
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Mengxue Zhang, Sami Baral, Neil Heffernan, Andrew Lan
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai