Automatic Task Requirements Writing Evaluation via Machine Reading Comprehension
Project Overview
The document presents a novel framework leveraging machine reading comprehension (MRC) to enhance the evaluation of student essays in English tests by focusing on Task Requirements (TRs). It critiques conventional automated grading systems for their inability to offer nuanced feedback, as they typically generate only overall scores. The proposed MRC framework improves upon these limitations by accurately assessing whether essays meet specific TRs and pinpointing the relevant sections of the essays that address these requirements. Experimental results demonstrate that this approach not only achieves high accuracy but also superior F1 scores compared to prior methods, highlighting its effectiveness in educational assessments. Overall, the implementation of generative AI in this context signifies a promising advancement in personalized feedback and grading efficiency, ultimately enhancing the learning experience for students.
Key Applications
Machine Reading Comprehension Framework for Task Requirements Evaluation
Context: This framework is used in assessing essays for Key English Test (KET) and Preliminary English Test (PET) among K-12 students.
Implementation: The framework consists of question normalization, an MRC module, and a response locating module, which together evaluate essays against task requirements.
Outcomes: Achieved 0.93 accuracy score and 0.85 F1 score on a real-world educational dataset, providing detailed feedback on student essays.
Challenges: Existing MRC models trained on general datasets like SQuAD 2.0 perform poorly on educational datasets due to differences in context and language use.
Implementation Barriers
Technical Barrier
Existing MRC models trained on general-purpose datasets do not perform well in educational scenarios due to the differences in data characteristics.
Proposed Solutions: Train models on specialized educational datasets and fine-tune them for optimal performance in the educational context.
Resource Barrier
Limited teacher resources hinder timely feedback on student essays.
Proposed Solutions: Implement automated systems like the proposed MRC framework to provide instant feedback.
Project Team
Shiting Xu
Researcher
Guowei Xu
Researcher
Peilei Jia
Researcher
Wenbiao Ding
Researcher
Zhongqin Wu
Researcher
Zitao Liu
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Shiting Xu, Guowei Xu, Peilei Jia, Wenbiao Ding, Zhongqin Wu, Zitao Liu
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai