Engineering an Intelligent Essay Scoring and Feedback System: An Experience Report
Project Overview
The document explores the development of an intelligent essay scoring and feedback system that leverages generative AI and machine learning (ML) to enhance educational assessment. It details the complexities involved in evolving client requirements into a robust cloud-based application capable of automating the scoring and feedback process for submitted essays. The system evaluates essays on various dimensions, generating scores and constructive textual feedback through the application of multiple ML models. While the implementation of such an automated system offers notable benefits, including increased scalability and efficiency, it also faces considerable challenges, particularly concerning data quality, the limitations of model training, and various deployment hurdles. Overall, the findings emphasize the potential of generative AI to transform educational assessment while highlighting the need to address these challenges to optimize its effectiveness in real-world applications.
Key Applications
Intelligent Essay Scoring and Feedback System
Context: Used by a recruitment support service for scoring essays submitted by job applicants.
Implementation: Developed a cloud-based framework with multiple ML models for scoring essays based on predetermined rubrics.
Outcomes: Automates the scoring process, generates feedback, and streamlines data collection from customers.
Challenges: Limited training data (1000 essays), ambiguity in open-ended text, deployment complexities, and high computational costs.
Implementation Barriers
Technical Barrier
Challenges in deploying complex ML models due to high resource demands and package versioning issues.
Proposed Solutions: Implementing a modular architecture that allows for independent updates and the use of cloud resources efficiently.
Data Barrier
Insufficient training data leading to potential overfitting and limited model effectiveness.
Proposed Solutions: Designing the system to evolve as more data becomes available, including online training capabilities.
Testing Barrier
Difficulty in applying traditional software testing techniques to ML systems due to their unique characteristics.
Proposed Solutions: Employing combinatorial testing and stress tests to measure performance and improve coverage.
Project Team
Akriti Chadda
Researcher
Kelly Song
Researcher
Raman Chandrasekar
Researcher
Ian Gorton
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Akriti Chadda, Kelly Song, Raman Chandrasekar, Ian Gorton
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai