Assessing UML Models by ChatGPT: Implications for Education
Project Overview
This document investigates the role of generative AI, particularly ChatGPT, in enhancing education within the field of software engineering, focusing on the assessment of UML models. By evaluating UML use case diagrams, class diagrams, and sequence diagrams, the study highlights ChatGPT's potential to automate grading processes and deliver feedback effectively. The findings indicate that ChatGPT's performance in evaluating these models is comparable to that of human experts; however, it tends to be more stringent and occasionally misinterpret the criteria, leading to slightly lower scores than those assigned by human evaluators. Overall, the research underscores the promise of generative AI in education for automating assessment tasks while also pointing out areas for improvement in its grading accuracy and understanding.
Key Applications
ChatGPT for assessing UML models
Context: Educational context involving undergraduate students in Software Engineering courses
Implementation: ChatGPT was prompted with grading criteria and student-generated UML models to provide automated assessments.
Outcomes: ChatGPT's grading scores were found to be similar to those of human experts, indicating its potential as an AI educator.
Challenges: ChatGPT exhibited overstrictness and misunderstandings in applying grading criteria, leading to discrepancies in scoring.
Implementation Barriers
Technical
ChatGPT struggles with understanding nuanced grading criteria and synonymous terms, often leading to overstrictness and evaluation discrepancies.
Proposed Solutions: Refinement of prompts and grading specifications to reduce overstrictness and improve understanding. Implementing more flexible grading criteria that allow for alternative valid answers.
Data Limitations
The study was limited to UML models created by a small group of students, affecting the generalizability of results.
Proposed Solutions: Expanding the sample size and including UML models from diverse educational settings.
Project Team
Chong Wang
Researcher
Beian Wang
Researcher
Peng Liang
Researcher
Jie Liang
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Chong Wang, Beian Wang, Peng Liang, Jie Liang
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai