"My Grade is Wrong!": A Contestable AI Framework for Interactive Feedback in Evaluating Student Essays
Project Overview
The document explores the integration of generative AI in education through the CAELF (Contestable AI Empowered LLM Framework), which automates interactive feedback for evaluating student essays. By utilizing large language models (LLMs) and a multi-agent system incorporating computational argumentation, CAELF fosters two-way feedback between teachers and students, addressing the inefficiencies of traditional feedback methods. This innovative approach not only streamlines the feedback process but also enhances the reasoning and interactive capabilities of LLMs. A case study highlighted in the document shows that CAELF significantly improves the quality of feedback provided to students and supports their development of critical thinking skills. Overall, the findings indicate that generative AI can play a transformative role in educational settings by making feedback more efficient and effective, ultimately benefiting both educators and learners.
Key Applications
CAELF (Contestable AI Empowered LLM Framework)
Context: Evaluating critical thinking essays in higher education settings.
Implementation: Implemented a multi-agent system where Teaching-Assistant Agents assess essays based on a rubric, followed by formal reasoning from a Teacher Agent to generate feedback and grades.
Outcomes: Improved accuracy in grading and feedback generation, enhanced interactive feedback capabilities, and supported students' critical thinking skills.
Challenges: Initial limitations of LLMs in reasoning and interaction; risk of inconsistent grading due to user challenges.
Implementation Barriers
Technical
LLMs struggle with reasoning and maintaining consistent evaluations during interactive feedback sessions. Students may exploit vulnerabilities in LLMs through manipulative prompts to receive favorable grades.
Proposed Solutions: Incorporating multi-agent discussions and formal argumentative reasoning to improve the reliability of feedback. Future research to enhance the safety of LLMs and develop detection mechanisms for AI-generated submissions.
Practical
Interactive feedback is often time-consuming for educators, limiting its widespread adoption.
Proposed Solutions: Automating the feedback process through the CAELF framework to reduce the time burden on educators.
Ethical
Students may exploit vulnerabilities in LLMs through manipulative prompts to receive favorable grades.
Proposed Solutions: Future research to enhance the safety of LLMs and develop detection mechanisms for AI-generated submissions.
Project Team
Shengxin Hong
Researcher
Chang Cai
Researcher
Sixuan Du
Researcher
Haiyue Feng
Researcher
Siyuan Liu
Researcher
Xiuyi Fan
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Shengxin Hong, Chang Cai, Sixuan Du, Haiyue Feng, Siyuan Liu, Xiuyi Fan
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai