Exploring LLM-Generated Feedback for Economics Essays: How Teaching Assistants Evaluate and Envision Its Use
Project Overview
The document examines the integration of generative AI in education, specifically focusing on its application in providing AI-generated feedback for a college-level introductory Economics course. It highlights the experiences of teaching assistants (TAs) regarding the effectiveness of AI feedback in enhancing the quality and efficiency of grading processes. The study indicates that AI-generated feedback aligns well with grading rubrics and can expedite grading while improving the quality of feedback provided to students. However, it also notes the limitations of AI, such as its rigidity and potential inaccuracies, which can affect the overall effectiveness of the feedback. The findings stress the necessity for clearly defined rubrics to facilitate the generation of effective AI feedback, suggesting that while generative AI holds significant potential in educational settings, careful implementation and oversight are crucial for maximizing its benefits and mitigating its shortcomings.
Key Applications
LLM-powered feedback engine for economics essays
Context: College-level introductory Economics course (ECON101) for students writing knowledge-intensive essays.
Implementation: The feedback engine generates feedback based on grading rubrics, utilizing a three-step process: identify relevant sentences, make judgments on rubric satisfaction, and generate feedback.
Outcomes: TAs found AI feedback improved grading consistency, provided effective praise and guidance, and helped identify key concepts faster. It also enhanced the overall quality of feedback.
Challenges: AI feedback can be overly rigid, misinterpret rubric requirements, and lacks the holistic understanding TAs provide. It may also produce errors when assessing specialized economic terms.
Implementation Barriers
Technical
AI-generated feedback can sometimes be inaccurate or overly rigid, leading to misjudgments about student responses.
Proposed Solutions: Establish clearly written rubrics, provide intermediate AI outputs for validation, and allow TAs to review AI judgments before finalizing feedback.
Human-AI Interaction
TAs may find AI feedback distracting and worry about becoming overly reliant on it, which could undermine their independent evaluation skills.
Proposed Solutions: Encourage TAs to read and evaluate student essays independently before considering AI feedback, ensuring they maintain control over the grading process.
Project Team
Xinyi Lu
Researcher
Aditya Mahesh
Researcher
Zejia Shen
Researcher
Mitchell Dudley
Researcher
Larissa Sano
Researcher
Xu Wang
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Xinyi Lu, Aditya Mahesh, Zejia Shen, Mitchell Dudley, Larissa Sano, Xu Wang
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai