PyEvalAI: AI-assisted evaluation of Jupyter Notebooks for immediate personalized feedback
Project Overview
The document highlights the advancements in educational technology through the introduction of PyEvalAI, an AI-assisted evaluation system tailored for STEM education. This innovative tool aims to tackle the common challenges associated with grading, such as prolonged feedback cycles and the necessity for educators to oversee the evaluation process. By providing immediate, personalized feedback on Jupyter notebook assignments, PyEvalAI enhances the learning experience for students, enabling them to iterate quickly on their work. Furthermore, it alleviates the grading burden on tutors, allowing them to focus on more complex instructional tasks. The implementation of a locally hosted language model ensures that data privacy is maintained, addressing concerns about student information security. Overall, the system exemplifies the effective use of generative AI in education, demonstrating positive outcomes in both student engagement and educator efficiency.
Key Applications
PyEvalAI - AI-assisted evaluation of Jupyter notebooks
Context: University-level numerics course for Bachelor of Science in Computer Science students
Implementation: Implemented as a web interface using a Tornado server to manage interactions between students, tutors, and administrators.
Outcomes: Improved speed of feedback and grading efficiency, with reports of better student performance through multiple attempts on assignments.
Challenges: Grading accuracy, dependency on AI feedback, potential for incorrect grading, and ensuring tutors can correct AI-generated grades.
Implementation Barriers
Technical
Challenges related to grading accuracy and consistency of AI-generated feedback.
Proposed Solutions: Incorporate human oversight in grading to adjust and correct AI feedback as needed.
Privacy
Concerns over data privacy when using external AI services.
Proposed Solutions: Host language models locally to maintain control over student data and ensure compliance with privacy policies.
Financial
High costs associated with proprietary AI models and tools.
Proposed Solutions: Develop and utilize open-source solutions to mitigate costs.
Project Team
Nils Wandel
Researcher
David Stotko
Researcher
Alexander Schier
Researcher
Reinhard Klein
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Nils Wandel, David Stotko, Alexander Schier, Reinhard Klein
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai