Ethel: A Virtual Teaching Assistant
Project Overview
Generative AI is making significant strides in education, exemplified by the use of a virtual teaching assistant named Ethel at ETH Zurich, which aids students in physics by solving problems, providing feedback on assessments, and grading exams. Ethel leverages course-specific materials and advanced tools like Retrieval Augmented Generation (RAG) to facilitate student interactions. While students generally appreciate the assistance provided by Ethel, challenges persist, including instances of incorrect outputs, handwriting recognition issues, and concerns about misplaced trust in AI technologies. Additionally, questions about the effectiveness of RAG for studying purposes have been raised. The project's ongoing development aims to enhance the system’s functionality and better integrate it with existing campus resources, despite considerations around implementation costs. Overall, the findings suggest that while generative AI has the potential to improve educational experiences, careful attention must be paid to its limitations and the need for continuous refinement.
Key Applications
AI-assisted feedback and grading system
Context: This system has been utilized across various educational contexts, including large introductory physics courses, homework feedback for handwritten solutions, and grading thermodynamics exams for a total of over 1800 students. Students submitted handwritten solutions scanned as PDFs to receive feedback, and the system was also used for grading free-form exam responses.
Implementation: The implementation involves using Retrieval Augmented Generation (RAG) and Large Language Models (LLMs) to provide interactive feedback and grading. Handwritten solutions are converted to LaTeX using tools like Mathpix and GPT-4V, which then generate feedback based on sample solutions and grading rubrics. The system requires about 400 lines of code.
Outcomes: Feedback from the AI-assisted system was rated helpful by students in approximately 75% of cases. The system demonstrated high precision in identifying correct solutions, particularly in grading, but also faced challenges with handwriting recognition accuracy and the identification of passing solutions.
Challenges: Key challenges included inaccuracies in handwriting recognition (around 50% accuracy), misinterpretation of student solutions due to these errors, and a tendency to produce responses that could be perceived as patronizing. There was also concern regarding misplaced trust in AI responses, especially in grading and feedback contexts.
Implementation Barriers
Technical barrier
Challenges with handwriting recognition and the accuracy of generated outputs.
Proposed Solutions: Future improvements in handwriting and graphics recognition technologies.
Trust barrier
Students may develop misplaced trust in AI responses, affecting their learning.
Proposed Solutions: Educating students about the limitations of AI and maintaining a healthy skepticism towards AI outputs.
Cost barrier
The operational cost of using Azure AI Services for the virtual assistant.
Proposed Solutions: Exploring open-weight options and possibilities for on-premises service deployment.
Project Team
Gerd Kortemeyer
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Gerd Kortemeyer
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai