Integrating AI Tutors in a Programming Course
Project Overview
The document examines the implementation of RAGMan, an AI tutoring system that leverages Retrieval-Augmented Generation (RAG) and Large Language Models (LLMs) in an introductory programming course. This system provides assistance to students by addressing homework-specific queries without offering direct answers, thereby promoting independent problem-solving. Student feedback reveals that these AI tutors significantly enhance the learning experience by creating a supportive, judgment-free environment, which is associated with a decrease in failing grades. However, there are concerns regarding the potential adverse effects on students achieving the highest grades. The study underscores the exploratory nature of integrating generative AI into educational settings and highlights the necessity for further research to fully understand its implications and optimize its applications in learning environments. Overall, the findings suggest that while generative AI can positively impact student engagement and performance, careful consideration and additional investigation are required to address the diverse effects it may have on different student demographics.
Key Applications
RAGMan - LLM-powered tutoring system
Context: Introductory programming course (ICS32) at a university, targeting first-year Computer Science students.
Implementation: Deployed RAGMan as an optional resource for homework-specific AI tutors, configured to provide assistance without giving direct solutions.
Outcomes: 98% accuracy in responses to legitimate homework questions; 78% of students reported that the tutors helped their learning; statistically significant decrease in failing grades.
Challenges: Some students reported dissatisfaction with the speed of the AI tutors and the occurrence of contradictory responses.
Implementation Barriers
Technical Limitation
AI tutors were reported to be slow, affecting user experience.
Proposed Solutions: The speed issue was identified and subsequently addressed, though not in time for the ICS32 course.
User Experience
Students experienced contradictions in the responses given by the AI tutors, which affected their interaction.
Proposed Solutions: Efforts to improve AI tutors will focus on minimizing the generation of contradictory responses.
Project Team
Iris Ma
Researcher
Alberto Krone Martins
Researcher
Cristina Videira Lopes
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Iris Ma, Alberto Krone Martins, Cristina Videira Lopes
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai