AI-TA: Towards an Intelligent Question-Answer Teaching Assistant using Open-Source LLMs
Project Overview
The document discusses the implementation of AI-TA, an intelligent teaching assistant designed to improve the educational experience in computing courses through generative AI. Utilizing open-source LLaMA-2 models, AI-TA enhances the efficiency and accuracy of responses to student inquiries on platforms such as Piazza by leveraging advanced techniques including Retrieval-Augmented Generation (RAG), Supervised Fine-Tuning (SFT), and Direct Preference Optimization (DPO). The system was rigorously evaluated using data from an introductory programming course, demonstrating a significant improvement in answer quality while effectively addressing key challenges such as scalability and data privacy. Overall, the findings highlight the potential of generative AI to transform educational contexts by providing timely and accurate assistance to students, thereby enhancing their learning outcomes.
Key Applications
AI-TA: Intelligent Question-Answering Teaching Assistant
Context: Used in computing courses to answer student questions on online platforms like Piazza.
Implementation: Leveraged open-source LLaMA-2 models, incorporating techniques like RAG, SFT, and DPO to improve response quality.
Outcomes: Achieved a 30% improvement in answer quality, providing timely and accurate responses while ensuring data privacy.
Challenges: Challenges included data processing, ensuring model accuracy, and managing the complexity of interactions.
Implementation Barriers
Technical Barriers
Challenges related to model training and data processing, such as deduplication of queries and managing complex datasets.
Proposed Solutions: Implementing advanced data processing techniques and optimizing model configurations to handle complexities.
Privacy Concerns
Using third-party providers raises data privacy issues, particularly for educational data.
Proposed Solutions: Adopting open-source LLMs to maintain control over data privacy and security.
Project Team
Yann Hicke
Researcher
Anmol Agarwal
Researcher
Qianou Ma
Researcher
Paul Denny
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Yann Hicke, Anmol Agarwal, Qianou Ma, Paul Denny
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai