Battling Botpoop using GenAI for Higher Education: A Study of a Retrieval Augmented Generation Chatbots Impact on Learning
Project Overview
The document examines the use of a custom generative AI chatbot, Professor Leodar, developed for undergraduate engineering students at Nanyang Technological University, aiming to enhance educational experiences through advanced technology. Utilizing Retrieval Augmented Generation (RAG), the chatbot delivers contextually relevant information, leading to high user satisfaction, with 97.1% of participants reporting positive interactions. Key applications of the chatbot include providing personalized guidance, fostering greater student engagement, and enhancing exam preparedness. Despite the positive outcomes, the study highlights challenges such as the need for faster response times and the integration of multimodal capabilities, underscoring the necessity for tailored AI solutions in educational settings. Overall, the findings suggest that generative AI holds significant potential to transform learning experiences, although ongoing improvements and adaptations are essential for maximizing its effectiveness in education.
Key Applications
Professor Leodar, a custom-built generative AI chatbot using Retrieval Augmented Generation (RAG)
Context: Undergraduate engineering students in the course MS0003: Introduction to Data Science and Artificial Intelligence at Nanyang Technological University.
Implementation: The chatbot was deployed in class, providing personalized, context-specific support based on course materials and real-time updates.
Outcomes: Significant enhancement in students' learning outcomes, engagement, and exam preparedness, with positive feedback from 97.1% of users.
Challenges: Response speed during peak usage times and the need for improved multimodal input capabilities.
Implementation Barriers
Technical Barrier
Slow response times during high-demand periods such as examinations.
Proposed Solutions: Transitioning to a more efficient language model (from GPT-4 to Claude 3) improved speed, but further optimizations are needed.
User Experience Barrier
Inconsistencies in the level of detail provided by the chatbot.
Proposed Solutions: Refinement of the chatbot’s scaffolding mechanisms to ensure consistent quality in responses.
Cultural Barrier
Use of Singlish in responses may confuse international students.
Proposed Solutions: Incorporating multilingual support to accommodate diverse language backgrounds.
Project Team
Maung Thway
Researcher
Jose Recatala-Gomez
Researcher
Fun Siong Lim
Researcher
Kedar Hippalgaonkar
Researcher
Leonard W. T. Ng
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Maung Thway, Jose Recatala-Gomez, Fun Siong Lim, Kedar Hippalgaonkar, Leonard W. T. Ng
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai