Retrieval-augmented Generation to Improve Math Question-Answering: Trade-offs Between Groundedness and Human Preference
Project Overview
The document explores the use of generative AI in education, specifically focusing on a Retrieval-Augmented Generation (RAG) system designed to enhance math question-answering for middle-school students. By incorporating external knowledge sources, this innovative approach aims to improve the quality of responses produced by large language models (LLMs). The study assesses various prompt guidance conditions to determine their influence on human preferences for the AI-generated answers, highlighting the importance of striking a balance between groundedness—ensuring the responses are based on reliable information—and user preference to optimize educational interactions. The findings underscore the potential of generative AI to support learning by providing tailored, context-aware assistance that meets students' needs while fostering engagement and understanding in complex subjects like math. Overall, this research demonstrates the effectiveness of integrating AI into educational frameworks, paving the way for more personalized and effective learning experiences.
Key Applications
Retrieval-augmented generation (RAG) system for math question-answering
Context: Middle-school math education, particularly for low-income students in Sierra Leone, Liberia, Ghana, and Rwanda.
Implementation: The RAG system uses a vetted open-source math textbook (OpenStax Prealgebra) to retrieve relevant information for answering student queries. Prompts were designed to influence the LLM's responses based on the retrieved content.
Outcomes: Improved response quality and student preferences for responses that are neither too grounded nor too vague. Preferences indicated that students favor responses that balance guidance and relevance.
Challenges: Responses can sometimes be incorrect or misaligned with the curriculum. There are trade-offs between groundedness and perceived usefulness of the responses.
Implementation Barriers
Technical Barrier
LLMs can generate incorrect answers or 'hallucinate' plausible yet factually incorrect information.
Proposed Solutions: Implement retrieval-augmented generation to draw from vetted educational materials and improve the correctness of the responses.
Implementation Barrier
Challenges in ensuring responses are well-aligned with specific educational contexts and curricula.
Proposed Solutions: Incorporate a variety of educational resources and maintain a focus on curriculum alignment during system design.
Project Team
Zachary Levonian
Researcher
Chenglu Li
Researcher
Wangda Zhu
Researcher
Anoushka Gade
Researcher
Owen Henkel
Researcher
Millie-Ellen Postle
Researcher
Wanli Xing
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Zachary Levonian, Chenglu Li, Wangda Zhu, Anoushka Gade, Owen Henkel, Millie-Ellen Postle, Wanli Xing
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai