Improving Socratic Question Generation using Data Augmentation and Preference Optimization
Project Overview
The document explores the application of generative AI, particularly large language models (LLMs), in enhancing educational practices through improved Socratic question generation. It identifies the challenges educators face in manually crafting questions that foster critical thinking and problem-solving skills without providing direct answers. To address these challenges, the authors propose a method that employs data augmentation to produce invalid question samples and preference optimization to refine the quality of questions generated by LLMs. Experimental results demonstrate that the DPO-optimized LLaMA 2 model significantly outperforms existing techniques in generating effective Socratic questions, thereby promoting better learning outcomes for students. This innovative approach underscores the potential of generative AI to enhance educational methodologies by facilitating deeper student engagement and understanding.
Key Applications
Socratic question generation using LLMs
Context: Educational context focused on helping students debug code; target audience includes novice programmers.
Implementation: The method involves two phases: data augmentation to generate invalid Socratic questions and preference optimization of LLaMA 2 using direct preference optimization (DPO).
Outcomes: Outperformed existing state-of-the-art prompting methods on common text similarity metrics, improving the validity of generated Socratic questions.
Challenges: Challenges included the generation of invalid questions that can confuse students, such as irrelevant or direct questions.
Implementation Barriers
Technical
Large language models like GPT-3.5 and GPT-4 can produce invalid or irrelevant questions.
Proposed Solutions: Implementing data augmentation methods to generate invalid question samples for training, and preference optimization to refine question generation.
Resource
Using proprietary models like GPT-3.5 and GPT-4 is expensive and not accessible for all educational contexts.
Proposed Solutions: Focusing on open-source models like LLaMA to provide a cost-effective, privacy-preserving alternative.
Project Team
Nischal Ashok Kumar
Researcher
Andrew Lan
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Nischal Ashok Kumar, Andrew Lan
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai