Math Multiple Choice Question Generation via Human-Large Language Model Collaboration
Project Overview
The document explores the integration of a prototype tool named HEDGE, designed to enhance collaboration between educators and large language models (LLMs) in the creation of math multiple choice questions (MCQs). It highlights the effectiveness of LLMs in generating question stems and correct answers, while also identifying their limitations in crafting valid distractors that accurately reflect common student misconceptions. This underscores the importance of human oversight in the question generation process to maintain the quality and validity of educational content. Overall, the findings suggest that while generative AI can aid in educational tasks, its current capabilities require human expertise to ensure that the generated materials meet pedagogical standards and effectively support student learning.
Key Applications
Human Enhanced Distractor Generation Engine (HEDGE)
Context: Educational context for math educators targeting middle and high school students creating MCQs.
Implementation: Educators use a two-step process where they first evaluate and edit LLM-generated stems and explanations, then guide the LLM to generate distractors and feedback based on anticipated student errors.
Outcomes: 70% of the generated stems were considered valid, but only 37% of the distractors and feedback were deemed valid, highlighting LLM limitations.
Challenges: LLMs struggle to generate valid distractors that capture common student errors and misconceptions.
Implementation Barriers
Technical Limitations and Content Quality
LLMs have difficulty generating distractors that accurately reflect common student errors and misconceptions. Additionally, the generated questions often reflect a low level of Bloom’s Taxonomy, lacking in-depth understanding.
Proposed Solutions: Involve human educators in the process to guide LLMs and improve the quality of generated distractors. Encourage educators to provide in-context examples with higher difficulty levels and more complex question types.
Project Team
Jaewook Lee
Researcher
Digory Smith
Researcher
Simon Woodhead
Researcher
Andrew Lan
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Jaewook Lee, Digory Smith, Simon Woodhead, Andrew Lan
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai