A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education
Project Overview
The document explores the application of generative AI, particularly GPT-4, in automating the creation of multiple-choice questions (MCQs) for programming education. It evaluates the effectiveness of AI-generated MCQs in comparison to those created by humans, emphasizing their alignment with educational learning objectives. The findings indicate that the quality of AI-generated questions is on par with that of human-created ones; however, there are areas for improvement, particularly in ensuring that each question has one clear correct answer and that distractors are of high quality. By utilizing AI tools for MCQ generation, educators can alleviate their workload and improve the design of assessments, ultimately enhancing the educational experience. This study highlights the potential of generative AI in streamlining assessment processes while still addressing the need for refinement in question quality.
Key Applications
LLM-powered MCQ generation system using GPT-4
Context: Higher education programming courses, specifically Python programming classes
Implementation: Developed a pipeline that uses high-level course context and specific learning objectives to generate MCQs.
Outcomes: Generated MCQs were found to be clear, well-aligned with learning objectives, and comparable in quality to human-crafted MCQs.
Challenges: Issues with generating multiple correct answers and distractors that could give away the correct answer.
Implementation Barriers
Quality Control and Alignment
AI-generated MCQs sometimes contain multiple correct answers or distractors that reveal the correct answer, while human-crafted MCQs were often aligned with course topics but not specific learning objectives.
Proposed Solutions: Future work should focus on prompt engineering techniques to improve the generation quality, and implementing AI tools could help ensure better alignment between assessments and learning objectives.
Project Team
Jacob Doughty
Researcher
Zipiao Wan
Researcher
Anishka Bompelli
Researcher
Jubahed Qayum
Researcher
Taozhi Wang
Researcher
Juran Zhang
Researcher
Yujia Zheng
Researcher
Aidan Doyle
Researcher
Pragnya Sridhar
Researcher
Arav Agarwal
Researcher
Christopher Bogart
Researcher
Eric Keylor
Researcher
Can Kultur
Researcher
Jaromir Savelka
Researcher
Majd Sakr
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Jacob Doughty, Zipiao Wan, Anishka Bompelli, Jubahed Qayum, Taozhi Wang, Juran Zhang, Yujia Zheng, Aidan Doyle, Pragnya Sridhar, Arav Agarwal, Christopher Bogart, Eric Keylor, Can Kultur, Jaromir Savelka, Majd Sakr
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai