It is Too Many Options: Pitfalls of Multiple-Choice Questions in Generative AI and Medical Education

Project Overview

The document evaluates the role of Large Language Models (LLMs) in medical education, particularly their performance on multiple-choice questions (MCQs) versus free-response questions. It identifies the limitations of MCQs in effectively assessing the medical knowledge and reasoning skills of LLMs, indicating that existing benchmarks may inflate their actual capabilities. To address this issue, the authors developed a new benchmark called FreeMedQA, which facilitates a comparison of LLM performance across both MCQ and free-response formats. The findings reveal a significant decline in LLM performance when faced with the more demanding free-response questions, underscoring the necessity for improved evaluation methods in medical education that prioritize authentic comprehension over mere pattern recognition. This study highlights critical implications for the integration of generative AI in educational assessments, suggesting that while AI can assist in learning, reliance solely on traditional testing formats may not adequately measure true understanding or skill mastery.

Key Applications

Implementation Barriers

Project Team

Contact Information

For information about the paper, please contact the authors.

Authors: Shrutika Singh, Anton Alyakin, Daniel Alexander Alber, Jaden Stryker, Ai Phuong S Tong, Karl Sangwon, Nicolas Goff, Mathew de la Paz, Miguel Hernandez-Rovira, Ki Yun Park, Eric Claude Leuthardt, Eric Karl Oermann

Source Publication: View Original PaperLink opens in a new window