Evaluating Large Language Models on the Spanish Medical Intern Resident (MIR) Examination 2024/2025:A Comparative Analysis of Clinical Reasoning and Knowledge Application
Project Overview
The document presents a comprehensive analysis of the use of generative AI, specifically large language models (LLMs), in the context of the Spanish Medical Intern Resident (MIR) examination for 2024 and 2025. It highlights the significant advancements in clinical reasoning, image interpretation, and epidemiological calculations, demonstrating that fine-tuned models, such as Miri Pro, outperform both general-purpose LLMs and human candidates in examination scores. Despite these promising results, the study identifies ongoing challenges related to ethical reasoning and the models' ability to navigate complex clinical scenarios. Furthermore, it underscores the necessity for a transformation in medical education that emphasizes AI literacy, the establishment of ethical frameworks, and the promotion of continuous professional development. This adaptation aims to ensure that AI tools are effectively integrated into clinical practice, ultimately enhancing educational outcomes and preparing future medical professionals to work alongside AI innovations.
Key Applications
Evaluation of large language models on the MIR examination
Context: Medical education and assessment for Spanish medical graduates
Implementation: Comparative analysis of LLMs' performance on the 2024 and 2025 MIR exams, focusing on clinical reasoning and image interpretation.
Outcomes: Fine-tuned LLMs like Miri Pro achieved high accuracy, outperforming general-purpose models and human candidates in clinical reasoning tasks.
Challenges: General models struggled with complex clinical scenarios requiring nuanced decision-making and ethical reasoning.
Implementation Barriers
Limitations of Generative AI in Education
Project Team
Carlos Luengo Vera
Researcher
Ignacio Ferro Picon
Researcher
M. Teresa del Val Nunez
Researcher
Jose Andres Gomez Gandia
Researcher
Antonio de Lucas Ancillo
Researcher
Victor Ramos Arroyo
Researcher
Carlos Milan Figueredo
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Carlos Luengo Vera, Ignacio Ferro Picon, M. Teresa del Val Nunez, Jose Andres Gomez Gandia, Antonio de Lucas Ancillo, Victor Ramos Arroyo, Carlos Milan Figueredo
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai