Repurposing the scientific literature with vision-language models
Project Overview
The document explores the integration of generative AI in education, particularly through the development of specialized models like CNS-Obsidian, which utilizes vision-language models (VLMs) trained on a multimodal dataset of neurosurgery publications (NeuroPubs). This model was evaluated against a leading generalist AI, GPT-4o, in a randomized controlled trial, showcasing its effectiveness in generating diagnostic assistance and educational content tailored to neurosurgery. Furthermore, the document emphasizes the training configurations of generative AI models for creating multiple-choice questions (MCQs), revealing that careful alignment and task-specific fine-tuning significantly enhance model accuracy, especially when leveraging diverse datasets. The findings suggest that extended training durations correlate with improved performance metrics. Overall, the implementation of domain-specific generative AI tools demonstrates promising potential to elevate educational practices, enhance scientific publishing, and support clinical decision-making in specialized fields while also addressing challenges related to data sourcing and model efficacy.
Key Applications
Generative AI for Educational Content Generation and Assessment
Context: Used in educational and clinical settings for neurosurgery residents and practitioners, as well as in educational assessments for students and educators, focusing on the creation of high-quality educational content and evaluation tools.
Implementation: Developed a pipeline that extracts and converts domain-specific journal data into multimodal datasets for training vision-language models and generative models. The training approach involves alignment, general fine-tuning, and task-specific fine-tuning with varying epochs to improve performance for specific educational tasks.
Outcomes: Achieved performance on par with advanced models like GPT-4o in diagnostic tasks and improved accuracy metrics in assessment tools (e.g., up to 79.48% on GPT-Test). Generated high-quality educational content including graphical abstracts, multiple-choice questions (MCQs), and other evaluation materials.
Challenges: Limited access to peer-reviewed journals for training data; ensuring the quality and relevance of AI-generated content; and optimizing the duration for training stages to maximize model performance.
Implementation Barriers
Data Access
Challenges in obtaining high-quality, peer-reviewed data for training AI models, as many valuable journals are behind paywalls.
Proposed Solutions: Securing permissions from publishers and focusing on domain-specific datasets to enhance the quality of training data.
User Engagement
Low utilization rates of AI tools by highly trained specialists who may prefer automation rather than decision-support systems.
Proposed Solutions: Developing user-friendly interfaces and passive AI-assisted workflows that better align with specialists' needs.
Technical Barrier
Determining the optimal training duration for different stages of model fine-tuning.
Proposed Solutions: Conducting ablation experiments to evaluate the impact of various training configurations on model performance.
Project Team
Anton Alyakin
Researcher
Jaden Stryker
Researcher
Daniel Alexander Alber
Researcher
Karl L. Sangwon
Researcher
Jin Vivian Lee
Researcher
Brandon Duderstadt
Researcher
Akshay Save
Researcher
David Kurland
Researcher
Spencer Frome
Researcher
Shrutika Singh
Researcher
Jeff Zhang
Researcher
Eunice Yang
Researcher
Ki Yun Park
Researcher
Cordelia Orillac
Researcher
Aly A. Valliani
Researcher
Sean Neifert
Researcher
Albert Liu
Researcher
Aneek Patel
Researcher
Christopher Livia
Researcher
Darryl Lau
Researcher
Ilya Laufer
Researcher
Peter A. Rozman
Researcher
Eveline Teresa Hidalgo
Researcher
Howard Riina
Researcher
Rui Feng
Researcher
Todd Hollon
Researcher
Yindalon Aphinyanaphongs
Researcher
John G. Golfinos
Researcher
Laura Snyder
Researcher
Eric Leuthardt
Researcher
Douglas Kondziolka
Researcher
Eric Karl Oermann
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Anton Alyakin, Jaden Stryker, Daniel Alexander Alber, Karl L. Sangwon, Jin Vivian Lee, Brandon Duderstadt, Akshay Save, David Kurland, Spencer Frome, Shrutika Singh, Jeff Zhang, Eunice Yang, Ki Yun Park, Cordelia Orillac, Aly A. Valliani, Sean Neifert, Albert Liu, Aneek Patel, Christopher Livia, Darryl Lau, Ilya Laufer, Peter A. Rozman, Eveline Teresa Hidalgo, Howard Riina, Rui Feng, Todd Hollon, Yindalon Aphinyanaphongs, John G. Golfinos, Laura Snyder, Eric Leuthardt, Douglas Kondziolka, Eric Karl Oermann
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai