A Survey of Large Language Models in Medicine: Progress, Application, and Challenge

AI-Enhanced Healthcare Support

Context: Assisting healthcare professionals and students through content generation, personalized learning, patient interaction, and psychological support by leveraging language models for various educational and clinical applications.

Implementation: Utilizes fine-tuned LLMs (such as ChatGPT and RoBERTa) for generating educational materials, automating medical coding, summarizing clinical reports, facilitating patient interactions, and providing psychological support through conversational interfaces. The implementations involve dynamic prompts, knowledge graphs, and training on clinical notes.

Outcomes: Improved educational experiences, enhanced patient care, and accurate clinical documentation. Achieved metrics include 30.72 ROUGE-L for diagnosis summarization, 0.926 AUC in ICD code prediction, and 47.93 ROUGE-L in report summarization.

Challenges: Potential for hallucinations and biases in AI outputs, need for expert review for clinical relevance, risks of misinformation, and lack of emotional understanding in conversational agents.

AI-Enhanced Medical Education Tools

Context: Providing comprehensive training and resources for healthcare professionals and students to improve learning outcomes through the integration of AI technologies into medical education.

Implementation: Employs LLMs to deliver personalized educational content, assessments, and resources, enhancing the understanding of both medical and AI concepts.

Outcomes: Results in better-informed healthcare professionals and improved integration of AI technology in healthcare practice.

Challenges: Ensuring the accuracy and reliability of AI-generated content in medical training and the need for effective frameworks for interdisciplinary collaboration.

Technical

Hallucination of LLMs leading to inaccurate information generation, and challenges in developing effective frameworks for sharing data between rural clinics and AI systems.

Proposed Solutions: Implement training-time correction, generation-time correction, retrieval-augmented correction, and research into interdisciplinary frameworks to facilitate data sharing.

Evaluation

Lack of comprehensive benchmarks and metrics to evaluate LLM performance in medical contexts.

Proposed Solutions: Develop domain-specific benchmarks that assess trustworthiness, helpfulness, and explainability.

Data Limitations

Limited access to high-quality medical datasets for training LLMs.

Proposed Solutions: Generate high-quality synthetic datasets and use smaller open-sourced datasets for fine-tuning.

Knowledge Adaptation

Challenges in updating LLMs with new medical knowledge efficiently.

Proposed Solutions: Utilize model editing and retrieval-augmented generation to incorporate new knowledge.

Ethical

Concerns regarding the ethical use of LLMs in medical settings, including accountability and safety.

Proposed Solutions: Establish guidelines for ethical practices and ensure transparency in AI decision-making.

Regulatory

Complex regulatory landscape for the integration of LLMs in healthcare.

Proposed Solutions: Develop adaptive and robust regulatory frameworks to ensure safety and ethical standards.

Educational Barrier

The need for bilingual professionals who are versed in both medicine and AI technology.

Proposed Solutions: Creation of bilingual education programs that train healthcare professionals in both fields.

Privacy Barrier

Concerns about patient data protection when integrating LLM technology in healthcare.

Proposed Solutions: Implementing effective in-house development methods to safeguard patient data.

Project Contact: Dr. Jianhua Yang

LLM Model Version: gpt-4o-mini-2024-07-18

Analysis Provider: Openai