The ELEVATE-AI LLMs Framework: An Evaluation Framework for Use of Large Language Models in HEOR: an ISPOR Working Group Report

LLM-assisted Research and Analysis Framework

Context: Applicable to various tasks in health economics and biomedical research, including systematic literature reviews, health economic modeling, and health technology assessment. The use cases demonstrate a focus on improving efficiency and accuracy in evaluating health-related data and outcomes.

Implementation: Leveraging large language models (LLMs) like GPT-4 for automating processes such as title and abstract screening in systematic literature reviews and replicating health economic analyses. This includes detailed prompts and validation against expert-annotated datasets to ensure accuracy and reliability.

Outcomes: Enhanced efficiency in research processes, accurate replication of health economic models, and improved screening accuracy, promoting transparency and reproducibility in health sciences research.

Challenges: Generalizability across diverse tasks requires further testing; metrics for fairness and bias monitoring are still in development, with some reporting domains rated ambiguous or not reported.

Technical

Challenges in establishing standards for evaluating LLMs in diverse HEOR tasks.

Proposed Solutions: Developing and validating specific metrics tailored to HEOR applications.

Methodological

Complexity in distinguishing between accuracy and comprehensiveness in reporting.

Proposed Solutions: Refining definitions and thresholds for metrics to enhance clarity in evaluation.

Ethical

Concerns regarding fairness and bias in AI outputs, especially in underrepresented populations.

Proposed Solutions: Prioritizing the development of fairness metrics and providing guidance on evaluating demographic representation.