Holmes: Automated Fact Check with Large Language Models
Project Overview
The document explores the innovative use of generative AI in education through the development of 'Holmes', an automated fact-checking framework that utilizes Large Language Models (LLMs) to combat disinformation. It addresses the limitations of current methods in verifying multimodal disinformation and introduces a novel evidence retrieval approach designed to enhance the performance of LLMs in providing precise verdicts and justifications. The findings from extensive experiments reveal that Holmes significantly outperforms traditional disinformation detection techniques, showcasing its potential as a valuable educational tool for improving students' critical thinking and media literacy skills. By leveraging advanced AI capabilities, Holmes not only aids in identifying false information but also fosters an educational environment where learners can engage with reliable data and develop their analytical abilities, ultimately contributing to a more informed society.
Key Applications
Holmes - Automated Fact Check Framework
Context: Educational institutions, public policy organizations, and media outlets focused on misinformation detection.
Implementation: Holmes employs LLMs to verify disinformation claims by retrieving relevant evidence from the internet, summarizing it, and evaluating the claim's truthfulness.
Outcomes: Holmes achieved an accuracy of 90.2% in real-time verification tasks and improved detection accuracy by 30.8% compared to existing methods.
Challenges: Challenges include the need for reliable evidence retrieval, handling multimodal disinformation, and the limitations of LLMs in accessing updated information.
Implementation Barriers
Technical Barrier
LLMs struggle to autonomously search for accurate and relevant evidence, which can lead to hallucinations and unreliable outputs.
Proposed Solutions: Implementing a structured evidence retrieval methodology, as seen in Holmes, to guide LLMs and improve accuracy.
Resource Barrier
Deep learning-based methods often require significant computational resources and labeled datasets for training, which limits scalability.
Proposed Solutions: Holmes reduces development costs by leveraging off-the-shelf LLMs without needing extensive training or data labeling.
Project Team
Haoran Ou
Researcher
Gelei Deng
Researcher
Xingshuo Han
Researcher
Jie Zhang
Researcher
Xinlei He
Researcher
Han Qiu
Researcher
Shangwei Guo
Researcher
Tianwei Zhang
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Haoran Ou, Gelei Deng, Xingshuo Han, Jie Zhang, Xinlei He, Han Qiu, Shangwei Guo, Tianwei Zhang
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai