Extracting Research Instruments from Educational Literature Using LLMs
Project Overview
The document explores the application of Large Language Models (LLMs) in education, particularly focusing on their role in enhancing information extraction related to research instruments. By employing a multi-step prompting approach and a specialized data schema, the proposed system aims to improve the accuracy and reliability of data extraction, thereby aiding knowledge management in educational research. The study reveals that LLMs can significantly streamline the process of identifying and categorizing research instruments, ultimately facilitating informed decision-making for both researchers and educators. However, it also acknowledges ongoing challenges with precision and context sensitivity in the outputs generated. Overall, the findings underscore the potential of generative AI to transform educational research practices, although there is still room for refinement to overcome existing limitations.
Key Applications
LLM-based system for extracting research instruments
Context: Educational research, targeting researchers, practitioners, and education leaders.
Implementation: The system uses a structured three-step prompt design with a domain-specific schema to extract information from educational literature.
Outcomes: The system significantly outperformed traditional methods in identifying instrument names and details, improving accessibility of research instrument information.
Challenges: Challenges include issues with precision in distinguishing constructs and the system's tendency to over-extract instruments.
Implementation Barriers
Technical barrier
The system struggles with context sensitivity, leading to false positives and over-extraction of instruments.
Proposed Solutions: Refined ontological rules and human-in-the-loop validation are suggested to mitigate these issues.
Operational barrier
High cost and manual effort associated with the curation of educational research instruments.
Proposed Solutions: Automation of instrument extraction to reduce manual curation efforts.
Project Team
Jiseung Yoo
Researcher
Curran Mahowald
Researcher
Meiyu Li
Researcher
Wei Ai
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Jiseung Yoo, Curran Mahowald, Meiyu Li, Wei Ai
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai