Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding
Project Overview
The document examines the integration of generative AI, particularly large language models (LLMs) like GPT-3, in educational research, focusing on their application in qualitative coding. It underscores the typically labor-intensive process of qualitative analysis and demonstrates how LLMs can automate portions of the deductive coding process, achieving a level of agreement with expert-coded results that ranges from fair to substantial. This potential is particularly beneficial for researchers who may not possess deep expertise in AI or access to extensive resources. However, the study also highlights significant challenges, including the propensity of these models to generate errors and the necessity for meticulous design in coding tasks to ensure reliability. Overall, the findings point to the promising yet complex role of generative AI in enhancing educational research methodologies, revealing both the opportunities for efficiency and the critical considerations for effective implementation.
Key Applications
Using GPT-3 for Deductive Coding in Qualitative Analysis
Context: Qualitative analysis of children's curiosity-driven questions to inform educational psychology.
Implementation: Combined GPT-3 with expert-developed codebooks for analyzing children's questions, measuring performance through Cohen’s Kappa statistics.
Outcomes: Achieved fair to substantial agreement with expert coding results, indicating feasibility of LLMs in qualitative analysis.
Challenges: LLMs are error-prone and may not capture nuanced meanings, which are essential in qualitative research.
Implementation Barriers
Technical
LLMs can produce incorrect labels and are limited in understanding nuanced meanings in qualitative data.
Proposed Solutions: Conduct detailed error analyses, design interfaces that encourage appropriate reliance on AI, and explore better methods for prompt design.
Project Team
Ziang Xiao
Researcher
Xingdi Yuan
Researcher
Q. Vera Liao
Researcher
Rania Abdelghani
Researcher
Pierre-Yves Oudeyer
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Ziang Xiao, Xingdi Yuan, Q. Vera Liao, Rania Abdelghani, Pierre-Yves Oudeyer
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai