Human-AI Collaboration in Thematic Analysis using ChatGPT: A User Study and Design Recommendations
Project Overview
This document examines the role of generative AI, particularly ChatGPT, in the field of education, focusing on its applications in qualitative research and thematic analysis. It highlights how the collaboration between qualitative researchers and AI can improve efficiency and offer valuable quantitative insights, aiding in language comprehension and data interpretation. However, the findings also underscore significant challenges, including concerns related to the trustworthiness and accuracy of AI-generated analyses, as well as the potential lack of contextual understanding that may arise from relying solely on AI. Overall, the document presents a balanced view of how generative AI can serve as a powerful tool in educational research while also emphasizing the necessity for careful consideration of its limitations and the need for human oversight to ensure reliability in qualitative data analysis.
Key Applications
ChatGPT for thematic analysis
Context: Qualitative research, specifically thematic analysis conducted by doctoral students and academics.
Implementation: Conducted a user study where participants collaborated with ChatGPT during thematic analysis exercises.
Outcomes: Improved efficiency in processing data, enhanced coding accuracy, provided quantitative insights, and assisted language comprehension for non-native speakers.
Challenges: Concerns about trustworthiness and accuracy of AI outputs, limited contextual understanding, and inconsistencies in results.
Implementation Barriers
Trust and Reliability
Participants expressed skepticism regarding the accuracy and reliability of ChatGPT's outputs, necessitating manual verification. Concerns about the acceptance of AI-assisted research methods within the academic community.
Proposed Solutions: Incorporating transparent explanatory mechanisms and validation checks. Encouraging transparency in AI usage and establishing guidelines for its validation in research.
Data Processing Limitations
ChatGPT has limitations in handling long data inputs, which can affect the quality of thematic analysis. AI's ability to infer context is limited, affecting its performance in thematic analysis.
Proposed Solutions: Model advancements and the development of chunking strategies to manage longer transcripts. Embedding user inputs for contextual information and developing AI systems that can learn from user feedback.
Interface Challenges
The current interface of ChatGPT is not optimized for thematic analysis, making collaboration inefficient.
Proposed Solutions: Redesigning the user interface to enhance usability and integrate better with research workflows.
Project Team
Lixiang Yan
Researcher
Vanessa Echeverria
Researcher
Gloria Fernandez Nieto
Researcher
Yueqiao Jin
Researcher
Zachari Swiecki
Researcher
Linxuan Zhao
Researcher
Dragan Gašević
Researcher
Roberto Martinez-Maldonado
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Lixiang Yan, Vanessa Echeverria, Gloria Fernandez Nieto, Yueqiao Jin, Zachari Swiecki, Linxuan Zhao, Dragan Gašević, Roberto Martinez-Maldonado
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai