Towards A Human-in-the-Loop LLM Approach to Collaborative Discourse Analysis
Project Overview
The document examines the utilization of Large Language Models (LLMs), particularly GPT-4-Turbo, in the educational sector, focusing on their role in analyzing collaborative discourse among high school students within STEM+C learning environments. It highlights a human-in-the-loop methodology that aims to deepen insights into students' synergistic learning experiences in subjects like physics and computing through automated discourse analysis. Preliminary findings indicate that LLMs can effectively characterize collaborative discourse similarly to human evaluators, showcasing their capability to identify complex discourse categories. However, the document also addresses certain limitations of LLMs, such as the occurrence of hallucinations and challenges related to contextual integration. Overall, the findings suggest that while LLMs present significant potential for enhancing educational discourse analysis, careful consideration of their limitations is essential for effective implementation in educational settings.
Key Applications
Human-in-the-loop LLM approach for discourse analysis
Context: High school students (aged 14-15) learning kinematics in a collaborative environment
Implementation: Used GPT-4-Turbo to summarize and categorize students' collaborative discourse during model-building tasks in a C2STEM curriculum
Outcomes: LLM-generated summaries were comparable to human-generated summaries, providing detailed insights into students' problem-solving processes and identifying when teacher assistance may be needed.
Challenges: LLMs can struggle with contextual integration and may produce hallucinations, misclassifying discourse segments.
Implementation Barriers
Technical barrier
LLMs can generate hallucinations that propagate errors in discourse analysis, impacting the quality of summaries. Additionally, there is a limited ability of LLMs to integrate environmental actions and temporal dynamics into discourse analysis.
Proposed Solutions: Implementing a human-in-the-loop approach to validate and refine LLM outputs, ensuring context and accuracy. Enhancing prompts to include more contextual information and utilizing prosodic audio data to capture nuances in student discourse.
Project Team
Clayton Cohn
Researcher
Caitlin Snyder
Researcher
Justin Montenegro
Researcher
Gautam Biswas
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Clayton Cohn, Caitlin Snyder, Justin Montenegro, Gautam Biswas
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai