"Is ChatGPT a Better Explainer than My Professor?": Evaluating the Explanation Capabilities of LLMs in Conversation Compared to a Human Baseline
Project Overview
The document explores the role of Generative AI, particularly Large Language Models (LLMs) like GPT-4, in enhancing educational practices through effective explanations in science communication. It emphasizes the importance of conversational explanations and examines how LLMs can complement human experts by augmenting their explanatory capabilities. Utilizing a dataset of explanatory acts, the study evaluates the quality of responses generated by LLMs in comparison to human responses, revealing that LLMs can outperform humans in specific contexts. However, the findings advocate for a collaborative approach, suggesting that while LLMs possess significant potential to engage and tailor explanations for diverse audiences, they should not be seen as replacements for human explainers but rather as supportive tools that enhance the educational experience. Overall, the document underscores the promising applications of generative AI in education, indicating that a synergistic relationship between AI and human educators could lead to more effective learning outcomes.
Key Applications
Evaluation of LLMs for explanation generation
Context: STEM education for college-level students
Implementation: Comparison of human explainer responses with LLM-generated responses using different prompting strategies.
Outcomes: LLM-generated responses preferred for their engagement and effectiveness in explanation dialogues.
Challenges: Conciseness and engagement in LLM responses can vary; potential for overwhelming explainees with information.
Implementation Barriers
Performance Barrier
LLMs may generate responses that are overly verbose and complex, making it difficult for explainees to understand.
Proposed Solutions: Use prompting strategies to guide LLM responses towards conciseness and relevance.
Engagement Barrier
LLM responses may lack the ability to actively engage the explainee in the conversation.
Proposed Solutions: Incorporate explicit instructions for LLMs to include follow-up questions and prompts to enhance engagement.
Project Team
Grace Li
Researcher
Milad Alshomary
Researcher
Smaranda Muresan
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Grace Li, Milad Alshomary, Smaranda Muresan
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai