Is ChatGPT a Good Teacher Coach? Measuring Zero-Shot Performance For Scoring and Providing Actionable Insights on Classroom Instruction
Project Overview
The document examines the role of generative AI, particularly ChatGPT, in enhancing education through teacher training and coaching. It identifies three key applications for generative AI in this context: scoring classroom observation transcripts, pinpointing strengths and weaknesses in teaching strategies, and offering actionable recommendations to improve student reasoning. While the study finds that ChatGPT can produce relevant feedback for educators, it notes that the insights generated are frequently lacking in novelty and depth, suggesting that the tool may serve more as a supplementary resource rather than a standalone solution. The findings emphasize the necessity for ongoing research to refine the effectiveness of generative AI in educational settings, ultimately aiming to enhance the quality of teaching and learning experiences.
Key Applications
ChatGPT as an automated teacher coach
Context: Elementary math classroom instruction, targeting teachers and instructional coaches
Implementation: Evaluated zero-shot performance of ChatGPT on scoring transcripts, identifying highlights, and providing actionable suggestions based on classroom observation protocols.
Outcomes: ChatGPT generated relevant suggestions but often repeated existing teacher strategies, indicating potential for improvement in novelty and insightfulness.
Challenges: Limited novelty in suggestions, low correlation with human ratings, and challenges in providing insightful and truthful feedback.
Implementation Barriers
Technical Barrier
ChatGPT struggles to produce novel and insightful feedback, often suggesting what teachers already do. The model's training data may not include sufficient examples of effective teacher coaching, leading to repetitive suggestions.
Proposed Solutions: Future work should involve reinforcement learning with feedback from coaches and better prompting methods to enhance the model's ability to generate actionable insights. Involving educators in the model fine-tuning process and expanding the dataset to include diverse teaching scenarios could mitigate this.
Ethical Barrier
Concerns regarding privacy and data security when using classroom transcripts with AI.
Proposed Solutions: Implementing safeguards to prevent the direct use of identifiable student data in AI training.
Project Team
Rose E. Wang
Researcher
Dorottya Demszky
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Rose E. Wang, Dorottya Demszky
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai