Will Your Next Pair Programming Partner Be Human? An Empirical Evaluation of Generative AI as a Collaborative Teammate in a Semester-Long Classroom Setting
Project Overview
The document explores the integration of Generative AI (GenAI), specifically Large Language Models (LLMs), in computer science education, focusing on their role in enhancing pair programming. It reveals that students collaborating with LLM-based tools alongside human partners experienced the most significant improvements in collaboration, learning outcomes, and overall performance, achieving the highest scores. In contrast, those programming independently with LLMs tended to score lower, emphasizing the importance of human interaction. The study highlights a shift in students' attitudes towards GenAI, showing that as they gained experience, their perceptions of its programming capabilities improved. Despite these positive findings, the document notes challenges such as contextual limitations and differing expectations between human and AI partners, which may affect the effectiveness of GenAI tools in educational settings. Overall, the integration of GenAI in education shows promise for enhancing student engagement and performance, though it necessitates careful consideration of its application and the dynamics of human-AI collaboration.
Key Applications
Pair Programming with Gen AI (PAI) using tools like GitHub Copilot and ChatGPT
Context: Undergraduate computer science course with 39 students working on pair programming assignments
Implementation: Students completed assignments under three conditions: Traditional Pair Programming (PP), Pair Programming with Gen AI (PAI), and Solo Programming with Gen AI (SAI). Each student experienced each condition once across three assignments.
Outcomes: Students in the PAI condition had the highest assignment scores, and their attitudes towards LLMs improved significantly. They primarily relied on LLMs for syntax clarification and conceptual guidance.
Challenges: Students noted limitations such as contextual constraints, outdated knowledge bases, and different expectations for AI compared to human partners.
Implementation Barriers
Technical Barrier
Contextual limitations and outdated knowledge bases of LLM-based tools hindered their effectiveness in complex programming tasks.
Proposed Solutions: Future tools need stronger contextual awareness and memory, ideally integrated directly into coding environments.
Expectational Barrier
Students had varying expectations of LLMs, viewing them as technical assistants rather than collaborators for deeper problem-solving.
Proposed Solutions: Design implications suggest differentiating LLM roles to provide quick technical support while allowing for deeper advisory interactions.
Project Team
Wenhan Lyu
Researcher
Yimeng Wang
Researcher
Yifan Sun
Researcher
Yixuan Zhang
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Wenhan Lyu, Yimeng Wang, Yifan Sun, Yixuan Zhang
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai