How Far Are We? The Triumphs and Trials of Generative AI in Learning Software Engineering
Project Overview
Generative AI, especially conversational models like ChatGPT, holds significant promise for transforming education, particularly in fields like Software Engineering by offering contextual support to learners. However, a recent study assessing ChatGPT's effectiveness found no noteworthy enhancements in student productivity or self-efficacy when compared to traditional educational resources. Instead, users reported heightened frustration and cognitive load due to various shortcomings in ChatGPT's interactions, such as its limited guidance on specialized topics, difficulty understanding complex problems, and failure to provide complete assistance. These issues contravene established Human-AI interaction principles, underscoring the necessity for ongoing refinement of generative AI tools to better meet educational needs and improve user experiences. The findings emphasize that while generative AI has potential, its current limitations must be addressed to realize meaningful benefits in educational contexts.
Key Applications
ChatGPT as a conversational generative AI tool
Context: Undergraduate software engineering courses, targeting students learning software engineering tasks.
Implementation: A between-subjects study where one group used ChatGPT exclusively while the control group used traditional resources. Participants completed three software engineering tasks.
Outcomes: No significant difference in productivity or self-efficacy, but increased frustration levels reported by the experimental group.
Challenges: Participants experienced frustration due to ChatGPT's inability to understand problems, provide complete assistance, and its tendency to hallucinate or give wrong guidance.
Implementation Barriers
Technical
ChatGPT often fails to provide relevant advice on niche topics and can misinterpret user queries.
Proposed Solutions: Incorporate better guidelines for Human-AI interaction to improve the system's performance and transparency.
User Experience
Users reported increased cognitive load and frustration when interacting with ChatGPT.
Proposed Solutions: Develop mechanisms for efficient correction and clearer expectations about the system's capabilities.
Project Team
Rudrajit Choudhuri
Researcher
Dylan Liu
Researcher
Igor Steinmacher
Researcher
Marco Gerosa
Researcher
Anita Sarma
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Rudrajit Choudhuri, Dylan Liu, Igor Steinmacher, Marco Gerosa, Anita Sarma
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai