ChatGPT and Software Testing Education: Promises & Perils
Project Overview
The document explores the integration of generative AI, particularly ChatGPT, in the realm of education, focusing on its application in software testing. It reports that ChatGPT successfully provides accurate or partially accurate responses to 55.6% of inquiries within the software testing curriculum, shedding light on both the advantages and limitations of this technology. While the findings suggest significant educational benefits, including personalized learning and immediate feedback, the study also points out the challenges related to the correctness and reliability of AI-generated information. It underscores the necessity for thoughtful implementation of AI tools in academic environments to maximize their potential while addressing the risks of misinformation. Overall, the document advocates for a balanced approach to employing generative AI in education, encouraging educators to harness its capabilities while remaining vigilant about its shortcomings to ultimately enhance learning outcomes.
Key Applications
ChatGPT as a conversational agent for answering software testing questions
Context: Software testing education for undergraduate and graduate students
Implementation: ChatGPT was tasked with answering questions from a popular software testing textbook, using both shared and separate context prompting strategies.
Outcomes: ChatGPT correctly answered 55.6% of questions and provided correct or partially correct explanations 53.0% of the time.
Challenges: ChatGPT's confidence in its answers is a poor predictor of correctness, and it struggles with questions that involve both coding and conceptual components.
Implementation Barriers
Technical Limitations
ChatGPT lacks knowledge on certain topics and may produce incorrect answers due to assumptions or lack of context.
Proposed Solutions: Providing more contextual information in prompts can improve the accuracy of ChatGPT's responses.
Assessment Integrity
Concerns about students using ChatGPT to circumvent learning and assessments.
Proposed Solutions: Design assessments that require deeper understanding and context to minimize reliance on AI tools.
Project Team
Sajed Jalil
Researcher
Suzzana Rafi
Researcher
Thomas D. LaToza
Researcher
Kevin Moran
Researcher
Wing Lam
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Sajed Jalil, Suzzana Rafi, Thomas D. LaToza, Kevin Moran, Wing Lam
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai