WIP: Assessing the Effectiveness of ChatGPT in Preparatory Testing Activities
Project Overview
The document explores the role of generative AI, specifically ChatGPT, in software testing education, emphasizing its effectiveness in generating testing artifacts more efficiently than traditional human methods. Conducted with postgraduate students at a community college, the study demonstrated that while ChatGPT significantly streamlined the creation of preparatory testing materials, human expertise is still crucial; students were required to apply critical thinking skills to refine their prompts for optimal results. The findings indicate that AI can substantially enhance educational practices in software testing by improving the efficiency of artifact generation, yet it also highlights persistent challenges, particularly concerning the accuracy and reliability of the AI-generated content. Overall, the integration of generative AI into education presents a promising avenue for innovation, though it necessitates a balanced approach that combines AI capabilities with human oversight and critical engagement.
Key Applications
ChatGPT for generating testing artifacts
Context: Postgraduate students in a Capstone Project course at a community college focused on software testing.
Implementation: Students manually created testing artifacts and then used ChatGPT to generate the same artifacts, comparing the results.
Outcomes: Students found ChatGPT useful for speeding up test case generation and improving coverage but noted limitations in accuracy and contextual relevance.
Challenges: ChatGPT-generated artifacts sometimes lacked accuracy and detail; students had to refine their prompts to achieve better results.
Implementation Barriers
Technical Barrier
ChatGPT may generate inaccurate or irrelevant information if not provided with sufficient context.
Proposed Solutions: Educators can teach prompt engineering techniques to guide students in crafting effective prompts.
Privacy and Security Barrier
Sensitive information could be inadvertently shared when using ChatGPT.
Proposed Solutions: Caution should be exercised to prevent sharing sensitive information and to educate students on safe usage practices.
Project Team
Susmita Haldar
Researcher
Mary Pierce
Researcher
Luiz Fernando Capretz
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Susmita Haldar, Mary Pierce, Luiz Fernando Capretz
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai