An Empirical Study on Usage and Perceptions of LLMs in a Software Engineering Project
Project Overview
This document explores the integration of Large Language Models (LLMs) in a software engineering course at the National University of Singapore, illustrating their effectiveness in enhancing student productivity in coding tasks such as syntax assistance, debugging, and generating foundational code structures. Although LLMs were not specifically designed for code generation, their application has proven beneficial, particularly during the initial phases of software development. The study reveals that the impact of LLMs varies according to students' coding skills and prior experience with AI tools, suggesting a need for tailored approaches to maximize their utility. Furthermore, the findings underscore the importance of preparing students for productive human-AI collaboration while also addressing concerns regarding the potential adverse effects on the development of coding skills. Overall, the research highlights the promise of generative AI in educational settings, advocating for thoughtful integration to enrich learning experiences without undermining foundational skill acquisition.
Key Applications
LLMs (ChatGPT, GitHub Copilot) for code generation
Context: Academic software engineering course with undergraduate students
Implementation: Students were encouraged to integrate LLMs into their development tool-chain during a semester-long group project.
Outcomes: Students found LLMs helpful for creating basic design patterns, debugging, and improving code quality.
Challenges: Concerns about reducing coding skill development and the impact on the job market.
Implementation Barriers
Skill barrier
Students with lower coding skills were less willing to use AI code generation tools, fearing they wouldn't understand or correctly assess AI-generated code.
Proposed Solutions: Educators should enhance students' foundational coding skills and provide training on how to effectively use and evaluate AI-generated code.
Access barrier
Some students may have had restricted access to paid versions of LLMs, limiting their usage and creating inequality in access to AI tools.
Proposed Solutions: Ensure all students have equal access to AI tools, possibly by providing institutional licenses or funding for students in need.
Project Team
Sanka Rasnayaka
Researcher
Guanlin Wang
Researcher
Ridwan Shariffdeen
Researcher
Ganesh Neelakanta Iyer
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Sanka Rasnayaka, Guanlin Wang, Ridwan Shariffdeen, Ganesh Neelakanta Iyer
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai