Evaluating the Effectiveness of LLMs in Introductory Computer Science Education: A Semester-Long Field Study
Project Overview
The document outlines a semester-long field study assessing the impact of CodeTutor, a generative AI-powered assistant, on student learning outcomes in introductory computer science courses. Results indicated that students utilizing CodeTutor experienced notable improvements in their final scores compared to their peers who did not engage with the tool. Participants valued its support for programming syntax and task completion; however, they raised concerns regarding its potential to inhibit critical thinking skills, expressing a preference for interaction with human teaching assistants as the course progressed. These findings highlight the importance of incorporating generative AI literacy into educational curricula, suggesting that while AI tools can enhance certain learning aspects, they must be carefully integrated to ensure they complement rather than replace essential cognitive skills.
Key Applications
CodeTutor, an LLM-powered assistant
Context: Introductory programming course for undergraduate students
Implementation: A semester-long field study with 50 participants, comparing a control group using traditional methods to an experimental group using CodeTutor.
Outcomes: Experimental group achieved statistically significant improvements in final scores; students found CodeTutor helpful for syntax comprehension and debugging.
Challenges: Students expressed skepticism about CodeTutor's ability to enhance critical thinking; preference for human TA support increased over time.
Implementation Barriers
Educational Impact
Concerns about the misuse of LLMs compromising academic integrity and critical thinking skills.
Proposed Solutions: Integrate Generative AI literacy into curricula to teach responsible usage and critical engagement with AI tools.
Prompt Quality
63% of student prompts were poor quality, limiting the effectiveness of CodeTutor.
Proposed Solutions: Implement workshops to improve students' skills in formulating detailed and clear prompts for AI tools.
Project Team
Wenhan Lyu
Researcher
Yimeng Wang
Researcher
Tingting
Researcher
Chung
Researcher
Yifan Sun
Researcher
Yixuan Zhang
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Wenhan Lyu, Yimeng Wang, Tingting, Chung, Yifan Sun, Yixuan Zhang
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai