"Give me the code" -- Log Analysis of First-Year CS Students' Interactions With GPT
Project Overview
The document explores the role of generative AI, particularly large language models (LLMs) like ChatGPT, in enhancing computer science education, specifically among first-year students learning programming. It highlights the dual impact of these AI tools: while they can significantly improve students' critical thinking and problem-solving abilities, there are concerns about students becoming overly dependent on AI-generated solutions, which may hinder their learning process. The study underscores the importance of implementing structured exercises that not only leverage the capabilities of LLMs but also encourage students to develop their programming skills independently. By providing guidance on how to interact with these AI tools effectively, educators can foster a balanced approach that maximizes the educational benefits of generative AI while mitigating the risks of over-reliance.
Key Applications
Interacting with ChatGPT for code generation
Context: First-year Computer Science students in a Data Structures and Algorithms course
Implementation: Students used a structured template for interacting with ChatGPT to generate code solutions for a project assignment, documenting their prompts and responses.
Outcomes: 72.2% of students incorporated ChatGPT's solutions into their projects, with 47.2% explicitly choosing one of the AI-generated solutions. The exercise fostered critical thinking skills.
Challenges: Students often had unsophisticated prompting techniques, lacked context in their requests, and struggled to incorporate AI solutions effectively without formal training.
Implementation Barriers
Technical barrier
Students' limited training in prompt engineering resulted in ineffective interactions with LLMs.
Proposed Solutions: Integrate prompt training into the curriculum, encouraging richer information in prompts and structured exercises.
Cognitive barrier
Some students exhibited an over-reliance on LLMs, misunderstanding its limitations.
Proposed Solutions: Encourage critical thinking and evaluation of AI-generated solutions, emphasizing the importance of understanding the code produced.
Project Team
Pedro Alves
Researcher
Bruno Pereira Cipriano
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Pedro Alves, Bruno Pereira Cipriano
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai