Navigating Compiler Errors with AI Assistance -- A Study of GPT Hints in an Introductory Programming Course
Project Overview
The document explores the application of generative AI, particularly the GPT-4 model, in an introductory programming course to provide personalized hints for resolving compiler errors. The study revealed that students appreciated the usefulness of the hints generated by GPT, resulting in increased focus and decreased confusion and frustration among those in the experimental group. However, despite these positive subjective experiences, the impact of the AI-generated hints on actual academic performance was inconsistent, particularly with more complex programming errors, where students did not always perform better when using the hints. Overall, while generative AI shows promise in enhancing student engagement and support in learning programming, its effectiveness in improving performance remains variable and suggests the need for further investigation into its optimal use in educational settings.
Key Applications
GPT-4 model for generating personalized hints for compiler errors
Context: Introductory programming course for computer science students at a university in Poland
Implementation: Hints generated via OpenAI API when a compiler error was detected during automated assessment of programming assignments.
Outcomes: Students rated the hints as useful, reported increased focus and decreased confusion; experimental group performed better on complex tasks after hints were disabled.
Challenges: Mixed results in performance; some students found hints unhelpful; challenges in automating hint generation due to the variety of potential compiler errors.
Implementation Barriers
Technical
The complexity and variety of compiler error messages make it difficult to provide relevant hints.
Proposed Solutions: Refined prompt engineering and potential fine-tuning of the AI model to improve hint relevance.
User Experience
Approximately 20% of students found the hints not beneficial.
Proposed Solutions: Gathering user feedback to tailor hints more closely to student needs.
Project Team
Maciej Pankiewicz
Researcher
Ryan S. Baker
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Maciej Pankiewicz, Ryan S. Baker
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai