Kattis vs. ChatGPT: Assessment and Evaluation of Programming Tasks in the Age of Artificial Intelligence
Project Overview
The document investigates the role of generative AI, particularly ChatGPT, in introductory programming education, focusing on its effectiveness in solving programming tasks. Utilizing Kattis, an automated grading system, the analysis reveals that ChatGPT managed to solve only 15% of the programming challenges, with better performance on simpler problems. This limited success raises significant concerns regarding academic integrity and the risk of students becoming overly reliant on AI for solutions. The findings emphasize the necessity for educators to incorporate AI tools judiciously into their teaching methodologies, ensuring that they foster critical thinking and encourage self-regulated learning among students. The study ultimately calls for a balanced approach to integrating AI in education, highlighting both its potential benefits and the challenges it poses.
Key Applications
ChatGPT for solving programming tasks
Context: Introductory programming courses in higher education
Implementation: ChatGPT was tested on 127 programming problems provided by Kattis, an automated grading tool.
Outcomes: ChatGPT solved 19 out of 127 tasks (15%), performing better on easier tasks.
Challenges: Limited capability for solving complex programming tasks; concerns about academic integrity.
Implementation Barriers
Technical Barrier
ChatGPT struggles with complex programming tasks and often produces incorrect solutions.
Proposed Solutions: Educators should integrate ChatGPT with other teaching methods to promote understanding.
Ethical Barrier
Increased potential for academic dishonesty as students may rely on AI to complete assignments. This includes the need for critical reflection on AI outputs.
Proposed Solutions: Encouraging critical reflection on AI outputs and training educators on AI use.
Project Team
Nora Dunder
Researcher
Saga Lundborg
Researcher
Olga Viberg
Researcher
Jacqueline Wong
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Nora Dunder, Saga Lundborg, Olga Viberg, Jacqueline Wong
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai