Let's Ask AI About Their Programs: Exploring ChatGPT's Answers To Program Comprehension Questions
Project Overview
The document examines the role of Large Language Models (LLMs) in computing education, particularly their effectiveness in addressing program comprehension questions based on self-generated code. It emphasizes the strengths and weaknesses of models such as GPT-3.5 and GPT-4, noting their proficiency in solving programming tasks and generating educational materials, while also acknowledging their propensity for errors akin to those made by novice programmers. The findings suggest that while LLMs can serve as valuable educational tools, there are notable limitations that necessitate further investigation into their pedagogical applications. Overall, the research underscores the potential of generative AI to enhance learning in computing education, while highlighting the need for careful integration and evaluation of these technologies to maximize their effectiveness in educational settings.
Key Applications
Questions about Learners’ Code (QLCs) generation and response
Context: Introductory programming courses for novice programmers
Implementation: LLMs were prompted to generate code solutions for specific programming tasks, followed by generating QLCs to assess comprehension of that code.
Outcomes: GPT-4 showed an average success rate of 88% in answering QLCs, while GPT-3.5 had a success rate of 69%. The study indicates the potential for LLMs to assist in educational assessments and improve understanding of programming.
Challenges: LLMs occasionally make errors such as misinterpreting questions, incorrectly counting line numbers, and generating hallucinated justifications for incorrect answers.
Implementation Barriers
Technical limitations of LLMs
LLMs can generate incorrect answers, struggle with understanding code execution details, and may require improvements in prompting techniques.
Proposed Solutions: Future research could focus on enhancing model performance and accuracy through improved prompting techniques.
Educational integration challenges
Incorporating LLMs into existing curricula may face resistance or require adjustments in teaching methodologies.
Proposed Solutions: Developing clear guidelines and training for educators on how to effectively use LLMs in the classroom.
Project Team
Teemu Lehtinen
Researcher
Charles Koutcheme
Researcher
Arto Hellas
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Teemu Lehtinen, Charles Koutcheme, Arto Hellas
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai