Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses
Project Overview
The document explores the transformative role of generative AI, specifically GPT-4, in education, particularly in introductory and intermediate Python programming courses. It reveals that GPT-4 demonstrates substantial advancements over earlier iterations of large language models (LLMs), achieving notable success in both multiple-choice questions and coding tasks. These improvements suggest a need for educators to reconsider traditional assessment methods, shifting from conventional testing approaches to fostering more meaningful and engaging learning experiences. The findings indicate that while AI can enhance educational outcomes, it also necessitates a reevaluation of pedagogical strategies to ensure that assessments align with the capabilities of modern generative AI, thereby promoting deeper understanding and practical application of knowledge among students.
Key Applications
GPT-4 for passing programming assessments
Context: Introductory and intermediate Python programming courses in higher education
Implementation: GPT-4 was tested against assessments from three Python courses with diverse question formats including MCQs and coding exercises.
Outcomes: GPT-4 showed significant improvement, successfully passing assessments that earlier models (GPT-3 and GPT-3.5) could not.
Challenges: GPT-4 struggled with certain types of MCQs, particularly those requiring multi-hop reasoning and coding tasks with external dependencies.
Implementation Barriers
Technical Limitation
GPT-4 has limitations in understanding multi-hop reasoning and struggles with code-related tasks that require external tools.
Proposed Solutions: Educators can redesign assessments to focus more on practical coding tasks that require integrating information from multiple sources.
Ethical Concerns
There is a concern about students relying too heavily on AI models to complete assessments, undermining their learning.
Proposed Solutions: Promote academic honesty and encourage a culture of original work and personal effort in education.
Project Team
Jaromir Savelka
Researcher
Arav Agarwal
Researcher
Marshall An
Researcher
Chris Bogart
Researcher
Majd Sakr
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Jaromir Savelka, Arav Agarwal, Marshall An, Chris Bogart, Majd Sakr
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai