Can ChatGPT pass a physics degree? Making a case for reformation of assessment of undergraduate degrees
Project Overview
The document examines the role of generative AI, specifically ChatGPT (GPT-4), in the context of education by evaluating its performance in passing a UK Physics undergraduate degree, which underscores both the potential and challenges of AI in academic settings. It highlights the impressive capabilities of AI in completing various academic tasks while also acknowledging its limitations, particularly in practical laboratory work and oral assessments. The authors argue that the rise of AI necessitates a reform in assessment practices to maintain academic integrity and effectively prepare students for future employment. They propose a dual approach that involves developing robust assessment methods alongside the integration of AI tools into the educational framework, thereby ensuring that the benefits of AI are harnessed while addressing its challenges. This strategic combination aims to enhance learning outcomes and equip students with the necessary skills to thrive in an increasingly AI-driven job market.
Key Applications
ChatGPT (GPT-4)
Context: Used in undergraduate assessments including coursework and examinations in Quantum Physics, Astrophysics, and programming tasks in Computer Science.
Implementation: GPT-4 was employed to generate responses to coursework and exam questions, as well as to provide solutions for programming tasks in Python. The approach included a 'maximal intelligent cheating' strategy for assessments and focused on generating content for both theoretical and practical questions.
Outcomes: GPT-4 achieved passing grades in various modules, performed exceptionally well in coding tasks, and scored reasonably in descriptive questions. However, it struggled with practical observation tasks, laboratory practicals, and complex logical problems.
Challenges: Inability to perform practical laboratory skills, conduct physical observations, and handle oral examinations (vivas). Additionally, there were limitations in multi-step reasoning and complex logical problems.
Implementation Barriers
Ethical and Integrity Concerns
The use of AI like ChatGPT raises significant concerns about academic integrity, the validity of assessments, and the difficulty in detecting AI-generated responses in student work, especially as outputs can be disguised.
Proposed Solutions: Implementing strict invigilated assessments and vivas to ensure authenticity, along with developing new detection tools and methods for identifying AI use in academic writing.
Technical Limitations
ChatGPT struggles with tasks requiring hands-on practical skills and complex diagrammatic questions.
Proposed Solutions: Focus on assessments that cannot be easily replicated by AI, such as laboratory work.
Project Team
Kevin A. Pimbblet
Researcher
Lesley J. Morrell
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Kevin A. Pimbblet, Lesley J. Morrell
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai