How critically can an AI think? A framework for evaluating the quality of thinking of generative artificial intelligence
Project Overview
The document explores the transformative role of generative AI, particularly large language models such as ChatGPT-4, in educational assessment design. It highlights the necessity for educators to grasp both the potential and limitations of generative AI in replicating critical thinking abilities. To address this, the MAGE framework is introduced as a tool to assess the susceptibility of various assessment tasks to completion by generative AI, ensuring that evaluations genuinely reflect students' critical thinking rather than simply their capacity for information retrieval. This approach is intended to foster authentic learning experiences, emphasizing the importance of designing assessments that challenge students to engage in deeper cognitive processes and develop essential skills. The findings suggest that while generative AI can enhance educational practices, it also necessitates a reevaluation of assessment strategies to maintain academic integrity and promote genuine learning outcomes.
Key Applications
MAGE Framework (Mapping, AI vulnerability testing, Grading, Evaluation)
Context: Higher education assessments, targeting educators designing assessments for students.
Implementation: Educators use the MAGE framework to map assessment tasks to cognitive skills, test AI vulnerability, grade responses, and evaluate task effectiveness.
Outcomes: Provides a method for educators to identify vulnerabilities in assessments, ensuring that tasks assess critical thinking skills rather than just factual recall.
Challenges: Challenges include the continuous evolution of generative AI capabilities and the difficulty in designing assessments that minimize AI vulnerability.
Implementation Barriers
Technical Barrier
Generative AI can produce assessment artifacts that fulfill task requirements without the requisite cognitive skills.
Proposed Solutions: Rethinking assessment tasks to focus on higher-order thinking skills that generative AI finds difficult to emulate.
Integrity Barrier
The authorship of assessment tasks is potentially compromised when students use generative AI.
Proposed Solutions: Implementing strategies such as invigilation and redesigning tasks to mitigate the risk of AI-based submissions.
Project Team
Luke Zaphir
Researcher
Jason M. Lodge
Researcher
Jacinta Lisec
Researcher
Dom McGrath
Researcher
Hassan Khosravi
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Luke Zaphir, Jason M. Lodge, Jacinta Lisec, Dom McGrath, Hassan Khosravi
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai