Learning Agent-based Modeling with LLM Companions: Experiences of Novices and Experts Using ChatGPT & NetLogo Chat
Project Overview
The document explores the transformative role of generative AI, particularly Large Language Models (LLMs) like ChatGPT, in enhancing education, with a specific focus on programming education and agent-based modeling (ABM) using NetLogo. A study involving both experts and novices reveals significant differences in their perceptions and experiences with LLMs, highlighting a knowledge gap and the distinct needs for guidance and personalization in AI interfaces. Generative AI tools are shown to facilitate learning by aiding students in understanding programming concepts, debugging, and improving coding skills, thereby creating more engaging and tailored learning experiences. However, the document also addresses challenges such as usability and the overall effectiveness of these AI-driven educational tools. Ultimately, the findings underscore the potential of generative AI to revolutionize programming education while also emphasizing the necessity of addressing its limitations to maximize its benefits for learners at all levels.
Key Applications
AI-Assisted Programming Tools
Context: Utilized in introductory programming education and software development courses, these tools support students and educators in learning coding skills and enhance productivity in software development tasks.
Implementation: Integrated into programming courses and development environments, these AI tools (including ChatGPT, GitHub Copilot, and others) assist users in generating code, debugging, and troubleshooting by providing real-time feedback and suggestions. They are designed to be supportive tools that enhance the learning experience through interactive guidance.
Outcomes: Improved student engagement, enhanced productivity, and assistance in solving programming problems. Users report better support for coding and troubleshooting, with experts finding greater benefits from these tools compared to novices.
Challenges: Users face difficulties with prompt design, understanding AI limitations, maintaining code quality, and potential over-reliance on AI-generated outputs. Both novices and experts encounter issues with LLM hallucinations, particularly in evaluating and debugging AI-generated code.
Conversational Learning Agents
Context: Applied in online education environments to engage students through personalized interactions, providing support and guidance during their learning process.
Implementation: These scaffolding-based conversational agents (such as Sara) are used in online courses to create an interactive learning atmosphere, encouraging students to ask questions and engage with the material in a conversational manner.
Outcomes: Improved learning outcomes through personalized interactions, facilitating a deeper understanding of the course material.
Challenges: Challenges include maintaining conversational context and accurately understanding user queries to provide relevant and helpful responses.
Implementation Barriers
Knowledge Gap
Novices lack the conceptual knowledge of modeling, basic programming concepts, and debugging skills, making it difficult for them to use LLMs effectively. Additionally, students may struggle to effectively use AI tools or understand their limitations.
Proposed Solutions: Implement design interventions to provide clearer guidance, context-sensitive assistance, and personalized learning paths for novices. Provide training and resources to help users learn to interact with AI tools.
User Experience
LLMs often misinterpret user intentions, leading to less efficient interactions and reliance on AI-generated outputs without understanding.
Proposed Solutions: Incorporate more probing questions from LLMs to clarify user intentions and provide tailored responses.
Technical Barrier
Challenges with AI usability and effectiveness in providing accurate coding assistance.
Proposed Solutions: Improving AI training data and refining user interface designs.
Project Team
John Chen
Researcher
Xi Lu
Researcher
Michael Rejtig
Researcher
David Du
Researcher
Ruth Bagley
Researcher
Michael S. Horn
Researcher
Uri J. Wilensky
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: John Chen, Xi Lu, Michael Rejtig, David Du, Ruth Bagley, Michael S. Horn, Uri J. Wilensky
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai