The Robots are Here: Navigating the Generative AI Revolution in Computing Education

Generative AI Tools for Programming Assistance

Context: Various programming education contexts, including introductory courses, collaborative coding assignments, live coding sessions, and lab exercises, targeting students and educators.

Implementation: Integration of Large Language Models (LLMs) like GitHub Copilot, ChatGPT, and specialized tools such as CodeHelp into programming curricula. This includes using LLMs for source code generation, interpretation, collaborative coding, automated generation of personalized teaching materials, and providing assistance during live coding sessions and lab exercises.

Outcomes: ['Improved coding assistance and personalized learning experiences.', 'Enhanced student engagement and understanding of error messages.', 'Increased productivity and understanding of course material.', 'Promotion of independent learning while providing support.']

Challenges: ['Potential for over-reliance on AI tools, leading to confusion and hindering skill development.', 'Quality control of AI-generated content and potential inaccuracies.', 'Concerns regarding academic integrity and misuse of AI in secure assessments.']

Evaluation of LLM Performance in Programming

Context: Introductory programming courses, specifically for undergraduate students at institutions like the United States Air Force Academy.

Implementation: Utilization of datasets like the FalconCode dataset to evaluate LLMs (e.g., GPT-3.5 and GPT-4) by extracting problem statements and unit tests, generating solutions using LLMs, and assessing correctness against the unit tests.

Outcomes: ['Insights into LLM capabilities and performance metrics.', 'Identification of factors affecting LLM success rates in programming tasks.']

Challenges: ['Low performance rates due to missing instructions and inadequate problem statements.', 'Need for improved data quality to facilitate better model outputs.']

Guidelines for Appropriate Use of Generative AI Tools

Context: Educational settings where students are learning programming and software development, particularly during lab exercises and assessments.

Implementation: Establishing clear guidelines for the use of generative AI tools, allowing them for learning during lab exercises and assignments while prohibiting their use in secure assessments to prevent academic misconduct.

Outcomes: ['Students learn to utilize AI tools responsibly in their future work.', 'Encouragement of ethical practices in programming education.']

Challenges: ['Ensuring students do not misuse AI tools during assessments.', 'Balancing the benefits of AI assistance with the need for independent skill development.']

Implementation Challenge

Difficulty in ensuring academic integrity as students may misuse generative AI tools for assignments, alongside concerns about the potential for cheating with LLM tools.

Proposed Solutions: Educators need to develop new teaching strategies and assessment methods to maintain academic integrity, including clear guidelines for appropriate use of LLMs in assessments and emphasis on learning processes over final products.

Ethical Concern

Concerns regarding the quality and reliability of AI-generated content, including potential inaccuracies and ethical use of AI tools in education.

Proposed Solutions: Establish guidelines for using AI tools in education, with explicit policies on when and how GenAI tools can be used, alongside educational content on the risks and appropriate uses of GenAI.

Student Learning Issues

Students may become over-reliant on AI tools, hindering their understanding of programming concepts and leading to a lack of motivation to engage with learning tasks.

Proposed Solutions: Instructors should provide scaffolding and explicit teaching about the appropriate use of AI tools, encouraging mastery of foundational tasks before using AI tools for more complex problems.

Implementation Barrier

Difficulty in integrating LLMs into existing curricula and assessment methods due to their rapid evolution.

Proposed Solutions: Regular updates to curricula and assessment methods to include LLM capabilities; professional development for educators on using LLMs effectively.

Resource Barrier

Access inequalities, as not all students have the same access to LLM tools due to subscription costs or internet availability.

Proposed Solutions: Institutional support for providing access to LLM tools for all students; consideration of access issues in course design.

Data Quality & Dataset Availability

Insufficient information in datasets, such as missing starter code and external resources, limits the ability of LLMs to generate correct solutions, and there is difficulty in finding quality datasets specifically designed for LLM code generation research.

Proposed Solutions: Future datasets should include full problem descriptions, necessary resources, and clear test cases to allow for effective evaluation of LLM capabilities; encouragement for the community to curate and maintain diverse and comprehensive datasets that cater to various programming tasks and languages.

Quality Barrier

Content generated by AI may be of poor quality, contain bugs, or not meet assignment requirements.

Proposed Solutions: Students should double-check AI-generated content and ensure they understand the material before submission.

Motivational Barrier

Over-reliance on AI tools can lead to a lack of motivation to engage with learning tasks.

Proposed Solutions: Encouraging mastery of foundational tasks before using AI tools for more complex problems.

Project Contact: Dr. Jianhua Yang

LLM Model Version: gpt-4o-mini-2024-07-18

Analysis Provider: Openai