Survey Results Analysis

Introduction

The impact of AI technologies on education has traditionally been most evident in essay-based subjects, where generative AI tools have been used to assist in drafting, editing, and analysing text. However, recent advancements in AI models, particularly in mathematical reasoning and complex problem-solving, signal a significant shift towards affecting mathematics education as well. These new capabilities raise important questions about AI’s role in mathematical assignments, which rely heavily on precision, logical structure, and accuracy.

For example, OpenAI's latest reasoning model has demonstrated advanced mathematical problem-solving skills by scoring in the 83rd percentile on the International Mathematics Olympiad (IMO) qualifying exams and achieving the 89th percentile in Codeforces coding competitions (Source). Such performances, previously unimaginable for AI, suggest that AI tools can now handle complex mathematical tasks with a high degree of competence, prompting a need to explore their impact on student learning and assessment in mathematics.

This study, conducted in June 2024, focuses specifically on how these new AI capabilities are influencing the use of AI in mathematics & statistics coursework. At the time of the survey, GPT-4o was the most advanced model available, though it had only recently been released. The most commonly used model for assignments was the free, now-legacy GPT-3.5-Turbo, utilised by 74 out of 145 students. Given the release date of GPT-3.5-Turbo in 2022 and its known limitations in mathematical accuracy, its prevalence may contribute to a cautious view of AI in this field by students. Nevertheless, this wariness may also reflect a critical understanding of large language models' general tendency to produce "hallucinations," where the model generates confidently incorrect answers.

This preface frames the survey's purpose: to examine the shifting role of AI in mathematics education and its impact on student use and perceptions of AI tool use for graded assignments.

How the Survey Data is Arranged

The survey data is organised into six key sections, each addressing a specific aspect of AI use in mathematics education. These sections are designed to provide a comprehensive analysis of the survey responses, focusing on the following areas: ethical considerations, academic integrity, the impact on degree value, student attitudes towards AI, integration of AI in assignments, usage patterns, and future concerns. Each section contains detailed analysis and insights based on the survey responses, highlighting the different perspectives of AI users and non-users.

To facilitate a deeper understanding, each section includes data-driven analysis and specific recommendations tailored to address the unique challenges and opportunities presented by AI in mathematics education. At the bottom of this landing page, these recommendations are discussed more holistically, providing a broader context for their implementation and impact.

Key Sections of the Analysis

Navigate through the detailed survey findings by exploring the following sections. You can access the pages by clicking on the titles.

Conclusion

The survey of 145 Mathematics and Statistics students at Warwick University reveals a complex and evolving landscape regarding the use of AI in academic assignments. The data highlights a significant divide between AI users and non-users, with profound implications for education and assessment practices.

Key Insights:

• Ethical Concerns and Perceptions of Cheating: A majority of students, especially non-AI users (76%), perceive the use of AI in assignments as cheating (Ethical Considerations and Academic Integrity). This reflects deep-seated concerns about fairness and academic honesty.
• Fears of Degree Devaluation: Many students believe that AI could undermine the value of their degrees, with 83% of non-AI users expressing this concern (Impact on Academic Standing and Degree Value). This anxiety persists even as AI capabilities advance.
• Skepticism About AI's Accuracy: Despite modern AI models like GPT-4o achieving around 70% correctness on assignment work (AI University Assignments Performance), a significant proportion of students—77% of AI users and 64% of non-AI users—believe that AI usually gives wrong answers in maths and statistics (Concerns and Future Directions). This indicates a gap between AI's actual capabilities and student perceptions.
• Continued Use Despite Concerns: Despite concerns, many students continue to use AI tools for assignments. While 59% of students reported using AI to help complete assignments, their usage patterns vary, with most using it only sometimes or rarely (Usage Patterns and Trends).
• Lack of AI Literacy and Awareness: Students are largely uninformed about the capabilities of modern AI models. The reliance on legacy tools like GPT-3.5 Turbo by 74 students suggests a need for improved AI literacy and awareness of current technologies (Usage Patterns and Trends).
• Desire for Clear Guidelines and Ethical Frameworks: There is a strong demand for clear policies on AI use in education. Both AI users and non-users express the need for guidance to navigate ethical concerns and academic integrity (Ethical Considerations and Academic Integrity).
• Uncertainty About AI's Role in Education: Approximately 35% of students are unsure about how they feel regarding the use of AI in assignments (Concerns and Future Directions). This ambivalence highlights the need for open dialogue and education.

Moving Forward: Navigating the Challenges and Seizing the Opportunities

The integration of AI into mathematics and statistics education presents a complex and evolving challenge. As AI technologies advance rapidly, they bring both significant opportunities and profound concerns. Recognising the seriousness and longevity of these challenges is essential. There are no clear-cut answers or simple solutions; instead, we must engage in continuous dialogue, experimentation, and adaptation involving all stakeholders. Educators, students, and institutions must collaboratively navigate this uncertain path, acknowledging that the landscape will continue to shift.

In response to these complexities, we propose a set of comprehensive recommendations that encompass the insights and suggestions from our detailed analysis. These recommendations are interconnected and require a collective effort to implement effectively. They are designed to address the multifaceted nature of AI integration in education, ensuring that we harness its benefits while maintaining academic integrity and educational excellence.

1. Develop Comprehensive Guidelines and Policies on AI Use
   • Establish Clear Guidelines on AI Use: Create detailed policies that explicitly outline acceptable and unacceptable uses of AI in assignments. Provide concrete examples and scenarios to clarify expectations for both students and educators.
   • Communicate Effectively: Ensure that these guidelines are disseminated widely and understood by all stakeholders to maintain consistency and fairness across all academic settings.
   • Develop Flexible Policies: Design policies that are adaptable to the rapid advancements in AI technology, allowing for timely updates and adjustments as needed.
2. Foster Open Dialogue and Collaboration
   • Encourage Open Dialogue Among Students and Educators: Facilitate ongoing discussions to align perceptions and expectations about AI use. Provide platforms such as forums, workshops, and collaborative projects to bridge understanding gaps.
   • Promote Collaboration: Involve all stakeholders in developing strategies for AI integration, acknowledging the complexities and working together to find balanced solutions.
3. Continuously Monitor and Adapt AI Integration Strategies
   • Evaluate and Adapt AI-Proofing Strategies: Regularly assess how AI tools perform on current assignments. Adjust assignment designs and assessment methods to ensure they promote critical thinking and are less susceptible to AI replication.
   • Implement Regular Feedback Mechanisms: Establish continuous feedback loops with students and faculty to monitor the effectiveness of policies and practices, making data-driven adjustments as necessary.
   • Stay Informed on AI Developments: Keep abreast of rapid AI advancements to adapt strategies proactively, recognising that this is an ongoing process without a definitive endpoint.
4. Enhance AI Literacy and Awareness
   • Enhance Education and Awareness Initiatives: Offer workshops, seminars, and resources to increase familiarity with AI tools. Address both the capabilities and limitations of AI, promoting informed and responsible use.
   • Improve AI Literacy Through Education: Educate students and faculty on effective and ethical AI use, helping them leverage these tools while upholding academic integrity.
   • Implement Programmes to Inform About AI Developments: Regularly update the academic community on the latest AI advancements to reduce misconceptions and scepticism.
5. Integrate AI Thoughtfully into the Curriculum
   • Redesign Assignments to Promote Critical Engagement: Craft assignments that require unique applications of concepts, critical analysis of AI-generated content, and reflection on the learning process.
   • Encourage Critical Engagement: Teach students to critically assess AI outputs, fostering skills that AI cannot easily replicate, such as critical thinking and originality.
   • Evaluate the Impact of AI Tools: Systematically assess how AI interacts with academic assignments to inform curriculum design and maintain academic rigour.
6. Support Research on AI's Impact on Education
   • Promote Further Research: Invest in studies exploring the long-term implications of AI in education to anticipate future challenges and inform policy development.
   • Support Ongoing Research: Align institutional strategies with broader research efforts to understand AI's effects on academic integrity and student perceptions.
7. Address Ethical Concerns and Ensure Academic Integrity
   • Develop Flexible Policies to Maintain Standards: Create adaptable policies that respond to AI advancements while upholding rigorous academic standards.
   • Address Accessibility and Equity Issues: Ensure equal access to AI tools and support, providing accommodations as needed to maintain fairness.
   • Provide Clear Guidelines on Ethical AI Usage: Alleviate fears related to academic integrity by defining acceptable AI use and addressing concerns about cheating.
8. Prepare Students for Future Careers
   • Provide Career Guidance: Offer support and information on how AI might impact future career paths in mathematics and statistics, helping students adapt to industry changes.
   • Enhance Skills Relevant in an AI-Influenced Job Market: Focus on developing critical thinking, problem-solving, and ethical reasoning skills that remain valuable despite AI advancements.
9. Maintain a Balanced Assessment Approach
   • Develop Flexible Integration Policies: Implement adaptable AI integration policies that respect both traditional and AI-supported assessment models, accommodating diverse comfort levels with AI.
   • Retain a Mix of Assessment Methods: Use a combination of assignments, exams, and practical evaluations to cater to different learning styles and mitigate AI misuse risks.
   • Evaluate the Impact of AI on Assessments: Regularly assess how AI tools affect assignment formats and adjust strategies to maintain academic integrity.

Conclusion

Addressing the challenges and opportunities presented by AI in mathematics and statistics education requires a multifaceted and adaptive approach. By acknowledging the complexities and engaging in continuous dialogue, we can develop strategies that harness the benefits of AI while safeguarding academic integrity. This process involves thoughtful policy development, collaborative efforts, and a commitment to ongoing research and adaptation.

It is crucial to proceed without making hasty, drastic changes. Instead, we should invest in experimental and flexible strategies, regularly updating our understanding and practices in response to the evolving AI landscape. Educators need support to stay informed about AI's performance on assignments and to engage critically with students and stakeholders in adjusting educational approaches accordingly.

This journey is unprecedented and challenging, but by recognising the seriousness and longevity of the problem, we can navigate this unclear path with diligence and innovation. Through collective effort and evidence-based strategies, we can ensure that the integration of AI enhances education while preparing students for a future where AI plays a significant role.

For a more detailed exploration of the nuances and recommendations discussed, it is important to acknowledge that there are no simple solutions to these concerns. There is a prevailing sentiment that changes must occur to safeguard academic integrity, but these changes should be measured and thoroughly explored rather than implemented hastily. Our recommendations aim to address these complexities and serve as a starting point for deeper discussions. While it is difficult to make broad generalisations without more data, this report provides an essential foundation for further exploration and consideration. Please refer to our comprehensive report: AI in Mathematics and Statistics Education: Recommendations and Future Directions.