Towards Responsible Development of Generative AI for Education: An Evaluation-Driven Approach

LearnLM-Tutor

Context: AI tutoring for learners in various educational settings, including programming courses at Arizona State University, academic support, and interactions with educational videos. The AI engages learners through conversational methods, providing support tailored to their specific needs and learning contexts.

Implementation: LearnLM-Tutor was developed through participatory methods, incorporating supervised fine-tuning with pedagogically informed data. The development process involved user feedback and safety evaluations to enhance its effectiveness and adaptability in educational scenarios.

Outcomes: The implementation has led to improved learner engagement, satisfaction, and pedagogical effectiveness, showing higher ratings on clarity, engagement, and adaptability in AI tutor interactions. There is a reported fostering of a supportive learning environment.

Challenges: Challenges faced include mitigating risks of biased outputs, preventing premature direct answers, managing potential emotional dependency on the AI, and accurately assessing the complexity of human-like pedagogical qualities.

Technical Barrier

Difficulty in verbalizing pedagogical intuitions into effective AI prompts and accurately simulating human-like pedagogical interactions.

Proposed Solutions: Developing a comprehensive set of pedagogical benchmarks, collaborating with educators to inform AI development, utilizing comprehensive evaluation criteria, and making iterative improvements based on learner feedback.

Evaluation Barrier

Lack of established evaluation practices for assessing AI in educational contexts.

Proposed Solutions: Creating a suite of pedagogical benchmarks that include both quantitative and qualitative measures to assess AI tutors.

Access Barrier

Equitable access to quality education and AI tools remains a significant challenge.

Proposed Solutions: Working towards democratizing access to AI technology and ensuring it meets the needs of diverse learners.

Technical

The AI tutor may produce biased or incorrect outputs, leading to misinformation.

Proposed Solutions: Implementing safety fine-tuning, regular evaluations against bias benchmarks, and curating high-quality training data.

Privacy and Ethical

Concerns over data privacy and potential surveillance of student interactions.

Proposed Solutions: Establishing strict data usage policies, ensuring anonymization of user data, and conducting transparency audits.

Pedagogical

Difficulty in maintaining effective pedagogical practices in AI responses, such as promoting critical thinking.

Proposed Solutions: Incorporating feedback from educators and continuous adjustments to the AI's pedagogical strategies.

Engagement Barrier

Learner engagement can be inconsistent, affecting the quality of interactions.

Proposed Solutions: Implementing strategies to promote active learning and adapt to the student's emotional state.

Project Contact: Dr. Jianhua Yang

LLM Model Version: gpt-4o-mini-2024-07-18

Analysis Provider: Openai