Real-World Deployment and Evaluation of Kwame for Science, An AI Teaching Assistant for Science Education in West Africa
Project Overview
The document examines the implementation and assessment of Kwame for Science, an AI teaching assistant aimed at enhancing science education in West Africa, particularly in the context of high student-to-teacher ratios. By offering students immediate access to answers for science inquiries and past examination papers, Kwame employs a bilingual AI model to provide tailored educational support. User evaluations indicate a positive reception, marked by high accuracy rates in the assistance provided. However, the deployment of this AI system has encountered challenges, including copyright concerns, technical limitations, and issues related to user engagement. Overall, the findings suggest that generative AI has the potential to significantly improve access to educational resources and personalized learning experiences in regions facing educational resource constraints, although further efforts are needed to address the obstacles identified during its implementation.
Key Applications
Kwame for Science
Context: Science education for high school students in West Africa, specifically for the Integrated Science subject of the WASSCE.
Implementation: Deployed as a web app, it uses a question-answering feature powered by a neural network model (SBERT) to provide answers to science questions and access to past exam questions.
Outcomes: Achieved 87.2% top 3 accuracy in providing useful answers; engaged 750 users across 32 countries with 1.5K questions asked.
Challenges: Issues with copyright permissions for local textbooks, OCR limitations for scientific symbols, and low user feedback rates.
Implementation Barriers
Copyright Issues
Difficulty in obtaining access to local textbooks due to copyright concerns.
Proposed Solutions: Hired experts to provide answers to past national exam questions to supplement knowledge sources.
Technical Limitations
Challenges with formatting scientific and mathematical symbols and equations due to OCR technology limitations.
Proposed Solutions: Manual inspection and correction of scanned documents and hiring individuals for quality control.
User Engagement
Low rates of user feedback on answers provided (only 7.2% of questions received feedback), necessitating more effective feedback mechanisms.
Proposed Solutions: Plans to implement a forced rating feature to gather more feedback effectively.
Project Team
George Boateng
Researcher
Samuel John
Researcher
Samuel Boateng
Researcher
Philemon Badu
Researcher
Patrick Agyeman-Budu
Researcher
Victor Kumbol
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: George Boateng, Samuel John, Samuel Boateng, Philemon Badu, Patrick Agyeman-Budu, Victor Kumbol
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai