Lost in Cultural Translation: Do LLMs Struggle with Math Across Cultural Contexts?
Project Overview
The document explores the application of generative AI, specifically Large Language Models (LLMs), in the educational domain, focusing on their performance with culturally adapted math problems from the GSM8K dataset. It reveals that while LLMs generally excel in mathematical tasks, they face significant challenges when the cultural context of the problems changes, leading to inaccuracies in reasoning despite the mathematical content being constant. The findings indicate that larger models tend to perform better across various contexts, though they still exhibit drops in accuracy, underscoring the crucial role of cultural familiarity in mathematical problem-solving. This highlights the necessity for diverse training data to enhance the robustness of LLMs, suggesting that integrating multiple cultural perspectives could improve their effectiveness in real-world educational applications. Overall, the document emphasizes the potential of generative AI in education while also pointing out the limitations that arise from cultural differences, ultimately advocating for more inclusive and contextually aware AI training practices to improve educational outcomes.
Key Applications
Culturally adapted math problems using GSM8K dataset
Context: Educational assessment in diverse cultural contexts, targeting researchers and educators evaluating AI performance.
Implementation: Synthetic datasets created by modifying the GSM8K dataset to include cultural references relevant to different countries.
Outcomes: The study revealed that LLMs perform better on the original GSM8K dataset compared to its culturally adapted versions, demonstrating the impact of cultural familiarity on reasoning.
Challenges: LLMs exhibited significant performance drops on culturally adapted math problems, indicating sensitivity to cultural shifts, tokenization issues, and biases in training data.
Implementation Barriers
Technical
LLMs struggle with tokenization and reasoning when faced with culturally adapted questions.
Proposed Solutions: Improved representation of diverse cultural contexts in training datasets and better tokenization strategies.
Cultural Bias
Models trained primarily on Western-centric data fail to generalize well to culturally distinct scenarios. It is essential to incorporate more diverse and representative training data to enhance model adaptability to various cultural contexts.
Proposed Solutions: Incorporate more diverse and representative training data to enhance model adaptability to various cultural contexts.
Project Team
Aabid Karim
Researcher
Abdul Karim
Researcher
Bhoomika Lohana
Researcher
Matt Keon
Researcher
Jaswinder Singh
Researcher
Abdul Sattar
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Aabid Karim, Abdul Karim, Bhoomika Lohana, Matt Keon, Jaswinder Singh, Abdul Sattar
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai