Testing the effectiveness of saliency-based explainability in NLP using randomized survey-based experiments
Project Overview
The document explores the role of generative AI in education, emphasizing the significance of explainability, particularly in Natural Language Processing (NLP) applications such as essay scoring. It underscores the necessity for transparency in AI models to foster trust among educators, students, and other stakeholders while addressing potential biases inherent in these systems. The study assesses the effectiveness of saliency-based explainability methods through survey responses, indicating that while these techniques can enhance users' comprehension of model predictions, they may inadvertently lead to misinterpretations, causing users to accept inaccurate explanations due to cognitive biases. Overall, the findings stress the need for careful implementation of explainability measures in AI-driven educational tools to ensure they are not only effective but also trustworthy and free from misleading information.
Key Applications
NLP-based essay scoring
Context: Educational context where AI models evaluate student essays, targeting educators and students.
Implementation: The study implemented a survey to assess the effectiveness of saliency-based explanations using a pre-trained RoBERTa model for sentiment analysis.
Outcomes: Participants generally agreed with model predictions when provided with saliency-based explanations, indicating that these explanations helped in understanding model behavior.
Challenges: Participants often agreed with incorrect predictions due to biases, suggesting limitations in the effectiveness of current explainability methods.
Implementation Barriers
Cognitive Bias
Users may agree with incorrect model predictions due to cognitive biases, such as the tendency to fill in blanks or the anchoring effect.
Proposed Solutions: Future work should explore more verifiable and trustworthy model explanations that can help users assess model accuracy accurately.
Project Team
Adel Rahimi
Researcher
Shaurya Jain
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Adel Rahimi, Shaurya Jain
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai