Metadata Augmentation using NLP, Machine Learning and AI chatbots: A comparison
Project Overview
The document explores the integration of generative AI and machine learning in education, particularly focusing on the application of AI chatbots and large language models (LLMs) to improve metadata augmentation in academic libraries. It evaluates these AI tools against traditional natural language processing (NLP) and machine learning methods for document classification, revealing that AI chatbots demonstrate superior accuracy but face challenges related to reliability and consistency, especially in tasks involving counting and labeling. The findings highlight the promising potential of incorporating AI technologies into data curation workflows, while also underscoring the necessity for human oversight to ensure quality and reliability. Overall, the document underscores the transformative role of generative AI in educational settings, advocating for a balanced approach that leverages the strengths of AI while mitigating its limitations through human intervention.
Key Applications
AI Approaches for Metadata Classification
Context: Academic libraries for metadata classification tasks, particularly for organizing and classifying theses based on content representation. The implementations span various AI techniques aimed at improving the accuracy and efficiency of document classification.
Implementation: A combination of AI chatbots, BERT fine-tuning, and XGBoost applied to metadata classification. AI chatbots utilized a zero-shot approach for classification, while BERT was fine-tuned on thesis datasets to enhance classification accuracy, and XGBoost was applied using a TF-IDF matrix for feature extraction. Each method was used to evaluate document classification performance in academic settings.
Outcomes: The implementations demonstrated varying degrees of effectiveness: AI chatbots showed higher accuracy than traditional machine learning methods, the fine-tuned BERT model achieved a reliable balance of accuracy, and XGBoost served as a benchmark but underperformed compared to LLMs.
Challenges: Common challenges included inconsistencies in AI chatbot responses, limited performance of BERT with few training samples for certain labels, and XGBoost's poor performance with limited training data, highlighting the need for larger datasets to improve accuracy.
Implementation Barriers
Technical
AI chatbots exhibited conceptual errors and inconsistencies in output, such as miscounting lines.
Proposed Solutions: Potential solutions include using API versions of chatbots for more reliable processing and implementing better prompt strategies.
User Experience
The user experience varied significantly between different AI models, affecting ease of use.
Proposed Solutions: Providing clear documentation and training for users on chatbot interactions could enhance usability.
Project Team
Alfredo González-Espinoza
Researcher
Dom Jebbia
Researcher
Haoyong Lan
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Alfredo González-Espinoza, Dom Jebbia, Haoyong Lan
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai