ProfiLLM: An LLM-Based Framework for Implicit Profiling of Chatbot Users
Project Overview
The document introduces ProfiLLM, an innovative framework designed for implicit user profiling within chatbot interactions, particularly in the IT and cybersecurity sectors. It addresses the shortcomings of current chatbots in delivering personalized responses that reflect user expertise and proposes a dynamic taxonomy for better user profiling. The findings reveal notable enhancements in profiling accuracy, with a substantial decrease in prediction errors observed after a single interaction. This flexibility of the ProfiLLM framework enables its application across various domains by simply adjusting the taxonomy, showcasing its potential to improve user engagement and education through tailored interactions. Overall, the document emphasizes the transformative role of generative AI in education by enhancing the adaptability and responsiveness of chatbots to meet individual learning needs effectively.
Key Applications
ProfiLLM
Context: IT/cybersecurity domain for users seeking technical support
Implementation: Developed an ITSec-adapted chatbot using the ProfiLLM framework to infer user profiles through dynamic interactions.
Outcomes: Achieved a 55-65% improvement in accuracy of user proficiency predictions after one interaction.
Challenges: Initial profiling may not be accurate without prior user data; users may have varied responses leading to fluctuations in accuracy.
Implementation Barriers
Technical and User Engagement Barrier
Chatbots often fail to personalize responses based on users' technical expertise and learning styles, leading to misalignment between user expertise and chatbot responses. This can result in frustration and reduced engagement.
Proposed Solutions: Implementing dynamic and implicit user profiling frameworks like ProfiLLM to adapt to user interactions and enhance adaptive responses to fit users' knowledge levels, thereby improving engagement and satisfaction.
Project Team
Shahaf David
Researcher
Yair Meidan
Researcher
Ido Hersko
Researcher
Daniel Varnovitzky
Researcher
Dudu Mimran
Researcher
Yuval Elovici
Researcher
Asaf Shabtai
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Shahaf David, Yair Meidan, Ido Hersko, Daniel Varnovitzky, Dudu Mimran, Yuval Elovici, Asaf Shabtai
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai