KCluster: An LLM-based Clustering Approach to Knowledge Component Discovery
Project Overview
The document explores the application of generative AI in education through the introduction of KCluster, an innovative algorithm designed for knowledge component (KC) discovery. By utilizing large language models (LLMs), KCluster enhances the process of mapping assessment questions to KCs, addressing the inefficiencies and limitations of traditional manual KC modeling approaches. It significantly reduces the need for human intervention while simultaneously generating more accurate predictive models of student performance. Evaluations of KCluster demonstrate its superiority over existing methods, indicating improved alignment with expert-designed KC models and a positive impact on student learning outcomes. Overall, the findings underscore the transformative potential of generative AI in refining educational assessment and enhancing student success.
Key Applications
KCluster
Context: Educational settings involving large question banks for assessments, targeting instructors and educational data analysts.
Implementation: KCluster uses a large language model (Phi-2) to calculate question congruity and applies clustering algorithms to group similar questions into knowledge components with minimal human involvement.
Outcomes: KCluster demonstrates superior performance in predicting student responses compared to expert-designed models and reduces the number of redundant KC labels, enabling better instructional design.
Challenges: The reliance on LLMs means that performance can vary based on the model used; there can also be issues with the specificity and clarity of the generated KC labels.
Implementation Barriers
Technical
The challenge of maintaining high-quality KC models in the face of rapid question generation by generative AI tools.
Proposed Solutions: Developing robust algorithms like KCluster that automate KC discovery and reduce the burden on instructors.
Resource-related
Limited access to high-quality, expert-designed KC models for large datasets, which hinders effective data analysis.
Proposed Solutions: Enhancing automated methods to create KC models from existing data without needing expert intervention.
Project Team
Yumou Wei
Researcher
Paulo Carvalho
Researcher
John Stamper
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Yumou Wei, Paulo Carvalho, John Stamper
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai