CIKMar: A Dual-Encoder Approach to Prompt-Based Reranking in Educational Dialogue Systems
Project Overview
The document explores the application of generative AI in education through the development and assessment of CIKMar, an educational dialogue system powered by the Gemma Language model and a Dual-Encoder ranking system. CIKMar is designed to produce relevant and accurate responses tailored to educational needs, demonstrating the promise of smaller language models in making advanced educational AI more accessible. However, the study also notes specific challenges, particularly the Dual-Encoder's inclination to favor theoretical responses over practical ones, which could impact the effectiveness of the system in real-world educational scenarios. Overall, the findings underscore both the potential benefits and the limitations of integrating generative AI in educational settings, highlighting the importance of continuous refinement to enhance the practical applicability of AI-generated content in learning environments.
Key Applications
CIKMar - an educational dialogue system powered by the Gemma Language model
Context: This system is designed for educational dialogue, particularly in teacher-student interactions to provide personalized learning experiences.
Implementation: CIKMar uses a Dual-Encoder ranking system with BERT and SBERT to rerank responses generated by the Gemma model based on context relevance.
Outcomes: Achieved a recall and F1-score of 0.70, indicating effective response generation and context relevance.
Challenges: The Dual-Encoder may prioritize theoretical responses over practical application, leading to less relevant rankings.
Implementation Barriers
Technical Barrier
Large language models are often impractical due to their size and memory requirements, making them inaccessible for many users.
Proposed Solutions: Utilizing smaller models like Gemma that require less computational power and RAM.
Response Quality Barrier
Inconsistently accurate and contextually appropriate responses from the model.
Proposed Solutions: Implementing a prompt-and-rerank approach to optimize response generation based on contextual understanding.
Project Team
Joanito Agili Lopo
Researcher
Marina Indah Prasasti
Researcher
Alma Permatasari
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Joanito Agili Lopo, Marina Indah Prasasti, Alma Permatasari
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai