MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMs

AI-driven personalized learning and instruction tuning

Context: K-12 education and higher education, targeting students with varying learning paces and researchers focusing on AI model improvement.

Implementation: Integration of AI algorithms to adapt content based on individual performance and development of instruction tuning datasets for enhancing AI model capabilities in handling dialogues and personalized learning.

Outcomes: Improved student engagement and learning outcomes through tailored content, alongside significant improvements in performance metrics for AI models handling long-context dialogues and multi-image recognition.

Challenges: Ensuring data privacy, managing the digital divide among students, and maintaining quality control during dataset construction.

Generative writing assistants

Context: Higher education courses focusing on creative writing, where students utilize AI tools.

Implementation: Students use AI tools to generate writing prompts and refine their narratives, enhancing their writing skills.

Outcomes: Enhanced creativity and confidence in writing skills among students.

Challenges: Potential over-reliance on AI for creative processes, which may hinder original thought.

Multi-turn multi-image dialog understanding benchmarks

Context: Educational context involves training and evaluation of AI models, targeting researchers and developers in AI technology.

Implementation: Developed a comprehensive benchmark with extensive multi-turn and multi-image dialogues sourced from Wikipedia, utilizing clustering algorithms and human annotation.

Outcomes: Enhanced AI's ability to engage in coherent multi-turn conversations with visual context, significantly improving performance metrics in comparison to existing models.

Challenges: Existing models struggle with multi-turn and multi-image contexts, leading to performance gaps between open-source and proprietary models.

Technical Barrier

The performance of existing open-source LVLMs is significantly lower than that of closed-source counterparts, primarily due to a lack of quality instruction-tuning data. Additionally, integration of AI tools into existing educational systems may be complex and resource-intensive.

Proposed Solutions: The introduction of MMDU-45k aims to provide a substantial dataset for finetuning open-source models, addressing the need for high-quality training data. Investing in training for educators and providing technical support for seamless integration.

Social Barrier

The datasets focus primarily on English and may perpetuate biases present in the source material, limiting their effectiveness across diverse populations.

Proposed Solutions: Future expansions of the benchmark should consider multilingual support and efforts to minimize biases in the dataset.

Ethical Barrier

Concerns around data privacy and the ethical implications of using AI in educational settings.

Proposed Solutions: Establishing clear guidelines and policies for data usage and ethical AI deployment in education.

Project Contact: Dr. Jianhua Yang

LLM Model Version: gpt-4o-mini-2024-07-18

Analysis Provider: Openai