An Interaction Design Toolkit for Physical Task Guidance with Artificial Intelligence and Mixed Reality
Project Overview
The document explores the application of generative AI in education through the development of the Interaction Design Toolkit for Physical Task Guidance (PTG), known as MixITS-Kit. This innovative toolkit combines Artificial Intelligence and Mixed Reality to improve the acquisition of physical skills by addressing user interaction challenges in real-world settings. It provides a structured framework that includes design considerations, design patterns, and an interaction canvas, which together facilitate the creation of intelligent task support systems tailored for educational and training environments. By promoting effective design practices, the MixITS-Kit aims to enhance the user experience and effectiveness of AI-driven MR applications, ultimately leading to better learning outcomes and skill development in various educational contexts. The findings highlight the potential of integrating advanced technologies like AI and MR to support learners in practical tasks, showcasing a significant advancement in educational methodologies that leverage generative AI for skill enhancement.
Key Applications
MixITS-Kit
Context: Graduate design course for Physical Task Guidance systems
Implementation: Developed through a 10-week course where students created low-fidelity prototypes using the toolkit to address design challenges.
Outcomes: Improved understanding of user-AI interactions, creation of design patterns, and enhanced collaborative problem-solving among participants.
Challenges: Complexity of integrating AI and MR, user experience issues, and the need for ongoing adaptation of designs based on user feedback.
Implementation Barriers
Technical Barriers
Challenges associated with the engineering complexity and high costs of integrating AI into real-time MR applications.
Proposed Solutions: Using low-fidelity prototypes to explore design challenges and leveraging existing technologies and infrastructures.
User-Centric Barriers
Difficulty in shifting from a technology-driven mindset to a user-centered design approach.
Proposed Solutions: Engaging students in role-playing exercises to better understand user needs and technology limitations.
Project Team
Arthur Caetano
Researcher
Alejandro Aponte
Researcher
Misha Sra
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Arthur Caetano, Alejandro Aponte, Misha Sra
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai