Debug Smarter, Not Harder: AI Agents for Error Resolution in Computational Notebooks
Project Overview
The document explores the integration of generative AI in education through an AI agent implemented in JetBrains' Datalore, which focuses on resolving errors in computational notebooks. This agent employs large language models (LLMs) to autonomously identify and fix errors by generating, editing, and executing code cells. A user study highlighted that participants found the agent's error resolution capabilities superior to traditional single-action solutions, indicating a positive reception of its functionality. However, users reported challenges with the interface, which negatively impacted their overall satisfaction and productivity. The findings suggest that while generative AI tools can enhance the educational experience by automating complex tasks, user interface design remains a critical aspect that needs improvement to fully leverage the benefits of such technology in educational settings.
Key Applications
AI Agent for error resolution in computational notebooks
Context: Educational context involving data science and programming for students and developers using Datalore.
Implementation: The agent is integrated into Datalore and allows users to initiate error resolution through a button once an error occurs in a code cell.
Outcomes: Users reported higher satisfaction with error resolution capabilities, and the agent effectively resolved most errors within one or two steps.
Challenges: Users found the user interface overly complex and difficult to track the agent's actions, leading to a feeling of loss of control.
Implementation Barriers
User Experience
Users reported difficulties in understanding the agent's actions and maintaining control over the notebook due to the fast-paced changes made by the AI.
Proposed Solutions: Improving the user interface for better clarity on which cells were edited, and providing more explicit indications of the agent's actions.
Project Team
Konstantin Grotov
Researcher
Artem Borzilov
Researcher
Maksim Krivobok
Researcher
Timofey Bryksin
Researcher
Yaroslav Zharov
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Konstantin Grotov, Artem Borzilov, Maksim Krivobok, Timofey Bryksin, Yaroslav Zharov
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai