Hints-In-Browser: Benchmarking Language Models for Programming Feedback Generation
Project Overview
The document examines the role of generative AI, particularly large language models, in programming education, emphasizing their ability to provide personalized feedback and hints to students. It introduces an innovative deployment method called 'hints-in-browser,' which enables these models to operate directly within users' browsers, thereby improving cost-efficiency and safeguarding data privacy. A benchmarking study evaluates several language models, including GPT-4 and Llama-3, assessing them based on quality, cost, and performance metrics. The results reveal substantial advantages of utilizing in-browser models for enhancing the educational experience in programming, while also acknowledging challenges such as the limitations posed by model size and consumer hardware capabilities. Overall, the findings underscore the potential of generative AI to transform programming education by delivering tailored support to learners.
Key Applications
Hints-in-Browser workflow for programming feedback generation
Context: Programming education targeting novice learners
Implementation: The model runs in the browser, generating feedback based on the learner's code and requests for hints.
Outcomes: No running costs for educators or learners, complete data privacy, and improved feedback quality with in-browser inference.
Challenges: Model size limitations and hardware requirements for effective inference.
Implementation Barriers
Cost and Time
High running costs for educators to provide feedback generation services using external models, along with long waiting times for learners while feedback is generated, which hampers interactivity.
Proposed Solutions: Utilizing in-browser models significantly reduces waiting times for feedback and eliminates the need for server-side processing, thereby reducing costs.
Data Privacy
Concerns over learner data being sent to external servers for processing.
Proposed Solutions: In-browser models keep user data local, enhancing privacy.
Project Team
Nachiket Kotalwar
Researcher
Alkis Gotovos
Researcher
Adish Singla
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Nachiket Kotalwar, Alkis Gotovos, Adish Singla
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai