Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025
Project Overview
The document explores the transformative role of generative AI in education, particularly through the implementation of a Review Feedback Agent that leverages large language models (LLMs) to improve peer review processes at AI conferences. A randomized control study conducted at ICLR 2025 revealed significant positive outcomes from AI-generated feedback, noting enhancements in review clarity and engagement, which resulted in longer and more informative reviews. Impressively, 27% of reviewers modified their assessments after receiving AI suggestions, indicating that the technology not only aided in refining the content of reviews but also facilitated improved communication between authors and reviewers. These findings underscore the potential of generative AI to augment educational practices, enhance collaborative feedback mechanisms, and ultimately contribute to a more effective and interactive academic environment.
Key Applications
Review Feedback Agent
Context: Peer review process at ICLR 2025 involving over 20,000 reviews.
Implementation: Implemented as a randomized control study providing automated feedback on reviews using LLMs.
Outcomes: Improved review clarity and actionability, increased engagement, and longer reviews.
Challenges: Ensuring feedback quality and addressing potential reviewer biases.
Implementation Barriers
Quality Control
Maintaining high-quality feedback that is actionable and specific.
Proposed Solutions: Developing reliability tests for feedback generated by LLMs.
Reviewer Engagement
Potential resistance from reviewers in adopting AI feedback.
Proposed Solutions: Making AI feedback optional and ensuring reviewers have control over their final review.
Project Team
Nitya Thakkar
Researcher
Mert Yuksekgonul
Researcher
Jake Silberg
Researcher
Animesh Garg
Researcher
Nanyun Peng
Researcher
Fei Sha
Researcher
Rose Yu
Researcher
Carl Vondrick
Researcher
James Zou
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Nitya Thakkar, Mert Yuksekgonul, Jake Silberg, Animesh Garg, Nanyun Peng, Fei Sha, Rose Yu, Carl Vondrick, James Zou
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai