Which LLMs are Difficult to Detect? A Detailed Analysis of Potential Factors Contributing to Difficulties in LLM Text Detection
Project Overview
The document explores the role of generative AI in education, specifically addressing the challenges associated with detecting AI-generated text across various writing domains, such as scientific writing and student essays. It highlights the performance of classifiers designed to differentiate between human-written and AI-generated content, revealing that detection efficacy varies significantly depending on the writing context. The findings underscore that OpenAI's models, known for their advanced human-like text generation abilities, present unique challenges for detection in educational settings. Overall, the document emphasizes the implications of these challenges for educators and the need for effective strategies to identify AI-generated work, as the integration of generative AI continues to evolve within educational frameworks.
Key Applications
Detection of AI-generated student essays
Context: Educational context focusing on student assessments and academic integrity
Implementation: Using AWS Bedrock and OpenAI APIs to generate a dataset of rewritten student essays, then training classifiers using the LibAUC library for AUC optimization.
Outcomes: Classifiers demonstrated varying success in detecting AI-generated texts, with OpenAI models being particularly hard to distinguish from human-written essays.
Challenges: Difficulty in detection due to the high quality of AI-generated texts and their similarity to human writing.
Implementation Barriers
Technical Barrier
The advanced capabilities of LLMs lead to high-quality AI-generated texts that are difficult to distinguish from human writing.
Proposed Solutions: Developing diverse datasets for training classifiers to improve detection accuracy across different text types and implementing robust detection systems that minimize false positives in AI detection.
Project Team
Shantanu Thorat
Researcher
Tianbao Yang
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Shantanu Thorat, Tianbao Yang
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai