Artificial Intelligence versus Maya Angelou: Experimental evidence that people cannot differentiate AI-generated from human-written poetry
Project Overview
The document explores the application of generative AI, particularly the GPT-2 Natural Language Generation algorithm, in the field of education through the creation of human-like text, such as poetry. It investigates the ability of individuals to distinguish between AI-generated and human-written poetry, revealing that participants often find it challenging to make this distinction. Interestingly, while a general preference for human-written poetry is noted, this preference tends to fluctuate based on whether individuals are made aware of the text's source. The findings underscore the phenomenon of algorithm aversion, where users may favor human input over AI-generated content, emphasizing the importance of human involvement in the selection and use of AI outputs. Overall, the document highlights both the potential of generative AI in educational settings and the nuanced attitudes of users toward AI-generated materials, suggesting that while AI can produce sophisticated texts, human judgment remains crucial in the educational context.
Key Applications
GPT-2 algorithm for generating poetry
Context: Experimental study assessing preferences and detection accuracy of AI-generated versus human-written poetry among participants
Implementation: Participants judged pairs of poems, one from GPT-2 and one human-written, under conditions of transparency and opacity regarding the poems' origins.
Outcomes: Participants preferred human-written poetry and struggled to accurately identify the origin of the poems. Human involvement in poem selection affected preferences and detection accuracy.
Challenges: Participants exhibited algorithm aversion and overconfidence in their ability to detect AI-generated text.
Implementation Barriers
Perception barrier
Participants generally showed aversion to algorithm-generated poetry, preferring human-written texts. Many participants were unable to reliably distinguish between human and AI-generated poetry.
Proposed Solutions: Increasing transparency regarding the algorithmic process and educating users about the capabilities of AI may mitigate aversion. Additionally, incentivizing accuracy in detection tasks may improve participants' performance and confidence in distinguishing AI-generated texts.
Project Team
Nils Köbis
Researcher
Luca Mossink
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Nils Köbis, Luca Mossink
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai