Impact of a Deployed LLM Survey Creation Tool through the IS Success Model
Project Overview
The document explores the application of generative AI, specifically a large language model (LLM)-powered survey creation tool, in the realm of Information Systems (IS) research. It emphasizes how this technology can automate and enhance survey design, a process typically reliant on considerable expert knowledge. By utilizing the DeLone and McLean IS Success Model, the tool effectively evaluates critical dimensions such as service quality, user satisfaction, system use, and information quality. Among its significant contributions, the document presents a hybrid evaluation framework that integrates both human and automated assessments, thereby improving the reliability of survey outcomes. Additionally, it outlines the implementation of safeguards designed to address potential risks associated with the use of generative AI. Overall, the findings indicate that generative AI not only streamlines the survey creation process but also contributes to more robust research methodologies in IS, ultimately leading to improved decision-making and insights in the field.
Key Applications
LLM-powered survey creation system
Context: Used in education for evaluating student satisfaction and experience
Implementation: Users input prompts describing their survey objectives, which the LLM interprets to generate relevant survey questions.
Outcomes: Achieved strong adoption; users reported the tool as easy to use, helpful, and worth recommending, leading to higher survey sharing rates.
Challenges: Usability barriers, such as system unreliability and UI design issues, which impacted user satisfaction.
Implementation Barriers
Usability Barrier
Issues related to system unreliability, including long wait times and fragmented UI design.
Proposed Solutions: Feedback from users highlighted the need for improved UI design and system reliability enhancements.
Transparency
Users were unaware of system-imposed limits on the number of prompts that can be submitted.
Proposed Solutions: Implement visual countdowns and clearer communication about system limitations.
Project Team
Peng Jiang
Researcher
Vinicius Cezar Monteiro de Lira
Researcher
Antonio Maiorino
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Peng Jiang, Vinicius Cezar Monteiro de Lira, Antonio Maiorino
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai