Assessing Generative AI value in a public sector context: evidence from a field experiment
Project Overview
The document examines the role of Generative AI (Gen AI) in the public sector, particularly in education, emphasizing its effectiveness in knowledge-driven tasks like document comprehension and data analysis. A randomized control trial at the Central Bank of Ireland revealed that Gen AI significantly enhanced performance on document tasks, yielding a 17% improvement in quality and a 34% reduction in completion time. Conversely, the technology led to a 12% decline in quality for data-related tasks, suggesting that Gen AI's efficacy is dependent on the specific type of task being performed. These findings underscore the importance of thoughtful implementation that takes into account the nature of the tasks, the necessity of user training, and the value of incorporating human oversight to optimize outcomes. Overall, while Gen AI shows promise in enhancing educational processes, its varied effectiveness necessitates a strategic approach to its integration in educational settings.
Key Applications
Generative AI for Document Comprehension and Data Analysis
Context: Central Bank of Ireland staff performing both document comprehension and data analysis tasks. Participants in these tasks were randomly assigned to either a treatment group with Generative AI assistance or a control group without it.
Implementation: A randomized control trial setup was employed for both document comprehension and data analysis tasks, focusing on assessing quality and completion time metrics. Participants' performance was compared across tasks to evaluate the impact of Generative AI assistance.
Outcomes: For document comprehension, there was a 17% improvement in quality and a 34% reduction in completion time for the treatment group compared to the control group. In contrast, the data analysis task showed a 12% reduction in quality in the Generative AI treatment group, with no significant difference in completion time.
Challenges: The effectiveness of Generative AI varied by task complexity. While it significantly improved performance in document comprehension tasks, it struggled with more complex document questions and yielded mixed results in data analysis due to simplicity in the task design.
Implementation Barriers
Technical
Access issues and outages with Gen AI applications during the trial led to delays.
Proposed Solutions: Enhance IT support and reliability of the applications used in trials.
User Training
Participants expressed a need for better training on effective prompting techniques for Gen AI.
Proposed Solutions: Implement structured training programs on prompt engineering and refining outputs.
Organizational
Self-selection bias in task assignments may have skewed results, particularly in data tasks.
Proposed Solutions: Future trials should randomize task assignments to ensure representativeness across participant capabilities.
Project Team
Trevor Fitzpatrick
Researcher
Seamus Kelly
Researcher
Patrick Carey
Researcher
David Walsh
Researcher
Ruairi Nugent
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Trevor Fitzpatrick, Seamus Kelly, Patrick Carey, David Walsh, Ruairi Nugent
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai