Evidence-centered Assessment for Writing with Generative AI
Project Overview
The document explores the integration of generative AI (GAI) in education, particularly through a methodology that assesses collaborative writing between humans and GAI, utilizing learning analytics grounded in the evidence-centered design (ECD) framework. Emphasizing the importance of evaluating writing processes rather than solely the final outputs, the study employs the CoAuthor writing tool to examine user interactions during collaborative writing tasks. It uncovers significant variations in writing processes influenced by factors such as user ownership and the type of prompts provided, indicating that traditional assessment methods may be inadequate for measuring learning outcomes in this new context. The findings advocate for adaptive assessments that align with the collaborative nature of writing involving GAI, highlighting the need for educational frameworks to evolve in response to the capabilities and implications of generative AI in collaborative learning environments.
Key Applications
CoAuthor writing tool
Context: Higher education, targeting students engaged in collaborative writing using AI tools.
Implementation: Utilized CoAuthor to collect trace data from writing sessions, focusing on user interactions with GAI.
Outcomes: Identified differences in writing processes based on user ownership and prompt type; provided a proof of concept for assessment methods in human-AI collaboration.
Challenges: Limited by the dataset (self-selected participants, outdated AI), complexity of implementation, and the need for generalizable methods.
Implementation Barriers
Technical Barrier
The complexity of tracking and analyzing user interactions with GAI tools limits scalability and accessibility.
Proposed Solutions: Develop an adaptable API for customizable parameters to facilitate diverse task requirements.
Data Barrier
Data collected may not be representative due to self-selection bias and limited interaction types in the CoAuthor tool.
Proposed Solutions: Future research should expand the dataset to include a broader range of participants and GAI tools.
Project Team
Yixin Cheng
Researcher
Kayley Lyons
Researcher
Guanliang Chen
Researcher
Dragan Gasevic
Researcher
Zachari Swiecki
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Yixin Cheng, Kayley Lyons, Guanliang Chen, Dragan Gasevic, Zachari Swiecki
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai