-
Ink and Algorithm: Exploring Temporal Dynamics in Human-AI Collaborative Writing
Authors:
Kaixun Yang,
Yixin Cheng,
Linxuan Zhao,
Mladen Raković,
Zachari Swiecki,
Dragan Gašević,
Guanliang Chen
Abstract:
The advent of Generative Artificial Intelligence (GAI) has revolutionized the field of writing, marking a shift towards human-AI collaborative writing in education. However, the dynamics of human-AI interaction in the collaborative writing process are not well understood, and thus it remains largely unknown how human learning can be effectively supported with such cutting-edge GAI technologies. In…
▽ More
The advent of Generative Artificial Intelligence (GAI) has revolutionized the field of writing, marking a shift towards human-AI collaborative writing in education. However, the dynamics of human-AI interaction in the collaborative writing process are not well understood, and thus it remains largely unknown how human learning can be effectively supported with such cutting-edge GAI technologies. In this study, we aim to bridge this gap by investigating how humans employ GAI in collaborative writing and examining the interplay between the patterns of GAI usage and human writing behaviors. Considering the potential varying degrees to which people rely on GAI usage, we proposed to use Dynamic Time War** time-series clustering for the identification and analysis of common temporal patterns in AI usage during the human-AI collaborative writing processes. Additionally, we incorporated Epistemic Network Analysis to reveal the correlation between GAI usage and human writing behaviors that reflect cognitive processes (i.e., knowledge telling, knowledge transformation, and cognitive presence), aiming to offer insights for develo** better approaches and tools to support human to learn effectively via such human-AI collaborative writing activities. Our findings reveal four major distinct temporal patterns in AI utilization and highlight significant correlations between these patterns and human writing behaviors. These findings have significant implications for effectively supporting human learning with GAI in educational writing tasks.
△ Less
Submitted 21 June, 2024;
originally announced June 2024.
-
Putting Our Minds Together: Iterative Exploration for Collaborative Mind Map**
Authors:
Ying Yang,
Tim Dwyer,
Zachari Swiecki,
Benjamin Lee,
Michael Wybrow,
Maxime Cordeil,
Teresa Wulandari,
Bruce H. Thomas,
Mark Billinghurst
Abstract:
We delineate the development of a mind-map** system designed concurrently for both VR and desktop platforms. Employing an iterative methodology with groups of users, we systematically examined and improved various facets of our system, including interactions, communication mechanisms and gamification elements, to streamline the mind-map** process while augmenting situational awareness and prom…
▽ More
We delineate the development of a mind-map** system designed concurrently for both VR and desktop platforms. Employing an iterative methodology with groups of users, we systematically examined and improved various facets of our system, including interactions, communication mechanisms and gamification elements, to streamline the mind-map** process while augmenting situational awareness and promoting active engagement among collaborators. We also report our observational findings on these facets from this iterative design process.
△ Less
Submitted 23 March, 2024; v1 submitted 20 March, 2024;
originally announced March 2024.
-
Evidence-centered Assessment for Writing with Generative AI
Authors:
Yixin Cheng,
Kayley Lyons,
Guanliang Chen,
Dragan Gasevic,
Zachari Swiecki
Abstract:
We propose a learning analytics-based methodology for assessing the collaborative writing of humans and generative artificial intelligence. Framed by the evidence-centered design, we used elements of knowledge-telling, knowledge transformation, and cognitive presence to identify assessment claims; we used data collected from the CoAuthor writing tool as potential evidence for these claims; and we…
▽ More
We propose a learning analytics-based methodology for assessing the collaborative writing of humans and generative artificial intelligence. Framed by the evidence-centered design, we used elements of knowledge-telling, knowledge transformation, and cognitive presence to identify assessment claims; we used data collected from the CoAuthor writing tool as potential evidence for these claims; and we used epistemic network analysis to make inferences from the data about the claims. Our findings revealed significant differences in the writing processes of different groups of CoAuthor users, suggesting that our method is a plausible approach to assessing human-AI collaborative writing.
△ Less
Submitted 16 January, 2024;
originally announced January 2024.
-
Human-Centred Learning Analytics and AI in Education: a Systematic Literature Review
Authors:
Riordan Alfredo,
Vanessa Echeverria,
Yueqiao **,
Lixiang Yan,
Zachari Swiecki,
Dragan Gašević,
Roberto Martinez-Maldonado
Abstract:
The rapid expansion of Learning Analytics (LA) and Artificial Intelligence in Education (AIED) offers new scalable, data-intensive systems but also raises concerns about data privacy and agency. Excluding stakeholders -- like students and teachers -- from the design process can potentially lead to mistrust and inadequately aligned tools. Despite a shift towards human-centred design in recent LA an…
▽ More
The rapid expansion of Learning Analytics (LA) and Artificial Intelligence in Education (AIED) offers new scalable, data-intensive systems but also raises concerns about data privacy and agency. Excluding stakeholders -- like students and teachers -- from the design process can potentially lead to mistrust and inadequately aligned tools. Despite a shift towards human-centred design in recent LA and AIED research, there remain gaps in our understanding of the importance of human control, safety, reliability, and trustworthiness in the design and implementation of these systems. We conducted a systematic literature review to explore these concerns and gaps. We analysed 108 papers to provide insights about i) the current state of human-centred LA/AIED research; ii) the extent to which educational stakeholders have contributed to the design process of human-centred LA/AIED systems; iii) the current balance between human control and computer automation of such systems; and iv) the extent to which safety, reliability and trustworthiness have been considered in the literature. Results indicate some consideration of human control in LA/AIED system design, but limited end-user involvement in actual design. Based on these findings, we recommend: 1) carefully balancing stakeholders' involvement in designing and deploying LA/AIED systems throughout all design phases, 2) actively involving target end-users, especially students, to delineate the balance between human control and automation, and 3) exploring safety, reliability, and trustworthiness as principles in future human-centred LA/AIED systems.
△ Less
Submitted 19 December, 2023;
originally announced December 2023.
-
Human-AI Collaboration in Thematic Analysis using ChatGPT: A User Study and Design Recommendations
Authors:
Lixiang Yan,
Vanessa Echeverria,
Gloria Fernandez Nieto,
Yueqiao **,
Zachari Swiecki,
Linxuan Zhao,
Dragan Gašević,
Roberto Martinez-Maldonado
Abstract:
Generative artificial intelligence (GenAI) offers promising potential for advancing human-AI collaboration in qualitative research. However, existing works focused on conventional machine-learning and pattern-based AI systems, and little is known about how researchers interact with GenAI in qualitative research. This work delves into researchers' perceptions of their collaboration with GenAI, spec…
▽ More
Generative artificial intelligence (GenAI) offers promising potential for advancing human-AI collaboration in qualitative research. However, existing works focused on conventional machine-learning and pattern-based AI systems, and little is known about how researchers interact with GenAI in qualitative research. This work delves into researchers' perceptions of their collaboration with GenAI, specifically ChatGPT. Through a user study involving ten qualitative researchers, we found ChatGPT to be a valuable collaborator for thematic analysis, enhancing coding efficiency, aiding initial data exploration, offering granular quantitative insights, and assisting comprehension for non-native speakers and non-experts. Yet, concerns about its trustworthiness and accuracy, reliability and consistency, limited contextual understanding, and broader acceptance within the research community persist. We contribute five actionable design recommendations to foster effective human-AI collaboration. These include incorporating transparent explanatory mechanisms, enhancing interface and integration capabilities, prioritising contextual understanding and customisation, embedding human-AI feedback loops and iterative functionality, and strengthening trust through validation mechanisms.
△ Less
Submitted 7 November, 2023;
originally announced November 2023.
