-
ReviewFlow: Intelligent Scaffolding to Support Academic Peer Reviewing
Authors:
Lu Sun,
Aaron Chan,
Yun Seo Chang,
Steven P. Dow
Abstract:
Peer review is a cornerstone of science. Research communities conduct peer reviews to assess contributions and to improve the overall quality of science work. Every year, new community members are recruited as peer reviewers for the first time. How could technology help novices adhere to their community's practices and standards for peer reviewing? To better understand peer review practices and ch…
▽ More
Peer review is a cornerstone of science. Research communities conduct peer reviews to assess contributions and to improve the overall quality of science work. Every year, new community members are recruited as peer reviewers for the first time. How could technology help novices adhere to their community's practices and standards for peer reviewing? To better understand peer review practices and challenges, we conducted a formative study with 10 novices and 10 experts. We found that many experts adopt a workflow of annotating, note-taking, and synthesizing notes into well-justified reviews that align with community standards. Novices lack timely guidance on how to read and assess submissions and how to structure paper reviews. To support the peer review process, we developed ReviewFlow -- an AI-driven workflow that scaffolds novices with contextual reflections to critique and annotate submissions, in-situ knowledge support to assess novelty, and notes-to-outline synthesis to help align peer reviews with community expectations. In a within-subjects experiment, 16 inexperienced reviewers wrote reviews in two conditions: using ReviewFlow and using a baseline environment with minimal guidance. With ReviewFlow, participants produced more comprehensive reviews, identifying more pros and cons. While participants appreciated the streamlined process support from ReviewFlow, they also expressed concerns about using AI as part of the scientific review process. We discuss the implications of using AI to scaffold the peer review process on scientific work and beyond.
△ Less
Submitted 26 February, 2024; v1 submitted 5 February, 2024;
originally announced February 2024.
-
Graphologue: Exploring Large Language Model Responses with Interactive Diagrams
Authors:
Peiling Jiang,
Jude Rayan,
Steven P. Dow,
Haijun Xia
Abstract:
Large language models (LLMs) have recently soared in popularity due to their ease of access and the unprecedented ability to synthesize text responses to diverse user questions. However, LLMs like ChatGPT present significant limitations in supporting complex information tasks due to the insufficient affordances of the text-based medium and linear conversational structure. Through a formative study…
▽ More
Large language models (LLMs) have recently soared in popularity due to their ease of access and the unprecedented ability to synthesize text responses to diverse user questions. However, LLMs like ChatGPT present significant limitations in supporting complex information tasks due to the insufficient affordances of the text-based medium and linear conversational structure. Through a formative study with ten participants, we found that LLM interfaces often present long-winded responses, making it difficult for people to quickly comprehend and interact flexibly with various pieces of information, particularly during more complex tasks. We present Graphologue, an interactive system that converts text-based responses from LLMs into graphical diagrams to facilitate information-seeking and question-answering tasks. Graphologue employs novel prompting strategies and interface designs to extract entities and relationships from LLM responses and constructs node-link diagrams in real-time. Further, users can interact with the diagrams to flexibly adjust the graphical presentation and to submit context-specific prompts to obtain more information. Utilizing diagrams, Graphologue enables graphical, non-linear dialogues between humans and LLMs, facilitating information exploration, organization, and comprehension.
△ Less
Submitted 4 August, 2023; v1 submitted 19 May, 2023;
originally announced May 2023.
-
Freeform Templates: Combining Freeform Curation with Structured Templates
Authors:
Stephen MacNeil,
Ziheng Huang,
Kenneth Chen,
Zijian Ding,
Alex Yu,
Kendall Nakai,
Steven P. Dow
Abstract:
Online whiteboards are becoming a popular way to facilitate collaborative design work, providing a free-form environment to curate ideas. However, as templates are increasingly being used to scaffold contributions from non-experts designers, it is crucial to understand their impact on the creative process. In this paper, we present the results from a study with 114 students in a large introductory…
▽ More
Online whiteboards are becoming a popular way to facilitate collaborative design work, providing a free-form environment to curate ideas. However, as templates are increasingly being used to scaffold contributions from non-experts designers, it is crucial to understand their impact on the creative process. In this paper, we present the results from a study with 114 students in a large introductory design course. Our results confirm prior findings that templates benefit students by providing a starting point, a shared process, and the ability to access their own work from previous steps. While prior research has criticized templates for being too rigid, we discovered that using templates within a free-form environment resulted in visual patterns of free-form curation where concepts were spatially organized, clustered, color-coded, and connected using arrows and lines. We introduce the concept of "Free-form Templates" to illustrate how templates and free-form curation can be synergistic.
△ Less
Submitted 1 May, 2023;
originally announced May 2023.
