-
Facilitating Human Feedback for GenAI Prompt Optimization
Authors:
Jacob Sherson,
Florent Vinchon
Abstract:
This study investigates the optimization of Generative AI (GenAI) systems through human feedback, focusing on how varying feedback mechanisms influence the quality of GenAI outputs. We devised a Human-AI training loop where 32 students, divided into two groups, evaluated AI-generated responses based on a single prompt. One group assessed a single output, while the other compared two outputs. Preli…
▽ More
This study investigates the optimization of Generative AI (GenAI) systems through human feedback, focusing on how varying feedback mechanisms influence the quality of GenAI outputs. We devised a Human-AI training loop where 32 students, divided into two groups, evaluated AI-generated responses based on a single prompt. One group assessed a single output, while the other compared two outputs. Preliminary results from this small-scale experiment suggest that comparative feedback might encourage more nuanced evaluations, highlighting the potential for improved human-AI collaboration in prompt optimization. Future research with larger samples is recommended to validate these findings and further explore effective feedback strategies for GenAI systems.
△ Less
Submitted 29 March, 2024;
originally announced April 2024.
-
SciNote: Collaborative Problem Solving and Argumentation Tool
Authors:
Janet Rafner,
Arthur Hjorth,
Carrie Weidner,
Shaeema Zaman Ahmed,
Christian Poulsen,
Clemens Klokmose,
Jacob Sherson
Abstract:
As educators push for students to learn science by doing science, there is a need for computational scaffolding to assist students' evaluation of scientific evidence and argument building. In this paper, we present a pilot study of SciNote, a CSCL tool allowing educators to integrate third-party software into a flexible and collaborative workspace. We explore how SciNote enables teams to build dat…
▽ More
As educators push for students to learn science by doing science, there is a need for computational scaffolding to assist students' evaluation of scientific evidence and argument building. In this paper, we present a pilot study of SciNote, a CSCL tool allowing educators to integrate third-party software into a flexible and collaborative workspace. We explore how SciNote enables teams to build data-driven arguments during inquiry-based learning activities.
△ Less
Submitted 7 April, 2021;
originally announced May 2021.
-
Revisiting Citizen Science Through the Lens of Hybrid Intelligence
Authors:
Janet Rafner,
Miroslav Gajdacz,
Gitte Kragh,
Arthur Hjorth,
Anna Gander,
Blanka Palfi,
Aleks Berditchevskaia,
François Grey,
Kobi Gal,
Avi Segal,
Mike Walmsley,
Josh Aaron Miller,
Dominik Dellerman,
Muki Haklay,
Pietro Michelucci,
Jacob Sherson
Abstract:
Artificial Intelligence (AI) can augment and sometimes even replace human cognition. Inspired by efforts to value human agency alongside productivity, we discuss the benefits of solving Citizen Science (CS) tasks with Hybrid Intelligence (HI), a synergetic mixture of human and artificial intelligence. Currently there is no clear framework or methodology on how to create such an effective mixture.…
▽ More
Artificial Intelligence (AI) can augment and sometimes even replace human cognition. Inspired by efforts to value human agency alongside productivity, we discuss the benefits of solving Citizen Science (CS) tasks with Hybrid Intelligence (HI), a synergetic mixture of human and artificial intelligence. Currently there is no clear framework or methodology on how to create such an effective mixture. Due to the unique participant-centered set of values and the abundance of tasks drawing upon both human common sense and complex 21st century skills, we believe that the field of CS offers an invaluable testbed for the development of HI and human-centered AI of the 21st century, while benefiting CS as well. In order to investigate this potential, we first relate CS to adjacent computational disciplines. Then, we demonstrate that CS projects can be grouped according to their potential for HI-enhancement by examining two key dimensions: the level of digitization and the amount of knowledge or experience required for participation. Finally, we propose a framework for types of human-AI interaction in CS based on established criteria of HI. This "HI lens" provides the CS community with an overview of several ways to utilize the combination of AI and human intelligence in their projects. It also allows the AI community to gain ideas on how develo** AI in CS projects can further their own field.
△ Less
Submitted 30 April, 2021;
originally announced April 2021.
-
The power of pictures: using ML assisted image generation to engage the crowd in complex socioscientific problems
Authors:
Janet Rafner,
Lotte Philipsen,
Sebastian Risi,
Joel Simon,
Jacob Sherson
Abstract:
Human-computer image generation using Generative Adversarial Networks (GANs) is becoming a well-established methodology for casual entertainment and open artistic exploration. Here, we take the interaction a step further by weaving in carefully structured design elements to transform the activity of ML-assisted imaged generation into a catalyst for large-scale popular dialogue on complex socioscie…
▽ More
Human-computer image generation using Generative Adversarial Networks (GANs) is becoming a well-established methodology for casual entertainment and open artistic exploration. Here, we take the interaction a step further by weaving in carefully structured design elements to transform the activity of ML-assisted imaged generation into a catalyst for large-scale popular dialogue on complex socioscientific problems such as the United Nations Sustainable Development Goals (SDGs) and as a gateway for public participation in research.
△ Less
Submitted 28 December, 2020; v1 submitted 15 October, 2020;
originally announced October 2020.
-
crea.blender: A Neural Network-Based Image Generation Game to Assess Creativity
Authors:
Janet Rafner,
Arthur Hjorth,
Sebastian Risi,
Lotte Philipsen,
Charles Dumas,
Michael Mose Biskjær,
Lior Noy,
Kristian Tylén,
Carsten Bergenholtz,
Jesse Lynch,
Blanka Zana,
Jacob Sherson
Abstract:
We present a pilot study on crea.blender, a novel co-creative game designed for large-scale, systematic assessment of distinct constructs of human creativity. Co-creative systems are systems in which humans and computers (often with Machine Learning) collaborate on a creative task. This human-computer collaboration raises questions about the relevance and level of human creativity and involvement…
▽ More
We present a pilot study on crea.blender, a novel co-creative game designed for large-scale, systematic assessment of distinct constructs of human creativity. Co-creative systems are systems in which humans and computers (often with Machine Learning) collaborate on a creative task. This human-computer collaboration raises questions about the relevance and level of human creativity and involvement in the process. We expand on, and explore aspects of these questions in this pilot study. We observe participants play through three different play modes in crea.blender, each aligned with established creativity assessment methods. In these modes, players "blend" existing images into new images under varying constraints. Our study indicates that crea.blender provides a playful experience, affords players a sense of control over the interface, and elicits different types of player behavior, supporting further study of the tool for use in a scalable, playful, creativity assessment.
△ Less
Submitted 17 August, 2020; v1 submitted 13 August, 2020;
originally announced August 2020.
-
Remote optimization of an ultra-cold atoms experiment by experts and citizen scientists
Authors:
Robert Heck,
Oana Vuculescu,
Jens Jakob Sørensen,
Jonathan Zoller,
Morten G. Andreasen,
Mark G. Bason,
Poul Ejlertsen,
Ottó Elíasson,
Pinja Haikka,
Jens S. Laustsen,
Lærke L. Nielsen,
Andrew Mao,
Romain Müller,
Mario Napolitano,
Mads K. Pedersen,
Aske R. Thorsen,
Carsten Bergenholtz,
Tommaso Calarco,
Simone Montangero,
Jacob F. Sherson
Abstract:
We introduce a novel remote interface to control and optimize the experimental production of Bose-Einstein condensates (BECs) and find improved solutions using two distinct implementations. First, a team of theoreticians employed a Remote version of their dCRAB optimization algorithm (RedCRAB), and second a gamified interface allowed 600 citizen scientists from around the world to participate in r…
▽ More
We introduce a novel remote interface to control and optimize the experimental production of Bose-Einstein condensates (BECs) and find improved solutions using two distinct implementations. First, a team of theoreticians employed a Remote version of their dCRAB optimization algorithm (RedCRAB), and second a gamified interface allowed 600 citizen scientists from around the world to participate in real-time optimization. Quantitative studies of player search behavior demonstrated that they collectively engage in a combination of local and global search. This form of adaptive search prevents premature convergence by the explorative behavior of low-performing players while high-performing players locally refine their solutions. In addition, many successful citizen science games have relied on a problem representation that directly engaged the visual or experiential intuition of the players. Here we demonstrate that citizen scientists can also be successful in an entirely abstract problem visualization. This gives encouragement that a much wider range of challenges could potentially be open to gamification in the future.
△ Less
Submitted 27 March, 2018; v1 submitted 7 September, 2017;
originally announced September 2017.
-
Play or science?: a study of learning and framing in crowdscience games
Authors:
Andreas Lieberoth,
Mads Kock Pedersen,
Jacob Sherson
Abstract:
Crowdscience games may hold unique potentials as learning opportunities compared to games made for fun or education. They are part of an actual science problem solving process: By playing, players help scientists, and thereby interact with real continuous research processes. This mixes the two worlds of play and science in new ways. During usability testing we discovered that users of the crowdsci…
▽ More
Crowdscience games may hold unique potentials as learning opportunities compared to games made for fun or education. They are part of an actual science problem solving process: By playing, players help scientists, and thereby interact with real continuous research processes. This mixes the two worlds of play and science in new ways. During usability testing we discovered that users of the crowdscience game Quantum Dreams tended to answer questions in game terms, even when directed explicitly to give science explanations.We then examined these competing frames of understanding through a mixed correlational and grounded theory analysis. This essay presents the core ideas of crowdscience games as learning opportunities, and reports how a group of players used "game", "science" and "conceptual" frames to interpret their experience. Our results suggest that oscillating between the frames instead of sticking to just one led to the largest number of correct science interpretations, as players could participate legitimately and autonomously at multiple levels of understanding.
△ Less
Submitted 23 October, 2015;
originally announced October 2015.
-
Getting Humans to do Quantum Optimization - User Acquisition, Engagement and Early Results from the Citizen Cyberscience Game Quantum Moves
Authors:
Andreas Lieberoth,
Mads Kock Pedersen,
Andreea Catalina Marin,
Tilo Planke,
Jacob Friis Sherson
Abstract:
The game Quantum Moves was designed to pit human players against computer algorithms, combining their solutions into hybrid optimization to control a scalable quantum computer. In this midstream report, we open our design process and describe the series of constitutive building stages going into a quantum physics citizen science game. We present our approach from designing a core gameplay around q…
▽ More
The game Quantum Moves was designed to pit human players against computer algorithms, combining their solutions into hybrid optimization to control a scalable quantum computer. In this midstream report, we open our design process and describe the series of constitutive building stages going into a quantum physics citizen science game. We present our approach from designing a core gameplay around quantum simulations, to putting extra game elements in place in order to frame, structure, and motivate players' difficult path from curious visitors to competent science contributors. The player base is extremely diverse - for instance, two top players are a 40 year old female accountant and a male taxi driver. Among statistical predictors for retention and in-game high scores, the data from our first year suggest that people recruited based on real-world physics interest and via real-world events, but only with an intermediate science education, are more likely to become engaged and skilled contributors. Interestingly, female players tended to perform better than male players, even though men played more games per day. To understand this relationship, we explore the profiles of our top players in more depth. We discuss in-world and in-game performance factors departing in psychological theories of intrinsic and extrinsic motivation, and the implications for using real live humans to do hybrid optimization via initially simple, but ultimately very cognitively complex games.
△ Less
Submitted 26 June, 2015;
originally announced June 2015.
