-
Safeguarding Crowdsourcing Surveys from ChatGPT with Prompt Injection
Authors:
Chaofan Wang,
Samuel Kernan Freire,
Mo Zhang,
**g Wei,
Jorge Goncalves,
Vassilis Kostakos,
Zhanna Sarsenbayeva,
Christina Schneegass,
Alessandro Bozzon,
Evangelos Niforatos
Abstract:
ChatGPT and other large language models (LLMs) have proven useful in crowdsourcing tasks, where they can effectively annotate machine learning training data. However, this means that they also have the potential for misuse, specifically to automatically answer surveys. LLMs can potentially circumvent quality assurance measures, thereby threatening the integrity of methodologies that rely on crowds…
▽ More
ChatGPT and other large language models (LLMs) have proven useful in crowdsourcing tasks, where they can effectively annotate machine learning training data. However, this means that they also have the potential for misuse, specifically to automatically answer surveys. LLMs can potentially circumvent quality assurance measures, thereby threatening the integrity of methodologies that rely on crowdsourcing surveys. In this paper, we propose a mechanism to detect LLM-generated responses to surveys. The mechanism uses "prompt injection", such as directions that can mislead LLMs into giving predictable responses. We evaluate our technique against a range of question scenarios, types, and positions, and find that it can reliably detect LLM-generated responses with more than 93% effectiveness. We also provide an open-source software to help survey designers use our technique to detect LLM responses. Our work is a step in ensuring that survey methodologies remain rigorous vis-a-vis LLMs.
△ Less
Submitted 14 June, 2023;
originally announced June 2023.
-
Comparing Concepts for Embedding Second-Language Vocabulary Acquisition into Everyday Smartphone Interactions
Authors:
Christina Schneegass,
Sophia Sigethy,
Malin Eiband,
Daniel Buschek
Abstract:
We present a three-week within-subject field study comparing three mobile language learning (MLL) applications with varying levels of integration into everyday smartphone interactions: We designed a novel (1) UnlockApp that presents a vocabulary task with each authentication event, nudging users towards short frequent learning sessions. We compare it with a (2) NotificationApp that displays vocabu…
▽ More
We present a three-week within-subject field study comparing three mobile language learning (MLL) applications with varying levels of integration into everyday smartphone interactions: We designed a novel (1) UnlockApp that presents a vocabulary task with each authentication event, nudging users towards short frequent learning sessions. We compare it with a (2) NotificationApp that displays vocabulary tasks in a push notification in the status bar, which is always visible but learning needs to be user-initiated, and a (3) StandardApp that requires users to start in-app learning actively. Our study is the first to directly compare these embedding concepts for MLL, showing that integrating vocabulary learning into everyday smartphone interactions via UnlockApp and NotificationApp increases the number of answers. However, users show individual subjective preferences. Based on our results, we discuss the trade-off between higher content exposure and disturbance, and the related challenges and opportunities of embedding learning seamlessly into everyday mobile interactions.
△ Less
Submitted 16 September, 2021;
originally announced September 2021.
-
What is "Intelligent" in Intelligent User Interfaces? A Meta-Analysis of 25 Years of IUI
Authors:
Sarah Theres Völkel,
Christina Schneegass,
Malin Eiband,
Daniel Buschek
Abstract:
This reflection paper takes the 25th IUI conference milestone as an opportunity to analyse in detail the understanding of intelligence in the community: Despite the focus on intelligent UIs, it has remained elusive what exactly renders an interactive system or user interface "intelligent", also in the fields of HCI and AI at large. We follow a bottom-up approach to analyse the emergent meaning of…
▽ More
This reflection paper takes the 25th IUI conference milestone as an opportunity to analyse in detail the understanding of intelligence in the community: Despite the focus on intelligent UIs, it has remained elusive what exactly renders an interactive system or user interface "intelligent", also in the fields of HCI and AI at large. We follow a bottom-up approach to analyse the emergent meaning of intelligence in the IUI community: In particular, we apply text analysis to extract all occurrences of "intelligent" in all IUI proceedings. We manually review these with regard to three main questions: 1) What is deemed intelligent? 2) How (else) is it characterised? and 3) What capabilities are attributed to an intelligent entity? We discuss the community's emerging implicit perspective on characteristics of intelligence in intelligent user interfaces and conclude with ideas for stating one's own understanding of intelligence more explicitly.
△ Less
Submitted 6 March, 2020;
originally announced March 2020.
