-
The Lab vs The Crowd: An Investigation into Data Quality for Neural Dialogue Models
Authors:
José Lopes,
Francisco J. Chiyah Garcia,
Helen Hastie
Abstract:
Challenges around collecting and processing quality data have hampered progress in data-driven dialogue models. Previous approaches are moving away from costly, resource-intensive lab settings, where collection is slow but where the data is deemed of high quality. The advent of crowd-sourcing platforms, such as Amazon Mechanical Turk, has provided researchers with an alternative cost-effective and…
▽ More
Challenges around collecting and processing quality data have hampered progress in data-driven dialogue models. Previous approaches are moving away from costly, resource-intensive lab settings, where collection is slow but where the data is deemed of high quality. The advent of crowd-sourcing platforms, such as Amazon Mechanical Turk, has provided researchers with an alternative cost-effective and rapid way to collect data. However, the collection of fluid, natural spoken or textual interaction can be challenging, particularly between two crowd-sourced workers. In this study, we compare the performance of dialogue models for the same interaction task but collected in two different settings: in the lab vs. crowd-sourced. We find that fewer lab dialogues are needed to reach similar accuracy, less than half the amount of lab data as crowd-sourced data. We discuss the advantages and disadvantages of each data collection method.
△ Less
Submitted 7 December, 2020;
originally announced December 2020.
-
Robots in the Danger Zone: Exploring Public Perception through Engagement
Authors:
David A. Robb,
Muneeb I. Ahmad,
Carlo Tiseo,
Simona Aracri,
Alistair C. McConnell,
Vincent Page,
Christian Dondrup,
Francisco J. Chiyah Garcia,
Hai-Nguyen Nguyen,
Èric Pairet,
Paola Ardón Ramírez,
Tushar Semwal,
Hazel M. Taylor,
Lindsay J. Wilson,
David Lane,
Helen Hastie,
Katrin Lohan
Abstract:
Public perceptions of Robotics and Artificial Intelligence (RAI) are important in the acceptance, uptake, government regulation and research funding of this technology. Recent research has shown that the public's understanding of RAI can be negative or inaccurate. We believe effective public engagement can help ensure that public opinion is better informed. In this paper, we describe our first ite…
▽ More
Public perceptions of Robotics and Artificial Intelligence (RAI) are important in the acceptance, uptake, government regulation and research funding of this technology. Recent research has shown that the public's understanding of RAI can be negative or inaccurate. We believe effective public engagement can help ensure that public opinion is better informed. In this paper, we describe our first iteration of a high throughput in-person public engagement activity. We describe the use of a light touch quiz-format survey instrument to integrate in-the-wild research participation into the engagement, allowing us to probe both the effectiveness of our engagement strategy, and public perceptions of the future roles of robots and humans working in dangerous settings, such as in the off-shore energy sector. We critique our methods and share interesting results into generational differences within the public's view of the future of Robotics and AI in hazardous environments. These findings include that older peoples' views about the future of robots in hazardous environments were not swayed by exposure to our exhibit, while the views of younger people were affected by our exhibit, leading us to consider carefully in future how to more effectively engage with and inform older people.
△ Less
Submitted 1 April, 2020;
originally announced April 2020.
-
CRWIZ: A Framework for Crowdsourcing Real-Time Wizard-of-Oz Dialogues
Authors:
Francisco J. Chiyah Garcia,
José Lopes,
Xingkun Liu,
Helen Hastie
Abstract:
Large corpora of task-based and open-domain conversational dialogues are hugely valuable in the field of data-driven dialogue systems. Crowdsourcing platforms, such as Amazon Mechanical Turk, have been an effective method for collecting such large amounts of data. However, difficulties arise when task-based dialogues require expert domain knowledge or rapid access to domain-relevant information, s…
▽ More
Large corpora of task-based and open-domain conversational dialogues are hugely valuable in the field of data-driven dialogue systems. Crowdsourcing platforms, such as Amazon Mechanical Turk, have been an effective method for collecting such large amounts of data. However, difficulties arise when task-based dialogues require expert domain knowledge or rapid access to domain-relevant information, such as databases for tourism. This will become even more prevalent as dialogue systems become increasingly ambitious, expanding into tasks with high levels of complexity that require collaboration and forward planning, such as in our domain of emergency response. In this paper, we propose CRWIZ: a framework for collecting real-time Wizard of Oz dialogues through crowdsourcing for collaborative, complex tasks. This framework uses semi-guided dialogue to avoid interactions that breach procedures and processes only known to experts, while enabling the capture of a wide variety of interactions. The framework is available at https://github.com/JChiyah/crwiz
△ Less
Submitted 12 March, 2020;
originally announced March 2020.
-
Natural Language Interaction to Facilitate Mental Models of Remote Robots
Authors:
Francisco J. Chiyah Garcia,
José Lopes,
Helen Hastie
Abstract:
Increasingly complex and autonomous robots are being deployed in real-world environments with far-reaching consequences. High-stakes scenarios, such as emergency response or offshore energy platform and nuclear inspections, require robot operators to have clear mental models of what the robots can and can't do. However, operators are often not the original designers of the robots and thus, they do…
▽ More
Increasingly complex and autonomous robots are being deployed in real-world environments with far-reaching consequences. High-stakes scenarios, such as emergency response or offshore energy platform and nuclear inspections, require robot operators to have clear mental models of what the robots can and can't do. However, operators are often not the original designers of the robots and thus, they do not necessarily have such clear mental models, especially if they are novice users. This lack of mental model clarity can slow adoption and can negatively impact human-machine teaming. We propose that interaction with a conversational assistant, who acts as a mediator, can help the user with understanding the functionality of remote robots and increase transparency through natural language explanations, as well as facilitate the evaluation of operators' mental models.
△ Less
Submitted 12 March, 2020;
originally announced March 2020.
-
Towards Self-Explainable Cyber-Physical Systems
Authors:
Mathias Blumreiter,
Joel Greenyer,
Francisco Javier Chiyah Garcia,
Verena Klös,
Maike Schwammberger,
Christoph Sommer,
Andreas Vogelsang,
Andreas Wortmann
Abstract:
With the increasing complexity of CPSs, their behavior and decisions become increasingly difficult to understand and comprehend for users and other stakeholders. Our vision is to build self-explainable systems that can, at run-time, answer questions about the system's past, current, and future behavior. As hitherto no design methodology or reference framework exists for building such systems, we p…
▽ More
With the increasing complexity of CPSs, their behavior and decisions become increasingly difficult to understand and comprehend for users and other stakeholders. Our vision is to build self-explainable systems that can, at run-time, answer questions about the system's past, current, and future behavior. As hitherto no design methodology or reference framework exists for building such systems, we propose the MAB-EX framework for building self-explainable systems that leverage requirements- and explainability models at run-time. The basic idea of MAB-EX is to first Monitor and Analyze a certain behavior of a system, then Build an explanation from explanation models and convey this EXplanation in a suitable way to a stakeholder. We also take into account that new explanations can be learned, by updating the explanation models, should new and yet un-explainable behavior be detected by the system.
△ Less
Submitted 13 August, 2019;
originally announced August 2019.
-
A Natural Language Interface with Relayed Acoustic Communications for Improved Command and Control of AUVs
Authors:
David A. Robb,
Jonatan Scharff Willners,
Nicolas Valeyrie,
Francisco J. Chiyah Garcia,
Atanas Laskov,
Xingkun Liu,
Pedro Patron,
Helen Hastie,
Yvan R. Petillot
Abstract:
Autonomous underwater vehicles (AUVs) are being tasked with increasingly complex missions. The acoustic communications required for AUVs are, by the nature of the medium, low bandwidth while adverse environmental conditions underwater often mean they are also intermittent. This has motivated development of highly autonomous systems, which can operate independently of their operators for considerab…
▽ More
Autonomous underwater vehicles (AUVs) are being tasked with increasingly complex missions. The acoustic communications required for AUVs are, by the nature of the medium, low bandwidth while adverse environmental conditions underwater often mean they are also intermittent. This has motivated development of highly autonomous systems, which can operate independently of their operators for considerable periods of time. These missions often involve multiple vehicles leading not only to challenges in communications but also in command and control (C2). Specifically operators face complexity in controlling multi-objective, multi-vehicle missions, whilst simultaneously facing uncertainty over the current status and safety of several remote high value assets. Additionally, it may be required to perform command and control of these complex missions in a remote control room. In this paper, we propose a combination of an intuitive, natural language operator interface combined with communications that use platforms from multiple domains to relay data over different mediums and transmission modes, improving command and control of collaborative and fully autonomous missions. In trials, we have demonstrated an integrated system combining working prototypes with established commercial C2 software that enables the use of a natural language interface to monitor an AUV survey mission in an on-shore command and control centre.
△ Less
Submitted 9 November, 2018; v1 submitted 8 November, 2018;
originally announced November 2018.
-
MIRIAM: A Multimodal Chat-Based Interface for Autonomous Systems
Authors:
Helen Hastie,
Francisco J. Chiyah Garcia,
David A. Robb,
Pedro Patron,
Atanas Laskov
Abstract:
We present MIRIAM (Multimodal Intelligent inteRactIon for Autonomous systeMs), a multimodal interface to support situation awareness of autonomous vehicles through chat-based interaction. The user is able to chat about the vehicle's plan, objectives, previous activities and mission progress. The system is mixed initiative in that it pro-actively sends messages about key events, such as fault warni…
▽ More
We present MIRIAM (Multimodal Intelligent inteRactIon for Autonomous systeMs), a multimodal interface to support situation awareness of autonomous vehicles through chat-based interaction. The user is able to chat about the vehicle's plan, objectives, previous activities and mission progress. The system is mixed initiative in that it pro-actively sends messages about key events, such as fault warnings. We will demonstrate MIRIAM using SeeByte's SeeTrack command and control interface and Neptune autonomy simulator.
△ Less
Submitted 6 March, 2018;
originally announced March 2018.
-
Explain Yourself: A Natural Language Interface for Scrutable Autonomous Robots
Authors:
Francisco J. Chiyah Garcia,
David A. Robb,
Xingkun Liu,
Atanas Laskov,
Pedro Patron,
Helen Hastie
Abstract:
Autonomous systems in remote locations have a high degree of autonomy and there is a need to explain what they are doing and why in order to increase transparency and maintain trust. Here, we describe a natural language chat interface that enables vehicle behaviour to be queried by the user. We obtain an interpretable model of autonomy through having an expert 'speak out-loud' and provide explanat…
▽ More
Autonomous systems in remote locations have a high degree of autonomy and there is a need to explain what they are doing and why in order to increase transparency and maintain trust. Here, we describe a natural language chat interface that enables vehicle behaviour to be queried by the user. We obtain an interpretable model of autonomy through having an expert 'speak out-loud' and provide explanations during a mission. This approach is agnostic to the type of autonomy model and as expert and operator are from the same user-group, we predict that these explanations will align well with the operator's mental model, increase transparency and assist with operator training.
△ Less
Submitted 6 March, 2018;
originally announced March 2018.
