-
A Survey of Language-Based Communication in Robotics
Authors:
William Hunt,
Sarvapali D. Ramchurn,
Mohammad D. Soorati
Abstract:
Embodied robots which can interact with their environment and neighbours are increasingly being used as a test case to develop Artificial Intelligence. This creates a need for multimodal robot controllers which can operate across different types of information including text. Large Language Models are able to process and generate textual as well as audiovisual data and, more recently, robot action…
▽ More
Embodied robots which can interact with their environment and neighbours are increasingly being used as a test case to develop Artificial Intelligence. This creates a need for multimodal robot controllers which can operate across different types of information including text. Large Language Models are able to process and generate textual as well as audiovisual data and, more recently, robot actions. Language Models are increasingly being applied to robotic systems; these Language-Based robots leverage the power of language models in a variety of ways. Additionally, the use of language opens up multiple forms of information exchange between members of a human-robot team. This survey motivates the use of language models in robotics, and then delineates works based on the part of the overall control flow in which language is incorporated. Language can be used by human to task a robot, by a robot to inform a human, between robots as a human-like communication medium, and internally for a robot's planning and control. Applications of language-based robots are explored, and finally numerous limitations and challenges are discussed to provide a summary of the development needed for language-based robotics moving forward. Links to each paper and, if available, source code are made available in the accompanying site at https://uos-haris.online/sooratilab/papers/WillSurvey/LangRobotSurvey.php
△ Less
Submitted 6 June, 2024;
originally announced June 2024.
-
Adaptive Human-Swarm Interaction based on Workload Measurement using Functional Near-Infrared Spectroscopy
Authors:
Ayodeji O. Abioye,
Aleksandra Landowska,
William Hunt,
Horia Maior,
Sarvapali D. Ramchurn,
Mohammad Naiseh,
Alec Banks,
Mohammad D. Soorati
Abstract:
One of the challenges of human-swarm interaction (HSI) is how to manage the operator's workload. In order to do this, we propose a novel neurofeedback technique for the real-time measurement of workload using functional near-infrared spectroscopy (fNIRS). The objective is to develop a baseline for workload measurement in human-swarm interaction using fNIRS and to develop an interface that dynamica…
▽ More
One of the challenges of human-swarm interaction (HSI) is how to manage the operator's workload. In order to do this, we propose a novel neurofeedback technique for the real-time measurement of workload using functional near-infrared spectroscopy (fNIRS). The objective is to develop a baseline for workload measurement in human-swarm interaction using fNIRS and to develop an interface that dynamically adapts to the operator's workload. The proposed method consists of using fNIRS device to measure brain activity, process this through a machine learning algorithm, and pass it on to the HSI interface. By dynamically adapting the HSI interface, the swarm operator's workload could be reduced and the performance improved.
△ Less
Submitted 13 May, 2024;
originally announced May 2024.
-
Collective Decision-Making on Task Allocation Feasibility
Authors:
Samratul Fuady,
Danesh Tarapore,
Shoaib Ehsan,
Mohammad D. Soorati
Abstract:
Robot swarms offer the potential to bring several advantages to the real-world applications but deploying them presents challenges in ensuring feasibility across diverse environments. Assessing the feasibility of new tasks for swarms is crucial to ensure the effective utilisation of resources, as well as to provide awareness of the suitability of a swarm solution for a particular task. In this pap…
▽ More
Robot swarms offer the potential to bring several advantages to the real-world applications but deploying them presents challenges in ensuring feasibility across diverse environments. Assessing the feasibility of new tasks for swarms is crucial to ensure the effective utilisation of resources, as well as to provide awareness of the suitability of a swarm solution for a particular task. In this paper, we introduce the concept of distributed feasibility, where the swarm collectively assesses the feasibility of task allocation based on local observations and interactions. We apply Direct Modulation of Majority-based Decisions as our collective decision-making strategy and show that, in a homogeneous setting, the swarm is able to collectively decide whether a given setup has a high or low feasibility as long as the robot-to-task ratio is not near one.
△ Less
Submitted 13 May, 2024;
originally announced May 2024.
-
Conversational Language Models for Human-in-the-Loop Multi-Robot Coordination
Authors:
William Hunt,
Toby Godfrey,
Mohammad D. Soorati
Abstract:
With the increasing prevalence and diversity of robots interacting in the real world, there is need for flexible, on-the-fly planning and cooperation. Large Language Models are starting to be explored in a multimodal setup for communication, coordination, and planning in robotics. Existing approaches generally use a single agent building a plan, or have multiple homogeneous agents coordinating for…
▽ More
With the increasing prevalence and diversity of robots interacting in the real world, there is need for flexible, on-the-fly planning and cooperation. Large Language Models are starting to be explored in a multimodal setup for communication, coordination, and planning in robotics. Existing approaches generally use a single agent building a plan, or have multiple homogeneous agents coordinating for a simple task. We present a decentralised, dialogical approach in which a team of agents with different abilities plans solutions through peer-to-peer and human-robot discussion. We suggest that argument-style dialogues are an effective way to facilitate adaptive use of each agent's abilities within a cooperative team. Two robots discuss how to solve a cleaning problem set by a human, define roles, and agree on paths they each take. Each step can be interrupted by a human advisor and agents check their plans with the human. Agents then execute this plan in the real world, collecting rubbish from people in each room. Our implementation uses text at every step, maintaining transparency and effective human-multi-robot interaction.
△ Less
Submitted 29 February, 2024;
originally announced February 2024.
-
The Effect of Predictive Formal Modelling at Runtime on Performance in Human-Swarm Interaction
Authors:
Ayodeji O. Abioye,
William Hunt,
Yue Gu,
Eike Schneiders,
Mohammad Naiseh,
Joel E. Fischer,
Sarvapali D. Ramchurn,
Mohammad D. Soorati,
Blair Archibald,
Michele Sevegnani
Abstract:
Formal Modelling is often used as part of the design and testing process of software development to ensure that components operate within suitable bounds even in unexpected circumstances. In this paper, we use predictive formal modelling (PFM) at runtime in a human-swarm mission and show that this integration can be used to improve the performance of human-swarm teams. We recruited 60 participants…
▽ More
Formal Modelling is often used as part of the design and testing process of software development to ensure that components operate within suitable bounds even in unexpected circumstances. In this paper, we use predictive formal modelling (PFM) at runtime in a human-swarm mission and show that this integration can be used to improve the performance of human-swarm teams. We recruited 60 participants to operate a simulated aerial swarm to deliver parcels to target locations. In the PFM condition, operators were informed of the estimated completion times given the number of drones deployed, whereas in the No-PFM condition, operators did not have this information. The operators could control the mission by adding or removing drones from the mission and thereby, increasing or decreasing the overall mission cost. The evaluation of human-swarm performance relied on four key metrics: the time taken to complete tasks, the number of agents involved, the total number of tasks accomplished, and the overall cost associated with the human-swarm task. Our results show that PFM modelling at runtime improves mission performance without significantly affecting the operator's workload or the system's usability.
△ Less
Submitted 22 January, 2024;
originally announced January 2024.
-
Outlining the design space of eXplainable swarm (xSwarm): experts perspective
Authors:
Mohammad Naiseh,
Mohammad D. Soorati,
Sarvapali Ramchurn
Abstract:
In swarm robotics, agents interact through local roles to solve complex tasks beyond an individual's ability. Even though swarms are capable of carrying out some operations without the need for human intervention, many safety-critical applications still call for human operators to control and monitor the swarm. There are novel challenges to effective Human-Swarm Interaction (HSI) that are only beg…
▽ More
In swarm robotics, agents interact through local roles to solve complex tasks beyond an individual's ability. Even though swarms are capable of carrying out some operations without the need for human intervention, many safety-critical applications still call for human operators to control and monitor the swarm. There are novel challenges to effective Human-Swarm Interaction (HSI) that are only beginning to be addressed. Explainability is one factor that can facilitate effective and trustworthy HSI and improve the overall performance of Human-Swarm team. Explainability was studied across various Human-AI domains, such as Human-Robot Interaction and Human-Centered ML. However, it is still ambiguous whether explanations studied in Human-AI literature would be beneficial in Human-Swarm research and development. Furthermore, the literature lacks foundational research on the prerequisites for explainability requirements in swarm robotics, i.e., what kind of questions an explainable swarm is expected to answer, and what types of explanations a swarm is expected to generate. By surveying 26 swarm experts, we seek to answer these questions and identify challenges experts faced to generate explanations in Human-Swarm environments. Our work contributes insights into defining a new area of research of eXplainable Swarm (xSwarm) which looks at how explainability can be implemented and developed in swarm systems. This paper opens the discussion on xSwarm and paves the way for more research in the field.
△ Less
Submitted 3 September, 2023;
originally announced September 2023.
-
The Effect of Data Visualisation Quality and Task Density on Human-Swarm Interaction
Authors:
Ayodeji O. Abioye,
Mohammad Naiseh,
William Hunt,
Jediah Clark,
Sarvapali D. Ramchurn,
Mohammad D. Soorati
Abstract:
Despite the advantages of having robot swarms, human supervision is required for real-world applications. The performance of the human-swarm system depends on several factors including the data availability for the human operators. In this paper, we study the human factors aspect of the human-swarm interaction and investigate how having access to high-quality data can affect the performance of the…
▽ More
Despite the advantages of having robot swarms, human supervision is required for real-world applications. The performance of the human-swarm system depends on several factors including the data availability for the human operators. In this paper, we study the human factors aspect of the human-swarm interaction and investigate how having access to high-quality data can affect the performance of the human-swarm system - the number of tasks completed and the human trust level in operation. We designed an experiment where a human operator is tasked to operate a swarm to identify casualties in an area within a given time period. One group of operators had the option to request high-quality pictures while the other group had to base their decision on the available low-quality images. We performed a user study with 120 participants and recorded their success rate (directly logged via the simulation platform) as well as their workload and trust level (measured through a questionnaire after completing a human-swarm scenario). The findings from our study indicated that the group granted access to high-quality data exhibited an increased workload and placed greater trust in the swarm, thus confirming our initial hypothesis. However, we also found that the number of accurately identified casualties did not significantly vary between the two groups, suggesting that data quality had no impact on the successful completion of tasks.
△ Less
Submitted 17 July, 2023;
originally announced July 2023.
-
Demonstrating Performance Benefits of Human-Swarm Teaming
Authors:
William Hunt,
Jack Ryan,
Ayodeji O. Abioye,
Sarvapali D. Ramchurn,
Mohammad D. Soorati
Abstract:
Autonomous swarms of robots can bring robustness, scalability and adaptability to safety-critical tasks such as search and rescue but their application is still very limited. Using semi-autonomous swarms with human control can bring robot swarms to real-world applications. Human operators can define goals for the swarm, monitor their performance and interfere with, or overrule, the decisions and b…
▽ More
Autonomous swarms of robots can bring robustness, scalability and adaptability to safety-critical tasks such as search and rescue but their application is still very limited. Using semi-autonomous swarms with human control can bring robot swarms to real-world applications. Human operators can define goals for the swarm, monitor their performance and interfere with, or overrule, the decisions and behaviour. We present the ``Human And Robot Interactive Swarm'' simulator (HARIS) that allows multi-user interaction with a robot swarm and facilitates qualitative and quantitative user studies through simulation of robot swarms completing tasks, from package delivery to search and rescue, with varying levels of human control. In this demonstration, we showcase the simulator by using it to study the performance gain offered by maintaining a ``human-in-the-loop'' over a fully autonomous system as an example. This is illustrated in the context of search and rescue, with an autonomous allocation of resources to those in need.
△ Less
Submitted 23 March, 2023; v1 submitted 22 March, 2023;
originally announced March 2023.
-
From Intelligent Agents to Trustworthy Human-Centred Multiagent Systems
Authors:
Mohammad Divband Soorati,
Enrico H. Gerding,
Enrico Marchioni,
Pavel Naumov,
Timothy J. Norman,
Sarvapali D. Ramchurn,
Bahar Rastegari,
Adam Sobey,
Sebastian Stein,
Danesh Tarpore,
Vahid Yazdanpanah,
Jie Zhang
Abstract:
The Agents, Interaction and Complexity research group at the University of Southampton has a long track record of research in multiagent systems (MAS). We have made substantial scientific contributions across learning in MAS, game-theoretic techniques for coordinating agent systems, and formal methods for representation and reasoning. We highlight key results achieved by the group and elaborate on…
▽ More
The Agents, Interaction and Complexity research group at the University of Southampton has a long track record of research in multiagent systems (MAS). We have made substantial scientific contributions across learning in MAS, game-theoretic techniques for coordinating agent systems, and formal methods for representation and reasoning. We highlight key results achieved by the group and elaborate on recent work and open research challenges in develo** trustworthy autonomous systems and deploying human-centred AI systems that aim to support societal good.
△ Less
Submitted 5 October, 2022;
originally announced October 2022.
-
Collective Decision Making in Communication-Constrained Environments
Authors:
Thomas G. Kelly,
Mohammad Divband Soorati,
Klaus-Peter Zauner,
Sarvapali D. Ramchurn,
Danesh Tarapore
Abstract:
One of the main tasks for autonomous robot swarms is to collectively decide on the best available option. Achieving that requires a high quality communication between the agents that may not be always available in a real world environment. In this paper we introduce the communication-constrained collective decision-making problem where some areas of the environment limit the agents' ability to com…
▽ More
One of the main tasks for autonomous robot swarms is to collectively decide on the best available option. Achieving that requires a high quality communication between the agents that may not be always available in a real world environment. In this paper we introduce the communication-constrained collective decision-making problem where some areas of the environment limit the agents' ability to communicate, either by reducing success rate or blocking the communication channels. We propose a decentralised algorithm for map** environmental features for robot swarms as well as improving collective decision making in communication-limited environments without prior knowledge of the communication landscape. Our results show that making a collective aware of the communication environment can improve the speed of convergence in the presence of communication limitations, at least 3 times faster, without sacrificing accuracy.
△ Less
Submitted 19 July, 2022;
originally announced July 2022.
-
Industry Led Use-Case Development for Human-Swarm Operations
Authors:
Jediah R. Clark,
Mohammad Naiseh,
Joel Fischer,
Marise Galvez Trigo,
Katie Parnell,
Mario Brito,
Adrian Bodenmann,
Sarvapali D. Ramchurn,
Mohammad Divband Soorati
Abstract:
In the domain of unmanned vehicles, autonomous robotic swarms promise to deliver increased efficiency and collective autonomy. How these swarms will operate in the future, and what communication requirements and operational boundaries will arise are yet to be sufficiently defined. A workshop was conducted with 11 professional unmanned-vehicle operators and designers with the objective of identifyi…
▽ More
In the domain of unmanned vehicles, autonomous robotic swarms promise to deliver increased efficiency and collective autonomy. How these swarms will operate in the future, and what communication requirements and operational boundaries will arise are yet to be sufficiently defined. A workshop was conducted with 11 professional unmanned-vehicle operators and designers with the objective of identifying use-cases for develo** and testing robotic swarms. Three scenarios were defined by experts and were then compiled to produce a single use case outlining the scenario, objectives, agents, communication requirements and stages of operation when collaborating with highly autonomous swarms. Our compiled use case is intended for researchers, designers, and manufacturers alike to test and tailor their design pipeline to accommodate for some of the key issues in human-swarm ininteraction. Examples of application include informing simulation development, forming the basis of further design workshops, and identifying trust issues that may arise between human operators and the swarm.
△ Less
Submitted 24 July, 2022; v1 submitted 19 July, 2022;
originally announced July 2022.
-
Repository for Reusing Artifacts of Artificial Neural Networks
Authors:
Javad Ghofrani,
Ehsan Kozegar,
Mohammad Divband Soorati,
Arezoo Bozorgmehr,
Hongfei Chen,
Maximilian Naake
Abstract:
Artificial Neural Networks (ANNs) replaced conventional software systems in various domains such as machine translation, natural language processing, and image processing. So, why do we need an repository for artificial neural networks? Those systems are developed with labeled data and we have strong dependencies between the data that is used for training and testing our network. Another challenge…
▽ More
Artificial Neural Networks (ANNs) replaced conventional software systems in various domains such as machine translation, natural language processing, and image processing. So, why do we need an repository for artificial neural networks? Those systems are developed with labeled data and we have strong dependencies between the data that is used for training and testing our network. Another challenge is the data quality as well as reuse-ability. There we are trying to apply concepts from classic software engineering that is not limited to the model, while data and code haven't been dealt with mostly in other projects. The first question that comes to mind might be, why don't we use GitHub, a well known widely spread tool for reuse, for our issue. And the reason why is that GitHub, although very good in its class is not developed for machine learning appliances and focuses more on software reuse. In addition to that GitHub does not allow to execute the code directly on the platform which would be very convenient for collaborative work on one project.
△ Less
Submitted 30 March, 2020;
originally announced March 2020.
-
Flora robotica -- An Architectural System Combining Living Natural Plants and Distributed Robots
Authors:
Heiko Hamann,
Mohammad Divband Soorati,
Mary Katherine Heinrich,
Daniel Nicolas Hofstadler,
Igor Kuksin,
Frank Veenstra,
Mostafa Wahby,
Stig Anton Nielsen,
Sebastian Risi,
Tomasz Skrzypczak,
Payam Zahadat,
Przemyslaw Wojtaszek,
Kasper Støy,
Thomas Schmickl,
Serge Kernbach,
Phil Ayres
Abstract:
Key to our project flora robotica is the idea of creating a bio-hybrid system of tightly coupled natural plants and distributed robots to grow architectural artifacts and spaces. Our motivation with this ground research project is to lay a principled foundation towards the design and implementation of living architectural systems that provide functionalities beyond those of orthodox building pract…
▽ More
Key to our project flora robotica is the idea of creating a bio-hybrid system of tightly coupled natural plants and distributed robots to grow architectural artifacts and spaces. Our motivation with this ground research project is to lay a principled foundation towards the design and implementation of living architectural systems that provide functionalities beyond those of orthodox building practice, such as self-repair, material accumulation and self-organization. Plants and robots work together to create a living organism that is inhabited by human beings. User-defined design objectives help to steer the directional growth of the plants, but also the system's interactions with its inhabitants determine locations where growth is prohibited or desired (e.g., partitions, windows, occupiable space). We report our plant species selection process and aspects of living architecture. A leitmotif of our project is the rich concept of braiding: braids are produced by robots from continuous material and serve as both scaffolds and initial architectural artifacts before plants take over and grow the desired architecture. We use light and hormones as attraction stimuli and far-red light as repelling stimulus to influence the plants. Applied sensors range from simple proximity sensing to detect the presence of plants to sophisticated sensing technology, such as electrophysiology and measurements of sap flow. We conclude by discussing our anticipated final demonstrator that integrates key features of flora robotica, such as the continuous growth process of architectural artifacts and self-repair of living architecture.
△ Less
Submitted 13 September, 2017;
originally announced September 2017.
