-
Adaptive Human-Swarm Interaction based on Workload Measurement using Functional Near-Infrared Spectroscopy
Authors:
Ayodeji O. Abioye,
Aleksandra Landowska,
William Hunt,
Horia Maior,
Sarvapali D. Ramchurn,
Mohammad Naiseh,
Alec Banks,
Mohammad D. Soorati
Abstract:
One of the challenges of human-swarm interaction (HSI) is how to manage the operator's workload. In order to do this, we propose a novel neurofeedback technique for the real-time measurement of workload using functional near-infrared spectroscopy (fNIRS). The objective is to develop a baseline for workload measurement in human-swarm interaction using fNIRS and to develop an interface that dynamica…
▽ More
One of the challenges of human-swarm interaction (HSI) is how to manage the operator's workload. In order to do this, we propose a novel neurofeedback technique for the real-time measurement of workload using functional near-infrared spectroscopy (fNIRS). The objective is to develop a baseline for workload measurement in human-swarm interaction using fNIRS and to develop an interface that dynamically adapts to the operator's workload. The proposed method consists of using fNIRS device to measure brain activity, process this through a machine learning algorithm, and pass it on to the HSI interface. By dynamically adapting the HSI interface, the swarm operator's workload could be reduced and the performance improved.
△ Less
Submitted 13 May, 2024;
originally announced May 2024.
-
Hierarchical Distribution-Aware Testing of Deep Learning
Authors:
Wei Huang,
Xingyu Zhao,
Alec Banks,
Victoria Cox,
Xiaowei Huang
Abstract:
Deep Learning (DL) is increasingly used in safety-critical applications, raising concerns about its reliability. DL suffers from a well-known problem of lacking robustness, especially when faced with adversarial perturbations known as Adversarial Examples (AEs). Despite recent efforts to detect AEs using advanced attack and testing methods, these approaches often overlook the input distribution an…
▽ More
Deep Learning (DL) is increasingly used in safety-critical applications, raising concerns about its reliability. DL suffers from a well-known problem of lacking robustness, especially when faced with adversarial perturbations known as Adversarial Examples (AEs). Despite recent efforts to detect AEs using advanced attack and testing methods, these approaches often overlook the input distribution and perceptual quality of the perturbations. As a result, the detected AEs may not be relevant in practical applications or may appear unrealistic to human observers. This can waste testing resources on rare AEs that seldom occur during real-world use, limiting improvements in DL model dependability.
In this paper, we propose a new robustness testing approach for detecting AEs that considers both the feature level distribution and the pixel level distribution, capturing the perceptual quality of adversarial perturbations. The two considerations are encoded by a novel hierarchical mechanism. First, we select test seeds based on the density of feature level distribution and the vulnerability of adversarial robustness. The vulnerability of test seeds are indicated by the auxiliary information, that are highly correlated with local robustness. Given a test seed, we then develop a novel genetic algorithm based local test case generation method, in which two fitness functions work alternatively to control the perceptual quality of detected AEs. Finally, extensive experiments confirm that our holistic approach considering hierarchical distributions is superior to the state-of-the-arts that either disregard any input distribution or only consider a single (non-hierarchical) distribution, in terms of not only detecting imperceptible AEs but also improving the overall robustness of the DL model under testing.
△ Less
Submitted 1 September, 2023; v1 submitted 17 May, 2022;
originally announced May 2022.
-
Reliability Assessment and Safety Arguments for Machine Learning Components in System Assurance
Authors:
Yi Dong,
Wei Huang,
Vibhav Bharti,
Victoria Cox,
Alec Banks,
Sen Wang,
Xingyu Zhao,
Sven Schewe,
Xiaowei Huang
Abstract:
The increasing use of Machine Learning (ML) components embedded in autonomous systems -- so-called Learning-Enabled Systems (LESs) -- has resulted in the pressing need to assure their functional safety. As for traditional functional safety, the emerging consensus within both, industry and academia, is to use assurance cases for this purpose. Typically assurance cases support claims of reliability…
▽ More
The increasing use of Machine Learning (ML) components embedded in autonomous systems -- so-called Learning-Enabled Systems (LESs) -- has resulted in the pressing need to assure their functional safety. As for traditional functional safety, the emerging consensus within both, industry and academia, is to use assurance cases for this purpose. Typically assurance cases support claims of reliability in support of safety, and can be viewed as a structured way of organising arguments and evidence generated from safety analysis and reliability modelling activities. While such assurance activities are traditionally guided by consensus-based standards developed from vast engineering experience, LESs pose new challenges in safety-critical application due to the characteristics and design of ML models. In this article, we first present an overall assurance framework for LESs with an emphasis on quantitative aspects, e.g., breaking down system-level safety targets to component-level requirements and supporting claims stated in reliability metrics. We then introduce a novel model-agnostic Reliability Assessment Model (RAM) for ML classifiers that utilises the operational profile and robustness verification evidence. We discuss the model assumptions and the inherent challenges of assessing ML reliability uncovered by our RAM and propose solutions to practical use. Probabilistic safety argument templates at the lower ML component-level are also developed based on the RAM. Finally, to evaluate and demonstrate our methods, we not only conduct experiments on synthetic/benchmark datasets but also scope our methods with case studies on simulated Autonomous Underwater Vehicles and physical Unmanned Ground Vehicles.
△ Less
Submitted 14 October, 2022; v1 submitted 30 November, 2021;
originally announced December 2021.
-
Assessing the Reliability of Deep Learning Classifiers Through Robustness Evaluation and Operational Profiles
Authors:
Xingyu Zhao,
Wei Huang,
Alec Banks,
Victoria Cox,
David Flynn,
Sven Schewe,
Xiaowei Huang
Abstract:
The utilisation of Deep Learning (DL) is advancing into increasingly more sophisticated applications. While it shows great potential to provide transformational capabilities, DL also raises new challenges regarding its reliability in critical functions. In this paper, we present a model-agnostic reliability assessment method for DL classifiers, based on evidence from robustness evaluation and the…
▽ More
The utilisation of Deep Learning (DL) is advancing into increasingly more sophisticated applications. While it shows great potential to provide transformational capabilities, DL also raises new challenges regarding its reliability in critical functions. In this paper, we present a model-agnostic reliability assessment method for DL classifiers, based on evidence from robustness evaluation and the operational profile (OP) of a given application. We partition the input space into small cells and then "assemble" their robustness (to the ground truth) according to the OP, where estimators on the cells' robustness and OPs are provided. Reliability estimates in terms of the probability of misclassification per input (pmi) can be derived together with confidence levels. A prototype tool is demonstrated with simplified case studies. Model assumptions and extension to real-world applications are also discussed. While our model easily uncovers the inherent difficulties of assessing the DL dependability (e.g. lack of data with ground truth and scalability issues), we provide preliminary/compromised solutions to advance in this research direction.
△ Less
Submitted 2 June, 2021;
originally announced June 2021.
-
Remote Attestation: A Literature Review
Authors:
Alexander Sprogø Banks,
Marek Kisiel,
Philip Korsholm
Abstract:
With the rising number of IoT devices, the security of such devices becomes increasingly important. Remote attestation (RA) is a distinct security service that allows a remote verifer to reason about the state of an untrusted remote prover (device).
Paradigms of remote attestation span from exclusively software, in software-based attestation, to exclusively hardware-based. In between the extreme…
▽ More
With the rising number of IoT devices, the security of such devices becomes increasingly important. Remote attestation (RA) is a distinct security service that allows a remote verifer to reason about the state of an untrusted remote prover (device).
Paradigms of remote attestation span from exclusively software, in software-based attestation, to exclusively hardware-based. In between the extremes are hybrid attestation that utilize the enhanced security of secure hardware components in combination with the lower cost of purely software-based implementations.
Traditional remote attestation protocols are concerned with reasoning about the state of a prover. However, extensions to remote attestation also exist, such as code updates, device resets, erasure and attestation of the device's run-time state. Furthermore, as interconnected IoT devices are becoming increasingly more popular, so is the need for attestation of device swarms.
We will describe and evaluate the state-of-the-art for remote attestation, which covers singular attestation of devices as well as newer research in the area of formally verified RA protocols, swarm attestation and control-flow attestation.
△ Less
Submitted 12 May, 2021; v1 submitted 6 May, 2021;
originally announced May 2021.
-
A Safety Framework for Critical Systems Utilising Deep Neural Networks
Authors:
Xingyu Zhao,
Alec Banks,
James Sharp,
Valentin Robu,
David Flynn,
Michael Fisher,
Xiaowei Huang
Abstract:
Increasingly sophisticated mathematical modelling processes from Machine Learning are being used to analyse complex data. However, the performance and explainability of these models within practical critical systems requires a rigorous and continuous verification of their safe utilisation. Working towards addressing this challenge, this paper presents a principled novel safety argument framework f…
▽ More
Increasingly sophisticated mathematical modelling processes from Machine Learning are being used to analyse complex data. However, the performance and explainability of these models within practical critical systems requires a rigorous and continuous verification of their safe utilisation. Working towards addressing this challenge, this paper presents a principled novel safety argument framework for critical systems that utilise deep neural networks. The approach allows various forms of predictions, e.g., future reliability of passing some demands, or confidence on a required reliability level. It is supported by a Bayesian analysis using operational data and the recent verification and validation techniques for deep learning. The prediction is conservative -- it starts with partial prior knowledge obtained from lifecycle activities and then determines the worst-case prediction. Open challenges are also identified.
△ Less
Submitted 6 June, 2020; v1 submitted 7 March, 2020;
originally announced March 2020.
-
Interpretation as a factor in understanding flawed spreadsheets
Authors:
David A. Banks,
Ann Monday
Abstract:
The spreadsheet has been used by the business community for many years and yet still raises a number of significant concerns. As educators our concern is to try to develop the students skills in both the development of spreadsheets and in taking a critical view of their potential defects. In this paper we consider both the problems of mechanical production and the problems of translation of prob…
▽ More
The spreadsheet has been used by the business community for many years and yet still raises a number of significant concerns. As educators our concern is to try to develop the students skills in both the development of spreadsheets and in taking a critical view of their potential defects. In this paper we consider both the problems of mechanical production and the problems of translation of problem to spreadsheet representation.
△ Less
Submitted 11 January, 2008;
originally announced January 2008.
