-
Towards Continuous Assurance Case Creation for ADS with the Evidential Tool Bus
Authors:
Lev Sorokin,
Radouane Bouchekir,
Tewodros A. Beyene,
Brian Hsuan-Cheng Liao,
Adam Molin
Abstract:
An assurance case has become an integral component for the certification of safety-critical systems. While manually defining assurance case patterns can be not avoided, system-specific instantiations of assurance case patterns are both costly and time-consuming. It becomes especially complex to maintain an assurance case for a system when the requirements of the System-Under-Assurance change, or a…
▽ More
An assurance case has become an integral component for the certification of safety-critical systems. While manually defining assurance case patterns can be not avoided, system-specific instantiations of assurance case patterns are both costly and time-consuming. It becomes especially complex to maintain an assurance case for a system when the requirements of the System-Under-Assurance change, or an assurance claim becomes invalid due to, e.g., degradation of a systems component, as common when deploying learning-enabled components. In this paper, we report on our preliminary experience leveraging the tool integration framework Evidential Tool Bus (ETB) for the construction and continuous maintenance of an assurance case from a predefined assurance case pattern. Specifically, we demonstrate the assurance process on an industrial Automated Valet Parking system from the automotive domain. We present the formalization of the provided assurance case pattern in the ETB processable logical specification language of workflows. Our findings show that ETB is able to create and maintain evidence required for the construction of an assurance case.
△ Less
Submitted 4 March, 2024;
originally announced March 2024.
-
Guiding the Search Towards Failure-Inducing Test Inputs Using Support Vector Machines
Authors:
Lev Sorokin,
Niklas Kerscher
Abstract:
In this paper, we present NSGA-II-SVM (Non-dominated Sorting Genetic Algorithm with Support Vector Machine Guidance), a novel learnable evolutionary and search-based testing algorithm that leverages Support Vector Machine (SVM) classification models to direct the search towards failure-revealing test inputs. Supported by genetic search, NSGA-II-SVM creates iteratively SVM-based models of the test…
▽ More
In this paper, we present NSGA-II-SVM (Non-dominated Sorting Genetic Algorithm with Support Vector Machine Guidance), a novel learnable evolutionary and search-based testing algorithm that leverages Support Vector Machine (SVM) classification models to direct the search towards failure-revealing test inputs. Supported by genetic search, NSGA-II-SVM creates iteratively SVM-based models of the test input space, learning which regions in the search space are promising to be explored. A subsequent sampling and repetition of evolutionary search iterations allow to refine and make the model more accurate in the prediction. Our preliminary evaluation of NSGA-II-SVM by testing an Automated Valet Parking system shows that NSGA-II-SVM is more effective in identifying more critical test cases than a state of the art learnable evolutionary testing technique as well as naive random search.
△ Less
Submitted 22 January, 2024;
originally announced January 2024.
-
Monitoring Auditable Claims in the Cloud
Authors:
Lev Sorokin,
Ulrich Schoepp
Abstract:
When deploying mission-critical systems in the cloud, where deviations may have severe consequences, the assurance of critical decisions becomes essential. Typical cloud systems are operated by third parties and are built on complex software stacks consisting of e.g., Kubernetes, Istio, or Kafka, which due to their size are difficult to be verified. Nevertheless, one needs to make sure that missio…
▽ More
When deploying mission-critical systems in the cloud, where deviations may have severe consequences, the assurance of critical decisions becomes essential. Typical cloud systems are operated by third parties and are built on complex software stacks consisting of e.g., Kubernetes, Istio, or Kafka, which due to their size are difficult to be verified. Nevertheless, one needs to make sure that mission-critical choices are made correctly. We propose a flexible runtime monitoring approach that is independent of the implementation of the observed system that allows to monitor safety and data-related properties. Our approach is based on combining distributed Datalog-based programs with tamper-proof storage based on Trillian to verify the premises of safety-critical actions. The approach can be seen as a generalization of the Certificate Transparency project. We apply our approach to an industrial use case that uses a cloud infrastructure for orchestrating unmanned air vehicles.
△ Less
Submitted 19 December, 2023;
originally announced December 2023.
-
OpenSBT: A Modular Framework for Search-based Testing of Automated Driving Systems
Authors:
Lev Sorokin,
Tiziano Munaro,
Damir Safin,
Brian Hsuan-Cheng Liao,
Adam Molin
Abstract:
Search-based software testing (SBT) is an effective and efficient approach for testing automated driving systems (ADS). However, testing pipelines for ADS testing are particularly challenging as they involve integrating complex driving simulation platforms and establishing communication protocols and APIs with the desired search algorithm. This complexity prevents a wide adoption of SBT and thorou…
▽ More
Search-based software testing (SBT) is an effective and efficient approach for testing automated driving systems (ADS). However, testing pipelines for ADS testing are particularly challenging as they involve integrating complex driving simulation platforms and establishing communication protocols and APIs with the desired search algorithm. This complexity prevents a wide adoption of SBT and thorough empirical comparative experiments with different simulators and search approaches. We present OpenSBT, an open-source, modular and extensible framework to facilitate the SBT of ADS. With OpenSBT, it is possible to integrate simulators with an embedded system under test, search algorithms and fitness functions for testing. We describe the architecture and show the usage of our framework by applying different search algorithms for testing Automated Emergency Braking Systems in CARLA as well in the high-fidelity Prescan simulator in collaboration with our industrial partner DENSO. OpenSBT is available at https://git.fortiss.org/opensbt. A demo video is provided here: https://youtu.be/6csl\_UAOD\_4.
△ Less
Submitted 2 November, 2023; v1 submitted 17 June, 2023;
originally announced June 2023.
-
Reflections on Surrogate-Assisted Search-Based Testing: A Taxonomy and Two Replication Studies based on Industrial ADAS and Simulink Models
Authors:
Shiva Nejati,
Lev Sorokin,
Damir Safin,
Federico Formica,
Mohammad Mahdi Mahboob,
Claudio Menghi
Abstract:
Surrogate-assisted search-based testing (SA-SBT) aims to reduce the computational time for testing compute-intensive systems. Surrogates enhance testing techniques by improving test case generation focusing the testing budget on the most critical portions of the input domain. In addition, they can serve as approximations of the system under test (SUT) to predict tests' results instead of executing…
▽ More
Surrogate-assisted search-based testing (SA-SBT) aims to reduce the computational time for testing compute-intensive systems. Surrogates enhance testing techniques by improving test case generation focusing the testing budget on the most critical portions of the input domain. In addition, they can serve as approximations of the system under test (SUT) to predict tests' results instead of executing the tests on compute-intensive SUTs. This article reflects on the existing SA-SBT techniques, particularly those applied to system-level testing and often facilitated using simulators or complex test beds. Our objective is to synthesize different heuristic algorithms and evaluation methods employed in existing SA-SBT techniques and present a comprehensive view of SA-SBT solutions. In addition, by critically reviewing our previous work on SA-SBT, we aim to identify the limitations in our proposed algorithms and evaluation methods and to propose potential improvements. We present a taxonomy that categorizes and contrasts existing SA-SBT solutions and highlights key research gaps. To identify the evaluation challenges, we conduct two replication studies of our past SA-SBT solutions: One study uses industrial advanced driver assistance system (ADAS) and the other relies on a Simulink model benchmark. We compare our results with those of the original studies and identify the difficulties in evaluating SA-SBT techniques, including the impact of different contextual factors on results generalization and the validity of our evaluation metrics. Based on our taxonomy and replication studies, we propose future research directions, including re-considerations in the current evaluation metrics used for SA-SBT solutions, utilizing surrogates for fault localization and repair in addition to testing, and creating frameworks for large-scale experiments by applying SA-SBT to multiple SUTs and simulators.
△ Less
Submitted 28 April, 2023;
originally announced May 2023.
-
Towards Auditable Distributed Systems
Authors:
Lev Sorokin
Abstract:
The emerging trend towards distributed (cloud) systems (DS) has widely arrived whether in the automotive, public or the financial sector, but the execution of services of heterogeneous service providers is exposed to several risks. Beside hardware/software faults or cyber attacks that can influence the correctness of the system, fraud is also an issue. In such case it is not only important to veri…
▽ More
The emerging trend towards distributed (cloud) systems (DS) has widely arrived whether in the automotive, public or the financial sector, but the execution of services of heterogeneous service providers is exposed to several risks. Beside hardware/software faults or cyber attacks that can influence the correctness of the system, fraud is also an issue. In such case it is not only important to verify the correctness of the system, but also have evidence which component and participant behaves faulty. This makes it possible, e.g. to claim for compensation after systems execution but also to assure information for verification can be trusted. The main goal of our research is to assure the monitoring of DS based on auditable information. We follow a decentralized monitoring strategy and envision a distributed monitoring approach of system properties based on distributedlogic programs that consider auditability. The expected contribution of this work is to establish with the application of our framework the mutual trust of distributed parties, as well as trust of clients in the systems execution. We showcase our ideas on a DS for booking services with unmanned air vehicles.
△ Less
Submitted 28 September, 2022;
originally announced September 2022.
