-
OpenSBT: A Modular Framework for Search-based Testing of Automated Driving Systems
Authors:
Lev Sorokin,
Tiziano Munaro,
Damir Safin,
Brian Hsuan-Cheng Liao,
Adam Molin
Abstract:
Search-based software testing (SBT) is an effective and efficient approach for testing automated driving systems (ADS). However, testing pipelines for ADS testing are particularly challenging as they involve integrating complex driving simulation platforms and establishing communication protocols and APIs with the desired search algorithm. This complexity prevents a wide adoption of SBT and thorou…
▽ More
Search-based software testing (SBT) is an effective and efficient approach for testing automated driving systems (ADS). However, testing pipelines for ADS testing are particularly challenging as they involve integrating complex driving simulation platforms and establishing communication protocols and APIs with the desired search algorithm. This complexity prevents a wide adoption of SBT and thorough empirical comparative experiments with different simulators and search approaches. We present OpenSBT, an open-source, modular and extensible framework to facilitate the SBT of ADS. With OpenSBT, it is possible to integrate simulators with an embedded system under test, search algorithms and fitness functions for testing. We describe the architecture and show the usage of our framework by applying different search algorithms for testing Automated Emergency Braking Systems in CARLA as well in the high-fidelity Prescan simulator in collaboration with our industrial partner DENSO. OpenSBT is available at https://git.fortiss.org/opensbt. A demo video is provided here: https://youtu.be/6csl\_UAOD\_4.
△ Less
Submitted 2 November, 2023; v1 submitted 17 June, 2023;
originally announced June 2023.
-
Reflections on Surrogate-Assisted Search-Based Testing: A Taxonomy and Two Replication Studies based on Industrial ADAS and Simulink Models
Authors:
Shiva Nejati,
Lev Sorokin,
Damir Safin,
Federico Formica,
Mohammad Mahdi Mahboob,
Claudio Menghi
Abstract:
Surrogate-assisted search-based testing (SA-SBT) aims to reduce the computational time for testing compute-intensive systems. Surrogates enhance testing techniques by improving test case generation focusing the testing budget on the most critical portions of the input domain. In addition, they can serve as approximations of the system under test (SUT) to predict tests' results instead of executing…
▽ More
Surrogate-assisted search-based testing (SA-SBT) aims to reduce the computational time for testing compute-intensive systems. Surrogates enhance testing techniques by improving test case generation focusing the testing budget on the most critical portions of the input domain. In addition, they can serve as approximations of the system under test (SUT) to predict tests' results instead of executing the tests on compute-intensive SUTs. This article reflects on the existing SA-SBT techniques, particularly those applied to system-level testing and often facilitated using simulators or complex test beds. Our objective is to synthesize different heuristic algorithms and evaluation methods employed in existing SA-SBT techniques and present a comprehensive view of SA-SBT solutions. In addition, by critically reviewing our previous work on SA-SBT, we aim to identify the limitations in our proposed algorithms and evaluation methods and to propose potential improvements. We present a taxonomy that categorizes and contrasts existing SA-SBT solutions and highlights key research gaps. To identify the evaluation challenges, we conduct two replication studies of our past SA-SBT solutions: One study uses industrial advanced driver assistance system (ADAS) and the other relies on a Simulink model benchmark. We compare our results with those of the original studies and identify the difficulties in evaluating SA-SBT techniques, including the impact of different contextual factors on results generalization and the validity of our evaluation metrics. Based on our taxonomy and replication studies, we propose future research directions, including re-considerations in the current evaluation metrics used for SA-SBT solutions, utilizing surrogates for fault localization and repair in addition to testing, and creating frameworks for large-scale experiments by applying SA-SBT to multiple SUTs and simulators.
△ Less
Submitted 28 April, 2023;
originally announced May 2023.
-
Numerical integration over implicitly defined domains for higher order unfitted finite element methods
Authors:
Maxim Olshanskii,
Danil Safin
Abstract:
The paper studies several approaches to numerical integration over a domain defined implicitly by an indicator function such as the level set function. The integration methods are based on subdivision, moment--fitting, local quasi-parametrization and Monte-Carlo techniques. As an application of these techniques, the paper addresses numerical solution of elliptic PDEs posed on domains and manifolds…
▽ More
The paper studies several approaches to numerical integration over a domain defined implicitly by an indicator function such as the level set function. The integration methods are based on subdivision, moment--fitting, local quasi-parametrization and Monte-Carlo techniques. As an application of these techniques, the paper addresses numerical solution of elliptic PDEs posed on domains and manifolds defined implicitly. A higher order unfitted finite element method (FEM) is assumed for the discretization. In such a method the underlying mesh is not fitted to the geometry, and hence the errors of numerical integration over curvilinear elements affect the accuracy of the finite element solution together with approximation errors. The paper studies the numerical complexity of the integration procedures and the performance of unfitted FEMs which employ these tools.
△ Less
Submitted 22 January, 2016;
originally announced January 2016.
-
A narrow-band unfitted finite element method for elliptic PDEs posed on surfaces
Authors:
Maxim A. Olshanskii,
Danil Safin
Abstract:
The paper studies a method for solving elliptic partial differential equations posed on hypersurfaces in $\mathbb{R}^N$, $N=2,3$. The method allows a surface to be given implicitly as a zero level of a level set function. A surface equation is extended to a narrow-band neighborhood of the surface. The resulting extended equation is a non-degenerate PDE and it is solved on a bulk mesh that is unali…
▽ More
The paper studies a method for solving elliptic partial differential equations posed on hypersurfaces in $\mathbb{R}^N$, $N=2,3$. The method allows a surface to be given implicitly as a zero level of a level set function. A surface equation is extended to a narrow-band neighborhood of the surface. The resulting extended equation is a non-degenerate PDE and it is solved on a bulk mesh that is unaligned to the surface. An unfitted finite element method is used to discretize extended equations. Error estimates are proved for finite element solutions in the bulk domain and restricted to the surface. The analysis admits finite elements of a higher order and gives sufficient conditions for archiving the optimal convergence order in the energy norm. Several numerical examples illustrate the properties of the method.
△ Less
Submitted 14 January, 2015; v1 submitted 29 January, 2014;
originally announced January 2014.
