-
Behavior Forests: Real-Time Discovery of Dynamic Behavior for Data Selection
Authors:
Philipp Reis,
Philipp Rigoll,
Eric Sax
Abstract:
Automated Driving Systems (ADS) development relies on utilizing real-world vehicle data. The volume of data generated by modern vehicles presents transmission, storage, and computational challenges. Focusing on Dynamic Behavior (DB) offers a promising approach to distinguish relevant from irrelevant information for ADS functionalities, thereby reducing data. Time series pattern recognition is bene…
▽ More
Automated Driving Systems (ADS) development relies on utilizing real-world vehicle data. The volume of data generated by modern vehicles presents transmission, storage, and computational challenges. Focusing on Dynamic Behavior (DB) offers a promising approach to distinguish relevant from irrelevant information for ADS functionalities, thereby reducing data. Time series pattern recognition is beneficial for this task as it can analyze the temporal context of vehicle driving behavior. However, existing state-of-the-art methods often lack the adaptability to identify variable-length patterns or provide analytical descriptions of discovered patterns. This contribution proposes a Behavior Forest framework for real-time data selection by constructing a Behavior Graph during vehicle operation, facilitating analytical descriptions without pre-training. The method demonstrates its performance using a synthetically generated and electrocardiogram data set. An automotive time series data set is used to evaluate the data reduction capabilities, in which this method discarded 96.01% of the incoming data stream, while relevant DB remain included.
△ Less
Submitted 4 July, 2024; v1 submitted 2 July, 2024;
originally announced July 2024.
-
GOOSE: Goal-Conditioned Reinforcement Learning for Safety-Critical Scenario Generation
Authors:
Joshua Ransiek,
Johannes Plaum,
Jacob Langner,
Eric Sax
Abstract:
Scenario-based testing is considered state-of-the-art for verifying and validating Advanced Driver Assistance Systems (ADASs) and Automated Driving Systems (ADSs). However, the practical application of scenario-based testing requires an efficient method to generate or collect the scenarios that are needed for the safety assessment. In this paper, we propose Goal-conditioned Scenario Generation (GO…
▽ More
Scenario-based testing is considered state-of-the-art for verifying and validating Advanced Driver Assistance Systems (ADASs) and Automated Driving Systems (ADSs). However, the practical application of scenario-based testing requires an efficient method to generate or collect the scenarios that are needed for the safety assessment. In this paper, we propose Goal-conditioned Scenario Generation (GOOSE), a goal-conditioned reinforcement learning (RL) approach that automatically generates safety-critical scenarios to challenge ADASs or ADSs. In order to simultaneously set up and optimize scenarios, we propose to control vehicle trajectories at the scenario level. Each step in the RL framework corresponds to a scenario simulation. We use Non-Uniform Rational B-Splines (NURBS) for trajectory modeling. To guide the goal-conditioned agent, we formulate test-specific, constraint-based goals inspired by the OpenScenario Domain Specific Language(DSL). Through experiments conducted on multiple pre-crash scenarios derived from UN Regulation No. 157 for Active Lane Kee** Systems (ALKS), we demonstrate the effectiveness of GOOSE in generating scenarios that lead to safety-critical events.
△ Less
Submitted 6 June, 2024;
originally announced June 2024.
-
CLIP** the Limits: Finding the Sweet Spot for Relevant Images in Automated Driving Systems Perception Testing
Authors:
Philipp Rigoll,
Laurenz Adolph,
Lennart Ries,
Eric Sax
Abstract:
Perception systems, especially cameras, are the eyes of automated driving systems. Ensuring that they function reliably and robustly is therefore an important building block in the automation of vehicles. There are various approaches to test the perception of automated driving systems. Ultimately, however, it always comes down to the investigation of the behavior of perception systems under specif…
▽ More
Perception systems, especially cameras, are the eyes of automated driving systems. Ensuring that they function reliably and robustly is therefore an important building block in the automation of vehicles. There are various approaches to test the perception of automated driving systems. Ultimately, however, it always comes down to the investigation of the behavior of perception systems under specific input data. Camera images are a crucial part of the input data. Image data sets are therefore collected for the testing of automated driving systems, but it is non-trivial to find specific images in these data sets. Thanks to recent developments in neural networks, there are now methods for sorting the images in a data set according to their similarity to a prompt in natural language. In order to further automate the provision of search results, we make a contribution by automating the threshold definition in these sorted results and returning only the images relevant to the prompt as a result. Our focus is on preventing false positives and false negatives equally. It is also important that our method is robust and in the case that our assumptions are not fulfilled, we provide a fallback solution.
△ Less
Submitted 8 April, 2024;
originally announced April 2024.
-
Integrating Static Code Analysis Toolchains
Authors:
Matthias Kern,
Ferhat Erata,
Markus Iser,
Carsten Sinz,
Frederic Loiret,
Stefan Otten,
Eric Sax
Abstract:
This paper proposes an approach for a tool-agnostic and heterogeneous static code analysis toolchain in combination with an exchange format. This approach enhances both traceability and comparability of analysis results. State of the art toolchains support features for either test execution and build automation or traceability between tests, requirements and design information. Our approach combin…
▽ More
This paper proposes an approach for a tool-agnostic and heterogeneous static code analysis toolchain in combination with an exchange format. This approach enhances both traceability and comparability of analysis results. State of the art toolchains support features for either test execution and build automation or traceability between tests, requirements and design information. Our approach combines all those features and extends traceability to the source code level, incorporating static code analysis. As part of our approach we introduce the "ASSUME Static Code Analysis tool exchange format" that facilitates the comparability of different static code analysis results. We demonstrate how this approach enhances the usability and efficiency of static code analysis in a development process. On the one hand, our approach enables the exchange of results and evaluations between static code analysis tools. On the other hand, it enables a complete traceability between requirements, designs, implementation, and the results of static code analysis. Within our approach we also propose an OSLC specification for static code analysis tools and an OSLC communication framework.
△ Less
Submitted 9 March, 2024;
originally announced March 2024.
-
The Machine Vision Iceberg Explained: Advancing Dynamic Testing by Considering Holistic Environmental Relations
Authors:
Hubert Padusinski,
Christian Steinhauser,
Thilo Braun,
Lennart Ries,
Eric Sax
Abstract:
Machine Vision (MV) is essential for solving driving automation. This paper examines potential shortcomings in current MV testing strategies for highly automated driving (HAD) systems. We argue for a more comprehensive understanding of the performance factors that must be considered during the MV evaluation process, noting that neglecting these factors can lead to significant risks. This is not on…
▽ More
Machine Vision (MV) is essential for solving driving automation. This paper examines potential shortcomings in current MV testing strategies for highly automated driving (HAD) systems. We argue for a more comprehensive understanding of the performance factors that must be considered during the MV evaluation process, noting that neglecting these factors can lead to significant risks. This is not only relevant to MV component testing, but also to integration testing. To illustrate this point, we draw an analogy to a ship navigating towards an iceberg to show potential hidden challenges in current MV testing strategies. The main contribution is a novel framework for black-box testing which observes environmental relations. This means it is designed to enhance MV assessments by considering the attributes and surroundings of relevant individual objects. The framework provides the identification of seven general concerns about the object recognition of MV, which are not addressed adequately in established test processes. To detect these deficits based on their performance factors, we propose the use of a taxonomy called "granularity orders" along with a graphical representation. This allows an identification of MV uncertainties across a range of driving scenarios. This approach aims to advance the precision, efficiency, and completeness of testing procedures for MV.
△ Less
Submitted 30 April, 2024; v1 submitted 26 January, 2024;
originally announced January 2024.
-
EfficientPPS: Part-aware Panoptic Segmentation of Transparent Objects for Robotic Manipulation
Authors:
Benjamin Alt,
Minh Dang Nguyen,
Andreas Hermann,
Darko Katic,
Rainer Jäkel,
Rüdiger Dillmann,
Eric Sax
Abstract:
The use of autonomous robots for assistance tasks in hospitals has the potential to free up qualified staff and im-prove patient care. However, the ubiquity of deformable and transparent objects in hospital settings poses signif-icant challenges to vision-based perception systems. We present EfficientPPS, a neural architecture for part-aware panoptic segmentation that provides robots with semantic…
▽ More
The use of autonomous robots for assistance tasks in hospitals has the potential to free up qualified staff and im-prove patient care. However, the ubiquity of deformable and transparent objects in hospital settings poses signif-icant challenges to vision-based perception systems. We present EfficientPPS, a neural architecture for part-aware panoptic segmentation that provides robots with semantically rich visual information for gras** and ma-nipulation tasks. We also present an unsupervised data collection and labelling method to reduce the need for human involvement in the training process. EfficientPPS is evaluated on a dataset containing real-world hospital objects and demonstrated to be robust and efficient in gras** transparent transfusion bags with a collaborative robot arm.
△ Less
Submitted 21 December, 2023;
originally announced December 2023.
-
Unveiling Objects with SOLA: An Annotation-Free Image Search on the Object Level for Automotive Data Sets
Authors:
Philipp Rigoll,
Jacob Langner,
Eric Sax
Abstract:
Huge image data sets are the fundament for the development of the perception of automated driving systems. A large number of images is necessary to train robust neural networks that can cope with diverse situations. A sufficiently large data set contains challenging situations and objects. For testing the resulting functions, it is necessary that these situations and objects can be found and extra…
▽ More
Huge image data sets are the fundament for the development of the perception of automated driving systems. A large number of images is necessary to train robust neural networks that can cope with diverse situations. A sufficiently large data set contains challenging situations and objects. For testing the resulting functions, it is necessary that these situations and objects can be found and extracted from the data set. While it is relatively easy to record a large amount of unlabeled data, it is far more difficult to find demanding situations and objects. However, during the development of perception systems, it must be possible to access challenging data without having to perform lengthy and time-consuming annotations. A developer must therefore be able to search dynamically for specific situations and objects in a data set. Thus, we designed a method which is based on state-of-the-art neural networks to search for objects with certain properties within an image. For the ease of use, the query of this search is described using natural language. To determine the time savings and performance gains, we evaluated our method qualitatively and quantitatively on automotive data sets.
△ Less
Submitted 7 December, 2023; v1 submitted 4 December, 2023;
originally announced December 2023.
-
Clustering-based Criticality Analysis for Testing of Automated Driving Systems
Authors:
Barbara Schütt,
Stefan Otten,
Eric Sax
Abstract:
With the implementation of the new EU regulation 2022/1426 regarding the type-approval of the automated driving system (ADS) of fully automated vehicles, scenario-based testing has gained significant importance in evaluating the performance and safety of advanced driver assistance systems and automated driving systems. However, the exploration and generation of concrete scenarios from a single log…
▽ More
With the implementation of the new EU regulation 2022/1426 regarding the type-approval of the automated driving system (ADS) of fully automated vehicles, scenario-based testing has gained significant importance in evaluating the performance and safety of advanced driver assistance systems and automated driving systems. However, the exploration and generation of concrete scenarios from a single logical scenario can often lead to a number of similar or redundant scenarios, which may not contribute to the testing goals.
This paper focuses on the the goal to reduce the scenario set by clustering concrete scenarios from a single logical scenario. By employing clustering techniques, redundant and uninteresting scenarios can be identified and eliminated, resulting in a representative scenario set. This reduction allows for a more focused and efficient testing process, enabling the allocation of resources to the most relevant and critical scenarios. Furthermore, the identified clusters can provide valuable insights into the scenario space, revealing patterns and potential problems with the system's behavior.
△ Less
Submitted 24 July, 2023; v1 submitted 22 June, 2023;
originally announced June 2023.
-
1001 Ways of Scenario Generation for Testing of Self-driving Cars: A Survey
Authors:
Barbara Schütt,
Joshua Ransiek,
Thilo Braun,
Eric Sax
Abstract:
Scenario generation is one of the essential steps in scenario-based testing and, therefore, a significant part of the verification and validation of driver assistance functions and autonomous driving systems. However, the term scenario generation is used for many different methods, e.g., extraction of scenarios from naturalistic driving data or variation of scenario parameters. This survey aims to…
▽ More
Scenario generation is one of the essential steps in scenario-based testing and, therefore, a significant part of the verification and validation of driver assistance functions and autonomous driving systems. However, the term scenario generation is used for many different methods, e.g., extraction of scenarios from naturalistic driving data or variation of scenario parameters. This survey aims to give a systematic overview of different approaches, establish different categories of scenario acquisition and generation, and show that each group of methods has typical input and output types. It shows that although the term is often used throughout literature, the evaluated methods use different inputs and the resulting scenarios differ in abstraction level and from a systematical point of view. Additionally, recent research and literature examples are given to underline this categorization.
△ Less
Submitted 21 April, 2023;
originally announced April 2023.
-
Inverse Universal Traffic Quality -- a Criticality Metric for Crowded Urban Traffic Scenes
Authors:
Barbara Schütt,
Maximilian Zipfl,
J. Marius Zöllner,
Eric Sax
Abstract:
An essential requirement for scenario-based testing the identification of critical scenes and their associated scenarios. However, critical scenes, such as collisions, occur comparatively rarely. Accordingly, large amounts of data must be examined. A further issue is that recorded real-world traffic often consists of scenes with a high number of vehicles, and it can be challenging to determine whi…
▽ More
An essential requirement for scenario-based testing the identification of critical scenes and their associated scenarios. However, critical scenes, such as collisions, occur comparatively rarely. Accordingly, large amounts of data must be examined. A further issue is that recorded real-world traffic often consists of scenes with a high number of vehicles, and it can be challenging to determine which are the most critical vehicles regarding the safety of an ego vehicle. Therefore, we present the inverse universal traffic quality, a criticality metric for urban traffic independent of predefined adversary vehicles and vehicle constellations such as intersection trajectories or car-following scenarios. Our metric is universally applicable for different urban traffic situations, e.g., intersections or roundabouts, and can be adjusted to certain situations if needed. Additionally, in this paper, we evaluate the proposed metric and compares its result to other well-known criticality metrics of this field, such as time-to-collision or post-encroachment time.
△ Less
Submitted 21 April, 2023;
originally announced April 2023.
-
Focus on the Challenges: Analysis of a User-friendly Data Search Approach with CLIP in the Automotive Domain
Authors:
Philipp Rigoll,
Patrick Petersen,
Hanno Stage,
Lennart Ries,
Eric Sax
Abstract:
Handling large amounts of data has become a key for develo** automated driving systems. Especially for develo** highly automated driving functions, working with images has become increasingly challenging due to the sheer size of the required data. Such data has to satisfy different requirements to be usable in machine learning-based approaches. Thus, engineers need to fully understand their la…
▽ More
Handling large amounts of data has become a key for develo** automated driving systems. Especially for develo** highly automated driving functions, working with images has become increasingly challenging due to the sheer size of the required data. Such data has to satisfy different requirements to be usable in machine learning-based approaches. Thus, engineers need to fully understand their large image data sets for the development and test of machine learning algorithms. However, current approaches lack automatability, are not generic and are limited in their expressiveness. Hence, this paper aims to analyze a state-of-the-art text and image embedding neural network and guides through the application in the automotive domain. This approach enables the search for similar images and the search based on a human understandable text-based description. Our experiments show the automatability and generalizability of our proposed method for handling large data sets in the automotive domain.
△ Less
Submitted 21 April, 2023; v1 submitted 20 April, 2023;
originally announced April 2023.
-
Fingerprint of a Traffic Scene: an Approach for a Generic and Independent Scene Assessment
Authors:
Maximilian Zipfl,
Barbara Schütt,
J. Marius Zöllner,
Eric Sax
Abstract:
A major challenge in the safety assessment of automated vehicles is to ensure that risk for all traffic participants is as low as possible. A concept that is becoming increasingly popular for testing in automated driving is scenario-based testing. It is founded on the assumption that most time on the road can be seen as uncritical and in mainly critical situations contribute to the safety case. Me…
▽ More
A major challenge in the safety assessment of automated vehicles is to ensure that risk for all traffic participants is as low as possible. A concept that is becoming increasingly popular for testing in automated driving is scenario-based testing. It is founded on the assumption that most time on the road can be seen as uncritical and in mainly critical situations contribute to the safety case. Metrics describing the criticality are necessary to automatically identify the critical situations and scenarios from measurement data. However, established metrics lack universality or a concept for metric combination. In this work, we present a multidimensional evaluation model that, based on conventional metrics, can evaluate scenes independently of the scene type. Furthermore, we present two new, further enhanced evaluation approaches, which can additionally serve as universal metrics. The metrics we introduce are then evaluated and discussed using real data from a motion dataset.
△ Less
Submitted 24 November, 2022;
originally announced November 2022.
-
An Application of Scenario Exploration to Find New Scenarios for the Development and Testing of Automated Driving Systems in Urban Scenarios
Authors:
Barbara Schütt,
Marc Heinrich,
Sonja Marahrens,
J. Marius Zöllner,
Eric Sax
Abstract:
Verification and validation are major challenges for develo** automated driving systems. A concept that gets more and more recognized for testing in automated driving is scenario-based testing. However, it introduces the problem of what scenarios are relevant for testing and which are not. This work aims to find relevant, interesting, or critical parameter sets within logical scenarios by utiliz…
▽ More
Verification and validation are major challenges for develo** automated driving systems. A concept that gets more and more recognized for testing in automated driving is scenario-based testing. However, it introduces the problem of what scenarios are relevant for testing and which are not. This work aims to find relevant, interesting, or critical parameter sets within logical scenarios by utilizing Bayes optimization and Gaussian processes. The parameter optimization is done by comparing and evaluating six different metrics in two urban intersection scenarios. Finally, a list of ideas this work leads to and should be investigated further is presented.
△ Less
Submitted 17 May, 2022;
originally announced May 2022.
-
A taxonomy for quality in simulation-based development and testing of automated driving systems
Authors:
Barbara Schütt,
Markus Steimle,
Birte Kramer,
Danny Behnecke,
Eric Sax
Abstract:
Ensuring the quality of automated driving systems is a major challenge the automotive industry is facing. In this context, quality defines the degree to which an object meets expectations and requirements. Especially, automated vehicles at SAE level 4 and 5 will be expected to operate safely in various contexts and complex situations without misconduct. Thus, a systematic approach is needed to sho…
▽ More
Ensuring the quality of automated driving systems is a major challenge the automotive industry is facing. In this context, quality defines the degree to which an object meets expectations and requirements. Especially, automated vehicles at SAE level 4 and 5 will be expected to operate safely in various contexts and complex situations without misconduct. Thus, a systematic approach is needed to show their safe operation. A way to address this challenge is simulation-based testing as pure physical testing is not feasible. During simulation-based testing, the data used to evaluate the actual quality of an automated driving system are generated using a simulation. However, to rely on these simulation data, the overall simulation, which also includes its simulation models, must provide a certain quality level. This quality level depends on the intended purpose for which the generated simulation data should be used. Therefore, three categories of quality can be considered: quality of the automated driving system and simulation quality, consisting of simulation model quality and scenario quality. Hence, quality must be determined and evaluated in various process steps in develo** and testing automated driving systems, the overall simulation, and the simulation models used for the simulation. In this paper, we propose a taxonomy to serve a better understanding of the concept of quality in the development and testing process to have a clear separation and insight where further testing is needed -- both in terms of automated driving systems and simulation, including their simulation models and scenarios used for testing.
△ Less
Submitted 25 November, 2021; v1 submitted 12 February, 2021;
originally announced February 2021.
-
SceML - A Graphical Modeling Framework for Scenario-based Testing of Autonomous Vehicles
Authors:
Barbara Schuett,
Thilo Braun,
Stefan Otten,
Eric Sax
Abstract:
Ensuring the functional correctness and safety of autonomous vehicles is a major challenge for the automotive industry. However, exhaustive physical test drives are not feasible, as billions of driven kilometers would be required to obtain reliable results. Scenariobased testing is an approach to tackle this problem and reduce necessary test drives by replacing driven kilometers with simulations o…
▽ More
Ensuring the functional correctness and safety of autonomous vehicles is a major challenge for the automotive industry. However, exhaustive physical test drives are not feasible, as billions of driven kilometers would be required to obtain reliable results. Scenariobased testing is an approach to tackle this problem and reduce necessary test drives by replacing driven kilometers with simulations of relevant or interesting scenarios. These scenarios can be generated or extracted from recorded data with machine learning algorithms or created by experts. In this paper, we propose a novel graphical scenario modeling language. The graphical framework allows experts to create new scenarios or review ones designed by other experts or generated by machine learning algorithms. The scenario description is modeled as a graph and based on behavior trees. It supports different abstraction levels of scenario description during software and test development. Additionally, the graphbased structure provides modularity and reusable sub-scenarios, an important use case in scenario modeling. A graphical visualization of the scenario enhances comprehensibility for different users. The presented approach eases the scenario creation process and increases the usage of scenarios within development and testing processes.
△ Less
Submitted 8 February, 2021;
originally announced February 2021.
-
CAN Radar: Sensing Physical Devices in CAN Networks based on Time Domain Reflectometry
Authors:
Marcel Rumez,
Jürgen Dürrwang,
Tim Brecht,
Timo Steinshorn,
Peter Neugebauer,
Reiner Kriesten,
Eric Sax
Abstract:
The presence of security vulnerabilities in automotive networks has already been shown by various publications in recent years. Due to the specification of the Controller Area Network (CAN) as a broadcast medium without security mechanisms, attackers are able to read transmitted messages without being noticed and to inject malicious messages. In order to detect potential attackers within a network…
▽ More
The presence of security vulnerabilities in automotive networks has already been shown by various publications in recent years. Due to the specification of the Controller Area Network (CAN) as a broadcast medium without security mechanisms, attackers are able to read transmitted messages without being noticed and to inject malicious messages. In order to detect potential attackers within a network or software system as early as possible, Intrusion Detection Systems (IDSs) are prevalent. Many approaches for vehicles are based on techniques which are able to detect deviations from specified CAN network behaviour regarding protocol or payload properties. However, it is challenging to detect attackers who secretly connect to CAN networks and do not actively participate in bus traffic. In this paper, we present an approach that is capable of successfully detecting unknown CAN devices and determining the distance (cable length) between the attacker device and our sensing unit based on Time Domain Reflectometry (TDR) technique. We evaluated our approach on a real vehicle network.
△ Less
Submitted 7 October, 2019;
originally announced October 2019.
