-
Requirements-driven Test Generation for Autonomous Vehicles with Machine Learning Components
Authors:
Cumhur Erkan Tuncali,
Georgios Fainekos,
Danil Prokhorov,
Hisahiro Ito,
James Kapinski
Abstract:
Autonomous vehicles are complex systems that are challenging to test and debug. A requirements-driven approach to the development process can decrease the resources required to design and test these systems, while simultaneously increasing the reliability. We present a testing framework that uses signal temporal logic (STL), which is a precise and unambiguous requirements language. Our framework e…
▽ More
Autonomous vehicles are complex systems that are challenging to test and debug. A requirements-driven approach to the development process can decrease the resources required to design and test these systems, while simultaneously increasing the reliability. We present a testing framework that uses signal temporal logic (STL), which is a precise and unambiguous requirements language. Our framework evaluates test cases against the STL formulae and additionally uses the requirements to automatically identify test cases that fail to satisfy the requirements. One of the key features of our tool is the support for machine learning (ML) components in the system design, such as deep neural networks. The framework allows evaluation of the control algorithms, including the ML components, and it also includes models of CCD camera, lidar, and radar sensors, as well as the vehicle environment. We use multiple methods to generate test cases, including covering arrays, which is an efficient method to search discrete variable spaces. The resulting test cases can be used to debug the controller design by identifying controller behaviors that do not satisfy requirements. The test cases can also enhance the testing phase of development by identifying critical corner cases that correspond to the limits of the system's allowed behaviors. We present STL requirements for an autonomous vehicle system, which capture both component-level and system-level behaviors. Additionally, we present three driving scenarios and demonstrate how our requirements-driven testing framework can be used to identify critical system behaviors, which can be used to support the development process.
△ Less
Submitted 2 August, 2019;
originally announced August 2019.
-
A Tutorial on Sim-ATAV: Simulation-based Adversarial Testing Framework for Autonomous Vehicles
Authors:
Cumhur Erkan Tuncali
Abstract:
Testing autonomous vehicles in simulation environments is crucial. Sim-ATAV is an open-source framework developed for experimenting with different test generation techniques in simulation environments for research purposes. This document provides a tutorial on Sim-ATAV with a running example.
Testing autonomous vehicles in simulation environments is crucial. Sim-ATAV is an open-source framework developed for experimenting with different test generation techniques in simulation environments for research purposes. This document provides a tutorial on Sim-ATAV with a running example.
△ Less
Submitted 25 March, 2019;
originally announced March 2019.
-
Rapidly-exploring Random Trees-based Test Generation for Autonomous Vehicles
Authors:
Cumhur Erkan Tuncali,
Georgios Fainekos
Abstract:
Autonomous vehicles are in an intensive research and development stage, and the organizations develo** these systems are targeting to deploy them on public roads in a very near future. One of the expectations from fully-automated vehicles is never to cause an accident. However, an automated vehicle may not be able to avoid all collisions, e.g., the collisions caused by other road occupants. Henc…
▽ More
Autonomous vehicles are in an intensive research and development stage, and the organizations develo** these systems are targeting to deploy them on public roads in a very near future. One of the expectations from fully-automated vehicles is never to cause an accident. However, an automated vehicle may not be able to avoid all collisions, e.g., the collisions caused by other road occupants. Hence, it is important for the system designers to understand the boundary case scenarios where an autonomous vehicle can no longer avoid a collision. In this paper, an automated test generation approach that utilizes Rapidly-exploring Random Trees is presented. A comparison of the proposed approach with an optimization-guided falsification approach from the literature is provided. Furthermore, a cost function that guides the test generation toward almost-avoidable collisions or near-misses is proposed.
△ Less
Submitted 25 March, 2019;
originally announced March 2019.
-
Simulation-based Adversarial Test Generation for Autonomous Vehicles with Machine Learning Components
Authors:
Cumhur Erkan Tuncali,
Georgios Fainekos,
Hisahiro Ito,
James Kapinski
Abstract:
Many organizations are develo** autonomous driving systems, which are expected to be deployed at a large scale in the near future. Despite this, there is a lack of agreement on appropriate methods to test, debug, and certify the performance of these systems. One of the main challenges is that many autonomous driving systems have machine learning components, such as deep neural networks, for whic…
▽ More
Many organizations are develo** autonomous driving systems, which are expected to be deployed at a large scale in the near future. Despite this, there is a lack of agreement on appropriate methods to test, debug, and certify the performance of these systems. One of the main challenges is that many autonomous driving systems have machine learning components, such as deep neural networks, for which formal properties are difficult to characterize. We present a testing framework that is compatible with test case generation and automatic falsification methods, which are used to evaluate cyber-physical systems. We demonstrate how the framework can be used to evaluate closed-loop properties of an autonomous driving system model that includes the ML components, all within a virtual environment. We demonstrate how to use test case generation methods, such as covering arrays, as well as requirement falsification methods to automatically identify problematic test scenarios. The resulting framework can be used to increase the reliability of autonomous driving systems.
△ Less
Submitted 7 January, 2019; v1 submitted 18 April, 2018;
originally announced April 2018.
-
Reasoning about Safety of Learning-Enabled Components in Autonomous Cyber-physical Systems
Authors:
Cumhur Erkan Tuncali,
James Kapinski,
Hisahiro Ito,
Jyotirmoy V. Deshmukh
Abstract:
We present a simulation-based approach for generating barrier certificate functions for safety verification of cyber-physical systems (CPS) that contain neural network-based controllers. A linear programming solver is utilized to find a candidate generator function from a set of simulation traces obtained by randomly selecting initial states for the CPS model. A level set of the generator function…
▽ More
We present a simulation-based approach for generating barrier certificate functions for safety verification of cyber-physical systems (CPS) that contain neural network-based controllers. A linear programming solver is utilized to find a candidate generator function from a set of simulation traces obtained by randomly selecting initial states for the CPS model. A level set of the generator function is then selected to act as a barrier certificate for the system, meaning it demonstrates that no unsafe system states are reachable from a given set of initial states. The barrier certificate properties are verified with an SMT solver. This approach is demonstrated on a case study in which a Dubins car model of an autonomous vehicle is controlled by a neural network to follow a given path.
△ Less
Submitted 11 April, 2018;
originally announced April 2018.
-
An Efficient Algorithm for Monitoring Practical TPTL Specifications
Authors:
Adel Dokhanchi,
Bardh Hoxha,
Cumhur Erkan Tuncali,
Georgios Fainekos
Abstract:
We provide a dynamic programming algorithm for the monitoring of a fragment of Timed Propositional Temporal Logic (TPTL) specifications. This fragment of TPTL, which is more expressive than Metric Temporal Logic, is characterized by independent time variables which enable the elicitation of complex real-time requirements. For this fragment, we provide an efficient polynomial time algorithm for off…
▽ More
We provide a dynamic programming algorithm for the monitoring of a fragment of Timed Propositional Temporal Logic (TPTL) specifications. This fragment of TPTL, which is more expressive than Metric Temporal Logic, is characterized by independent time variables which enable the elicitation of complex real-time requirements. For this fragment, we provide an efficient polynomial time algorithm for off-line monitoring of finite traces. Finally, we provide experimental results on a prototype implementation of our tool in order to demonstrate the feasibility of using our tool in practical applications.
△ Less
Submitted 9 December, 2016;
originally announced December 2016.
-
Modeling Concurrency and Reconfiguration in Vehicular Systems: A $π$-calculus Approach
Authors:
Joseph Campbell,
Cumhur Erkan Tuncali,
Theodore P. Pavlic,
Georgios Fainekos
Abstract:
As autonomous or semi-autonomous vehicles are deployed on the roads, they will have to eventually start communicating with each other in order to achieve increased efficiency and safety. Current approaches in the control of collaborative vehicles primarily consider homogeneous simplified vehicle dynamics and usually ignore any communication issues. This raises an important question of how systems…
▽ More
As autonomous or semi-autonomous vehicles are deployed on the roads, they will have to eventually start communicating with each other in order to achieve increased efficiency and safety. Current approaches in the control of collaborative vehicles primarily consider homogeneous simplified vehicle dynamics and usually ignore any communication issues. This raises an important question of how systems without the aforementioned limiting assumptions can be modeled, analyzed and certified for safe operation by both industry and governmental agencies. In this work, we propose a modeling framework where communication and system reconfiguration is modeled through $π$-calculus expressions while the closed-loop control systems are modeled through hybrid automata. We demonstrate how the framework can be utilized for modeling and simulation of platooning behaviors of heterogeneous vehicles.
△ Less
Submitted 7 April, 2016;
originally announced April 2016.
