-
ADS Standardization Landscape: Making Sense of its Status and of the Associated Research Questions
Authors:
Scott Schnelle,
Francesca M. Favaro
Abstract:
Automated Driving Systems (ADS) hold great potential to increase safety, mobility, and equity. However, without public acceptance, none of these promises can be fulfilled. To engender public trust, many entities in the ADS community participate in standards development organizations (SDOs) with the goal of enhancing safety for the entire industry through a collaborative approach. The breadth and d…
▽ More
Automated Driving Systems (ADS) hold great potential to increase safety, mobility, and equity. However, without public acceptance, none of these promises can be fulfilled. To engender public trust, many entities in the ADS community participate in standards development organizations (SDOs) with the goal of enhancing safety for the entire industry through a collaborative approach. The breadth and depth of the ADS safety standardization landscape is vast and constantly changing, as often is the case for novel technologies in rapid evolution. The pace of development of the ADS industry makes it hard for the public and interested parties to keep track of ongoing SDO efforts, including the topics touched by each standard and the committees addressing each topic, as well as make sense of the wealth of documentation produced. Therefore, the authors present here a simplified framework for abstracting and organizing the current landscape of ADS safety standards into high-level, long term themes. This framework is then utilized to develop and organize associated research questions that have not yet reached widely adopted industry positions, along with identifying potential gaps where further research and standardization is needed.
△ Less
Submitted 30 June, 2023;
originally announced June 2023.
-
Interpreting Safety Outcomes: Waymo's Performance Evaluation in the Context of a Broader Determination of Safety Readiness
Authors:
Francesca M. Favaro,
Trent Victor,
Henning Hohnhold,
Scott Schnelle
Abstract:
This paper frames recent publications from Waymo within the broader context of the safety readiness determination for an Automated Driving System (ADS). Starting from a brief overview of safety performance outcomes reported by Waymo (i.e., contact events experienced during fully autonomous operations), this paper highlights the need for a diversified approach to safety determination that complemen…
▽ More
This paper frames recent publications from Waymo within the broader context of the safety readiness determination for an Automated Driving System (ADS). Starting from a brief overview of safety performance outcomes reported by Waymo (i.e., contact events experienced during fully autonomous operations), this paper highlights the need for a diversified approach to safety determination that complements the analysis of observed safety outcomes with other estimation techniques. Our discussion highlights: the presentation of a "credibility paradox" within the comparison between ADS crash data and human-derived baselines; the recognition of continuous confidence growth through in-use monitoring; and the need to supplement any aggregate statistical analysis with appropriate event-level reasoning.
△ Less
Submitted 23 June, 2023;
originally announced June 2023.
-
Building a Credible Case for Safety: Waymo's Approach for the Determination of Absence of Unreasonable Risk
Authors:
Francesca Favaro,
Laura Fraade-Blanar,
Scott Schnelle,
Trent Victor,
Mauricio Peña,
Johan Engstrom,
John Scanlon,
Kris Kusano,
Dan Smith
Abstract:
This paper presents an overview of Waymo's approach to building a reliable case for safety - a novel and thorough blueprint for use by any company building fully autonomous driving systems. A safety case for fully autonomous operations is a formal way to explain how a company determines that an AV system is safe enough to be deployed on public roads without a human driver, and it includes evidence…
▽ More
This paper presents an overview of Waymo's approach to building a reliable case for safety - a novel and thorough blueprint for use by any company building fully autonomous driving systems. A safety case for fully autonomous operations is a formal way to explain how a company determines that an AV system is safe enough to be deployed on public roads without a human driver, and it includes evidence to support that determination. It involves an explanation of the system, the methodologies used to develop it, the metrics used to validate it and the actual results of validation tests. Yet, in order to develop a worthwhile safety case, it is first important to understand what makes one credible and well crafted, and align on evaluation criteria. This paper helps enabling such alignment by providing foundational thinking into not only how a system is determined to be ready for deployment but also into justifying that the set of acceptance criteria employed in such determination is sufficient and that their evaluation (and associated methods) is credible. The publication is structured around three complementary perspectives on safety that build upon content published by Waymo since 2020: a layered approach to safety; a dynamic approach to safety; and a credible approach to safety. The proposed approach is methodology-agnostic, so that anyone in the space could employ portions or all of it.
△ Less
Submitted 2 June, 2023;
originally announced June 2023.
-
Collision Avoidance Testing of the Waymo Automated Driving System
Authors:
Kristofer D. Kusano,
Kurt Beatty,
Scott Schnelle,
Francesca Favaro,
Cam Crary,
Trent Victor
Abstract:
This paper describes Waymo's Collision Avoidance Testing (CAT) methodology: a scenario-based testing method that evaluates the safety of the Waymo Driver Automated Driving Systems' (ADS) intended functionality in conflict situations initiated by other road users that require urgent evasive maneuvers. Because SAE Level 4 ADS are responsible for the dynamic driving task (DDT), when engaged, without…
▽ More
This paper describes Waymo's Collision Avoidance Testing (CAT) methodology: a scenario-based testing method that evaluates the safety of the Waymo Driver Automated Driving Systems' (ADS) intended functionality in conflict situations initiated by other road users that require urgent evasive maneuvers. Because SAE Level 4 ADS are responsible for the dynamic driving task (DDT), when engaged, without immediate human intervention, evaluating a Level 4 ADS using scenario-based testing is difficult due to the potentially infinite number of operational scenarios in which hazardous situations may unfold. To that end, in this paper we first describe the safety test objectives for the CAT methodology, including the collision and serious injury metrics and the reference behavior model representing a non-impaired eyes on conflict human driver used to form an acceptance criterion. Afterward, we introduce the process for identifying potentially hazardous situations from a combination of human data, ADS testing data, and expert knowledge about the product design and associated Operational Design Domain (ODD). The test allocation and execution strategy is presented next, which exclusively utilize simulations constructed from sensor data collected on a test track, real-world driving, or from simulated sensor data. The paper concludes with the presentation of results from applying CAT to the fully autonomous ride-hailing service that Waymo operates in San Francisco, California and Phoenix, Arizona. The iterative nature of scenario identification, combined with over ten years of experience of on-road testing, results in a scenario database that converges to a representative set of responder role scenarios for a given ODD. Using Waymo's virtual test platform, which is calibrated to data collected as part of many years of ADS development, the CAT methodology provides a robust and scalable safety evaluation.
△ Less
Submitted 15 December, 2022;
originally announced December 2022.
-
Waymo's Fatigue Risk Management Framework: Prevention, Monitoring, and Mitigation of Fatigue-Induced Risks while Testing Automated Driving Systems
Authors:
Francesca Favaro,
Keith Hutchings,
Philip Nemec,
Leticia Cavalcante,
Trent Victor
Abstract:
This report presents Waymo's proposal for a systematic fatigue risk management framework that addresses prevention, monitoring, and mitigation of fatigue-induced risks during on-road testing of ADS technology. The proposed framework remains flexible to incorporate continuous improvements, and was informed by state of the art practices, research, learnings, and experience (both internal and externa…
▽ More
This report presents Waymo's proposal for a systematic fatigue risk management framework that addresses prevention, monitoring, and mitigation of fatigue-induced risks during on-road testing of ADS technology. The proposed framework remains flexible to incorporate continuous improvements, and was informed by state of the art practices, research, learnings, and experience (both internal and external to Waymo). Fatigue is a recognized contributory factor in a substantial fraction of on-road crashes involving human drivers, and mitigation of fatigue-induced risks is still an open concern researched world-wide. While the proposed framework was specifically designed in relation to on-road testing of SAE Level 4 ADS technology, it has implications and applicability to lower levels of automation as well.
△ Less
Submitted 26 August, 2022;
originally announced August 2022.
-
System Architecture and Communication Infrastructure for the RoboVaaS project
Authors:
Emanuele Coccolo,
Cosmin Delea,
Fabian Steinmetz,
Roberto Francescon,
Alberto Signori,
Ching Nok Au,
Filippo Campagnaro,
Vincent Schneider,
Federico Favaro,
Johannes Oeffner,
Christian Renner,
Michele Zorzi
Abstract:
Current advancements in waterborne autonomous systems, together with the development of cloud-based service-oriented architectures and the recent availability of low-cost underwater acoustic modems and long-range above water wireless devices, enabled the development of new applications to support ships and port activities. Unmanned Surface Vehicle (USV) can, for instance, be used to perform bathym…
▽ More
Current advancements in waterborne autonomous systems, together with the development of cloud-based service-oriented architectures and the recent availability of low-cost underwater acoustic modems and long-range above water wireless devices, enabled the development of new applications to support ships and port activities. Unmanned Surface Vehicle (USV) can, for instance, be used to perform bathymetry and environmental data collection tasks to ensure under-keel clearance and to monitor the quality of the water. Similarly, Remotely Operated Vehicles (ROVs) can be deployed to inspect ship hulls and typical port infrastructure elements, such as quay and sheet pilling walls. In this paper we present the complete system deployed for the small-scale demonstrations of the Robotic Vessels as-a-Service (RoboVaaS) project, which introduces an on-demand service-based cloud system that dispatches Unmanned Vehicles (UVs) capable of performing the required service either autonomously or piloted. These vessels are able to interact with sensors deployed in the port and with the shore station through an integrated underwater and above water network. The developed system has been validated through sea trials and showcased through an underwater sensor data collection service. The results of the test presented in this paper provide a proof-of-concept of the system design and indicate its technical feasibility. It also shows the need for further developments for a mature technology allowing on-demand robotic maritime assistance services in real operational scenarios.
△ Less
Submitted 10 June, 2022;
originally announced June 2022.
-
Exploring the Relationship Between "Positive Risk Balance" and "Absence of Unreasonable Risk"
Authors:
Francesca Favaro
Abstract:
International discussions on the overarching topic of how to define and quantify what a "safe enough" Automated Driving System (ADS) is are currently hinged on the question of determining the relationship between "positive risk balance" (PRB) and "absence of unreasonable risk" (AUR). In order to advance the conversation on these important safety topics at the international level, it is first impor…
▽ More
International discussions on the overarching topic of how to define and quantify what a "safe enough" Automated Driving System (ADS) is are currently hinged on the question of determining the relationship between "positive risk balance" (PRB) and "absence of unreasonable risk" (AUR). In order to advance the conversation on these important safety topics at the international level, it is first important to start from a shared common understanding, grounded in clear definitions and terminology. To that end, this paper will start with an overview of the notions of PRB and AUR; it will then summarize different positions of the present debate; finally, it will conclude that two possible interpretations exist for PRB, and that failure to distinguish them can lead to misunderstanding different parties' positions. The argumentation in this paper is aimed at showing that the two interpretations for PRB can actually complement each other, but can be considered independently, and can both be subsumed within non-prescriptive guidelines toward ADS safety assurance.
△ Less
Submitted 20 October, 2021;
originally announced October 2021.
-
An Event-Based Stack For Data Transmission Through Underwater Multimodal Networks
Authors:
Roberto Francescon,
Filippo Campagnaro,
Emanuele Coccolo,
Alberto Signori,
Federico Guerra,
Federico Favaro,
Michele Zorzi
Abstract:
The DESERT Underwater framework (http://desert-underwater.dei.unipd.it/), originally designed for simulating and testing underwater acoustic networks in sea trials, has recently been extended to support real payload data transmission through underwater multimodal networks. Specifically, the new version of the framework is now able to transmit data in real time through the EvoLogics S2C low-rate an…
▽ More
The DESERT Underwater framework (http://desert-underwater.dei.unipd.it/), originally designed for simulating and testing underwater acoustic networks in sea trials, has recently been extended to support real payload data transmission through underwater multimodal networks. Specifically, the new version of the framework is now able to transmit data in real time through the EvoLogics S2C low-rate and high-rate acoustic modems, the SmartPORT low-cost acoustic underwater modem prototype (AHOI) for IoT applications, as well as Ethernet, surface WiFi, and the BlueComm optical modem. The system can also be tested in the lab by employing a simulated channel, and the EvoLogics S2C DMAC Emulator (DMACE)
△ Less
Submitted 2 March, 2021;
originally announced March 2021.
-
Waymo's Safety Methodologies and Safety Readiness Determinations
Authors:
Nick Webb,
Dan Smith,
Christopher Ludwick,
Trent Victor,
Qi Hommes,
Francesca Favaro,
George Ivanov,
Tom Daniel
Abstract:
Waymo's safety methodologies, which draw on well established engineering processes and address new safety challenges specific to Automated Vehicle technology, provide a firm foundation for safe deployment of Waymo's Level 4 ADS, which Waymo also refers to as the Waymo Driver. Waymo's determination of its readiness to deploy its AVs safely in different settings rests on that firm foundation and on…
▽ More
Waymo's safety methodologies, which draw on well established engineering processes and address new safety challenges specific to Automated Vehicle technology, provide a firm foundation for safe deployment of Waymo's Level 4 ADS, which Waymo also refers to as the Waymo Driver. Waymo's determination of its readiness to deploy its AVs safely in different settings rests on that firm foundation and on a thorough analysis of risks specific to a particular Operational Design Domain. Waymo's process for making these readiness determinations entails an ordered examination of the relevant outputs from all of its safety methodologies combined with careful safety and engineering judgment focused on the specific facts relevant for a particular determination. Waymo will approve when it determines the ADS is ready for the new conditions without creating any unreasonable risks to safety. This paper explains Waymo's methodologies as applied to the three layers of its technology: hardware, ADS behavior, and operations, and also explains Waymo's safety governance. Waymo will continue to apply and adapt those methodologies, and to learn from the important contributions of others in the AV industry, as Waymo continues to build an ever safer and more able ADS.
△ Less
Submitted 30 October, 2020;
originally announced November 2020.
-
Waymo Public Road Safety Performance Data
Authors:
Matthew Schwall,
Tom Daniel,
Trent Victor,
Francesca Favaro,
Henning Hohnhold
Abstract:
Waymo's mission to reduce traffic injuries and fatalities and improve mobility for all has led us to expand deployment of automated vehicles on public roads without a human driver behind the wheel. As part of this process, Waymo is committed to providing the public with informative and relevant data regarding the demonstrated safety of Waymo's automated driving system, which we call the Waymo Driv…
▽ More
Waymo's mission to reduce traffic injuries and fatalities and improve mobility for all has led us to expand deployment of automated vehicles on public roads without a human driver behind the wheel. As part of this process, Waymo is committed to providing the public with informative and relevant data regarding the demonstrated safety of Waymo's automated driving system, which we call the Waymo Driver. The data presented in this paper represents more than 6.1 million miles of automated driving in the Phoenix, Arizona metropolitan area, including operations with a trained operator behind the steering wheel from calendar year 2019 and 65,000 miles of driverless operation without a human behind the steering wheel from 2019 and the first nine months of 2020. The paper includes every collision and minor contact experienced during these operations as well as every predicted contact identified using Waymo's counterfactual, what if, simulation of events had the vehicle's trained operator not disengaged automated driving. There were 47 contact events that occurred over this time period, consisting of 18 actual and 29 simulated contact events, none of which would be expected to result in severe or life threatening injuries. This paper presents the collision typology and severity for each actual and simulated event, along with diagrams depicting each of the most significant events. Nearly all the events involved one or more road rule violations or other errors by a human driver or road user, including all eight of the most severe events, which we define as involving actual or expected airbag deployment in any involved vehicle. When compared to national collision statistics, the Waymo Driver completely avoided certain collision modes that human driven vehicles are frequently involved in, including road departure and collisions with fixed objects.
△ Less
Submitted 30 October, 2020;
originally announced November 2020.
