Showing 1–15 of 15 results for author: Liu, H X

Evaluating Roadside Perception for Autonomous Vehicles: Insights from Field Testing

Authors: Rusheng Zhang, Depu Meng, Shengyin Shen, Tinghan Wang, Tai Karir, Michael Maile, Henry X. Liu

Abstract: Roadside perception systems are increasingly crucial in enhancing traffic safety and facilitating cooperative driving for autonomous vehicles. Despite rapid technological advancements, a major challenge persists for this newly arising field: the absence of standardized evaluation methods and benchmarks for these systems. This limitation hampers the ability to effectively assess and compare the per… ▽ More Roadside perception systems are increasingly crucial in enhancing traffic safety and facilitating cooperative driving for autonomous vehicles. Despite rapid technological advancements, a major challenge persists for this newly arising field: the absence of standardized evaluation methods and benchmarks for these systems. This limitation hampers the ability to effectively assess and compare the performance of different systems, thus constraining progress in this vital field. This paper introduces a comprehensive evaluation methodology specifically designed to assess the performance of roadside perception systems. Our methodology encompasses measurement techniques, metric selection, and experimental trial design, all grounded in real-world field testing to ensure the practical applicability of our approach. We applied our methodology in Mcity\footnote{\url{https://mcity.umich.edu/}}, a controlled testing environment, to evaluate various off-the-shelf perception systems. This approach allowed for an in-depth comparative analysis of their performance in realistic scenarios, offering key insights into their respective strengths and limitations. The findings of this study are poised to inform the development of industry-standard benchmarks and evaluation methods, thereby enhancing the effectiveness of roadside perception system development and deployment for autonomous vehicles. We anticipate that this paper will stimulate essential discourse on standardizing evaluation methods for roadside perception systems, thus pushing the frontiers of this technology. Furthermore, our results offer both academia and industry a comprehensive understanding of the capabilities of contemporary infrastructure-based perception systems. △ Less

Submitted 22 January, 2024; originally announced January 2024.

Comments: 6 figures, 8 tables, 14 pages

Evaluation of automated driving system safety metrics with logged vehicle trajectory data

Authors: Xintao Yan, Shuo Feng, David J. LeBlanc, Carol Flannagan, Henry X. Liu

Abstract: Real-time safety metrics are important for the automated driving system (ADS) to assess the risk of driving situations and to assist the decision-making. Although a number of real-time safety metrics have been proposed in the literature, systematic performance evaluation of these safety metrics has been lacking. As different behavioral assumptions are adopted in different safety metrics, it is dif… ▽ More Real-time safety metrics are important for the automated driving system (ADS) to assess the risk of driving situations and to assist the decision-making. Although a number of real-time safety metrics have been proposed in the literature, systematic performance evaluation of these safety metrics has been lacking. As different behavioral assumptions are adopted in different safety metrics, it is difficult to compare the safety metrics and evaluate their performance. To overcome this challenge, in this study, we propose an evaluation framework utilizing logged vehicle trajectory data, in that vehicle trajectories for both subject vehicle (SV) and background vehicles (BVs) are obtained and the prediction errors caused by behavioral assumptions can be eliminated. Specifically, we examine whether the SV is in a collision unavoidable situation at each moment, given all near-future trajectories of BVs. In this way, we level the ground for a fair comparison of different safety metrics, as a good safety metric should always alarm in advance to the collision unavoidable moment. When trajectory data from a large number of trips are available, we can systematically evaluate and compare different metrics' statistical performance. In the case study, three representative real-time safety metrics, including the time-to-collision (TTC), the PEGASUS Criticality Metric (PCM), and the Model Predictive Instantaneous Safety Metric (MPrISM), are evaluated using a large-scale simulated trajectory dataset. The proposed evaluation framework is important for researchers, practitioners, and regulators to characterize different metrics, and to select appropriate metrics for different applications. Moreover, by conducting failure analysis on moments when a safety metric failed, we can identify its potential weaknesses which are valuable for its potential refinements and improvements. △ Less

Submitted 2 January, 2024; originally announced January 2024.

MSight: An Edge-Cloud Infrastructure-based Perception System for Connected Automated Vehicles

Authors: Rusheng Zhang, Depu Meng, Shengyin Shen, Zhengxia Zou, Houqiang Li, Henry X. Liu

Abstract: As vehicular communication and networking technologies continue to advance, infrastructure-based roadside perception emerges as a pivotal tool for connected automated vehicle (CAV) applications. Due to their elevated positioning, roadside sensors, including cameras and lidars, often enjoy unobstructed views with diminished object occlusion. This provides them a distinct advantage over onboard perc… ▽ More As vehicular communication and networking technologies continue to advance, infrastructure-based roadside perception emerges as a pivotal tool for connected automated vehicle (CAV) applications. Due to their elevated positioning, roadside sensors, including cameras and lidars, often enjoy unobstructed views with diminished object occlusion. This provides them a distinct advantage over onboard perception, enabling more robust and accurate detection of road objects. This paper presents MSight, a cutting-edge roadside perception system specifically designed for CAVs. MSight offers real-time vehicle detection, localization, tracking, and short-term trajectory prediction. Evaluations underscore the system's capability to uphold lane-level accuracy with minimal latency, revealing a range of potential applications to enhance CAV safety and efficiency. Presently, MSight operates 24/7 at a two-lane roundabout in the City of Ann Arbor, Michigan. △ Less

Submitted 8 October, 2023; originally announced October 2023.

Comments: Submitted to IEEE T-ITS

Robust Roadside Perception: an Automated Data Synthesis Pipeline Minimizing Human Annotation

Authors: Rusheng Zhang, Depu Meng, Lance Bassett, Shengyin Shen, Zhengxia Zou, Henry X. Liu

Abstract: Recently, advancements in vehicle-to-infrastructure communication technologies have elevated the significance of infrastructure-based roadside perception systems for cooperative driving. This paper delves into one of its most pivotal challenges: data insufficiency. The lacking of high-quality labeled roadside sensor data with high diversity leads to low robustness, and low transfer-ability of curr… ▽ More Recently, advancements in vehicle-to-infrastructure communication technologies have elevated the significance of infrastructure-based roadside perception systems for cooperative driving. This paper delves into one of its most pivotal challenges: data insufficiency. The lacking of high-quality labeled roadside sensor data with high diversity leads to low robustness, and low transfer-ability of current roadside perception systems. In this paper, a novel solution is proposed to address this problem that creates synthesized training data using Augmented Reality. A Generative Adversarial Network is then applied to enhance the reality further, that produces a photo-realistic synthesized dataset that is capable of training or fine-tuning a roadside perception detector which is robust to different weather and lighting conditions. Our approach was rigorously tested at two key intersections in Michigan, USA: the Mcity intersection and the State St./Ellsworth Rd roundabout. The Mcity intersection is located within the Mcity test field, a controlled testing environment. In contrast, the State St./Ellsworth Rd intersection is a bustling roundabout notorious for its high traffic flow and a significant number of accidents annually. Experimental results demonstrate that detectors trained solely on synthesized data exhibit commendable performance across all conditions. Furthermore, when integrated with labeled data, the synthesized data can notably bolster the performance of pre-existing detectors, especially in adverse conditions. △ Less

Submitted 8 February, 2024; v1 submitted 29 June, 2023; originally announced June 2023.

Comments: Accepted by IEEE Transactions on Intelligent Vehicles

ROCO: A Roundabout Traffic Conflict Dataset

Authors: Depu Meng, Owen Sayer, Rusheng Zhang, Shengyin Shen, Houqiang Li, Henry X. Liu

Abstract: Traffic conflicts have been studied by the transportation research community as a surrogate safety measure for decades. However, due to the rarity of traffic conflicts, collecting large-scale real-world traffic conflict data becomes extremely challenging. In this paper, we introduce and analyze ROCO - a real-world roundabout traffic conflict dataset. The data is collected at a two-lane roundabout… ▽ More Traffic conflicts have been studied by the transportation research community as a surrogate safety measure for decades. However, due to the rarity of traffic conflicts, collecting large-scale real-world traffic conflict data becomes extremely challenging. In this paper, we introduce and analyze ROCO - a real-world roundabout traffic conflict dataset. The data is collected at a two-lane roundabout at the intersection of State St. and W. Ellsworth Rd. in Ann Arbor, Michigan. We use raw video dataflow captured from four fisheye cameras installed at the roundabout as our input data source. We adopt a learning-based conflict identification algorithm from video to find potential traffic conflicts, and then manually label them for dataset collection and annotation. In total 557 traffic conflicts and 17 traffic crashes are collected from August 2021 to October 2021. We provide trajectory data of the traffic conflict scenes extracted using our roadside perception system. Taxonomy based on traffic conflict severity, reason for the traffic conflict, and its effect on the traffic flow is provided. With the traffic conflict data collected, we discover that failure to yield to circulating vehicles when entering the roundabout is the largest contributing reason for traffic conflicts. ROCO dataset will be made public in the short future. △ Less

Submitted 1 March, 2023; v1 submitted 1 March, 2023; originally announced March 2023.

Comments: Accepted by TRBAM 2023 presentation

"Curse of rarity" for autonomous vehicles

Authors: Henry X. Liu, Shuo Feng

Abstract: In this paper, we reveal that the rarity of safety-critical events in high-dimensional driving environments is the root cause of the safety challenge for autonomous vehicle development. We formulate it as "curse of rarity" (CoR) because it occurs ubiquitously in various safety-critical systems such as medical diagnosis and aerospace systems. In this paper, we reveal that the rarity of safety-critical events in high-dimensional driving environments is the root cause of the safety challenge for autonomous vehicle development. We formulate it as "curse of rarity" (CoR) because it occurs ubiquitously in various safety-critical systems such as medical diagnosis and aerospace systems. △ Less

Submitted 26 September, 2022; v1 submitted 6 July, 2022; originally announced July 2022.

Real-time Full-stack Traffic Scene Perception for Autonomous Driving with Roadside Cameras

Authors: Zhengxia Zou, Rusheng Zhang, Shengyin Shen, Gaurav Pandey, Punarjay Chakravarty, Armin Parchami, Henry X. Liu

Abstract: We propose a novel and pragmatic framework for traffic scene perception with roadside cameras. The proposed framework covers a full-stack of roadside perception pipeline for infrastructure-assisted autonomous driving, including object detection, object localization, object tracking, and multi-camera information fusion. Unlike previous vision-based perception frameworks rely upon depth offset or 3D… ▽ More We propose a novel and pragmatic framework for traffic scene perception with roadside cameras. The proposed framework covers a full-stack of roadside perception pipeline for infrastructure-assisted autonomous driving, including object detection, object localization, object tracking, and multi-camera information fusion. Unlike previous vision-based perception frameworks rely upon depth offset or 3D annotation at training, we adopt a modular decoupling design and introduce a landmark-based 3D localization method, where the detection and localization can be well decoupled so that the model can be easily trained based on only 2D annotations. The proposed framework applies to either optical or thermal cameras with pinhole or fish-eye lenses. Our framework is deployed at a two-lane roundabout located at Ellsworth Rd. and State St., Ann Arbor, MI, USA, providing 7x24 real-time traffic flow monitoring and high-precision vehicle trajectory extraction. The whole system runs efficiently on a low-power edge computing device with all-component end-to-end delay of less than 20ms. △ Less

Submitted 20 June, 2022; originally announced June 2022.

Comments: This paper is accepted and presented in ICRA 2022

A Novel Underwater Image Enhancement and Improved Underwater Biological Detection Pipeline

Authors: Zheng Liu, Yaoming Zhuang, Pengrun Jia, Chengdong Wu, Hongli Xu ang Zhanlin Liu

Abstract: For aquaculture resource evaluation and ecological environment monitoring, automatic detection and identification of marine organisms is critical. However, due to the low quality of underwater images and the characteristics of underwater biological, a lack of abundant features may impede traditional hand-designed feature extraction approaches or CNN-based object detection algorithms, particularly… ▽ More For aquaculture resource evaluation and ecological environment monitoring, automatic detection and identification of marine organisms is critical. However, due to the low quality of underwater images and the characteristics of underwater biological, a lack of abundant features may impede traditional hand-designed feature extraction approaches or CNN-based object detection algorithms, particularly in complex underwater environment. Therefore, the goal of this paper is to perform object detection in the underwater environment. This paper proposed a novel method for capturing feature information, which adds the convolutional block attention module (CBAM) to the YOLOv5 backbone. The interference of underwater creature characteristics on object characteristics is decreased, and the output of the backbone network to object information is enhanced. In addition, the self-adaptive global histogram stretching algorithm (SAGHS) is designed to eliminate the degradation problems such as low contrast and color loss caused by underwater environmental information to better restore image quality. Extensive experiments and comprehensive evaluation on the URPC2021 benchmark dataset demonstrate the effectiveness and adaptivity of our methods. Beyond that, this paper conducts an exhaustive analysis of the role of training data on performance. △ Less

Submitted 20 May, 2022; originally announced May 2022.

Comments: 14 pages,14 figures

A hierarchical behavior prediction framework at signalized intersections

Authors: Zhen Yang, Rusheng Zhang, Henry X. Liu

Abstract: Road user behavior prediction is one of the most critical components in trajectory planning for autonomous driving, especially in urban scenarios involving traffic signals. In this paper, a hierarchical framework is proposed to predict vehicle behaviors at a signalized intersection, using the traffic signal information of the intersection. The framework is composed of two phases: a discrete intent… ▽ More Road user behavior prediction is one of the most critical components in trajectory planning for autonomous driving, especially in urban scenarios involving traffic signals. In this paper, a hierarchical framework is proposed to predict vehicle behaviors at a signalized intersection, using the traffic signal information of the intersection. The framework is composed of two phases: a discrete intention prediction phase and a continuous trajectory prediction phase. In the discrete intention prediction phase, a Bayesian network is adopted to predict the vehicle's high-level intention, after that, maximum entropy inverse reinforcement learning is utilized to learn the human driving model offline; during the online trajectory prediction phase, a driver characteristic is designed and updated to capture the different driving preferences between human drivers. We applied the proposed framework to one of the most challenging scenarios in autonomous driving: the yellow light running scenario. Numerical experiment results are presented in the later part of the paper which show the viability of the method. The accuracy of the Bayesian network for discrete intention prediction is 91.1%, and the prediction results are getting more and more accurate as the yellow time elapses. The average Euclidean distance error in continuous trajectory prediction is only 0.85 m in the yellow light running scenario. △ Less

Submitted 28 October, 2021; originally announced October 2021.

Comments: This work has been accepted to the IEEE 2021 International Conference on Intelligent Transportation Systems

A probabilistic model for missing traffic volume reconstruction based on data fusion

Authors: Xintao Yan, Yan Zhao, Henry X. Liu

Abstract: Traffic volume information is critical for intelligent transportation systems. It serves as a key input to transportation planning, roadway design, and traffic signal control. However, the traffic volume data collected by fixed-location sensors, such as loop detectors, often suffer from the missing data problem and low coverage problem. The missing data problem could be caused by hardware malfunct… ▽ More Traffic volume information is critical for intelligent transportation systems. It serves as a key input to transportation planning, roadway design, and traffic signal control. However, the traffic volume data collected by fixed-location sensors, such as loop detectors, often suffer from the missing data problem and low coverage problem. The missing data problem could be caused by hardware malfunction. The low coverage problem is due to the limited coverage of fixed-location sensors in the transportation network, which restrains our understanding of the traffic at the network level. To tackle these problems, we propose a probabilistic model for traffic volume reconstruction by fusing fixed-location sensor data and probe vehicle data. We apply the probabilistic principal component analysis (PPCA) to capture the correlations in traffic volume data. An innovative contribution of this work is that we also integrate probe vehicle data into the framework, which allows the model to solve both of the above-mentioned two problems. Using a real-world traffic volume dataset, we show that the proposed method outperforms state-of-the-art methods for the extensively studied missing data problem. Moreover, for the low coverage problem, which cannot be handled by most existing methods, the proposed model can also achieve high accuracy. The experiments also show that even when the missing ratio reaches 80%, the proposed method can still give an accurate estimate of the unknown traffic volumes with only a 10% probe vehicle penetration rate. The results validate the effectiveness and robustness of the proposed model and demonstrate its potential for practical applications. △ Less

Submitted 6 May, 2021; originally announced May 2021.

Corner Case Generation and Analysis for Safety Assessment of Autonomous Vehicles

Authors: Haowei Sun, Shuo Feng, Xintao Yan, Henry X. Liu

Abstract: Testing and evaluation is a crucial step in the development and deployment of Connected and Automated Vehicles (CAVs). To comprehensively evaluate the performance of CAVs, it is of necessity to test the CAVs in safety-critical scenarios, which rarely happen in naturalistic driving environment. Therefore, how to purposely and systematically generate these corner cases becomes an important problem.… ▽ More Testing and evaluation is a crucial step in the development and deployment of Connected and Automated Vehicles (CAVs). To comprehensively evaluate the performance of CAVs, it is of necessity to test the CAVs in safety-critical scenarios, which rarely happen in naturalistic driving environment. Therefore, how to purposely and systematically generate these corner cases becomes an important problem. Most existing studies focus on generating adversarial examples for perception systems of CAVs, whereas limited efforts have been put on the decision-making systems, which is the highlight of this paper. As the CAVs need to interact with numerous background vehicles (BVs) for a long duration, variables that define the corner cases are usually high dimensional, which makes the generation a challenging problem. In this paper, a unified framework is proposed to generate corner cases for the decision-making systems. To address the challenge brought by high dimensionality, the driving environment is formulated based on Markov Decision Process, and the deep reinforcement learning techniques are applied to learn the behavior policy of BVs. With the learned policy, BVs will behave and interact with the CAVs more aggressively, resulting in more corner cases. To further analyze the generated corner cases, the techniques of feature extraction and clustering are utilized. By selecting representative cases of each cluster and outliers, the valuable corner cases can be identified from all generated corner cases. Simulation results of a highway driving environment show that the proposed methods can effectively generate and identify the valuable corner cases. △ Less

Submitted 5 February, 2021; originally announced February 2021.

Comments: 23 pages, 11 figures

A Learning-based Stochastic Driving Model for Autonomous Vehicle Testing

Authors: Lin Liu, Shuo Feng, Yiheng Feng, Xichan Zhu, Henry X. Liu

Abstract: In the simulation-based testing and evaluation of autonomous vehicles (AVs), how background vehicles (BVs) drive directly influences the AV's driving behavior and further impacts the testing result. Existing simulation platforms use either pre-determined trajectories or deterministic driving models to model the BVs' behaviors. However, pre-determined BV trajectories can not react to the AV's maneu… ▽ More In the simulation-based testing and evaluation of autonomous vehicles (AVs), how background vehicles (BVs) drive directly influences the AV's driving behavior and further impacts the testing result. Existing simulation platforms use either pre-determined trajectories or deterministic driving models to model the BVs' behaviors. However, pre-determined BV trajectories can not react to the AV's maneuvers, and deterministic models are different from real human drivers due to the lack of stochastic components and errors. Both methods lead to unrealistic traffic scenarios. This paper presents a learning-based stochastic driving model that meets the unique needs of AV testing, i.e. interactive and human-like. The model is built based on the long-short-term-memory (LSTM) architecture. By incorporating the concept of quantile-regression to the loss function of the model, the stochastic behaviors are reproduced without any prior assumption of human drivers. The model is trained with the large-scale naturalistic driving data (NDD) from the Safety Pilot Model Deployment(SPMD) project and then compared with a stochastic intelligent driving model (IDM). Analysis of individual trajectories shows that the proposed model can reproduce more similar trajectories to human drivers than IDM. To validate the ability of the proposed model in generating a naturalistic driving environment, traffic simulation experiments are implemented. The results show that the traffic flow parameters such as speed, range, and headway distribution match closely with the NDD, which is of significant importance for AV testing and evaluation. △ Less

Submitted 4 February, 2021; originally announced February 2021.

Distributionally Consistent Simulation of Naturalistic Driving Environment for Autonomous Vehicle Testing

Authors: Xintao Yan, Shuo Feng, Haowei Sun, Henry X. Liu

Abstract: Microscopic traffic simulation provides a controllable, repeatable, and efficient testing environment for autonomous vehicles (AVs). To evaluate AVs' safety performance unbiasedly, the probability distributions of environment statistics in the simulated naturalistic driving environment (NDE) need to be consistent with those from the real-world driving environment. However, although human driving b… ▽ More Microscopic traffic simulation provides a controllable, repeatable, and efficient testing environment for autonomous vehicles (AVs). To evaluate AVs' safety performance unbiasedly, the probability distributions of environment statistics in the simulated naturalistic driving environment (NDE) need to be consistent with those from the real-world driving environment. However, although human driving behaviors have been extensively investigated in the transportation engineering field, most existing models were developed for traffic flow analysis without considering the distributional consistency of driving behaviors, which could cause significant evaluation biasedness for AV testing. To fill this research gap, a distributional consistent NDE modeling framework is proposed in this paper. Using large-scale naturalistic driving data, empirical distributions are obtained to construct the stochastic human driving behavior models under different conditions. To address the error accumulation problem during the simulation, an optimization-based method is further designed to refine the empirical behavior models. Specifically, the vehicle state evolution is modeled as a Markov chain and its stationary distribution is twisted to match the distribution from the real-world driving environment. The framework is evaluated in the case study of a multi-lane highway driving simulation, where the distributional accuracy of the generated NDE is validated and the safety performance of an AV model is effectively evaluated. △ Less

Submitted 1 July, 2022; v1 submitted 7 January, 2021; originally announced January 2021.

Comments: 13 pages, 11 figures

Testing Scenario Library Generation for Connected and Automated Vehicles, Part II: Case Studies

Authors: Shuo Feng, Yiheng Feng, Haowei Sun, Shao Bao, Yi Zhang, Henry X. Liu

Abstract: Testing scenario library generation (TSLG) is a critical step for the development and deployment of connected and automated vehicles (CAVs). In Part I of this study, a general methodology for TSLG is proposed, and theoretical properties are investigated regarding the accuracy and efficiency of CAV evaluation. This paper aims to provide implementation examples and guidelines, and to enhance the pro… ▽ More Testing scenario library generation (TSLG) is a critical step for the development and deployment of connected and automated vehicles (CAVs). In Part I of this study, a general methodology for TSLG is proposed, and theoretical properties are investigated regarding the accuracy and efficiency of CAV evaluation. This paper aims to provide implementation examples and guidelines, and to enhance the proposed methodology under high-dimensional scenarios. Three typical cases, including cut-in, highway-exit, and car-following, are designed and studied in this paper. For each case, the process of library generation and CAV evaluation is elaborated. To address the challenges brought by high dimensions, the proposed methodology is further enhanced by reinforcement learning technique. For all three cases, results show that the proposed methods can accelerate the CAV evaluation process by multiple magnitudes with same evaluation accuracy, if compared with the on-road test method. △ Less

Submitted 21 February, 2020; v1 submitted 8 May, 2019; originally announced May 2019.

Comments: 12 pages, 13 figures

Journal ref: IEEE Transactions on Intelligent Transportation Systems, 2020

Testing Scenario Library Generation for Connected and Automated Vehicles, Part I: Methodology

Authors: Shuo Feng, Yiheng Feng, Chunhui Yu, Yi Zhang, Henry X. Liu

Abstract: Testing and evaluation is a critical step in the development and deployment of connected and automated vehicles (CAVs), and yet there is no systematic framework to generate testing scenario library. This study aims to provide a general framework for the testing scenario library generation (TSLG) problem with different operational design domains (ODDs), CAV models, and performance metrics. Given an… ▽ More Testing and evaluation is a critical step in the development and deployment of connected and automated vehicles (CAVs), and yet there is no systematic framework to generate testing scenario library. This study aims to provide a general framework for the testing scenario library generation (TSLG) problem with different operational design domains (ODDs), CAV models, and performance metrics. Given an ODD, the testing scenario library is defined as a critical set of scenarios that can be used for CAV test. Each testing scenario is evaluated by a newly proposed measure, scenario criticality, which can be computed as a combination of maneuver challenge and exposure frequency. To search for critical scenarios, an auxiliary objective function is designed, and a multi-start optimization method along with seed-filling is applied. The proposed framework is theoretically proved to obtain accurate evaluation results with much fewer number of tests, if compared with the on-road test method. In part II of the study, three case studies are investigated to demonstrate the proposed methodologies. Reinforcement learning based technique is applied to enhance the searching method under high-dimensional scenarios. △ Less

Submitted 5 February, 2020; v1 submitted 8 May, 2019; originally announced May 2019.

Comments: 11 pages,3 figures

Journal ref: IEEE Transactions on Intelligent Transportation Systems, 2020