Gesture Recognition with Keypoint and Radar Stream Fusion for Automated Vehicles
Authors:
Adrian Holzbock,
Nicolai Kern,
Christian Waldschmidt,
Klaus Dietmayer,
Vasileios Belagiannis
Abstract:
We present a joint camera and radar approach to enable autonomous vehicles to understand and react to human gestures in everyday traffic. Initially, we process the radar data with a PointNet followed by a spatio-temporal multilayer perceptron (stMLP). Independently, the human body pose is extracted from the camera frame and processed with a separate stMLP network. We propose a fusion neural networ…
▽ More
We present a joint camera and radar approach to enable autonomous vehicles to understand and react to human gestures in everyday traffic. Initially, we process the radar data with a PointNet followed by a spatio-temporal multilayer perceptron (stMLP). Independently, the human body pose is extracted from the camera frame and processed with a separate stMLP network. We propose a fusion neural network for both modalities, including an auxiliary loss for each modality. In our experiments with a collected dataset, we show the advantages of gesture recognition with two modalities. Motivated by adverse weather conditions, we also demonstrate promising performance when one of the sensors lacks functionality.
△ Less
Submitted 20 February, 2023;
originally announced February 2023.
Under the Sand: Navigation and Localization of a Micro Aerial Vehicle for Landmine Detection with Ground Penetrating Synthetic Aperture Radar
Authors:
Rik Bähnemann,
Nicholas Lawrance,
Lucas Streichenberg,
Jen Jen Chung,
Michael Pantic,
Alexander Grathwohl,
Christian Waldschmidt,
Roland Siegwart
Abstract:
Ground penetrating radar mounted on micro aerial vehicle (MAV) is a promising tool to assist humanitarian landmine clearance. However, the quality of synthetic aperture radar images depends on accurate and precise motion estimation of the radar antennas as well as generating informative viewpoints with the MAV. This paper presents a complete and automatic airborne ground-penetrating synthetic aper…
▽ More
Ground penetrating radar mounted on micro aerial vehicle (MAV) is a promising tool to assist humanitarian landmine clearance. However, the quality of synthetic aperture radar images depends on accurate and precise motion estimation of the radar antennas as well as generating informative viewpoints with the MAV. This paper presents a complete and automatic airborne ground-penetrating synthetic aperture radar (GPSAR) system. The system consists of a spatially calibrated and temporally synchronized industrial grade sensor suite that enables navigation above ground level, radar imaging, and optical imaging. A custom mission planning framework allows generation and automatic execution of stripmap and circular (GPSAR) trajectories controlled above ground level as well as aerial imaging survey flights. A factor graph based state estimator fuses measurements from dual receiver real-time kinematic (RTK) global navigation satellite system (GNSS) and inertial measurement unit (IMU) to obtain precise, high rate platform positions and orientations. Ground truth experiments showed sensor timing as accurate as 0.8 us and as precise as 0.1 us with localization rates of 1 kHz. The dual position factor formulation improves online localization accuracy up to 40% and batch localization accuracy up to 59% compared to a single position factor with uncertain heading initialization. Our field trials validated a localization accuracy and precision that enables coherent radar measurement addition and detection of radar targets buried in sand. This validates the potential as an aerial landmine detection system.
△ Less
Submitted 5 December, 2021; v1 submitted 18 June, 2021;
originally announced June 2021.
