Showing 1–6 of 6 results for author: Maggio, D

A-TEAM: Advanced Traffic Event Analysis and Management Platform for Transportation Data-Driven Problem Solving

Authors: Zilin Bian, Dachuan Zuo, **gqin Gao, Kaan Ozbay, Matthew D. Maggio

Abstract: The rapid growth in terms of the availability of transportation data provides great potential for the introduction of emerging data-driven methodologies into transportation-related research and development efforts. However, advanced data-driven models, such as artificial-intelligence based approaches, usually contain complicated modeling structures and require strict data formats along with a very… ▽ More The rapid growth in terms of the availability of transportation data provides great potential for the introduction of emerging data-driven methodologies into transportation-related research and development efforts. However, advanced data-driven models, such as artificial-intelligence based approaches, usually contain complicated modeling structures and require strict data formats along with a very complex execution environment. It is thus often challenging to deploy and implement such data-driven models in a real-world environment. Moreover, a full-fledged application requires not only well developed and calibrated models, but also efficient connections with back end infrastructure such as large databases and front end utilities, such as a user-interface. This paper introduces a novel platform which provides an integrated architecture for deploying multi-purpose real-time traffic management applications. Inspired by the concept of modular design in software system development, this paper proposes a modular platform allowing users to customize their mission specific needs and preferences. The developed platform is capable of incorporating flexible user-provided models and/or data with the ultimate goal of deploying them as a complete application ready for real-world use. To illustrate this novel modular software system concept, this paper presents a work zone management and coordination application that is built upon the developed implementation platform to provide useful decision support to traffic engineers. △ Less

Submitted 6 June, 2024; originally announced June 2024.

Clio: Real-time Task-Driven Open-Set 3D Scene Graphs

Authors: Dominic Maggio, Yun Chang, Nathan Hughes, Matthew Trang, Dan Griffith, Carlyn Dougherty, Eric Cristofalo, Lukas Schmid, Luca Carlone

Abstract: Modern tools for class-agnostic image segmentation (e.g., SegmentAnything) and open-set semantic understanding (e.g., CLIP) provide unprecedented opportunities for robot perception and map**. While traditional closed-set metric-semantic maps were restricted to tens or hundreds of semantic classes, we can now build maps with a plethora of objects and countless semantic variations. This leaves us… ▽ More Modern tools for class-agnostic image segmentation (e.g., SegmentAnything) and open-set semantic understanding (e.g., CLIP) provide unprecedented opportunities for robot perception and map**. While traditional closed-set metric-semantic maps were restricted to tens or hundreds of semantic classes, we can now build maps with a plethora of objects and countless semantic variations. This leaves us with a fundamental question: what is the right granularity for the objects (and, more generally, for the semantic concepts) the robot has to include in its map representation? While related work implicitly chooses a level of granularity by tuning thresholds for object detection, we argue that such a choice is intrinsically task-dependent. The first contribution of this paper is to propose a task-driven 3D scene understanding problem, where the robot is given a list of tasks in natural language and has to select the granularity and the subset of objects and scene structure to retain in its map that is sufficient to complete the tasks. We show that this problem can be naturally formulated using the Information Bottleneck (IB), an established information-theoretic framework. The second contribution is an algorithm for task-driven 3D scene understanding based on an Agglomerative IB approach, that is able to cluster 3D primitives in the environment into task-relevant objects and regions and executes incrementally. The third contribution is to integrate our task-driven clustering algorithm into a real-time pipeline, named Clio, that constructs a hierarchical 3D scene graph of the environment online using only onboard compute, as the robot explores it. Our final contribution is an extensive experimental campaign showing that Clio not only allows real-time construction of compact open-set 3D scene graphs, but also improves the accuracy of task execution by limiting the map to relevant semantic concepts. △ Less

Submitted 29 April, 2024; v1 submitted 21 April, 2024; originally announced April 2024.

VERF: Runtime Monitoring of Pose Estimation with Neural Radiance Fields

Authors: Dominic Maggio, Courtney Mario, Luca Carlone

Abstract: We present VERF, a collection of two methods (VERF-PnP and VERF-Light) for providing runtime assurance on the correctness of a camera pose estimate of a monocular camera without relying on direct depth measurements. We leverage the ability of NeRF (Neural Radiance Fields) to render novel RGB perspectives of a scene. We only require as input the camera image whose pose is being estimated, an estima… ▽ More We present VERF, a collection of two methods (VERF-PnP and VERF-Light) for providing runtime assurance on the correctness of a camera pose estimate of a monocular camera without relying on direct depth measurements. We leverage the ability of NeRF (Neural Radiance Fields) to render novel RGB perspectives of a scene. We only require as input the camera image whose pose is being estimated, an estimate of the camera pose we want to monitor, and a NeRF model containing the scene pictured by the camera. We can then predict if the pose estimate is within a desired distance from the ground truth and justify our prediction with a level of confidence. VERF-Light does this by rendering a viewpoint with NeRF at the estimated pose and estimating its relative offset to the sensor image up to scale. Since scene scale is unknown, the approach renders another auxiliary image and reasons over the consistency of the optical flows across the three images. VERF-PnP takes a different approach by rendering a stereo pair of images with NeRF and utilizing the Perspective-n-Point (PnP) algorithm. We evaluate both methods on the LLFF dataset, on data from a Unitree A1 quadruped robot, and on data collected from Blue Origin's sub-orbital New Shepard rocket to demonstrate the effectiveness of the proposed pose monitoring method across a range of scene scales. We also show monitoring can be completed in under half a second on a 3090 GPU. △ Less

Submitted 11 August, 2023; originally announced August 2023.

Vision-Based Terrain Relative Navigation on High-Altitude Balloon and Sub-Orbital Rocket

Authors: Dominic Maggio, Courtney Mario, Brett Streetman, Ted Steiner, Luca Carlone

Abstract: We present an experimental analysis on the use of a camera-based approach for high-altitude navigation by associating mapped landmarks from a satellite image database to camera images, and by leveraging inertial sensors between camera frames. We evaluate performance of both a sideways-tilted and downward-facing camera on data collected from a World View Enterprises high-altitude balloon with data… ▽ More We present an experimental analysis on the use of a camera-based approach for high-altitude navigation by associating mapped landmarks from a satellite image database to camera images, and by leveraging inertial sensors between camera frames. We evaluate performance of both a sideways-tilted and downward-facing camera on data collected from a World View Enterprises high-altitude balloon with data beginning at an altitude of 33 km and descending to near ground level (4.5 km) with 1.5 hours of flight time. We demonstrate less than 290 meters of average position error over a trajectory of more than 150 kilometers. In addition to showing performance across a range of altitudes, we also demonstrate the robustness of the Terrain Relative Navigation (TRN) method to rapid rotations of the balloon, in some cases exceeding 20 degrees per second, and to camera obstructions caused by both cloud coverage and cords swaying underneath the balloon. Additionally, we evaluate performance on data collected by two cameras inside the capsule of Blue Origin's New Shepard rocket on payload flight NS-23, traveling at speeds up to 880 km/hr, and demonstrate less than 55 meters of average position error. △ Less

Submitted 15 February, 2023; originally announced February 2023.

Comments: Published in 2023 AIAA SciTech

Journal ref: 2023 AIAA SciTech

A Correct-and-Certify Approach to Self-Supervise Object Pose Estimators via Ensemble Self-Training

Authors: **gnan Shi, Rajat Talak, Dominic Maggio, Luca Carlone

Abstract: Real-world robotics applications demand object pose estimation methods that work reliably across a variety of scenarios. Modern learning-based approaches require large labeled datasets and tend to perform poorly outside the training domain. Our first contribution is to develop a robust corrector module that corrects pose estimates using depth information, thus enabling existing methods to better g… ▽ More Real-world robotics applications demand object pose estimation methods that work reliably across a variety of scenarios. Modern learning-based approaches require large labeled datasets and tend to perform poorly outside the training domain. Our first contribution is to develop a robust corrector module that corrects pose estimates using depth information, thus enabling existing methods to better generalize to new test domains; the corrector operates on semantic keypoints (but is also applicable to other pose estimators) and is fully differentiable. Our second contribution is an ensemble self-training approach that simultaneously trains multiple pose estimators in a self-supervised manner. Our ensemble self-training architecture uses the robust corrector to refine the output of each pose estimator; then, it evaluates the quality of the outputs using observable correctness certificates; finally, it uses the observably correct outputs for further training, without requiring external supervision. As an additional contribution, we propose small improvements to a regression-based keypoint detection architecture, to enhance its robustness to outliers; these improvements include a robust pooling scheme and a robust centroid computation. Experiments on the YCBV and TLESS datasets show the proposed ensemble self-training outperforms fully supervised baselines while not requiring 3D annotations on real data. △ Less

Submitted 11 May, 2023; v1 submitted 12 February, 2023; originally announced February 2023.

Loc-NeRF: Monte Carlo Localization using Neural Radiance Fields

Authors: Dominic Maggio, Marcus Abate, **gnan Shi, Courtney Mario, Luca Carlone

Abstract: We present Loc-NeRF, a real-time vision-based robot localization approach that combines Monte Carlo localization and Neural Radiance Fields (NeRF). Our system uses a pre-trained NeRF model as the map of an environment and can localize itself in real-time using an RGB camera as the only exteroceptive sensor onboard the robot. While neural radiance fields have seen significant applications for visua… ▽ More We present Loc-NeRF, a real-time vision-based robot localization approach that combines Monte Carlo localization and Neural Radiance Fields (NeRF). Our system uses a pre-trained NeRF model as the map of an environment and can localize itself in real-time using an RGB camera as the only exteroceptive sensor onboard the robot. While neural radiance fields have seen significant applications for visual rendering in computer vision and graphics, they have found limited use in robotics. Existing approaches for NeRF-based localization require both a good initial pose guess and significant computation, making them impractical for real-time robotics applications. By using Monte Carlo localization as a workhorse to estimate poses using a NeRF map model, Loc-NeRF is able to perform localization faster than the state of the art and without relying on an initial pose estimate. In addition to testing on synthetic data, we also run our system using real data collected by a Clearpath Jackal UGV and demonstrate for the first time the ability to perform real-time global localization with neural radiance fields. We make our code publicly available at https://github.com/MIT-SPARK/Loc-NeRF. △ Less

Submitted 19 September, 2022; originally announced September 2022.