Showing 1–50 of 63 results for author: Tosi, F

Stereo-Depth Fusion through Virtual Pattern Projection

Authors: Luca Bartolomei, Matteo Poggi, Fabio Tosi, Andrea Conti, Stefano Mattoccia

Abstract: This paper presents a novel general-purpose stereo and depth data fusion paradigm that mimics the active stereo principle by replacing the unreliable physical pattern projector with a depth sensor. It works by projecting virtual patterns consistent with the scene geometry onto the left and right images acquired by a conventional stereo camera, using the sparse hints obtained from a depth sensor, t… ▽ More This paper presents a novel general-purpose stereo and depth data fusion paradigm that mimics the active stereo principle by replacing the unreliable physical pattern projector with a depth sensor. It works by projecting virtual patterns consistent with the scene geometry onto the left and right images acquired by a conventional stereo camera, using the sparse hints obtained from a depth sensor, to facilitate the visual correspondence. Purposely, any depth sensing device can be seamlessly plugged into our framework, enabling the deployment of a virtual active stereo setup in any possible environment and overcoming the severe limitations of physical pattern projection, such as the limited working range and environmental conditions. Exhaustive experiments on indoor and outdoor datasets featuring both long and close range, including those providing raw, unfiltered depth hints from off-the-shelf depth sensors, highlight the effectiveness of our approach in notably boosting the robustness and accuracy of algorithms and deep stereo without any code modification and even without re-training. Additionally, we assess the performance of our strategy on active stereo evaluation datasets with conventional pattern projection. Indeed, in all these scenarios, our virtual pattern projection paradigm achieves state-of-the-art performance. The source code is available at: https://github.com/bartn8/vppstereo. △ Less

Submitted 6 June, 2024; originally announced June 2024.

Comments: extended version of ICCV 2023: "Active Stereo Without Pattern Projector"

Calibration of MAJIS (Moons And Jupiter Imaging Spectrometer): III. Spectral Calibration

Authors: Paolo Haffoud, François Poulet, Mathieu Vincendon, Gianrico Filacchione, Alessandra Barbis, Pierre Guiot, Benoit Lecomte, Yves Langevin, Giuseppe Piccioni, Cydalise Dumesnil, Sébastien Rodriguez, John Carter, Stefani Stefania, Leonardo Tommasi, Federico Tosi, Cédric Pilorget

Abstract: The Moons And Jupiter Imaging Spectrometer (MAJIS) is the visible and near-infrared imaging spectrometer onboard ESA s Jupiter Icy Moons Explorer (JUICE) mission. Before its integration into the spacecraft, the instrument undergoes an extensive ground calibration to establish its baseline performances. This process prepares the imaging spectrometer for flight operations by characterizing the behav… ▽ More The Moons And Jupiter Imaging Spectrometer (MAJIS) is the visible and near-infrared imaging spectrometer onboard ESA s Jupiter Icy Moons Explorer (JUICE) mission. Before its integration into the spacecraft, the instrument undergoes an extensive ground calibration to establish its baseline performances. This process prepares the imaging spectrometer for flight operations by characterizing the behavior of the instrument under various operative conditions and uncovering instrumental distortions that may depend on instrumental commands. Two steps of the on-ground calibration campaigns were held at the instrument level to produce the data. Additional in-flight measurements have recently been obtained after launch during the Near-Earth Commissioning Phase. In this article, we present the analyses of these datasets, focusing on the characterization of the spectral performances. First, we describe and analyze the spectral calibration datasets obtained using both monochromatic sources and polychromatic sources coupled with solid and gas samples. Then, we derive the spectral sampling and the spectral response function over the entire field of view. These spectral characteristics are quantified for various operational parameters of MAJIS, such as temperature and spectral binning. The derived on-ground performances are then compared with in-flight measurements obtained after launch and presented in the framework of the MAJIS performance requirements. △ Less

Submitted 29 May, 2024; originally announced May 2024.

Journal ref: Rev. Sci. Instrum. 95, 031301 (2024)

Federated Online Adaptation for Deep Stereo

Authors: Matteo Poggi, Fabio Tosi

Abstract: We introduce a novel approach for adapting deep stereo networks in a collaborative manner. By building over principles of federated learning, we develop a distributed framework allowing for demanding the optimization process to a number of clients deployed in different environments. This makes it possible, for a deep stereo network running on resourced-constrained devices, to capitalize on the ada… ▽ More We introduce a novel approach for adapting deep stereo networks in a collaborative manner. By building over principles of federated learning, we develop a distributed framework allowing for demanding the optimization process to a number of clients deployed in different environments. This makes it possible, for a deep stereo network running on resourced-constrained devices, to capitalize on the adaptation process carried out by other instances of the same architecture, and thus improve its accuracy in challenging environments even when it cannot carry out adaptation on its own. Experimental results show how federated adaptation performs equivalently to on-device adaptation, and even better when dealing with challenging environments. △ Less

Submitted 23 May, 2024; originally announced May 2024.

Comments: CVPR 2024. Project page: https://fedstereo.github.io/

The Third Monocular Depth Estimation Challenge

Authors: Jaime Spencer, Fabio Tosi, Matteo Poggi, Ripudaman Singh Arora, Chris Russell, Simon Hadfield, Richard Bowden, GuangYuan Zhou, ZhengXin Li, Qiang Rao, Yi** Bao, Xiao Liu, Dohyeong Kim, **seong Kim, Myunghyun Kim, Mykola Lavreniuk, Rui Li, Qing Mao, Jiang Wu, Yu Zhu, **qiu Sun, Yanning Zhang, Suraj Patni, Aradhye Agarwal, Chetan Arora , et al. (16 additional authors not shown)

Abstract: This paper discusses the results of the third edition of the Monocular Depth Estimation Challenge (MDEC). The challenge focuses on zero-shot generalization to the challenging SYNS-Patches dataset, featuring complex scenes in natural and indoor settings. As with the previous edition, methods can use any form of supervision, i.e. supervised or self-supervised. The challenge received a total of 19 su… ▽ More This paper discusses the results of the third edition of the Monocular Depth Estimation Challenge (MDEC). The challenge focuses on zero-shot generalization to the challenging SYNS-Patches dataset, featuring complex scenes in natural and indoor settings. As with the previous edition, methods can use any form of supervision, i.e. supervised or self-supervised. The challenge received a total of 19 submissions outperforming the baseline on the test set: 10 among them submitted a report describing their approach, highlighting a diffused use of foundational models such as Depth Anything at the core of their method. The challenge winners drastically improved 3D F-Score performance, from 17.51% to 23.72%. △ Less

Submitted 27 April, 2024; v1 submitted 25 April, 2024; originally announced April 2024.

Comments: To appear in CVPRW2024

How NeRFs and 3D Gaussian Splatting are Resha** SLAM: a Survey

Authors: Fabio Tosi, Youmin Zhang, Ziren Gong, Erik Sandström, Stefano Mattoccia, Martin R. Oswald, Matteo Poggi

Abstract: Over the past two decades, research in the field of Simultaneous Localization and Map** (SLAM) has undergone a significant evolution, highlighting its critical role in enabling autonomous exploration of unknown environments. This evolution ranges from hand-crafted methods, through the era of deep learning, to more recent developments focused on Neural Radiance Fields (NeRFs) and 3D Gaussian Spla… ▽ More Over the past two decades, research in the field of Simultaneous Localization and Map** (SLAM) has undergone a significant evolution, highlighting its critical role in enabling autonomous exploration of unknown environments. This evolution ranges from hand-crafted methods, through the era of deep learning, to more recent developments focused on Neural Radiance Fields (NeRFs) and 3D Gaussian Splatting (3DGS) representations. Recognizing the growing body of research and the absence of a comprehensive survey on the topic, this paper aims to provide the first comprehensive overview of SLAM progress through the lens of the latest advancements in radiance fields. It sheds light on the background, evolutionary path, inherent strengths and limitations, and serves as a fundamental reference to highlight the dynamic progress and specific challenges. △ Less

Submitted 11 April, 2024; v1 submitted 20 February, 2024; originally announced February 2024.

Revisiting Depth Completion from a Stereo Matching Perspective for Cross-domain Generalization

Authors: Luca Bartolomei, Matteo Poggi, Andrea Conti, Fabio Tosi, Stefano Mattoccia

Abstract: This paper proposes a new framework for depth completion robust against domain-shifting issues. It exploits the generalization capability of modern stereo networks to face depth completion, by processing fictitious stereo pairs obtained through a virtual pattern projection paradigm. Any stereo network or traditional stereo matcher can be seamlessly plugged into our framework, allowing for the depl… ▽ More This paper proposes a new framework for depth completion robust against domain-shifting issues. It exploits the generalization capability of modern stereo networks to face depth completion, by processing fictitious stereo pairs obtained through a virtual pattern projection paradigm. Any stereo network or traditional stereo matcher can be seamlessly plugged into our framework, allowing for the deployment of a virtual stereo setup that is future-proof against advancement in the stereo field. Exhaustive experiments on cross-domain generalization support our claims. Hence, we argue that our framework can help depth completion to reach new deployment scenarios. △ Less

Submitted 14 December, 2023; originally announced December 2023.

Comments: 3DV 2024. Code: https://github.com/bartn8/vppdc - Project page: https://vppdc.github.io/

Identification of Ammonium Salts on Comet 67P/C-G Surface from Infrared VIRTIS/Rosetta Data Based on Laboratory Experiments. Implications and Perspectives

Authors: Olivier Poch, Istiqomah Istiqomah, Eric Quirico, Pierre Beck, Bernard Schmitt, Patrice Theulé, Alexandre Faure, Pierre Hily-Blant, Lydie Bonal, Andrea Raponi, Mauro Ciarniello, Batiste Rousseau, Sandra Potin, Olivier Brissaud, Laurène Flandinet, Gianrico Filacchione, Antoine Pommerol, Nicolas Thomas, David Kappel, Vito Mennella, Lyuba Moroz, Vassilissa Vinogradoff, Gabriele Arnold, Stéphane Erard, Dominique Bockelée-Morvan , et al. (7 additional authors not shown)

Abstract: The nucleus of comet 67P/Churyumov-Gerasimenko exhibits a broad spectral reflectance feature around 3.2 $μ$m, which is omnipresent in all spectra of the surface, and whose attribution has remained elusive since its discovery. Based on laboratory experiments, we have shown that most of this absorption feature is due to ammonium (NH4+) salts mixed with the dark surface material. The depth of the ban… ▽ More The nucleus of comet 67P/Churyumov-Gerasimenko exhibits a broad spectral reflectance feature around 3.2 $μ$m, which is omnipresent in all spectra of the surface, and whose attribution has remained elusive since its discovery. Based on laboratory experiments, we have shown that most of this absorption feature is due to ammonium (NH4+) salts mixed with the dark surface material. The depth of the band is compatible with semi-volatile ammonium salts being a major reservoir of nitrogen in the comet, which could dominate over refractory organic matter and volatile species. These salts may thus represent the long-sought reservoir of nitrogen in comets, possibly bringing their nitrogen-to-carbon ratio in agreement with the solar value. Moreover, the reflectance spectra of several asteroids are compatible with the presence of NH4+ salts at their surfaces. The presence of such salts, and other NH4+-bearing compounds on asteroids, comets, and possibly in proto-stellar environments, suggests that NH4+ may be a tracer of the incorporation and transformation of nitrogen in ices, minerals and organics, at different phases of the formation of the Solar System. △ Less

Submitted 20 November, 2023; originally announced November 2023.

Journal ref: European Conference on Laboratory Astrophysics ECLA2020, Sep 2021, Anacapri, Italy. pp.271 - 279

Composition and thermal properties of Ganymede's surface from JWST/NIRSpec and MIRI observations

Authors: D. Bockelee-Morvan, E. Lellouch, O. Poch, E. Quirico, S. Cazaux, I. de Pater, T. Fouchet, P. M. Fry, P. Rodriguez-Ovalle, F. Tosi, M. H. Wong, I. Boshuizen, K. de Kleer, L. N. Fletcher, L. Meunier, A. Mura, L. Roth, J. Saur, B. Schmitt, S. K. Trumbo, M. E. Brown, J. O'Donoghue, G. S. Orton, M. R. Showalter

Abstract: JWST NIRSpec IFU (2.9-5.3 mu) and MIRI MRS (4.9-28.5 mu) observations were performed on both the leading and trailing hemispheres of Ganymede with a spectral resolution of ~2700. Reflectance spectra show signatures of water ice, CO2 and H2O2. An absorption feature at 5.9 mu is revealed and is tentatively assigned to sulfuric acid hydrates. The CO2 4.26-mu band shows latitudinal and longitudinal va… ▽ More JWST NIRSpec IFU (2.9-5.3 mu) and MIRI MRS (4.9-28.5 mu) observations were performed on both the leading and trailing hemispheres of Ganymede with a spectral resolution of ~2700. Reflectance spectra show signatures of water ice, CO2 and H2O2. An absorption feature at 5.9 mu is revealed and is tentatively assigned to sulfuric acid hydrates. The CO2 4.26-mu band shows latitudinal and longitudinal variations in depth, shape and position over the two hemispheres, unveiling different CO2 physical states. In the ice-rich polar regions, which are the most exposed to Jupiter's plasma irradiation, the CO2 band is redshifted with respect to other terrains. In the leading northern polar cap, the CO2 band is dominated by a high wavelength component at ~4.27 mu, consistent with CO2 trapped in amorphous water ice. At equatorial latitudes (and especially on dark terrains) the observed band is broader and shifted towards the blue, suggesting CO2 adsorbed on non-icy materials. Amorphous ice is detected in the ice-rich polar regions, and is especially abundant on the leading northern polar cap. In both hemispheres the north polar cap ice appears to be more processed than the south polar cap. A longitudinal modification of the H2O ice molecular structure and/or nano/micrometre-scale texture, of diurnal or geographic origin, is observed in both hemispheres. Ice frost is observed on the morning limb of the trailing hemisphere, possibly formed during the night from the recondensation of water subliming from the warmer subsurface. Reflectance spectra of the dark terrains are compatible with the presence of Na-/Mg-sulfate salts, sulfuric acid hydrates, and possibly phyllosilicates mixed with fine-grained opaque minerals, having an highly porous texture. Mid-IR brightness temperatures indicate a rough surface and a very low thermal inertia of 20-40 J m-2 s-0.5 K-1, consistent with a porous surface. △ Less

Submitted 21 October, 2023; originally announced October 2023.

Comments: 35 pages, 34 figures

Journal ref: A&A 681, A27 (2024)

GasMono: Geometry-Aided Self-Supervised Monocular Depth Estimation for Indoor Scenes

Authors: Chaoqiang Zhao, Matteo Poggi, Fabio Tosi, Lei Zhou, Qiyu Sun, Yang Tang, Stefano Mattoccia

Abstract: This paper tackles the challenges of self-supervised monocular depth estimation in indoor scenes caused by large rotation between frames and low texture. We ease the learning process by obtaining coarse camera poses from monocular sequences through multi-view geometry to deal with the former. However, we found that limited by the scale ambiguity across different scenes in the training dataset, a n… ▽ More This paper tackles the challenges of self-supervised monocular depth estimation in indoor scenes caused by large rotation between frames and low texture. We ease the learning process by obtaining coarse camera poses from monocular sequences through multi-view geometry to deal with the former. However, we found that limited by the scale ambiguity across different scenes in the training dataset, a naïve introduction of geometric coarse poses cannot play a positive role in performance improvement, which is counter-intuitive. To address this problem, we propose to refine those poses during training through rotation and translation/scale optimization. To soften the effect of the low texture, we combine the global reasoning of vision transformers with an overfitting-aware, iterative self-distillation mechanism, providing more accurate depth guidance coming from the network itself. Experiments on NYUv2, ScanNet, 7scenes, and KITTI datasets support the effectiveness of each component in our framework, which sets a new state-of-the-art for indoor self-supervised monocular depth estimation, as well as outstanding generalization ability. Code and models are available at https://github.com/zxcqlf/GasMono △ Less

Submitted 26 September, 2023; originally announced September 2023.

Comments: ICCV 2023. Code: https://github.com/zxcqlf/GasMono

Active Stereo Without Pattern Projector

Authors: Luca Bartolomei, Matteo Poggi, Fabio Tosi, Andrea Conti, Stefano Mattoccia

Abstract: This paper proposes a novel framework integrating the principles of active stereo in standard passive camera systems without a physical pattern projector. We virtually project a pattern over the left and right images according to the sparse measurements obtained from a depth sensor. Any such devices can be seamlessly plugged into our framework, allowing for the deployment of a virtual active stere… ▽ More This paper proposes a novel framework integrating the principles of active stereo in standard passive camera systems without a physical pattern projector. We virtually project a pattern over the left and right images according to the sparse measurements obtained from a depth sensor. Any such devices can be seamlessly plugged into our framework, allowing for the deployment of a virtual active stereo setup in any possible environment, overcoming the limitation of pattern projectors, such as limited working range or environmental conditions. Experiments on indoor/outdoor datasets, featuring both long and close-range, support the seamless effectiveness of our approach, boosting the accuracy of both stereo algorithms and deep networks. △ Less

Submitted 21 September, 2023; originally announced September 2023.

Comments: ICCV 2023. Code: https://github.com/bartn8/vppstereo - Project page: https://vppstereo.github.io

GO-SLAM: Global Optimization for Consistent 3D Instant Reconstruction

Authors: Youmin Zhang, Fabio Tosi, Stefano Mattoccia, Matteo Poggi

Abstract: Neural implicit representations have recently demonstrated compelling results on dense Simultaneous Localization And Map** (SLAM) but suffer from the accumulation of errors in camera tracking and distortion in the reconstruction. Purposely, we present GO-SLAM, a deep-learning-based dense visual SLAM framework globally optimizing poses and 3D reconstruction in real-time. Robust pose estimation is… ▽ More Neural implicit representations have recently demonstrated compelling results on dense Simultaneous Localization And Map** (SLAM) but suffer from the accumulation of errors in camera tracking and distortion in the reconstruction. Purposely, we present GO-SLAM, a deep-learning-based dense visual SLAM framework globally optimizing poses and 3D reconstruction in real-time. Robust pose estimation is at its core, supported by efficient loop closing and online full bundle adjustment, which optimize per frame by utilizing the learned global geometry of the complete history of input frames. Simultaneously, we update the implicit and continuous surface representation on-the-fly to ensure global consistency of 3D reconstruction. Results on various synthetic and real-world datasets demonstrate that GO-SLAM outperforms state-of-the-art approaches at tracking robustness and reconstruction accuracy. Furthermore, GO-SLAM is versatile and can run with monocular, stereo, and RGB-D input. △ Less

Submitted 5 September, 2023; originally announced September 2023.

Comments: ICCV 2023. Code: https://github.com/youmi-zym/GO-SLAM - Project Page: https://youmi-zym.github.io/projects/GO-SLAM/

JIRAM Observations of Volcanic Flux on Io: Distribution and Comparison to Tidal Heat Flow Models

Authors: M. Pettine, S. Imbeah, J. Rathbun, A. Hayes, R. Lopes-Gautier, A. Mura, F. Tosi, F. Zambon, S. Bertolino

Abstract: The Jovian Infrared Auroral Mapper (JIRAM) instrument aboard Juno has allowed clear, high-resolution imaging of Io's polar volcanoes. We have used data from JIRAM's M-band (4.78 micron) imager from eleven Juno orbits to construct a global map of volcanic flux. This map provides insight into the spatial distribution of volcanoes and the ways in which high- and low-latitude volcanoes differ. Using s… ▽ More The Jovian Infrared Auroral Mapper (JIRAM) instrument aboard Juno has allowed clear, high-resolution imaging of Io's polar volcanoes. We have used data from JIRAM's M-band (4.78 micron) imager from eleven Juno orbits to construct a global map of volcanic flux. This map provides insight into the spatial distribution of volcanoes and the ways in which high- and low-latitude volcanoes differ. Using spherical harmonic analysis, we have quantitatively compared our volcanic flux map to the surface heat flow distribution expected by modeling data of Io's tidal heat deposition (de Kleer et al. 2019). Systems of bright volcanoes were confirmed at high latitudes. Our study finds that both poles are more active than previous studies have suggested, although the south pole was viewed too infrequently to establish reliable long-term trends. While none of the tidal heat flow models match well, the asthenospheric heating model agrees best with the observed volcanic flux. △ Less

Submitted 1 December, 2023; v1 submitted 10 August, 2023; originally announced August 2023.

Comments: The paper is in revision and this version does not reflect the current status of the paper. Once it's published, the newest version will be uploaded

Learning Depth Estimation for Transparent and Mirror Surfaces

Authors: Alex Costanzino, Pierluigi Zama Ramirez, Matteo Poggi, Fabio Tosi, Stefano Mattoccia, Luigi Di Stefano

Abstract: Inferring the depth of transparent or mirror (ToM) surfaces represents a hard challenge for either sensors, algorithms, or deep networks. We propose a simple pipeline for learning to estimate depth properly for such surfaces with neural networks, without requiring any ground-truth annotation. We unveil how to obtain reliable pseudo labels by in-painting ToM objects in images and processing them wi… ▽ More Inferring the depth of transparent or mirror (ToM) surfaces represents a hard challenge for either sensors, algorithms, or deep networks. We propose a simple pipeline for learning to estimate depth properly for such surfaces with neural networks, without requiring any ground-truth annotation. We unveil how to obtain reliable pseudo labels by in-painting ToM objects in images and processing them with a monocular depth estimation model. These labels can be used to fine-tune existing monocular or stereo networks, to let them learn how to deal with ToM surfaces. Experimental results on the Booster dataset show the dramatic improvements enabled by our remarkably simple proposal. △ Less

Submitted 27 July, 2023; originally announced July 2023.

Comments: Accepted at ICCV 2023. Project Page: https://cvlab-unibo.github.io/Depth4ToM

The Second Monocular Depth Estimation Challenge

Authors: Jaime Spencer, C. Stella Qian, Michaela Trescakova, Chris Russell, Simon Hadfield, Erich W. Graf, Wendy J. Adams, Andrew J. Schofield, James Elder, Richard Bowden, Ali Anwar, Hao Chen, Xiaozhi Chen, Kai Cheng, Yuchao Dai, Huynh Thai Hoa, Sadat Hossain, Jianmian Huang, Mohan **g, Bo Li, Chao Li, Baojun Li, Zhiwen Liu, Stefano Mattoccia, Siegfried Mercelis , et al. (18 additional authors not shown)

Abstract: This paper discusses the results for the second edition of the Monocular Depth Estimation Challenge (MDEC). This edition was open to methods using any form of supervision, including fully-supervised, self-supervised, multi-task or proxy depth. The challenge was based around the SYNS-Patches dataset, which features a wide diversity of environments with high-quality dense ground-truth. This includes… ▽ More This paper discusses the results for the second edition of the Monocular Depth Estimation Challenge (MDEC). This edition was open to methods using any form of supervision, including fully-supervised, self-supervised, multi-task or proxy depth. The challenge was based around the SYNS-Patches dataset, which features a wide diversity of environments with high-quality dense ground-truth. This includes complex natural environments, e.g. forests or fields, which are greatly underrepresented in current benchmarks. The challenge received eight unique submissions that outperformed the provided SotA baseline on any of the pointcloud- or image-based metrics. The top supervised submission improved relative F-Score by 27.62%, while the top self-supervised improved it by 16.61%. Supervised submissions generally leveraged large collections of datasets to improve data diversity. Self-supervised submissions instead updated the network architecture and pretrained backbones. These results represent a significant progress in the field, while highlighting avenues for future research, such as reducing interpolation artifacts at depth boundaries, improving self-supervised indoor performance and overall natural image accuracy. △ Less

Submitted 26 April, 2023; v1 submitted 14 April, 2023; originally announced April 2023.

Comments: Published at CVPRW2023

NeRF-Supervised Deep Stereo

Authors: Fabio Tosi, Alessio Tonioni, Daniele De Gregorio, Matteo Poggi

Abstract: We introduce a novel framework for training deep stereo networks effortlessly and without any ground-truth. By leveraging state-of-the-art neural rendering solutions, we generate stereo training data from image sequences collected with a single handheld camera. On top of them, a NeRF-supervised training procedure is carried out, from which we exploit rendered stereo triplets to compensate for occl… ▽ More We introduce a novel framework for training deep stereo networks effortlessly and without any ground-truth. By leveraging state-of-the-art neural rendering solutions, we generate stereo training data from image sequences collected with a single handheld camera. On top of them, a NeRF-supervised training procedure is carried out, from which we exploit rendered stereo triplets to compensate for occlusions and depth maps as proxy labels. This results in stereo networks capable of predicting sharp and detailed disparity maps. Experimental results show that models trained under this regime yield a 30-40% improvement over existing self-supervised methods on the challenging Middlebury dataset, filling the gap to supervised models and, most times, outperforming them at zero-shot generalization. △ Less

Submitted 30 March, 2023; originally announced March 2023.

Comments: CVPR 2023. Project page: https://nerfstereo.github.io/ Code: https://github.com/fabiotosi92/NeRF-Supervised-Deep-Stereo

Depth Super-Resolution from Explicit and Implicit High-Frequency Features

Authors: Xin Qiao, Chenyang Ge, Youmin Zhang, Yanhui Zhou, Fabio Tosi, Matteo Poggi, Stefano Mattoccia

Abstract: We propose a novel multi-stage depth super-resolution network, which progressively reconstructs high-resolution depth maps from explicit and implicit high-frequency features. The former are extracted by an efficient transformer processing both local and global contexts, while the latter are obtained by projecting color images into the frequency domain. Both are combined together with depth feature… ▽ More We propose a novel multi-stage depth super-resolution network, which progressively reconstructs high-resolution depth maps from explicit and implicit high-frequency features. The former are extracted by an efficient transformer processing both local and global contexts, while the latter are obtained by projecting color images into the frequency domain. Both are combined together with depth features by means of a fusion strategy within a multi-stage and multi-scale framework. Experiments on the main benchmarks, such as NYUv2, Middlebury, DIML and RGBDD, show that our approach outperforms existing methods by a large margin (~20% on NYUv2 and DIML against the contemporary work DADA, with 16x upsampling), establishing a new state-of-the-art in the guided depth super-resolution task. △ Less

Submitted 30 May, 2023; v1 submitted 16 March, 2023; originally announced March 2023.

Booster: a Benchmark for Depth from Images of Specular and Transparent Surfaces

Authors: Pierluigi Zama Ramirez, Alex Costanzino, Fabio Tosi, Matteo Poggi, Samuele Salti, Stefano Mattoccia, Luigi Di Stefano

Abstract: Estimating depth from images nowadays yields outstanding results, both in terms of in-domain accuracy and generalization. However, we identify two main challenges that remain open in this field: dealing with non-Lambertian materials and effectively processing high-resolution images. Purposely, we propose a novel dataset that includes accurate and dense ground-truth labels at high resolution, featu… ▽ More Estimating depth from images nowadays yields outstanding results, both in terms of in-domain accuracy and generalization. However, we identify two main challenges that remain open in this field: dealing with non-Lambertian materials and effectively processing high-resolution images. Purposely, we propose a novel dataset that includes accurate and dense ground-truth labels at high resolution, featuring scenes containing several specular and transparent surfaces. Our acquisition pipeline leverages a novel deep space-time stereo framework, enabling easy and accurate labeling with sub-pixel precision. The dataset is composed of 606 samples collected in 85 different scenes, each sample includes both a high-resolution pair (12 Mpx) as well as an unbalanced stereo pair (Left: 12 Mpx, Right: 1.1 Mpx), typical of modern mobile devices that mount sensors with different resolutions. Additionally, we provide manually annotated material segmentation masks and 15K unlabeled samples. The dataset is composed of a train set and two test sets, the latter devoted to the evaluation of stereo and monocular depth estimation networks. Our experiments highlight the open challenges and future research directions in this field. △ Less

Submitted 30 January, 2024; v1 submitted 19 January, 2023; originally announced January 2023.

Comments: Extension of the paper "Open Challenges in Deep Stereo: the Booster Dataset" presented at CVPR 2022. Accepted at TPAMI

The Monocular Depth Estimation Challenge

Authors: Jaime Spencer, C. Stella Qian, Chris Russell, Simon Hadfield, Erich Graf, Wendy Adams, Andrew J. Schofield, James Elder, Richard Bowden, Heng Cong, Stefano Mattoccia, Matteo Poggi, Zeeshan Khan Suri, Yang Tang, Fabio Tosi, Hao Wang, Youmin Zhang, Yusheng Zhang, Chaoqiang Zhao

Abstract: This paper summarizes the results of the first Monocular Depth Estimation Challenge (MDEC) organized at WACV2023. This challenge evaluated the progress of self-supervised monocular depth estimation on the challenging SYNS-Patches dataset. The challenge was organized on CodaLab and received submissions from 4 valid teams. Participants were provided a devkit containing updated reference implementati… ▽ More This paper summarizes the results of the first Monocular Depth Estimation Challenge (MDEC) organized at WACV2023. This challenge evaluated the progress of self-supervised monocular depth estimation on the challenging SYNS-Patches dataset. The challenge was organized on CodaLab and received submissions from 4 valid teams. Participants were provided a devkit containing updated reference implementations for 16 State-of-the-Art algorithms and 4 novel techniques. The threshold for acceptance for novel techniques was to outperform every one of the 16 SotA baselines. All participants outperformed the baseline in traditional metrics such as MAE or AbsRel. However, pointcloud reconstruction metrics were challenging to improve upon. We found predictions were characterized by interpolation artefacts at object boundaries and errors in relative object positioning. We hope this challenge is a valuable contribution to the community and encourage authors to participate in future editions. △ Less

Submitted 22 November, 2022; originally announced November 2022.

Comments: WACV-Workshops 2023

Cross-Spectral Neural Radiance Fields

Authors: Matteo Poggi, Pierluigi Zama Ramirez, Fabio Tosi, Samuele Salti, Stefano Mattoccia, Luigi Di Stefano

Abstract: We propose X-NeRF, a novel method to learn a Cross-Spectral scene representation given images captured from cameras with different light spectrum sensitivity, based on the Neural Radiance Fields formulation. X-NeRF optimizes camera poses across spectra during training and exploits Normalized Cross-Device Coordinates (NXDC) to render images of different modalities from arbitrary viewpoints, which a… ▽ More We propose X-NeRF, a novel method to learn a Cross-Spectral scene representation given images captured from cameras with different light spectrum sensitivity, based on the Neural Radiance Fields formulation. X-NeRF optimizes camera poses across spectra during training and exploits Normalized Cross-Device Coordinates (NXDC) to render images of different modalities from arbitrary viewpoints, which are aligned and at the same resolution. Experiments on 16 forward-facing scenes, featuring color, multi-spectral and infrared images, confirm the effectiveness of X-NeRF at modeling Cross-Spectral scene representations. △ Less

Submitted 1 September, 2022; originally announced September 2022.

Comments: 3DV 2022. Project page: https://cvlab-unibo.github.io/xnerf-web/

MonoViT: Self-Supervised Monocular Depth Estimation with a Vision Transformer

Authors: Chaoqiang Zhao, Youmin Zhang, Matteo Poggi, Fabio Tosi, Xianda Guo, Zheng Zhu, Guan Huang, Yang Tang, Stefano Mattoccia

Abstract: Self-supervised monocular depth estimation is an attractive solution that does not require hard-to-source depth labels for training. Convolutional neural networks (CNNs) have recently achieved great success in this task. However, their limited receptive field constrains existing network architectures to reason only locally, dampening the effectiveness of the self-supervised paradigm. In the light… ▽ More Self-supervised monocular depth estimation is an attractive solution that does not require hard-to-source depth labels for training. Convolutional neural networks (CNNs) have recently achieved great success in this task. However, their limited receptive field constrains existing network architectures to reason only locally, dampening the effectiveness of the self-supervised paradigm. In the light of the recent successes achieved by Vision Transformers (ViTs), we propose MonoViT, a brand-new framework combining the global reasoning enabled by ViT models with the flexibility of self-supervised monocular depth estimation. By combining plain convolutions with Transformer blocks, our model can reason locally and globally, yielding depth prediction at a higher level of detail and accuracy, allowing MonoViT to achieve state-of-the-art performance on the established KITTI dataset. Moreover, MonoViT proves its superior generalization capacities on other datasets such as Make3D and DrivingStereo. △ Less

Submitted 6 August, 2022; originally announced August 2022.

Comments: Accepted by 3DV 2022

RGB-Multispectral Matching: Dataset, Learning Methodology, Evaluation

Authors: Fabio Tosi, Pierluigi Zama Ramirez, Matteo Poggi, Samuele Salti, Stefano Mattoccia, Luigi Di Stefano

Abstract: We address the problem of registering synchronized color (RGB) and multi-spectral (MS) images featuring very different resolution by solving stereo matching correspondences. Purposely, we introduce a novel RGB-MS dataset framing 13 different scenes in indoor environments and providing a total of 34 image pairs annotated with semi-dense, high-resolution ground-truth labels in the form of disparity… ▽ More We address the problem of registering synchronized color (RGB) and multi-spectral (MS) images featuring very different resolution by solving stereo matching correspondences. Purposely, we introduce a novel RGB-MS dataset framing 13 different scenes in indoor environments and providing a total of 34 image pairs annotated with semi-dense, high-resolution ground-truth labels in the form of disparity maps. To tackle the task, we propose a deep learning architecture trained in a self-supervised manner by exploiting a further RGB camera, required only during training data acquisition. In this setup, we can conveniently learn cross-modal matching in the absence of ground-truth labels by distilling knowledge from an easier RGB-RGB matching task based on a collection of about 11K unlabeled image triplets. Experiments show that the proposed pipeline sets a good performance bar (1.16 pixels average registration error) for future research on this novel, challenging task. △ Less

Submitted 14 June, 2022; originally announced June 2022.

Comments: CVPR 2022, New Orleans. Project page: https://cvlab-unibo.github.io/rgb-ms-web/

Open Challenges in Deep Stereo: the Booster Dataset

Authors: Pierluigi Zama Ramirez, Fabio Tosi, Matteo Poggi, Samuele Salti, Stefano Mattoccia, Luigi Di Stefano

Abstract: We present a novel high-resolution and challenging stereo dataset framing indoor scenes annotated with dense and accurate ground-truth disparities. Peculiar to our dataset is the presence of several specular and transparent surfaces, i.e. the main causes of failures for state-of-the-art stereo networks. Our acquisition pipeline leverages a novel deep space-time stereo framework which allows for ea… ▽ More We present a novel high-resolution and challenging stereo dataset framing indoor scenes annotated with dense and accurate ground-truth disparities. Peculiar to our dataset is the presence of several specular and transparent surfaces, i.e. the main causes of failures for state-of-the-art stereo networks. Our acquisition pipeline leverages a novel deep space-time stereo framework which allows for easy and accurate labeling with sub-pixel precision. We release a total of 419 samples collected in 64 different scenes and annotated with dense ground-truth disparities. Each sample include a high-resolution pair (12 Mpx) as well as an unbalanced pair (Left: 12 Mpx, Right: 1.1 Mpx). Additionally, we provide manually annotated material segmentation masks and 15K unlabeled samples. We evaluate state-of-the-art deep networks based on our dataset, highlighting their limitations in addressing the open challenges in stereo and drawing hints for future research. △ Less

Submitted 9 June, 2022; originally announced June 2022.

Comments: CVPR 2022, New Orleans. Project page: https://cvlab-unibo.github.io/booster-web/

On the origin of molecular oxygen on the surface of Ganymede

Authors: A. Migliorini, Z. Kanuchova, S. Ioppolo, M. Barbieri, N. C. Jones, S. V. Hoffmann, G. Strazzulla, F. Tosi, G. Piccioni

Abstract: Since its first identification on the surface of Ganymede in 1995, molecular oxygen (O2) ice has been at the center of a scientific debate as the surface temperature of the Jovian moon is on average well above the freezing point of O2. Laboratory evidence suggested that solid O2 may either exist in a cold (<50 K) subsurface layer of the icy surface of Ganymede, or it is in an atmospheric haze of t… ▽ More Since its first identification on the surface of Ganymede in 1995, molecular oxygen (O2) ice has been at the center of a scientific debate as the surface temperature of the Jovian moon is on average well above the freezing point of O2. Laboratory evidence suggested that solid O2 may either exist in a cold (<50 K) subsurface layer of the icy surface of Ganymede, or it is in an atmospheric haze of the moon. Alternatively, O2 is constantly replenished at the surface through ion irradiation of water-containing ices. A conclusive answer on the existence of solid O2 on the surface of Ganymede is hampered by the lack of detailed, extensive observational datasets. We present new ground-based, high-resolution spectroscopic observations of Ganymede's surface obtained at the Telescopio Nazionale Galileo. These are combined with dedicated laboratory measurements of ultraviolet-visible (UV-vis) photoabsorption spectra of O2 ice, both pure and mixed with other species of potential interest for the Galilean satellites. Our study confirms that the two bands identified in the visible spectra of Ganymede's surface are due to the (1,0) and (0,0) transition bands of O2 ice. Oxygen-rich ice mixtures including water (H2O) and carbon dioxide (CO2) can reproduce observational reflectance data of the Ganymede's surface better than pure O2 ice in the temperature range 20-35 K. Solid H2O and CO2 also provide an environment where O2 ice can be trapped at higher temperatures than its pure ice desorption under vacuum space conditions. Our experiments at different temperatures show also that the (1,0)/(0,0) ratio in case of the CO2:O2=1:2 ice mixture at 35 K has the closest value to observations, while at 30 K the (1,0)/(0,0) ratio seems to be mixture independent with the exception of the N2:O2=1:2 ice mixture. The present work will support the ESA/JUICE mission to the Jovian system. △ Less

Submitted 3 May, 2022; originally announced May 2022.

Comments: Accepted for publication in Icarus

Neural Disparity Refinement for Arbitrary Resolution Stereo

Authors: Filippo Aleotti, Fabio Tosi, Pierluigi Zama Ramirez, Matteo Poggi, Samuele Salti, Stefano Mattoccia, Luigi Di Stefano

Abstract: We introduce a novel architecture for neural disparity refinement aimed at facilitating deployment of 3D computer vision on cheap and widespread consumer devices, such as mobile phones. Our approach relies on a continuous formulation that enables to estimate a refined disparity map at any arbitrary output resolution. Thereby, it can handle effectively the unbalanced camera setup typical of nowaday… ▽ More We introduce a novel architecture for neural disparity refinement aimed at facilitating deployment of 3D computer vision on cheap and widespread consumer devices, such as mobile phones. Our approach relies on a continuous formulation that enables to estimate a refined disparity map at any arbitrary output resolution. Thereby, it can handle effectively the unbalanced camera setup typical of nowadays mobile phones, which feature both high and low resolution RGB sensors within the same device. Moreover, our neural network can process seamlessly the output of a variety of stereo methods and, by refining the disparity maps computed by a traditional matching algorithm like SGM, it can achieve unpaired zero-shot generalization performance compared to state-of-the-art end-to-end stereo models. △ Less

Submitted 28 October, 2021; originally announced October 2021.

Comments: 3DV 2021 Oral paper. Project page: https://cvlab-unibo.github.io/neural-disparity-refinement-web

The surface of (4) Vesta in visible light as seen by Dawn/VIR

Authors: B. Rousseau, M. C. De Sanctis, A. Raponi, M. Ciarniello, E. Ammannito, A. Frigeri, F. G. Carrozzo, F. Tosi, P. Scarica, S. Fonte, C. A. Raymond, C. T. Russel

Abstract: We analyzed the surface of Vesta at visible wavelengths, using the data of the Visible and InfraRed map** spectrometer (VIR) on board the Dawn spacecraft. We mapped the variations of various spectral parameters on the entire surface of the asteroid, and also derived a map of the lithology. We took advantage of the recent corrected VIR visible data to map the radiance factor at 550 nm, three colo… ▽ More We analyzed the surface of Vesta at visible wavelengths, using the data of the Visible and InfraRed map** spectrometer (VIR) on board the Dawn spacecraft. We mapped the variations of various spectral parameters on the entire surface of the asteroid, and also derived a map of the lithology. We took advantage of the recent corrected VIR visible data to map the radiance factor at 550 nm, three color composites, two spectral slopes, and a band area parameter relative to the 930 nm crystal field signature in pyroxene. Using the howardite-eucrite-diogenite (HED) meteorites data as a reference, we derived the lithology of Vesta using the variations of the 930 nm and 506 nm (spin-forbidden) band centers observed in the VIR dataset. Our spectral parameters highlight a significant spectral diversity at the surface of Vesta. This diversity is mainly evidenced by impact craters and illustrates the heterogeneous subsurface and upper crust of Vesta. Impact craters also participate directly in this spectral diversity by bringing dark exogenous material to an almost entire hemisphere. Our derived lithology agrees with previous results obtained using a combination of infrared and visible data. We therefore demonstrate that it is possible to obtain crucial mineralogical information from visible wavelengths alone. In addition to the 506 nm band, we identified the 550 nm spin-forbidden one. As reported by a laboratory study for synthetic pyroxenes, we also do not observe any shift of the band center of this feature across the surface of Vesta, and thus across different mineralogies, preventing use of the 550 nm spin-forbidden band for the lithology derivation. Finally, the largest previously identified olivine rich-spot shows a peculiar behavior in two color composites but not in the other spectral parameters. △ Less

Submitted 20 September, 2021; originally announced September 2021.

Comments: 23 pages, 11+7 figures

Journal ref: A&A 653, A118 (2021)

SMD-Nets: Stereo Mixture Density Networks

Authors: Fabio Tosi, Yiyi Liao, Carolin Schmitt, Andreas Geiger

Abstract: Despite stereo matching accuracy has greatly improved by deep learning in the last few years, recovering sharp boundaries and high-resolution outputs efficiently remains challenging. In this paper, we propose Stereo Mixture Density Networks (SMD-Nets), a simple yet effective learning framework compatible with a wide class of 2D and 3D architectures which ameliorates both issues. Specifically, we e… ▽ More Despite stereo matching accuracy has greatly improved by deep learning in the last few years, recovering sharp boundaries and high-resolution outputs efficiently remains challenging. In this paper, we propose Stereo Mixture Density Networks (SMD-Nets), a simple yet effective learning framework compatible with a wide class of 2D and 3D architectures which ameliorates both issues. Specifically, we exploit bimodal mixture densities as output representation and show that this allows for sharp and precise disparity estimates near discontinuities while explicitly modeling the aleatoric uncertainty inherent in the observations. Moreover, we formulate disparity estimation as a continuous problem in the image domain, allowing our model to query disparities at arbitrary spatial precision. We carry out comprehensive experiments on a new high-resolution and highly realistic synthetic stereo dataset, consisting of stereo pairs at 8Mpx resolution, as well as on real-world stereo datasets. Our experiments demonstrate increased depth accuracy near object boundaries and prediction of ultra high-resolution disparity maps on standard GPUs. We demonstrate the flexibility of our technique by improving the performance of a variety of stereo backbones. △ Less

Submitted 8 April, 2021; originally announced April 2021.

Comments: CVPR 2021. Project Page: https://github.com/fabiotosi92/SMD-Nets

On the confidence of stereo matching in a deep-learning era: a quantitative evaluation

Authors: Matteo Poggi, Seungryong Kim, Fabio Tosi, Sunok Kim, Filippo Aleotti, Dongbo Min, Kwanghoon Sohn, Stefano Mattoccia

Abstract: Stereo matching is one of the most popular techniques to estimate dense depth maps by finding the disparity between matching pixels on two, synchronized and rectified images. Alongside with the development of more accurate algorithms, the research community focused on finding good strategies to estimate the reliability, i.e. the confidence, of estimated disparity maps. This information proves to b… ▽ More Stereo matching is one of the most popular techniques to estimate dense depth maps by finding the disparity between matching pixels on two, synchronized and rectified images. Alongside with the development of more accurate algorithms, the research community focused on finding good strategies to estimate the reliability, i.e. the confidence, of estimated disparity maps. This information proves to be a powerful cue to naively find wrong matches as well as to improve the overall effectiveness of a variety of stereo algorithms according to different strategies. In this paper, we review more than ten years of developments in the field of confidence estimation for stereo matching. We extensively discuss and evaluate existing confidence measures and their variants, from hand-crafted ones to the most recent, state-of-the-art learning based methods. We study the different behaviors of each measure when applied to a pool of different stereo algorithms and, for the first time in literature, when paired with a state-of-the-art deep stereo network. Our experiments, carried out on five different standard datasets, provide a comprehensive overview of the field, highlighting in particular both strengths and limitations of learning-based strategies. △ Less

Submitted 30 March, 2021; v1 submitted 2 January, 2021; originally announced January 2021.

Comments: TPAMI final version

Correction of the VIR-visible data set from the Dawn mission at Vesta

Authors: B. Rousseau, M. C De Sanctis, A. Raponi, M. Ciarniello, E. Ammannito, P. Scarica, S. Fonte, A. Frigeri, F. G Carrozzo, F. Tosi

Abstract: This work describes the correction method applied to the dataset acquired at the asteroid (4) Vesta by the visible channel of the visible and infrared map** spectrometer. The rising detector temperature during data acquisitions in the visible wavelengths leads to a spectral slope increase over the whole spectral range. This limits the accuracy of the studies of the Vesta surface in this waveleng… ▽ More This work describes the correction method applied to the dataset acquired at the asteroid (4) Vesta by the visible channel of the visible and infrared map** spectrometer. The rising detector temperature during data acquisitions in the visible wavelengths leads to a spectral slope increase over the whole spectral range. This limits the accuracy of the studies of the Vesta surface in this wavelength range. Here, we detail an empirical method to correct for the visible detector temperature dependency while taking into account the specificity of the Vesta dataset. △ Less

Submitted 26 November, 2020; originally announced November 2020.

Comments: 6 pages, 5 figures and 1 table excluding supplementary material

Journal ref: Review of Scientific Instrument, 91, 2020

The surface of (1) Ceres in visible light as seen by Dawn/VIR

Authors: B. Rousseau, M. C. De Sanctis, A. Raponi, M. Ciarniello, E. Ammannito, A. Frigeri, M. Ferrari, S. De Angelis, F. Tosi, S. E. Schröder, C. A. Raymond, C. T. Russell

Abstract: We study the surface of Ceres at visible wavelengths, as observed by the Visible and InfraRed map** spectrometer (VIR) onboard the Dawn spacecraft, and analyze the variations of various spectral parameters across the whole surface. We also focus on several noteworthy areas of the surface of this dwarf planet. We made use of the newly corrected VIR visible data to build global maps of a calibrate… ▽ More We study the surface of Ceres at visible wavelengths, as observed by the Visible and InfraRed map** spectrometer (VIR) onboard the Dawn spacecraft, and analyze the variations of various spectral parameters across the whole surface. We also focus on several noteworthy areas of the surface of this dwarf planet. We made use of the newly corrected VIR visible data to build global maps of a calibrated radiance factor at 550 nm, with two color composites and three spectral slopes between 400 and 950 nm. We have made these maps available for the community via the Aladin Desktop software. Ceres surface shows diverse spectral behaviors in the visible range. The color composite and the spectral slope between 480 and 800 nm highlight fresh impact craters and young geologic formations of endogenous origin, which appear bluer than the rest of the surface. The steep slope before 465 nm displays very distinct variations and may be a proxy for the absorptions caused by the $O_2^{-}$ -> $Fe^{3+}$ or the $2Fe^{3+}$ -> $Fe^{2+} + Fe^{4+}$ charge transfer, if the latter are found to be responsible for the drop in this spectral range. We notice several similarities between the spectral slopes and the abundance of phyllosilicates detected in the infrared by the VIR, whereas no correlation can be clearly established with carbonate species. The region of the Dantu impact crater presents a peculiar spectral behavior (especially through the color and the spectral slope before 465 nm) suggesting a change in composition or in the surface physical properties that is not observed elsewhere on Ceres. △ Less

Submitted 7 October, 2020; originally announced October 2020.

Comments: 22 pages, 14 + 5 figures. Published online on Astronomy and Astrophysics on 07-10-2020

Journal ref: A&A 642, A74 (2020)

Infrared detection of aliphatic organics on a cometary nucleus

Authors: A. Raponi, M. Ciarniello, F. Capaccioni, V. Mennella, G. Filacchione, V. Vinogradoff, O. Poch, P. Beck, E. Quirico, M. C. De Sanctis, L. Moroz, D. Kappel, S. Erard, D. Bockelée-Morvan, A. Longobardo, F. Tosi, E. Palomba, J. -P. Combe, B. Rousseau, G. Arnold, R. W. Carlson, A. Pommerol, C. Pilorget, S. Fornasier, G. Bellucci , et al. (6 additional authors not shown)

Abstract: The ESA Rosetta mission has acquired unprecedented measurements of comet 67/P-Churyumov-Gerasimenko (hereafter 67P) nucleus surface, whose composition, as determined by in situ and remote sensing instruments including VIRTIS (Visible, InfraRed and Thermal Imaging Spectrometer) appears to be made by an assemblage of ices, minerals, and organic material. We performed a refined analysis of infrared o… ▽ More The ESA Rosetta mission has acquired unprecedented measurements of comet 67/P-Churyumov-Gerasimenko (hereafter 67P) nucleus surface, whose composition, as determined by in situ and remote sensing instruments including VIRTIS (Visible, InfraRed and Thermal Imaging Spectrometer) appears to be made by an assemblage of ices, minerals, and organic material. We performed a refined analysis of infrared observations of the nucleus of comet 67P carried out by the VIRTIS-M hyperspectral imager. We found that the overall shape of the 67P infrared spectrum is similar to that of other carbon-rich outer solar system objects suggesting a possible genetic link with them. More importantly, we are also able to confirm the complex spectral structure of the wide 2.8-3.6 micron absorption feature populated by fainter bands. Among these, we unambiguously identified the presence of aliphatic organics by their ubiquitous 3.38, 3.42 and 3.47 micron bands. This novel infrared detection of aliphatic species on a cometary surface has strong implications for the evolutionary history of the primordial solar system and give evidence that comets provide an evolutionary link between interstellar material and solar system bodies. △ Less

Submitted 30 September, 2020; originally announced September 2020.

Reversing the cycle: self-supervised deep stereo through enhanced monocular distillation

Authors: Filippo Aleotti, Fabio Tosi, Li Zhang, Matteo Poggi, Stefano Mattoccia

Abstract: In many fields, self-supervised learning solutions are rapidly evolving and filling the gap with supervised approaches. This fact occurs for depth estimation based on either monocular or stereo, with the latter often providing a valid source of self-supervision for the former. In contrast, to soften typical stereo artefacts, we propose a novel self-supervised paradigm reversing the link between th… ▽ More In many fields, self-supervised learning solutions are rapidly evolving and filling the gap with supervised approaches. This fact occurs for depth estimation based on either monocular or stereo, with the latter often providing a valid source of self-supervision for the former. In contrast, to soften typical stereo artefacts, we propose a novel self-supervised paradigm reversing the link between the two. Purposely, in order to train deep stereo networks, we distill knowledge through a monocular completion network. This architecture exploits single-image clues and few sparse points, sourced by traditional stereo algorithms, to estimate dense yet accurate disparity maps by means of a consensus mechanism over multiple estimations. We thoroughly evaluate with popular stereo datasets the impact of different supervisory signals showing how stereo networks trained with our paradigm outperform existing self-supervised frameworks. Finally, our proposal achieves notable generalization capabilities dealing with domain shift issues. Code available at https://github.com/FilippoAleotti/Reversing △ Less

Submitted 17 August, 2020; originally announced August 2020.

Comments: ECCV 2020

Self-adapting confidence estimation for stereo

Authors: Matteo Poggi, Filippo Aleotti, Fabio Tosi, Giulio Zaccaroni, Stefano Mattoccia

Abstract: Estimating the confidence of disparity maps inferred by a stereo algorithm has become a very relevant task in the years, due to the increasing number of applications leveraging such cue. Although self-supervised learning has recently spread across many computer vision tasks, it has been barely considered in the field of confidence estimation. In this paper, we propose a flexible and lightweight so… ▽ More Estimating the confidence of disparity maps inferred by a stereo algorithm has become a very relevant task in the years, due to the increasing number of applications leveraging such cue. Although self-supervised learning has recently spread across many computer vision tasks, it has been barely considered in the field of confidence estimation. In this paper, we propose a flexible and lightweight solution enabling self-adapting confidence estimation agnostic to the stereo algorithm or network. Our approach relies on the minimum information available in any stereo setup (i.e., the input stereo pair and the output disparity map) to learn an effective confidence measure. This strategy allows us not only a seamless integration with any stereo system, including consumer and industrial devices equipped with undisclosed stereo perception methods, but also, due to its self-adapting capability, for its out-of-the-box deployment in the field. Exhaustive experimental results with different standard datasets support our claims, showing how our solution is the first-ever enabling online learning of accurate confidence estimation for any stereo system and without any requirement for the end-user. △ Less

Submitted 24 November, 2020; v1 submitted 14 August, 2020; originally announced August 2020.

Comments: ECCV 2020 (errata corrige: eq.6, k domain)

Continual Adaptation for Deep Stereo

Authors: Matteo Poggi, Alessio Tonioni, Fabio Tosi, Stefano Mattoccia, Luigi Di Stefano

Abstract: Depth estimation from stereo images is carried out with unmatched results by convolutional neural networks trained end-to-end to regress dense disparities. Like for most tasks, this is possible if large amounts of labelled samples are available for training, possibly covering the whole data distribution encountered at deployment time. Being such an assumption systematically unmet in real applicati… ▽ More Depth estimation from stereo images is carried out with unmatched results by convolutional neural networks trained end-to-end to regress dense disparities. Like for most tasks, this is possible if large amounts of labelled samples are available for training, possibly covering the whole data distribution encountered at deployment time. Being such an assumption systematically unmet in real applications, the capacity of adapting to any unseen setting becomes of paramount importance. Purposely, we propose a continual adaptation paradigm for deep stereo networks designed to deal with challenging and ever-changing environments. We design a lightweight and modular architecture, Modularly ADaptive Network (MADNet), and formulate Modular ADaptation algorithms (MAD, MAD++) which permit efficient optimization of independent sub-portions of the entire network. In our paradigm, the learning signals needed to continuously adapt models online can be sourced from self-supervision via right-to-left image war** or from traditional stereo algorithms. With both sources, no other data than the input images being gathered at deployment time are needed. Thus, our network architecture and adaptation algorithms realize the first real-time self-adaptive deep stereo system and pave the way for a new paradigm that can facilitate practical deployment of end-to-end architectures for dense disparity regression. △ Less

Submitted 3 May, 2021; v1 submitted 10 July, 2020; originally announced July 2020.

Comments: Extended version of CVPR 2019 paper "Real-time self-adaptive deep stereo" - Accepted to TPAMI

Real-time single image depth perception in the wild with handheld devices

Authors: Filippo Aleotti, Giulio Zaccaroni, Luca Bartolomei, Matteo Poggi, Fabio Tosi, Stefano Mattoccia

Abstract: Depth perception is paramount to tackle real-world problems, ranging from autonomous driving to consumer applications. For the latter, depth estimation from a single image represents the most versatile solution, since a standard camera is available on almost any handheld device. Nonetheless, two main issues limit its practical deployment: i) the low reliability when deployed in-the-wild and ii) th… ▽ More Depth perception is paramount to tackle real-world problems, ranging from autonomous driving to consumer applications. For the latter, depth estimation from a single image represents the most versatile solution, since a standard camera is available on almost any handheld device. Nonetheless, two main issues limit its practical deployment: i) the low reliability when deployed in-the-wild and ii) the demanding resource requirements to achieve real-time performance, often not compatible with such devices. Therefore, in this paper, we deeply investigate these issues showing how they are both addressable adopting appropriate network design and training strategies -- also outlining how to map the resulting networks on handheld devices to achieve real-time performance. Our thorough evaluation highlights the ability of such fast networks to generalize well to new environments, a crucial feature required to tackle the extremely varied contexts faced in real applications. Indeed, to further support this evidence, we report experimental results concerning real-time depth-aware augmented reality and image blurring with smartphones in-the-wild. △ Less

Submitted 10 June, 2020; originally announced June 2020.

Comments: 11 pages, 9 figures

On the uncertainty of self-supervised monocular depth estimation

Authors: Matteo Poggi, Filippo Aleotti, Fabio Tosi, Stefano Mattoccia

Abstract: Self-supervised paradigms for monocular depth estimation are very appealing since they do not require ground truth annotations at all. Despite the astonishing results yielded by such methodologies, learning to reason about the uncertainty of the estimated depth maps is of paramount importance for practical applications, yet uncharted in the literature. Purposely, we explore for the first time how… ▽ More Self-supervised paradigms for monocular depth estimation are very appealing since they do not require ground truth annotations at all. Despite the astonishing results yielded by such methodologies, learning to reason about the uncertainty of the estimated depth maps is of paramount importance for practical applications, yet uncharted in the literature. Purposely, we explore for the first time how to estimate the uncertainty for this task and how this affects depth accuracy, proposing a novel peculiar technique specifically designed for self-supervised approaches. On the standard KITTI dataset, we exhaustively assess the performance of each method with different self-supervised paradigms. Such evaluation highlights that our proposal i) always improves depth accuracy significantly and ii) yields state-of-the-art results concerning uncertainty estimation when training on sequences and competitive results uniquely deploying stereo pairs. △ Less

Submitted 13 May, 2020; originally announced May 2020.

Comments: CVPR 2020. Code will be available https://github.com/mattpoggi/mono-uncertainty

On the Synergies between Machine Learning and Binocular Stereo for Depth Estimation from Images: a Survey

Authors: Matteo Poggi, Fabio Tosi, Konstantinos Batsos, Philippos Mordohai, Stefano Mattoccia

Abstract: Stereo matching is one of the longest-standing problems in computer vision with close to 40 years of studies and research. Throughout the years the paradigm has shifted from local, pixel-level decision to various forms of discrete and continuous optimization to data-driven, learning-based methods. Recently, the rise of machine learning and the rapid proliferation of deep learning enhanced stereo m… ▽ More Stereo matching is one of the longest-standing problems in computer vision with close to 40 years of studies and research. Throughout the years the paradigm has shifted from local, pixel-level decision to various forms of discrete and continuous optimization to data-driven, learning-based methods. Recently, the rise of machine learning and the rapid proliferation of deep learning enhanced stereo matching with new exciting trends and applications unthinkable until a few years ago. Interestingly, the relationship between these two worlds is two-way. While machine, and especially deep, learning advanced the state-of-the-art in stereo matching, stereo itself enabled new ground-breaking methodologies such as self-supervised monocular depth estimation based on deep networks. In this paper, we review recent research in the field of learning-based depth estimation from single and binocular images highlighting the synergies, the successes achieved so far and the open challenges the community is going to face in the immediate future. △ Less

Submitted 31 March, 2021; v1 submitted 18 April, 2020; originally announced April 2020.

Comments: Accepted to TPAMI. Paper version of our CVPR 2019 tutorial: "Learning-based depth estimation from stereo and monocular images: successes, limitations and future challenges" (https://sites.google.com/view/cvpr-2019-depth-from-image/home)

Distilled Semantics for Comprehensive Scene Understanding from Videos

Authors: Fabio Tosi, Filippo Aleotti, Pierluigi Zama Ramirez, Matteo Poggi, Samuele Salti, Luigi Di Stefano, Stefano Mattoccia

Abstract: Whole understanding of the surroundings is paramount to autonomous systems. Recent works have shown that deep neural networks can learn geometry (depth) and motion (optical flow) from a monocular video without any explicit supervision from ground truth annotations, particularly hard to source for these two tasks. In this paper, we take an additional step toward holistic scene understanding with mo… ▽ More Whole understanding of the surroundings is paramount to autonomous systems. Recent works have shown that deep neural networks can learn geometry (depth) and motion (optical flow) from a monocular video without any explicit supervision from ground truth annotations, particularly hard to source for these two tasks. In this paper, we take an additional step toward holistic scene understanding with monocular cameras by learning depth and motion alongside with semantics, with supervision for the latter provided by a pre-trained network distilling proxy ground truth images. We address the three tasks jointly by a) a novel training protocol based on knowledge distillation and self-supervision and b) a compact network architecture which enables efficient scene understanding on both power hungry GPUs and low-power embedded platforms. We thoroughly assess the performance of our framework and show that it yields state-of-the-art results for monocular depth estimation, optical flow and motion segmentation. △ Less

Submitted 31 March, 2020; originally announced March 2020.

Comments: CVPR 2020. Code will be available at https://github.com/CVLAB-Unibo/omeganet

Ammonium salts are a reservoir of nitrogen on a cometary nucleus and possibly on some asteroids

Authors: O. Poch, I. Istiqomah, E. Quirico, P. Beck, B. Schmitt, P. Theulé, A. Faure, P. Hily-Blant, L. Bonal, A. Raponi, M. Ciarniello, B. Rousseau, S. Potin, O. Brissaud, L. Flandinet, G. Filacchione, A. Pommerol, N. Thomas, D. Kappel, V. Mennella, L. Moroz, V. Vinogradoff, G. Arnold, S. Erard, D. Bockelée-Morvan , et al. (7 additional authors not shown)

Abstract: The measured nitrogen-to-carbon ratio in comets is lower than for the Sun, a discrepancy which could be alleviated if there is an unknown reservoir of nitrogen in comets. The nucleus of comet 67P/Churyumov-Gerasimenko exhibits an unidentified broad spectral reflectance feature around 3.2 micrometers, which is ubiquitous across its surface. On the basis of laboratory experiments, we attribute this… ▽ More The measured nitrogen-to-carbon ratio in comets is lower than for the Sun, a discrepancy which could be alleviated if there is an unknown reservoir of nitrogen in comets. The nucleus of comet 67P/Churyumov-Gerasimenko exhibits an unidentified broad spectral reflectance feature around 3.2 micrometers, which is ubiquitous across its surface. On the basis of laboratory experiments, we attribute this absorption band to ammonium salts mixed with dust on the surface. The depth of the band indicates that semivolatile ammonium salts are a substantial reservoir of nitrogen in the comet, potentially dominating over refractory organic matter and more volatile species. Similar absorption features appear in the spectra of some asteroids, implying a compositional link between asteroids, comets, and the parent interstellar cloud. △ Less

Submitted 12 March, 2020; originally announced March 2020.

Comments: Main manuscript and Supplementary material document, Accepted for publication in Science on February 14, 2020

Journal ref: Science 367, (2020) eaaw7462

Correction of the VIR-Visible data set from the Dawn mission

Authors: B. Rousseau, A. Raponi, M. Ciarniello, E. Ammannito, F. G. Carrozzo, M. C. De Sanctis, S. Fonte, A. Frigeri, F. Tosi

Abstract: Data acquired at Ceres by the visible channel of the Visible and InfraRed map** spectrometer (VIR) on board the NASA Dawn spacecraft are affected by the temperatures of both the visible (VIS) and the infrared (IR) sensors, which are respectively a CCD and a HgCdTe array. The variations of the visible channel temperatures measured during the sessions of acquisitions are correlated with variations… ▽ More Data acquired at Ceres by the visible channel of the Visible and InfraRed map** spectrometer (VIR) on board the NASA Dawn spacecraft are affected by the temperatures of both the visible (VIS) and the infrared (IR) sensors, which are respectively a CCD and a HgCdTe array. The variations of the visible channel temperatures measured during the sessions of acquisitions are correlated with variations in the spectral slope and shape for all the mission phases. The infrared channel (IR) temperature is more stable during the acquisitions, nonetheless it is characterized by a bi-modal distribution whether the cryocooler (and therefore the IR channel) is used or not during the visible channel operations. When the infrared channel temperature is high (175K, i.e. not in use and with crycooler off), an additional negative slope and a distortion are observed in the spectra of the visible channel. We developed an empirical correction based on a reference spectrum for the whole data set; it is designed to correct the two issues related to the sensor temperatures that we have identified. The reference spectrum is calculated to be representative of the global Ceres' surface. It is also made of data acquired when the visible and infrared channel temperatures are equal to the ones measured during an observation of the Arcturus star by VIR, which is consistent with several ground-based observations. The developed correction allows reliable analysis and map** to be performed by minimizing the artifacts induced by fluctuations of the VIS temperature. Thanks to this correction, a direct comparison between different mission phases during which VIR experienced different visible and infrared channel temperatures is now possible. △ Less

Submitted 19 December, 2019; originally announced December 2019.

Comments: The following article has been accepted by Review of Scientific Instruments on 24 November 2019. After it is published, it will be found at this link: https://publishing.aip.org/resources/librarians/products/journals/

Journal ref: Review of Scientific Instruments 2019 90:12

Learning End-To-End Scene Flow by Distilling Single Tasks Knowledge

Authors: Filippo Aleotti, Matteo Poggi, Fabio Tosi, Stefano Mattoccia

Abstract: Scene flow is a challenging task aimed at jointly estimating the 3D structure and motion of the sensed environment. Although deep learning solutions achieve outstanding performance in terms of accuracy, these approaches divide the whole problem into standalone tasks (stereo and optical flow) addressing them with independent networks. Such a strategy dramatically increases the complexity of the tra… ▽ More Scene flow is a challenging task aimed at jointly estimating the 3D structure and motion of the sensed environment. Although deep learning solutions achieve outstanding performance in terms of accuracy, these approaches divide the whole problem into standalone tasks (stereo and optical flow) addressing them with independent networks. Such a strategy dramatically increases the complexity of the training procedure and requires power-hungry GPUs to infer scene flow barely at 1 FPS. Conversely, we propose DWARF, a novel and lightweight architecture able to infer full scene flow jointly reasoning about depth and optical flow easily and elegantly trainable end-to-end from scratch. Moreover, since ground truth images for full scene flow are scarce, we propose to leverage on the knowledge learned by networks specialized in stereo or flow, for which much more data are available, to distill proxy annotations. Exhaustive experiments show that i) DWARF runs at about 10 FPS on a single high-end GPU and about 1 FPS on NVIDIA Jetson TX2 embedded at KITTI resolution, with moderate drop in accuracy compared to 10x deeper models, ii) learning from many distilled samples is more effective than from the few, annotated ones available. Code available at: https://github.com/FilippoAleotti/Dwarf-Tensorflow △ Less

Submitted 22 November, 2019; originally announced November 2019.

Comments: Accepted to AAAI 2020. Project page: https://vision.disi.unibo.it/~faleotti/dwarf.html

GAUSS -- A Sample Return Mission to Ceres

Authors: Xian Shi, Julie Castillo-Rogez, Henry Hsieh, Hejiu Hui, Wing-Huen Ip, Hanlun Lei, Jian-Yang Li, Federico Tosi, Liyong Zhou, Jessica Agarwal, Antonella Barucci, Pierre Beck, Adriano Campo Bagatin, Fabrizio Capaccioni, Andrew Coates, Gabriele Cremonese, Rene Duffard, Ralf Jaumann, Geraint Jones, Manuel Grande, Esa Kallio, Yangting Lin, Olivier Mousis, Andreas Nathues, Jürgen Oberst , et al. (4 additional authors not shown)

Abstract: The goal of Project GAUSS is to return samples from the dwarf planet Ceres. Ceres is the most accessible ocean world candidate and the largest reservoir of water in the inner solar system. It shows active cryovolcanism and hydrothermal activities in recent history that resulted in minerals not found in any other planets to date except for Earth's upper crust. The possible occurrence of recent subs… ▽ More The goal of Project GAUSS is to return samples from the dwarf planet Ceres. Ceres is the most accessible ocean world candidate and the largest reservoir of water in the inner solar system. It shows active cryovolcanism and hydrothermal activities in recent history that resulted in minerals not found in any other planets to date except for Earth's upper crust. The possible occurrence of recent subsurface ocean on Ceres and the complex geochemistry suggest possible past habitability and even the potential for ongoing habitability. Aiming to answer a broad spectrum of questions about the origin and evolution of Ceres and its potential habitability, GAUSS will return samples from this possible ocean world for the first time. The project will address the following top-level scientific questions: 1) What is the origin of Ceres and the origin and transfer of water and other volatiles in the inner solar system? 2) What are the physical properties and internal structure of Ceres? What do they tell us about the evolutionary and aqueous alteration history of icy dwarf planets? 3) What are the astrobiological implications of Ceres? Was it habitable in the past and is it still today? 4) What are the mineralogical connections between Ceres and our current collections of primitive meteorites? GAUSS will first perform a high-resolution global remote sensing investigation, characterizing the geophysical and geochemical properties of Ceres. Candidate sampling sites will then be identified, and observation campaigns will be run for an in-depth assessment of the candidate sites. Once the sampling site is selected, a lander will be deployed on the surface to collect samples and return them to Earth in cryogenic conditions that preserves the volatile and organic composition as well as the original physical status as much as possible. △ Less

Submitted 19 March, 2020; v1 submitted 21 August, 2019; originally announced August 2019.

Comments: Section 3.4 revised; List of team members updated; Typos corrected

Ice Giant Systems: The Scientific Potential of Orbital Missions to Uranus and Neptune

Authors: Leigh N. Fletcher, Ravit Helled, Elias Roussos, Geraint Jones, Sébastien Charnoz, Nicolas André, David Andrews, Michele Bannister, Emma Bunce, Thibault Cavalié, Francesca Ferri, Jonathan Fortney, Davide Grassi, Léa Griton, Paul Hartogh, Ricardo Hueso, Yohai Kaspi, Laurent Lamy, Adam Masters, Henrik Melin, Julianne Moses, Olivier Mousis, Nadine Nettleman, Christina Plainaki, Jürgen Schmidt , et al. (5 additional authors not shown)

Abstract: Uranus and Neptune, and their diverse satellite and ring systems, represent the least explored environments of our Solar System, and yet may provide the archetype for the most common outcome of planetary formation throughout our galaxy. Ice Giants will be the last remaining class of Solar System planet to have a dedicated orbital explorer, and international efforts are under way to realise such an… ▽ More Uranus and Neptune, and their diverse satellite and ring systems, represent the least explored environments of our Solar System, and yet may provide the archetype for the most common outcome of planetary formation throughout our galaxy. Ice Giants will be the last remaining class of Solar System planet to have a dedicated orbital explorer, and international efforts are under way to realise such an ambitious mission in the coming decades. In 2019, the European Space Agency released a call for scientific themes for its strategic science planning process for the 2030s and 2040s, known as Voyage 2050. We used this opportunity to review our present-day knowledge of the Uranus and Neptune systems, producing a revised and updated set of scientific questions and motivations for their exploration. This review article describes how such a mission could explore their origins, ice-rich interiors, dynamic atmospheres, unique magnetospheres, and myriad icy satellites, to address questions at the heart of modern planetary science. These two worlds are superb examples of how planets with shared origins can exhibit remarkably different evolutionary paths: Neptune as the archetype for Ice Giants, whereas Uranus may be atypical. Exploring Uranus' natural satellites and Neptune's captured moon Triton could reveal how Ocean Worlds form and remain active, redefining the extent of the habitable zone in our Solar System. For these reasons and more, we advocate that an Ice Giant System explorer should become a strategic cornerstone mission within ESA's Voyage 2050 programme, in partnership with international collaborators, and targeting launch opportunities in the early 2030s. △ Less

Submitted 11 June, 2020; v1 submitted 4 July, 2019; originally announced July 2019.

Comments: 34 pages, 9 figures, accepted to Planetary and Space Science

Guided Stereo Matching

Authors: Matteo Poggi, Davide Pallotti, Fabio Tosi, Stefano Mattoccia

Abstract: Stereo is a prominent technique to infer dense depth maps from images, and deep learning further pushed forward the state-of-the-art, making end-to-end architectures unrivaled when enough data is available for training. However, deep networks suffer from significant drops in accuracy when dealing with new environments. Therefore, in this paper, we introduce Guided Stereo Matching, a novel paradigm… ▽ More Stereo is a prominent technique to infer dense depth maps from images, and deep learning further pushed forward the state-of-the-art, making end-to-end architectures unrivaled when enough data is available for training. However, deep networks suffer from significant drops in accuracy when dealing with new environments. Therefore, in this paper, we introduce Guided Stereo Matching, a novel paradigm leveraging a small amount of sparse, yet reliable depth measurements retrieved from an external source enabling to ameliorate this weakness. The additional sparse cues required by our method can be obtained with any strategy (e.g., a LiDAR) and used to enhance features linked to corresponding disparity hypotheses. Our formulation is general and fully differentiable, thus enabling to exploit the additional sparse inputs in pre-trained deep stereo networks as well as for training a new instance from scratch. Extensive experiments on three standard datasets and two state-of-the-art deep architectures show that even with a small set of sparse input cues, i) the proposed paradigm enables significant improvements to pre-trained networks. Moreover, ii) training from scratch notably increases accuracy and robustness to domain shifts. Finally, iii) it is suited and effective even with traditional stereo algorithms such as SGM. △ Less

Submitted 24 May, 2019; originally announced May 2019.

Comments: CVPR 2019

The thermal, mechanical, structural, and dielectric properties of cometary nuclei after Rosetta

Authors: O. Groussin, N. Attree, Y. Brouet, V. Ciarletti, B. Davidsson, G. Filacchione, H. H. Fischer, B. Gundlach, M. Knapmeyer, J. Knollenberg, R. Kokotanekova, E. Kührt, C. Leyrat, D. Marshall, I. Pelivan, Y. Skorov, C. Snodgrass, T. Spohn, F. Tosi

Abstract: The physical properties of cometary nuclei observed today relate to their complex history and help to constrain their formation and evolution. In this article, we review some of the main physical properties of cometary nuclei and focus in particular on the thermal, mechanical, structural and dielectric properties, emphasizing the progress made during the Rosetta mission. Comets have a low density… ▽ More The physical properties of cometary nuclei observed today relate to their complex history and help to constrain their formation and evolution. In this article, we review some of the main physical properties of cometary nuclei and focus in particular on the thermal, mechanical, structural and dielectric properties, emphasizing the progress made during the Rosetta mission. Comets have a low density of 480 $\pm$ 220 kg m-3 and a low permittivity of 1.9 - 2.0, consistent with a high porosity of 70 - 80 %, are weak with a very low global tensile strength $<$100 Pa, and have a low bulk thermal inertia of 0 - 60 J K-1 m-2 s-1/2 that allowed them to preserve highly volatiles species (e.g. CO, CO2, CH4, N2) into their interior since their formation. As revealed by 67P/Churyumov-Gerasimenko, the above physical properties vary across the nucleus, spatially at its surface but also with depth. The broad picture is that the bulk of the nucleus consists of a weakly bonded, rather homogeneous material that preserved primordial properties under a thin shell of processed material, and possibly covered by a granular material; this cover might in places reach a thickness of several meters. The properties of the top layer (the first meter) are not representative of that of the bulk nucleus. More globally, strong nucleus heterogeneities at a scale of a few meters are ruled out on 67P small lobe. △ Less

Submitted 3 May, 2019; originally announced May 2019.

Comments: 66 pages, 13 figures, published in Space Science Reviews

Journal ref: Space Science Reviews 215 (2019)

Learning monocular depth estimation infusing traditional stereo knowledge

Authors: Fabio Tosi, Filippo Aleotti, Matteo Poggi, Stefano Mattoccia

Abstract: Depth estimation from a single image represents a fascinating, yet challenging problem with countless applications. Recent works proved that this task could be learned without direct supervision from ground truth labels leveraging image synthesis on sequences or stereo pairs. Focusing on this second case, in this paper we leverage stereo matching in order to improve monocular depth estimation. To… ▽ More Depth estimation from a single image represents a fascinating, yet challenging problem with countless applications. Recent works proved that this task could be learned without direct supervision from ground truth labels leveraging image synthesis on sequences or stereo pairs. Focusing on this second case, in this paper we leverage stereo matching in order to improve monocular depth estimation. To this aim we propose monoResMatch, a novel deep architecture designed to infer depth from a single input image by synthesizing features from a different point of view, horizontally aligned with the input image, performing stereo matching between the two cues. In contrast to previous works sharing this rationale, our network is the first trained end-to-end from scratch. Moreover, we show how obtaining proxy ground truth annotation through traditional stereo algorithms, such as Semi-Global Matching, enables more accurate monocular depth estimation still countering the need for expensive depth labels by kee** a self-supervised approach. Exhaustive experimental results prove how the synergy between i) the proposed monoResMatch architecture and ii) proxy-supervision attains state-of-the-art for self-supervised monocular depth estimation. The code is publicly available at https://github.com/fabiotosi92/monoResMatch-Tensorflow. △ Less

Submitted 8 April, 2019; originally announced April 2019.

Comments: accepted at CVPR 2019. Code available at https://github.com/fabiotosi92/monoResMatch-Tensorflow

Real-time self-adaptive deep stereo

Authors: Alessio Tonioni, Fabio Tosi, Matteo Poggi, Stefano Mattoccia, Luigi Di Stefano

Abstract: Deep convolutional neural networks trained end-to-end are the state-of-the-art methods to regress dense disparity maps from stereo pairs. These models, however, suffer from a notable decrease in accuracy when exposed to scenarios significantly different from the training set, e.g., real vs synthetic images, etc.). We argue that it is extremely unlikely to gather enough samples to achieve effective… ▽ More Deep convolutional neural networks trained end-to-end are the state-of-the-art methods to regress dense disparity maps from stereo pairs. These models, however, suffer from a notable decrease in accuracy when exposed to scenarios significantly different from the training set, e.g., real vs synthetic images, etc.). We argue that it is extremely unlikely to gather enough samples to achieve effective training/tuning in any target domain, thus making this setup impractical for many applications. Instead, we propose to perform unsupervised and continuous online adaptation of a deep stereo network, which allows for preserving its accuracy in any environment. However, this strategy is extremely computationally demanding and thus prevents real-time inference. We address this issue introducing a new lightweight, yet effective, deep stereo architecture, Modularly ADaptive Network (MADNet) and develo** a Modular ADaptation (MAD) algorithm, which independently trains sub-portions of the network. By deploying MADNet together with MAD we introduce the first real-time self-adaptive deep stereo system enabling competitive performance on heterogeneous datasets. △ Less

Submitted 5 April, 2019; v1 submitted 12 October, 2018; originally announced October 2018.

Comments: Accepted at CVPR2019 as oral presentation. Code Available https://github.com/CVLAB-Unibo/Real-time-self-adaptive-deep-stereo

Geometry meets semantics for semi-supervised monocular depth estimation

Authors: Pierluigi Zama Ramirez, Matteo Poggi, Fabio Tosi, Stefano Mattoccia, Luigi Di Stefano

Abstract: Depth estimation from a single image represents a very exciting challenge in computer vision. While other image-based depth sensing techniques leverage on the geometry between different viewpoints (e.g., stereo or structure from motion), the lack of these cues within a single image renders ill-posed the monocular depth estimation task. For inference, state-of-the-art encoder-decoder architectures… ▽ More Depth estimation from a single image represents a very exciting challenge in computer vision. While other image-based depth sensing techniques leverage on the geometry between different viewpoints (e.g., stereo or structure from motion), the lack of these cues within a single image renders ill-posed the monocular depth estimation task. For inference, state-of-the-art encoder-decoder architectures for monocular depth estimation rely on effective feature representations learned at training time. For unsupervised training of these models, geometry has been effectively exploited by suitable images war** losses computed from views acquired by a stereo rig or a moving camera. In this paper, we make a further step forward showing that learning semantic information from images enables to improve effectively monocular depth estimation as well. In particular, by leveraging on semantically labeled images together with unsupervised signals gained by geometry through an image war** loss, we propose a deep learning approach aimed at joint semantic segmentation and depth estimation. Our overall learning framework is semi-supervised, as we deploy groundtruth data only in the semantic domain. At training time, our network learns a common feature representation for both tasks and a novel cross-task loss function is proposed. The experimental findings show how, jointly tackling depth prediction and semantic segmentation, allows to improve depth estimation accuracy. In particular, on the KITTI dataset our network outperforms state-of-the-art methods for monocular depth estimation. △ Less

Submitted 26 October, 2018; v1 submitted 9 October, 2018; originally announced October 2018.

Comments: 16 pages, Accepted to ACCV 2018

Learning monocular depth estimation with unsupervised trinocular assumptions

Authors: Matteo Poggi, Fabio Tosi, Stefano Mattoccia

Abstract: Obtaining accurate depth measurements out of a single image represents a fascinating solution to 3D sensing. CNNs led to considerable improvements in this field, and recent trends replaced the need for ground-truth labels with geometry-guided image reconstruction signals enabling unsupervised training. Currently, for this purpose, state-of-the-art techniques rely on images acquired with a binocula… ▽ More Obtaining accurate depth measurements out of a single image represents a fascinating solution to 3D sensing. CNNs led to considerable improvements in this field, and recent trends replaced the need for ground-truth labels with geometry-guided image reconstruction signals enabling unsupervised training. Currently, for this purpose, state-of-the-art techniques rely on images acquired with a binocular stereo rig to predict inverse depth (i.e., disparity) according to the aforementioned supervision principle. However, these methods suffer from well-known problems near occlusions, left image border, etc inherited from the stereo setup. Therefore, in this paper, we tackle these issues by moving to a trinocular domain for training. Assuming the central image as the reference, we train a CNN to infer disparity representations pairing such image with frames on its left and right side. This strategy allows obtaining depth maps not affected by typical stereo artifacts. Moreover, being trinocular datasets seldom available, we introduce a novel interleaved training procedure enabling to enforce the trinocular assumption outlined from current binocular datasets. Exhaustive experimental results on the KITTI dataset confirm that our proposal outperforms state-of-the-art methods for unsupervised monocular depth estimation trained on binocular stereo pairs as well as any known methods relying on other cues. △ Less

Submitted 5 August, 2018; originally announced August 2018.

Comments: 14 pages, 7 figures, 4 tables. Accepted to 3DV 2018

Towards real-time unsupervised monocular depth estimation on CPU

Authors: Matteo Poggi, Filippo Aleotti, Fabio Tosi, Stefano Mattoccia

Abstract: Unsupervised depth estimation from a single image is a very attractive technique with several implications in robotic, autonomous navigation, augmented reality and so on. This topic represents a very challenging task and the advent of deep learning enabled to tackle this problem with excellent results. However, these architectures are extremely deep and complex. Thus, real-time performance can be… ▽ More Unsupervised depth estimation from a single image is a very attractive technique with several implications in robotic, autonomous navigation, augmented reality and so on. This topic represents a very challenging task and the advent of deep learning enabled to tackle this problem with excellent results. However, these architectures are extremely deep and complex. Thus, real-time performance can be achieved only by leveraging power-hungry GPUs that do not allow to infer depth maps in application fields characterized by low-power constraints. To tackle this issue, in this paper we propose a novel architecture capable to quickly infer an accurate depth map on a CPU, even of an embedded system, using a pyramid of features extracted from a single input image. Similarly to state-of-the-art, we train our network in an unsupervised manner casting depth estimation as an image reconstruction problem. Extensive experimental results on the KITTI dataset show that compared to the top performing approach our network has similar accuracy but a much lower complexity (about 6% of parameters) enabling to infer a depth map for a KITTI image in about 1.7 s on the Raspberry Pi 3 and at more than 8 Hz on a standard CPU. Moreover, by trading accuracy for efficiency, our network allows to infer maps at about 2 Hz and 40 Hz respectively, still being more accurate than most state-of-the-art slower methods. To the best of our knowledge, it is the first method enabling such performance on CPUs paving the way for effective deployment of unsupervised monocular depth estimation even on embedded systems. △ Less

Submitted 31 July, 2018; v1 submitted 29 June, 2018; originally announced June 2018.

Comments: 7 pages, 3 figures. Accepted to IROS 2018

Solar System Ice Giants: Exoplanets in our Backyard

Authors: Abigail Rymer, Kathleen Mandt, Dana Hurley, Carey Lisse, Noam Izenberg, H. Todd Smith, Joseph Westlake, Emma Bunce, Christopher Arridge, Adam Masters, Mark Hofstadter, Amy Simon, Pontus Brandt, George Clark, Ian Cohen, Robert Allen, Sarah Vine, Kenneth Hansen, George Hospodarsky, William Kurth, Paul Romani, Laurent Lamy, Philippe Zarka, Hao Cao, Carol Paty , et al. (88 additional authors not shown)

Abstract: Future remote sensing of exoplanets will be enhanced by a thorough investigation of our solar system Ice Giants (Neptune-size planets). What can the configuration of the magnetic field tell us (remotely) about the interior, and what implications does that field have for the structure of the magnetosphere; energy input into the atmosphere, and surface geophysics (for example surface weathering of s… ▽ More Future remote sensing of exoplanets will be enhanced by a thorough investigation of our solar system Ice Giants (Neptune-size planets). What can the configuration of the magnetic field tell us (remotely) about the interior, and what implications does that field have for the structure of the magnetosphere; energy input into the atmosphere, and surface geophysics (for example surface weathering of satellites that might harbour sub-surface oceans). How can monitoring of auroral emission help inform future remote observations of emission from exoplanets? Our Solar System provides the only laboratory in which we can perform in-situ experiments to understand exoplanet formation, dynamos, systems and magnetospheres. △ Less

Submitted 10 April, 2018; originally announced April 2018.

Comments: Exoplanet Science Strategy White Paper, submitted to the National Academies of Sciences, Engineering and Medicine, Space Studies Board, 9 March 2018

Journal ref: 2020n3i176