-
From-Ground-To-Objects: Coarse-to-Fine Self-supervised Monocular Depth Estimation of Dynamic Objects with Ground Contact Prior
Authors:
Jaeho Moon,
Juan Luis Gonzalez Bello,
Byeongjun Kwon,
Munchurl Kim
Abstract:
Self-supervised monocular depth estimation (DE) is an approach to learning depth without costly depth ground truths. However, it often struggles with moving objects that violate the static scene assumption during training. To address this issue, we introduce a coarse-to-fine training strategy leveraging the ground contacting prior based on the observation that most moving objects in outdoor scenes…
▽ More
Self-supervised monocular depth estimation (DE) is an approach to learning depth without costly depth ground truths. However, it often struggles with moving objects that violate the static scene assumption during training. To address this issue, we introduce a coarse-to-fine training strategy leveraging the ground contacting prior based on the observation that most moving objects in outdoor scenes contact the ground. In the coarse training stage, we exclude the objects in dynamic classes from the reprojection loss calculation to avoid inaccurate depth learning. To provide precise supervision on the depth of the objects, we present a novel Ground-contacting-prior Disparity Smoothness Loss (GDS-Loss) that encourages a DE network to align the depth of the objects with their ground-contacting points. Subsequently, in the fine training stage, we refine the DE network to learn the detailed depth of the objects from the reprojection loss, while ensuring accurate DE on the moving object regions by employing our regularization loss with a cost-volume-based weighting factor. Our overall coarse-to-fine training strategy can easily be integrated with existing DE methods without any modifications, significantly enhancing DE performance on challenging Cityscapes and KITTI datasets, especially in the moving object regions.
△ Less
Submitted 15 December, 2023;
originally announced December 2023.
-
ProNeRF: Learning Efficient Projection-Aware Ray Sampling for Fine-Grained Implicit Neural Radiance Fields
Authors:
Juan Luis Gonzalez Bello,
Minh-Quan Viet Bui,
Munchurl Kim
Abstract:
Recent advances in neural rendering have shown that, albeit slow, implicit compact models can learn a scene's geometries and view-dependent appearances from multiple views. To maintain such a small memory footprint but achieve faster inference times, recent works have adopted `sampler' networks that adaptively sample a small subset of points along each ray in the implicit neural radiance fields. A…
▽ More
Recent advances in neural rendering have shown that, albeit slow, implicit compact models can learn a scene's geometries and view-dependent appearances from multiple views. To maintain such a small memory footprint but achieve faster inference times, recent works have adopted `sampler' networks that adaptively sample a small subset of points along each ray in the implicit neural radiance fields. Although these methods achieve up to a 10$\times$ reduction in rendering time, they still suffer from considerable quality degradation compared to the vanilla NeRF. In contrast, we propose ProNeRF, which provides an optimal trade-off between memory footprint (similar to NeRF), speed (faster than HyperReel), and quality (better than K-Planes). ProNeRF is equipped with a novel projection-aware sampling (PAS) network together with a new training strategy for ray exploration and exploitation, allowing for efficient fine-grained particle sampling. Our ProNeRF yields state-of-the-art metrics, being 15-23x faster with 0.65dB higher PSNR than NeRF and yielding 0.95dB higher PSNR than the best published sampler-based method, HyperReel. Our exploration and exploitation training strategy allows ProNeRF to learn the full scenes' color and density distributions while also learning efficient ray sampling focused on the highest-density regions. We provide extensive experimental results that support the effectiveness of our method on the widely adopted forward-facing and 360 datasets, LLFF and Blender, respectively.
△ Less
Submitted 13 December, 2023;
originally announced December 2023.
-
Novel View Synthesis with View-Dependent Effects from a Single Image
Authors:
Juan Luis Gonzalez Bello,
Munchurl Kim
Abstract:
In this paper, we firstly consider view-dependent effects into single image-based novel view synthesis (NVS) problems. For this, we propose to exploit the camera motion priors in NVS to model view-dependent appearance or effects (VDE) as the negative disparity in the scene. By recognizing specularities "follow" the camera motion, we infuse VDEs into the input images by aggregating input pixel colo…
▽ More
In this paper, we firstly consider view-dependent effects into single image-based novel view synthesis (NVS) problems. For this, we propose to exploit the camera motion priors in NVS to model view-dependent appearance or effects (VDE) as the negative disparity in the scene. By recognizing specularities "follow" the camera motion, we infuse VDEs into the input images by aggregating input pixel colors along the negative depth region of the epipolar lines. Also, we propose a `relaxed volumetric rendering' approximation that allows computing the densities in a single pass, improving efficiency for NVS from single images. Our method can learn single-image NVS from image sequences only, which is a completely self-supervised learning method, for the first time requiring neither depth nor camera pose annotations. We present extensive experiment results and show that our proposed method can learn NVS with VDEs, outperforming the SOTA single-view NVS methods on the RealEstate10k and MannequinChallenge datasets.
△ Less
Submitted 13 December, 2023;
originally announced December 2023.
-
Segmentation of glioblastomas in early post-operative multi-modal MRI with deep neural networks
Authors:
Ragnhild Holden Helland,
Alexandros Ferles,
André Pedersen,
Ivar Kommers,
Hilko Ardon,
Frederik Barkhof,
Lorenzo Bello,
Mitchel S. Berger,
Tora Dunås,
Marco Conti Nibali,
Julia Furtner,
Shawn Hervey-Jumper,
Albert J. S. Idema,
Barbara Kiesel,
Rishi Nandoe Tewari,
Emmanuel Mandonnet,
Domenique M. J. Müller,
Pierre A. Robe,
Marco Rossi,
Lisa M. Sagberg,
Tommaso Sciortino,
Tom Aalders,
Michiel Wagemakers,
Georg Widhalm,
Marnix G. Witte
, et al. (8 additional authors not shown)
Abstract:
Extent of resection after surgery is one of the main prognostic factors for patients diagnosed with glioblastoma. To achieve this, accurate segmentation and classification of residual tumor from post-operative MR images is essential. The current standard method for estimating it is subject to high inter- and intra-rater variability, and an automated method for segmentation of residual tumor in ear…
▽ More
Extent of resection after surgery is one of the main prognostic factors for patients diagnosed with glioblastoma. To achieve this, accurate segmentation and classification of residual tumor from post-operative MR images is essential. The current standard method for estimating it is subject to high inter- and intra-rater variability, and an automated method for segmentation of residual tumor in early post-operative MRI could lead to a more accurate estimation of extent of resection. In this study, two state-of-the-art neural network architectures for pre-operative segmentation were trained for the task. The models were extensively validated on a multicenter dataset with nearly 1000 patients, from 12 hospitals in Europe and the United States. The best performance achieved was a 61\% Dice score, and the best classification performance was about 80\% balanced accuracy, with a demonstrated ability to generalize across hospitals. In addition, the segmentation performance of the best models was on par with human expert raters. The predicted segmentations can be used to accurately classify the patients into those with residual tumor, and those with gross total resection.
△ Less
Submitted 18 April, 2023;
originally announced April 2023.
-
Positional Information is All You Need: A Novel Pipeline for Self-Supervised SVDE from Videos
Authors:
Juan Luis Gonzalez Bello,
Jaeho Moon,
Munchurl Kim
Abstract:
Recently, much attention has been drawn to learning the underlying 3D structures of a scene from monocular videos in a fully self-supervised fashion. One of the most challenging aspects of this task is handling the independently moving objects as they break the rigid-scene assumption. For the first time, we show that pixel positional information can be exploited to learn SVDE (Single View Depth Es…
▽ More
Recently, much attention has been drawn to learning the underlying 3D structures of a scene from monocular videos in a fully self-supervised fashion. One of the most challenging aspects of this task is handling the independently moving objects as they break the rigid-scene assumption. For the first time, we show that pixel positional information can be exploited to learn SVDE (Single View Depth Estimation) from videos. Our proposed moving object (MO) masks, which are induced by shifted positional information (SPI) and referred to as `SPIMO' masks, are very robust and consistently remove the independently moving objects in the scenes, allowing for better learning of SVDE from videos. Additionally, we introduce a new adaptive quantization scheme that assigns the best per-pixel quantization curve for our depth discretization. Finally, we employ existing boosting techniques in a new way to further self-supervise the depth of the moving objects. With these features, our pipeline is robust against moving objects and generalizes well to high-resolution images, even when trained with small patches, yielding state-of-the-art (SOTA) results with almost 8.5x fewer parameters than the previous works that learn from videos. We present extensive experiments on KITTI and CityScapes that show the effectiveness of our method.
△ Less
Submitted 18 May, 2022;
originally announced May 2022.
-
Preoperative brain tumor imaging: models and software for segmentation and standardized reporting
Authors:
D. Bouget,
A. Pedersen,
A. S. Jakola,
V. Kavouridis,
K. E. Emblem,
R. S. Eijgelaar,
I. Kommers,
H. Ardon,
F. Barkhof,
L. Bello,
M. S. Berger,
M. C. Nibali,
J. Furtner,
S. Hervey-Jumper,
A. J. S. Idema,
B. Kiesel,
A. Kloet,
E. Mandonnet,
D. M. J. Müller,
P. A. Robe,
M. Rossi,
T. Sciortino,
W. Van den Brink,
M. Wagemakers,
G. Widhalm
, et al. (5 additional authors not shown)
Abstract:
For patients suffering from brain tumor, prognosis estimation and treatment decisions are made by a multidisciplinary team based on a set of preoperative MR scans. Currently, the lack of standardized and automatic methods for tumor detection and generation of clinical reports represents a major hurdle. In this study, we investigate glioblastomas, lower grade gliomas, meningiomas, and metastases, t…
▽ More
For patients suffering from brain tumor, prognosis estimation and treatment decisions are made by a multidisciplinary team based on a set of preoperative MR scans. Currently, the lack of standardized and automatic methods for tumor detection and generation of clinical reports represents a major hurdle. In this study, we investigate glioblastomas, lower grade gliomas, meningiomas, and metastases, through four cohorts of up to 4000 patients. Tumor segmentation models were trained using the AGU-Net architecture with different preprocessing steps and protocols. Segmentation performances were assessed in-depth using a wide-range of voxel and patient-wise metrics covering volume, distance, and probabilistic aspects. Finally, two software solutions have been developed, enabling an easy use of the trained models and standardized generation of clinical reports: Raidionics and Raidionics-Slicer. Segmentation performances were quite homogeneous across the four different brain tumor types, with an average true positive Dice ranging between 80% and 90%, patient-wise recall between 88% and 98%, and patient-wise precision around 95%. With our Raidionics software, running on a desktop computer with CPU support, tumor segmentation can be performed in 16 to 54 seconds depending on the dimensions of the MRI volume. For the generation of a standardized clinical report, including the tumor segmentation and features computation, 5 to 15 minutes are necessary. All trained models have been made open-access together with the source code for both software solutions and validation metrics computation. In the future, an automatic classification of the brain tumor type would be necessary to replace manual user input. Finally, the inclusion of post-operative segmentation in both software solutions will be key for generating complete post-operative standardized clinical reports.
△ Less
Submitted 29 April, 2022;
originally announced April 2022.
-
PLADE-Net: Towards Pixel-Level Accuracy for Self-Supervised Single-View Depth Estimation with Neural Positional Encoding and Distilled Matting Loss
Authors:
Juan Luis Gonzalez Bello,
Munchurl Kim
Abstract:
In this paper, we propose a self-supervised single-view pixel-level accurate depth estimation network, called PLADE-Net. The PLADE-Net is the first work that shows unprecedented accuracy levels, exceeding 95\% in terms of the $δ^1$ metric on the challenging KITTI dataset. Our PLADE-Net is based on a new network architecture with neural positional encoding and a novel loss function that borrows fro…
▽ More
In this paper, we propose a self-supervised single-view pixel-level accurate depth estimation network, called PLADE-Net. The PLADE-Net is the first work that shows unprecedented accuracy levels, exceeding 95\% in terms of the $δ^1$ metric on the challenging KITTI dataset. Our PLADE-Net is based on a new network architecture with neural positional encoding and a novel loss function that borrows from the closed-form solution of the matting Laplacian to learn pixel-level accurate depth estimation from stereo images. Neural positional encoding allows our PLADE-Net to obtain more consistent depth estimates by letting the network reason about location-specific image properties such as lens and projection distortions. Our novel distilled matting Laplacian loss allows our network to predict sharp depths at object boundaries and more consistent depths in highly homogeneous regions. Our proposed method outperforms all previous self-supervised single-view depth estimation methods by a large margin on the challenging KITTI dataset, with unprecedented levels of accuracy. Furthermore, our PLADE-Net, naively extended for stereo inputs, outperforms the most recent self-supervised stereo methods, even without any advanced blocks like 1D correlations, 3D convolutions, or spatial pyramid pooling. We present extensive ablation studies and experiments that support our method's effectiveness on the KITTI, CityScapes, and Make3D datasets.
△ Less
Submitted 12 March, 2021;
originally announced March 2021.
-
Relaxed Peephole Optimization: A Novel Compiler Optimization for Quantum Circuits
Authors:
Ji Liu,
Luciano Bello,
Huiyang Zhou
Abstract:
In this paper, we propose a novel quantum compiler optimization, named relaxed peephole optimization (RPO) for quantum computers. RPO leverages the single-qubit state information that can be determined statically by the compiler. We define that a qubit is in a basis state when, at a given point in time, its state is either in the X-, Y-, or Z-basis. When basis qubits are used as inputs to quantum…
▽ More
In this paper, we propose a novel quantum compiler optimization, named relaxed peephole optimization (RPO) for quantum computers. RPO leverages the single-qubit state information that can be determined statically by the compiler. We define that a qubit is in a basis state when, at a given point in time, its state is either in the X-, Y-, or Z-basis. When basis qubits are used as inputs to quantum gates, there exist opportunities for strength reduction, which replaces quantum operations with equivalent but less expensive ones. Compared to the existing peephole optimization for quantum programs, the difference is that our proposed optimization does not require an identical unitary matrix, thereby named `relaxed' peephole optimization. We also extend our approach to optimize the quantum gates when some input qubits are in known pure states. Both optimizations, namely the Quantum Basis-state Optimization (QBO) and the Quantum Pure-state Optimization (QPO), are implemented in the IBM's Qiskit transpiler. Our experimental results show that our proposed optimization pass is fast and effective. The circuits optimized with our compiler optimizations obtain up to 18.0% (11.7% on average) fewer CNOT gates and up to 8.2% (7.1% on average) lower transpilation time than that of the most aggressive optimization level in the Qiskit compiler. When running on real quantum computers, the success rates of 3-qubit quantum phase estimation algorithm improve by 2.30X due to the reduced gate counts.
△ Less
Submitted 14 December, 2020;
originally announced December 2020.
-
Deep 3D Pan via adaptive "t-shaped" convolutions with global and local adaptive dilations
Authors:
Juan Luis Gonzalez Bello,
Munchurl Kim
Abstract:
Recent advances in deep learning have shown promising results in many low-level vision tasks. However, solving the single-image-based view synthesis is still an open problem. In particular, the generation of new images at parallel camera views given a single input image is of great interest, as it enables 3D visualization of the 2D input scenery. We propose a novel network architecture to perform…
▽ More
Recent advances in deep learning have shown promising results in many low-level vision tasks. However, solving the single-image-based view synthesis is still an open problem. In particular, the generation of new images at parallel camera views given a single input image is of great interest, as it enables 3D visualization of the 2D input scenery. We propose a novel network architecture to perform stereoscopic view synthesis at arbitrary camera positions along the X-axis, or Deep 3D Pan, with "t-shaped" adaptive kernels equipped with globally and locally adaptive dilations. Our proposed network architecture, the monster-net, is devised with a novel "t-shaped" adaptive kernel with globally and locally adaptive dilation, which can efficiently incorporate global camera shift into and handle local 3D geometries of the target image's pixels for the synthesis of naturally looking 3D panned views when a 2-D input image is given. Extensive experiments were performed on the KITTI, CityScapes and our VICLAB_STEREO indoors dataset to prove the efficacy of our method. Our monster-net significantly outperforms the state-of-the-art method, SOTA, by a large margin in all metrics of RMSE, PSNR, and SSIM. Our proposed monster-net is capable of reconstructing more reliable image structures in synthesized images with coherent geometry. Moreover, the disparity information that can be extracted from the "t-shaped" kernel is much more reliable than that of the SOTA for the unsupervised monocular depth estimation task, confirming the effectiveness of our method.
△ Less
Submitted 20 October, 2019; v1 submitted 2 October, 2019;
originally announced October 2019.
-
Deep 3D-Zoom Net: Unsupervised Learning of Photo-Realistic 3D-Zoom
Authors:
Juan Luis Gonzalez Bello,
Munchurl Kim
Abstract:
The 3D-zoom operation is the positive translation of the camera in the Z-axis, perpendicular to the image plane. In contrast, the optical zoom changes the focal length and the digital zoom is used to enlarge a certain region of an image to the original image size. In this paper, we are the first to formulate an unsupervised 3D-zoom learning problem where images with an arbitrary zoom factor can be…
▽ More
The 3D-zoom operation is the positive translation of the camera in the Z-axis, perpendicular to the image plane. In contrast, the optical zoom changes the focal length and the digital zoom is used to enlarge a certain region of an image to the original image size. In this paper, we are the first to formulate an unsupervised 3D-zoom learning problem where images with an arbitrary zoom factor can be generated from a given single image. An unsupervised framework is convenient, as it is a challenging task to obtain a 3D-zoom dataset of natural scenes due to the need for special equipment to ensure camera movement is restricted to the Z-axis. In addition, the objects in the scenes should not move when being captured, which hinders the construction of a large dataset of outdoor scenes. We present a novel unsupervised framework to learn how to generate arbitrarily 3D-zoomed versions of a single image, not requiring a 3D-zoom ground truth, called the Deep 3D-Zoom Net. The Deep 3D-Zoom Net incorporates the following features: (i) transfer learning from a pre-trained disparity estimation network via a back re-projection reconstruction loss; (ii) a fully convolutional network architecture that models depth-image-based rendering (DIBR), taking into account high-frequency details without the need for estimating the intermediate disparity; and (iii) incorporating a discriminator network that acts as a no-reference penalty for unnaturally rendered areas. Even though there is no baseline to fairly compare our results, our method outperforms previous novel view synthesis research in terms of realistic appearance on large camera baselines. We performed extensive experiments to verify the effectiveness of our method on the KITTI and Cityscapes datasets.
△ Less
Submitted 2 October, 2019; v1 submitted 20 September, 2019;
originally announced September 2019.
-
A Novel Monocular Disparity Estimation Network with Domain Transformation and Ambiguity Learning
Authors:
Juan Luis Gonzalez Bello,
Munchurl Kim
Abstract:
Convolutional neural networks (CNN) have shown state-of-the-art results for low-level computer vision problems such as stereo and monocular disparity estimations, but still, have much room to further improve their performance in terms of accuracy, numbers of parameters, etc. Recent works have uncovered the advantages of using an unsupervised scheme to train CNN's to estimate monocular disparity, w…
▽ More
Convolutional neural networks (CNN) have shown state-of-the-art results for low-level computer vision problems such as stereo and monocular disparity estimations, but still, have much room to further improve their performance in terms of accuracy, numbers of parameters, etc. Recent works have uncovered the advantages of using an unsupervised scheme to train CNN's to estimate monocular disparity, where only the relatively-easy-to-obtain stereo images are needed for training. We propose a novel encoder-decoder architecture that outperforms previous unsupervised monocular depth estimation networks by (i) taking into account ambiguities, (ii) efficient fusion between encoder and decoder features with rectangular convolutions and (iii) domain transformations between encoder and decoder. Our architecture outperforms the Monodepth baseline in all metrics, even with a considerable reduction of parameters. Furthermore, our architecture is capable of estimating a full disparity map in a single forward pass, whereas the baseline needs two passes. We perform extensive experiments to verify the effectiveness of our method on the KITTI dataset.
△ Less
Submitted 20 March, 2019;
originally announced March 2019.
-
Qiskit Backend Specifications for OpenQASM and OpenPulse Experiments
Authors:
David C. McKay,
Thomas Alexander,
Luciano Bello,
Michael J. Biercuk,
Lev Bishop,
Jiayin Chen,
Jerry M. Chow,
Antonio D. Córcoles,
Daniel Egger,
Stefan Filipp,
Juan Gomez,
Michael Hush,
Ali Javadi-Abhari,
Diego Moreda,
Paul Nation,
Brent Paulovicks,
Erick Winston,
Christopher J. Wood,
James Wootton,
Jay M. Gambetta
Abstract:
As interest in quantum computing grows, there is a pressing need for standardized API's so that algorithm designers, circuit designers, and physicists can be provided a common reference frame for designing, executing, and optimizing experiments. There is also a need for a language specification that goes beyond gates and allows users to specify the time dynamics of a quantum experiment and recover…
▽ More
As interest in quantum computing grows, there is a pressing need for standardized API's so that algorithm designers, circuit designers, and physicists can be provided a common reference frame for designing, executing, and optimizing experiments. There is also a need for a language specification that goes beyond gates and allows users to specify the time dynamics of a quantum experiment and recover the time dynamics of the output. In this document we provide a specification for a common interface to backends (simulators and experiments) and a standarized data structure (Qobj --- quantum object) for sending experiments to those backends via Qiskit. We also introduce OpenPulse, a language for specifying pulse level control (i.e. control of the continuous time dynamics) of a general quantum device independent of the specific hardware implementation.
△ Less
Submitted 10 September, 2018;
originally announced September 2018.
-
A novel JXTA-based architecture for implementing heterogenous Networks of Things
Authors:
Filippo Battaglia,
Lucia Lo Bello
Abstract:
This paper presents EmbJXTAChord, a novel peer-to-peer (P2P) architecture that integrates the good features of different sources, such as JXTA, EXI, CoAP, combining and augmenting them to provide a framework that is specifically devised for develo** IoT applications over heterogeneous networks. EmbJXTAChord provides for several interesting properties, such as, distributed and fault-tolerant reso…
▽ More
This paper presents EmbJXTAChord, a novel peer-to-peer (P2P) architecture that integrates the good features of different sources, such as JXTA, EXI, CoAP, combining and augmenting them to provide a framework that is specifically devised for develo** IoT applications over heterogeneous networks. EmbJXTAChord provides for several interesting properties, such as, distributed and fault-tolerant resource discovery, transparent routing over subnetworks, application protocol independence from the transport protocol in narrowband WSN, thus eliminating the need for using dedicated software or configuring custom gateways to achieve these functionalities. Moreover, EmbJXTAChord offers native support not only for TCP/HTTP, but also for Bluetooth RFCOMM and 6LoWPAN, thus opening to a broad range of IoT devices in supernetworks composed of networks using different interconnection technologies, not necessarily IP-based. In addition, EmbJXTAChord offers security over heterogeneous networks providing support for secure peergroups (even nested) and for group encryption, thus allowing for unicast and multicast communication between groups of objects sharing the same resources. Finally, EmbJXTAChord provides jxCOAP-E, a new CoAP implementation that leverages on the transport mechanisms for heterogeneous networks offered by EmbJXTAChord. jxCOAP-E enables to realize a RESTful service architecture for peer-to-peer narrowband or broadband networks composed of devices connected via Ethernet, Wi-Fi, Bluetooth, BLE or IEEE 802.15.4. Differently from CoAP, jxCOAP-E provides a distributed and fault-tolerant service discovery mechanism and support for secure multicast communications. The paper presents EmbJXTAChord, discusses all the relevant design challenges and presents a comparative experimental performance assessment with state-of-the-art solutions on commercial-off-the-shelf devices.
△ Less
Submitted 22 November, 2017;
originally announced November 2017.
