-
MegaScenes: Scene-Level View Synthesis at Scale
Authors:
Joseph Tung,
Gene Chou,
Ruo** Cai,
Guandao Yang,
Kai Zhang,
Gordon Wetzstein,
Bharath Hariharan,
Noah Snavely
Abstract:
Scene-level novel view synthesis (NVS) is fundamental to many vision and graphics applications. Recently, pose-conditioned diffusion models have led to significant progress by extracting 3D information from 2D foundation models, but these methods are limited by the lack of scene-level training data. Common dataset choices either consist of isolated objects (Objaverse), or of object-centric scenes…
▽ More
Scene-level novel view synthesis (NVS) is fundamental to many vision and graphics applications. Recently, pose-conditioned diffusion models have led to significant progress by extracting 3D information from 2D foundation models, but these methods are limited by the lack of scene-level training data. Common dataset choices either consist of isolated objects (Objaverse), or of object-centric scenes with limited pose distributions (DTU, CO3D). In this paper, we create a large-scale scene-level dataset from Internet photo collections, called MegaScenes, which contains over 100K structure from motion (SfM) reconstructions from around the world. Internet photos represent a scalable data source but come with challenges such as lighting and transient objects. We address these issues to further create a subset suitable for the task of NVS. Additionally, we analyze failure cases of state-of-the-art NVS methods and significantly improve generation consistency. Through extensive experiments, we validate the effectiveness of both our dataset and method on generating in-the-wild scenes. For details on the dataset and code, see our project page at https://megascenes.github.io .
△ Less
Submitted 17 June, 2024;
originally announced June 2024.
-
DiffuBox: Refining 3D Object Detection with Point Diffusion
Authors:
Xiangyu Chen,
Zhenzhen Liu,
Katie Z Luo,
Siddhartha Datta,
Adhitya Polavaram,
Yan Wang,
Yurong You,
Boyi Li,
Marco Pavone,
Wei-Lun Chao,
Mark Campbell,
Bharath Hariharan,
Kilian Q. Weinberger
Abstract:
Ensuring robust 3D object detection and localization is crucial for many applications in robotics and autonomous driving. Recent models, however, face difficulties in maintaining high performance when applied to domains with differing sensor setups or geographic locations, often resulting in poor localization accuracy due to domain shift. To overcome this challenge, we introduce a novel diffusion-…
▽ More
Ensuring robust 3D object detection and localization is crucial for many applications in robotics and autonomous driving. Recent models, however, face difficulties in maintaining high performance when applied to domains with differing sensor setups or geographic locations, often resulting in poor localization accuracy due to domain shift. To overcome this challenge, we introduce a novel diffusion-based box refinement approach. This method employs a domain-agnostic diffusion model, conditioned on the LiDAR points surrounding a coarse bounding box, to simultaneously refine the box's location, size, and orientation. We evaluate this approach under various domain adaptation settings, and our results reveal significant improvements across different datasets, object classes and detectors.
△ Less
Submitted 24 May, 2024;
originally announced May 2024.
-
Learning to Detect and Segment Mobile Objects from Unlabeled Videos
Authors:
Yihong Sun,
Bharath Hariharan
Abstract:
Embodied agents must detect and localize objects of interest, e.g. traffic participants for self-driving cars. Supervision in the form of bounding boxes for this task is extremely expensive. As such, prior work has looked at unsupervised object segmentation, but in the absence of annotated boxes, it is unclear how pixels must be grouped into objects and which objects are of interest. This results…
▽ More
Embodied agents must detect and localize objects of interest, e.g. traffic participants for self-driving cars. Supervision in the form of bounding boxes for this task is extremely expensive. As such, prior work has looked at unsupervised object segmentation, but in the absence of annotated boxes, it is unclear how pixels must be grouped into objects and which objects are of interest. This results in over- / under-segmentation and irrelevant objects. Inspired both by the human visual system and by practical applications, we posit that the key missing cue is motion: objects of interest are typically mobile objects. We propose MOD-UV, a Mobile Object Detector learned from Unlabeled Videos only. We begin with pseudo-labels derived from motion segmentation, but introduce a novel training paradigm to progressively discover small objects and static-but-mobile objects that are missed by motion segmentation. As a result, though only learned from unlabeled videos, MOD-UV can detect and segment mobile objects from a single static image. Empirically, we achieve state-of-the-art performance in unsupervised mobile object detection on Waymo Open, nuScenes, and KITTI Dataset without using any external data or supervised models. Code is publicly available at https://github.com/YihongSun/MOD-UV.
△ Less
Submitted 23 May, 2024;
originally announced May 2024.
-
Better Monocular 3D Detectors with LiDAR from the Past
Authors:
Yurong You,
Cheng Perng Phoo,
Carlos Andres Diaz-Ruiz,
Katie Z Luo,
Wei-Lun Chao,
Mark Campbell,
Bharath Hariharan,
Kilian Q Weinberger
Abstract:
Accurate 3D object detection is crucial to autonomous driving. Though LiDAR-based detectors have achieved impressive performance, the high cost of LiDAR sensors precludes their widespread adoption in affordable vehicles. Camera-based detectors are cheaper alternatives but often suffer inferior performance compared to their LiDAR-based counterparts due to inherent depth ambiguities in images. In th…
▽ More
Accurate 3D object detection is crucial to autonomous driving. Though LiDAR-based detectors have achieved impressive performance, the high cost of LiDAR sensors precludes their widespread adoption in affordable vehicles. Camera-based detectors are cheaper alternatives but often suffer inferior performance compared to their LiDAR-based counterparts due to inherent depth ambiguities in images. In this work, we seek to improve monocular 3D detectors by leveraging unlabeled historical LiDAR data. Specifically, at inference time, we assume that the camera-based detectors have access to multiple unlabeled LiDAR scans from past traversals at locations of interest (potentially from other high-end vehicles equipped with LiDAR sensors). Under this setup, we proposed a novel, simple, and end-to-end trainable framework, termed AsyncDepth, to effectively extract relevant features from asynchronous LiDAR traversals of the same location for monocular 3D detectors. We show consistent and significant performance gain (up to 9 AP) across multiple state-of-the-art models and datasets with a negligible additional latency of 9.66 ms and a small storage cost.
△ Less
Submitted 9 April, 2024; v1 submitted 7 April, 2024;
originally announced April 2024.
-
Remote Sensing Vision-Language Foundation Models without Annotations via Ground Remote Alignment
Authors:
Utkarsh Mall,
Cheng Perng Phoo,
Meilin Kelsey Liu,
Carl Vondrick,
Bharath Hariharan,
Kavita Bala
Abstract:
We introduce a method to train vision-language models for remote-sensing images without using any textual annotations. Our key insight is to use co-located internet imagery taken on the ground as an intermediary for connecting remote-sensing images and language. Specifically, we train an image encoder for remote sensing images to align with the image encoder of CLIP using a large amount of paired…
▽ More
We introduce a method to train vision-language models for remote-sensing images without using any textual annotations. Our key insight is to use co-located internet imagery taken on the ground as an intermediary for connecting remote-sensing images and language. Specifically, we train an image encoder for remote sensing images to align with the image encoder of CLIP using a large amount of paired internet and satellite images. Our unsupervised approach enables the training of a first-of-its-kind large-scale vision language model (VLM) for remote sensing images at two different resolutions. We show that these VLMs enable zero-shot, open-vocabulary image classification, retrieval, segmentation and visual question answering for satellite images. On each of these tasks, our VLM trained without textual annotations outperforms existing VLMs trained with supervision, with gains of up to 20% for classification and 80% for segmentation.
△ Less
Submitted 11 December, 2023;
originally announced December 2023.
-
Accurate Differential Operators for Hybrid Neural Fields
Authors:
Aditya Chetan,
Guandao Yang,
Zichen Wang,
Steve Marschner,
Bharath Hariharan
Abstract:
Neural fields have become widely used in various fields, from shape representation to neural rendering, and for solving partial differential equations (PDEs). With the advent of hybrid neural field representations like Instant NGP that leverage small MLPs and explicit representations, these models train quickly and can fit large scenes. Yet in many applications like rendering and simulation, hybri…
▽ More
Neural fields have become widely used in various fields, from shape representation to neural rendering, and for solving partial differential equations (PDEs). With the advent of hybrid neural field representations like Instant NGP that leverage small MLPs and explicit representations, these models train quickly and can fit large scenes. Yet in many applications like rendering and simulation, hybrid neural fields can cause noticeable and unreasonable artifacts. This is because they do not yield accurate spatial derivatives needed for these downstream applications. In this work, we propose two ways to circumvent these challenges. Our first approach is a post hoc operator that uses local polynomial-fitting to obtain more accurate derivatives from pre-trained hybrid neural fields. Additionally, we also propose a self-supervised fine-tuning approach that refines the neural field to yield accurate derivatives directly while preserving the initial signal. We show the application of our method on rendering, collision simulation, and solving PDEs. We observe that using our approach yields more accurate derivatives, reducing artifacts and leading to more accurate simulations in downstream applications.
△ Less
Submitted 10 December, 2023;
originally announced December 2023.
-
Reward Finetuning for Faster and More Accurate Unsupervised Object Discovery
Authors:
Katie Z Luo,
Zhenzhen Liu,
Xiangyu Chen,
Yurong You,
Sagie Benaim,
Cheng Perng Phoo,
Mark Campbell,
Wen Sun,
Bharath Hariharan,
Kilian Q. Weinberger
Abstract:
Recent advances in machine learning have shown that Reinforcement Learning from Human Feedback (RLHF) can improve machine learning models and align them with human preferences. Although very successful for Large Language Models (LLMs), these advancements have not had a comparable impact in research for autonomous vehicles -- where alignment with human expectations can be imperative. In this paper,…
▽ More
Recent advances in machine learning have shown that Reinforcement Learning from Human Feedback (RLHF) can improve machine learning models and align them with human preferences. Although very successful for Large Language Models (LLMs), these advancements have not had a comparable impact in research for autonomous vehicles -- where alignment with human expectations can be imperative. In this paper, we propose to adapt similar RL-based methods to unsupervised object discovery, i.e. learning to detect objects from LiDAR points without any training labels. Instead of labels, we use simple heuristics to mimic human feedback. More explicitly, we combine multiple heuristics into a simple reward function that positively correlates its score with bounding box accuracy, i.e., boxes containing objects are scored higher than those without. We start from the detector's own predictions to explore the space and reinforce boxes with high rewards through gradient updates. Empirically, we demonstrate that our approach is not only more accurate, but also orders of magnitudes faster to train compared to prior works on object discovery.
△ Less
Submitted 5 November, 2023; v1 submitted 29 October, 2023;
originally announced October 2023.
-
Dynamo-Depth: Fixing Unsupervised Depth Estimation for Dynamical Scenes
Authors:
Yihong Sun,
Bharath Hariharan
Abstract:
Unsupervised monocular depth estimation techniques have demonstrated encouraging results but typically assume that the scene is static. These techniques suffer when trained on dynamical scenes, where apparent object motion can equally be explained by hypothesizing the object's independent motion, or by altering its depth. This ambiguity causes depth estimators to predict erroneous depth for moving…
▽ More
Unsupervised monocular depth estimation techniques have demonstrated encouraging results but typically assume that the scene is static. These techniques suffer when trained on dynamical scenes, where apparent object motion can equally be explained by hypothesizing the object's independent motion, or by altering its depth. This ambiguity causes depth estimators to predict erroneous depth for moving objects. To resolve this issue, we introduce Dynamo-Depth, an unifying approach that disambiguates dynamical motion by jointly learning monocular depth, 3D independent flow field, and motion segmentation from unlabeled monocular videos. Specifically, we offer our key insight that a good initial estimation of motion segmentation is sufficient for jointly learning depth and independent motion despite the fundamental underlying ambiguity. Our proposed method achieves state-of-the-art performance on monocular depth estimation on Waymo Open and nuScenes Dataset with significant improvement in the depth of moving objects. Code and additional results are available at https://dynamo-depth.github.io.
△ Less
Submitted 28 October, 2023;
originally announced October 2023.
-
Small-scale cosmic ray anisotropy observed by the GRAPES-3 experiment at TeV energies
Authors:
M. Chakraborty,
S. Ahmad,
A. Chandra,
S. R. Dugad,
U. D. Goswami,
S. K. Gupta,
B. Hariharan,
Y. Hayashi,
P. Jagadeesan,
A. Jain,
P. Jain,
S. Kawakami,
T. Koi,
H. Kojima,
S. Mahapatra,
P. K. Mohanty,
R. Moharana,
Y. Muraki,
P. K. Nayak,
T. Nonaka,
T. Nakamura,
A. Oshima,
B. P. Pant,
D. Pattanaik,
S. Paul
, et al. (13 additional authors not shown)
Abstract:
GRAPES-3 is a mid-altitude (2200 m) and near equatorial ($11.4^{\circ}$ North) air shower array, overlap** in its field of view for cosmic ray observations with experiments that are located in Northern and Southern hemispheres. We analyze a sample of $3.7\times10^9$ cosmic ray events collected by the GRAPES-3 experiment between 1 January 2013 and 31 December 2016 with a median energy of…
▽ More
GRAPES-3 is a mid-altitude (2200 m) and near equatorial ($11.4^{\circ}$ North) air shower array, overlap** in its field of view for cosmic ray observations with experiments that are located in Northern and Southern hemispheres. We analyze a sample of $3.7\times10^9$ cosmic ray events collected by the GRAPES-3 experiment between 1 January 2013 and 31 December 2016 with a median energy of $\sim16$ TeV for study of small-scale ($<60^{\circ}$) angular scale anisotropies. We observed two structures labeled as A and B, deviate from the expected isotropic distribution of cosmic rays in a statistically significant manner. Structure `A' spans $50^{\circ}$ to $80^{\circ}$ in the right ascension and $-15^{\circ}$ to $30^{\circ}$ in the declination coordinate. The relative excess observed in the structure A is at the level of $(6.5\pm1.3)\times10^{-4}$ with a statistical significance of 6.8 standard deviations. Structure `B' is observed in the right ascension range of $110^{\circ}$ to $140^{\circ}$. The relative excess observed in this region is at the level of $(4.9\pm1.4)\times10^{-4}$ with a statistical significance of 4.7 standard deviations. These structures are consistent with those reported by Milagro, ARGO-YBJ, and HAWC. These observations could provide a better understanding of the cosmic ray sources, propagation and the magnetic structures in our Galaxy.
△ Less
Submitted 23 October, 2023;
originally announced October 2023.
-
Pre-Training LiDAR-Based 3D Object Detectors Through Colorization
Authors:
Tai-Yu Pan,
Chenyang Ma,
Tianle Chen,
Cheng Perng Phoo,
Katie Z Luo,
Yurong You,
Mark Campbell,
Kilian Q. Weinberger,
Bharath Hariharan,
Wei-Lun Chao
Abstract:
Accurate 3D object detection and understanding for self-driving cars heavily relies on LiDAR point clouds, necessitating large amounts of labeled data to train. In this work, we introduce an innovative pre-training approach, Grounded Point Colorization (GPC), to bridge the gap between data and labels by teaching the model to colorize LiDAR point clouds, equip** it with valuable semantic cues. To…
▽ More
Accurate 3D object detection and understanding for self-driving cars heavily relies on LiDAR point clouds, necessitating large amounts of labeled data to train. In this work, we introduce an innovative pre-training approach, Grounded Point Colorization (GPC), to bridge the gap between data and labels by teaching the model to colorize LiDAR point clouds, equip** it with valuable semantic cues. To tackle challenges arising from color variations and selection bias, we incorporate color as "context" by providing ground-truth colors as hints during colorization. Experimental results on the KITTI and Waymo datasets demonstrate GPC's remarkable effectiveness. Even with limited labeled data, GPC significantly improves fine-tuning performance; notably, on just 20% of the KITTI dataset, GPC outperforms training from scratch with the entire dataset. In sum, we introduce a fresh perspective on pre-training for 3D object detection, aligning the objective with the model's intended role and ultimately advancing the accuracy and efficiency of 3D object detection for autonomous vehicles.
△ Less
Submitted 25 February, 2024; v1 submitted 23 October, 2023;
originally announced October 2023.
-
RealFill: Reference-Driven Generation for Authentic Image Completion
Authors:
Luming Tang,
Nataniel Ruiz,
Qinghao Chu,
Yuanzhen Li,
Aleksander Holynski,
David E. Jacobs,
Bharath Hariharan,
Yael Pritch,
Neal Wadhwa,
Kfir Aberman,
Michael Rubinstein
Abstract:
Recent advances in generative imagery have brought forth outpainting and inpainting models that can produce high-quality, plausible image content in unknown regions. However, the content these models hallucinate is necessarily inauthentic, since they are unaware of the true scene. In this work, we propose RealFill, a novel generative approach for image completion that fills in missing regions of a…
▽ More
Recent advances in generative imagery have brought forth outpainting and inpainting models that can produce high-quality, plausible image content in unknown regions. However, the content these models hallucinate is necessarily inauthentic, since they are unaware of the true scene. In this work, we propose RealFill, a novel generative approach for image completion that fills in missing regions of an image with the content that should have been there. RealFill is a generative inpainting model that is personalized using only a few reference images of a scene. These reference images do not have to be aligned with the target image, and can be taken with drastically varying viewpoints, lighting conditions, camera apertures, or image styles. Once personalized, RealFill is able to complete a target image with visually compelling contents that are faithful to the original scene. We evaluate RealFill on a new image completion benchmark that covers a set of diverse and challenging scenarios, and find that it outperforms existing approaches by a large margin. Project page: https://realfill.github.io
△ Less
Submitted 14 May, 2024; v1 submitted 28 September, 2023;
originally announced September 2023.
-
Unsupervised Domain Adaptation for Self-Driving from Past Traversal Features
Authors:
Travis Zhang,
Katie Luo,
Cheng Perng Phoo,
Yurong You,
Wei-Lun Chao,
Bharath Hariharan,
Mark Campbell,
Kilian Q. Weinberger
Abstract:
The rapid development of 3D object detection systems for self-driving cars has significantly improved accuracy. However, these systems struggle to generalize across diverse driving environments, which can lead to safety-critical failures in detecting traffic participants. To address this, we propose a method that utilizes unlabeled repeated traversals of multiple locations to adapt object detector…
▽ More
The rapid development of 3D object detection systems for self-driving cars has significantly improved accuracy. However, these systems struggle to generalize across diverse driving environments, which can lead to safety-critical failures in detecting traffic participants. To address this, we propose a method that utilizes unlabeled repeated traversals of multiple locations to adapt object detectors to new driving environments. By incorporating statistics computed from repeated LiDAR scans, we guide the adaptation process effectively. Our approach enhances LiDAR-based detection models using spatial quantized historical features and introduces a lightweight regression head to leverage the statistics for feature regularization. Additionally, we leverage the statistics for a novel self-training process to stabilize the training. The framework is detector model-agnostic and experiments on real-world datasets demonstrate significant improvements, achieving up to a 20-point performance gain, especially in detecting pedestrians and distant objects. Code is available at https://github.com/zhangtravis/Hist-DA.
△ Less
Submitted 21 September, 2023;
originally announced September 2023.
-
Doppelgangers: Learning to Disambiguate Images of Similar Structures
Authors:
Ruo** Cai,
Joseph Tung,
Qianqian Wang,
Hadar Averbuch-Elor,
Bharath Hariharan,
Noah Snavely
Abstract:
We consider the visual disambiguation task of determining whether a pair of visually similar images depict the same or distinct 3D surfaces (e.g., the same or opposite sides of a symmetric building). Illusory image matches, where two images observe distinct but visually similar 3D surfaces, can be challenging for humans to differentiate, and can also lead 3D reconstruction algorithms to produce er…
▽ More
We consider the visual disambiguation task of determining whether a pair of visually similar images depict the same or distinct 3D surfaces (e.g., the same or opposite sides of a symmetric building). Illusory image matches, where two images observe distinct but visually similar 3D surfaces, can be challenging for humans to differentiate, and can also lead 3D reconstruction algorithms to produce erroneous results. We propose a learning-based approach to visual disambiguation, formulating it as a binary classification task on image pairs. To that end, we introduce a new dataset for this problem, Doppelgangers, which includes image pairs of similar structures with ground truth labels. We also design a network architecture that takes the spatial distribution of local keypoints and matches as input, allowing for better reasoning about both local and global cues. Our evaluation shows that our method can distinguish illusory matches in difficult cases, and can be integrated into SfM pipelines to produce correct, disambiguated 3D reconstructions. See our project page for our code, datasets, and more results: http://doppelgangers-3d.github.io/.
△ Less
Submitted 5 September, 2023;
originally announced September 2023.
-
Tracking Everything Everywhere All at Once
Authors:
Qianqian Wang,
Yen-Yu Chang,
Ruo** Cai,
Zhengqi Li,
Bharath Hariharan,
Aleksander Holynski,
Noah Snavely
Abstract:
We present a new test-time optimization method for estimating dense and long-range motion from a video sequence. Prior optical flow or particle video tracking algorithms typically operate within limited temporal windows, struggling to track through occlusions and maintain global consistency of estimated motion trajectories. We propose a complete and globally consistent motion representation, dubbe…
▽ More
We present a new test-time optimization method for estimating dense and long-range motion from a video sequence. Prior optical flow or particle video tracking algorithms typically operate within limited temporal windows, struggling to track through occlusions and maintain global consistency of estimated motion trajectories. We propose a complete and globally consistent motion representation, dubbed OmniMotion, that allows for accurate, full-length motion estimation of every pixel in a video. OmniMotion represents a video using a quasi-3D canonical volume and performs pixel-wise tracking via bijections between local and canonical space. This representation allows us to ensure global consistency, track through occlusions, and model any combination of camera and object motion. Extensive evaluations on the TAP-Vid benchmark and real-world footage show that our approach outperforms prior state-of-the-art methods by a large margin both quantitatively and qualitatively. See our project page for more results: http://omnimotion.github.io/
△ Less
Submitted 12 September, 2023; v1 submitted 8 June, 2023;
originally announced June 2023.
-
Emergent Correspondence from Image Diffusion
Authors:
Luming Tang,
Menglin Jia,
Qianqian Wang,
Cheng Perng Phoo,
Bharath Hariharan
Abstract:
Finding correspondences between images is a fundamental problem in computer vision. In this paper, we show that correspondence emerges in image diffusion models without any explicit supervision. We propose a simple strategy to extract this implicit knowledge out of diffusion networks as image features, namely DIffusion FeaTures (DIFT), and use them to establish correspondences between real images.…
▽ More
Finding correspondences between images is a fundamental problem in computer vision. In this paper, we show that correspondence emerges in image diffusion models without any explicit supervision. We propose a simple strategy to extract this implicit knowledge out of diffusion networks as image features, namely DIffusion FeaTures (DIFT), and use them to establish correspondences between real images. Without any additional fine-tuning or supervision on the task-specific data or annotations, DIFT is able to outperform both weakly-supervised methods and competitive off-the-shelf features in identifying semantic, geometric, and temporal correspondences. Particularly for semantic correspondence, DIFT from Stable Diffusion is able to outperform DINO and OpenCLIP by 19 and 14 accuracy points respectively on the challenging SPair-71k benchmark. It even outperforms the state-of-the-art supervised methods on 9 out of 18 categories while remaining on par for the overall performance. Project page: https://diffusionfeatures.github.io
△ Less
Submitted 6 December, 2023; v1 submitted 6 June, 2023;
originally announced June 2023.
-
Distilling from Similar Tasks for Transfer Learning on a Budget
Authors:
Kenneth Borup,
Cheng Perng Phoo,
Bharath Hariharan
Abstract:
We address the challenge of getting efficient yet accurate recognition systems with limited labels. While recognition models improve with model size and amount of data, many specialized applications of computer vision have severe resource constraints both during training and inference. Transfer learning is an effective solution for training with few labels, however often at the expense of a comput…
▽ More
We address the challenge of getting efficient yet accurate recognition systems with limited labels. While recognition models improve with model size and amount of data, many specialized applications of computer vision have severe resource constraints both during training and inference. Transfer learning is an effective solution for training with few labels, however often at the expense of a computationally costly fine-tuning of large base models. We propose to mitigate this unpleasant trade-off between compute and accuracy via semi-supervised cross-domain distillation from a set of diverse source models. Initially, we show how to use task similarity metrics to select a single suitable source model to distill from, and that a good selection process is imperative for good downstream performance of a target model. We dub this approach DistillNearest. Though effective, DistillNearest assumes a single source model matches the target task, which is not always the case. To alleviate this, we propose a weighted multi-source distillation method to distill multiple source models trained on different domains weighted by their relevance for the target task into a single efficient model (named DistillWeighted). Our methods need no access to source data, and merely need features and pseudo-labels of the source models. When the goal is accurate recognition under computational constraints, both DistillNearest and DistillWeighted approaches outperform both transfer learning from strong ImageNet initializations as well as state-of-the-art semi-supervised techniques such as FixMatch. Averaged over 8 diverse target tasks our multi-source method outperforms the baselines by 5.6%-points and 4.5%-points, respectively.
△ Less
Submitted 24 April, 2023;
originally announced April 2023.
-
Unsupervised Adaptation from Repeated Traversals for Autonomous Driving
Authors:
Yurong You,
Cheng Perng Phoo,
Katie Z Luo,
Travis Zhang,
Wei-Lun Chao,
Bharath Hariharan,
Mark Campbell,
Kilian Q. Weinberger
Abstract:
For a self-driving car to operate reliably, its perceptual system must generalize to the end-user's environment -- ideally without additional annotation efforts. One potential solution is to leverage unlabeled data (e.g., unlabeled LiDAR point clouds) collected from the end-users' environments (i.e. target domain) to adapt the system to the difference between training and testing environments. Whi…
▽ More
For a self-driving car to operate reliably, its perceptual system must generalize to the end-user's environment -- ideally without additional annotation efforts. One potential solution is to leverage unlabeled data (e.g., unlabeled LiDAR point clouds) collected from the end-users' environments (i.e. target domain) to adapt the system to the difference between training and testing environments. While extensive research has been done on such an unsupervised domain adaptation problem, one fundamental problem lingers: there is no reliable signal in the target domain to supervise the adaptation process. To overcome this issue we observe that it is easy to collect unsupervised data from multiple traversals of repeated routes. While different from conventional unsupervised domain adaptation, this assumption is extremely realistic since many drivers share the same roads. We show that this simple additional assumption is sufficient to obtain a potent signal that allows us to perform iterative self-training of 3D object detectors on the target domain. Concretely, we generate pseudo-labels with the out-of-domain detector but reduce false positives by removing detections of supposedly mobile objects that are persistent across traversals. Further, we reduce false negatives by encouraging predictions in regions that are not persistent. We experiment with our approach on two large-scale driving datasets and show remarkable improvement in 3D object detection of cars, pedestrians, and cyclists, bringing us a step closer to generalizable autonomous driving.
△ Less
Submitted 27 March, 2023;
originally announced March 2023.
-
Polynomial Neural Fields for Subband Decomposition and Manipulation
Authors:
Guandao Yang,
Sagie Benaim,
Varun Jampani,
Kyle Genova,
Jonathan T. Barron,
Thomas Funkhouser,
Bharath Hariharan,
Serge Belongie
Abstract:
Neural fields have emerged as a new paradigm for representing signals, thanks to their ability to do it compactly while being easy to optimize. In most applications, however, neural fields are treated like black boxes, which precludes many signal manipulation tasks. In this paper, we propose a new class of neural fields called polynomial neural fields (PNFs). The key advantage of a PNF is that it…
▽ More
Neural fields have emerged as a new paradigm for representing signals, thanks to their ability to do it compactly while being easy to optimize. In most applications, however, neural fields are treated like black boxes, which precludes many signal manipulation tasks. In this paper, we propose a new class of neural fields called polynomial neural fields (PNFs). The key advantage of a PNF is that it can represent a signal as a composition of a number of manipulable and interpretable components without losing the merits of neural fields representation. We develop a general theoretical framework to analyze and design PNFs. We use this framework to design Fourier PNFs, which match state-of-the-art performance in signal representation tasks that use neural fields. In addition, we empirically demonstrate that Fourier PNFs enable signal manipulation applications such as texture transfer and scale-space interpolation. Code is available at https://github.com/stevenygd/PNF.
△ Less
Submitted 9 February, 2023;
originally announced February 2023.
-
Image-to-Image Translation for Autonomous Driving from Coarsely-Aligned Image Pairs
Authors:
Youya Xia,
Josephine Monica,
Wei-Lun Chao,
Bharath Hariharan,
Kilian Q Weinberger,
Mark Campbell
Abstract:
A self-driving car must be able to reliably handle adverse weather conditions (e.g., snowy) to operate safely. In this paper, we investigate the idea of turning sensor inputs (i.e., images) captured in an adverse condition into a benign one (i.e., sunny), upon which the downstream tasks (e.g., semantic segmentation) can attain high accuracy. Prior work primarily formulates this as an unpaired imag…
▽ More
A self-driving car must be able to reliably handle adverse weather conditions (e.g., snowy) to operate safely. In this paper, we investigate the idea of turning sensor inputs (i.e., images) captured in an adverse condition into a benign one (i.e., sunny), upon which the downstream tasks (e.g., semantic segmentation) can attain high accuracy. Prior work primarily formulates this as an unpaired image-to-image translation problem due to the lack of paired images captured under the exact same camera poses and semantic layouts. While perfectly-aligned images are not available, one can easily obtain coarsely-paired images. For instance, many people drive the same routes daily in both good and adverse weather; thus, images captured at close-by GPS locations can form a pair. Though data from repeated traversals are unlikely to capture the same foreground objects, we posit that they provide rich contextual information to supervise the image translation model. To this end, we propose a novel training objective leveraging coarsely-aligned image pairs. We show that our coarsely-aligned training scheme leads to a better image translation quality and improved downstream tasks, such as semantic segmentation, monocular depth estimation, and visual localization.
△ Less
Submitted 23 September, 2022;
originally announced September 2022.
-
Ithaca365: Dataset and Driving Perception under Repeated and Challenging Weather Conditions
Authors:
Carlos A. Diaz-Ruiz,
Youya Xia,
Yurong You,
Jose Nino,
Junan Chen,
Josephine Monica,
Xiangyu Chen,
Katie Luo,
Yan Wang,
Marc Emond,
Wei-Lun Chao,
Bharath Hariharan,
Kilian Q. Weinberger,
Mark Campbell
Abstract:
Advances in perception for self-driving cars have accelerated in recent years due to the availability of large-scale datasets, typically collected at specific locations and under nice weather conditions. Yet, to achieve the high safety requirement, these perceptual systems must operate robustly under a wide variety of weather conditions including snow and rain. In this paper, we present a new data…
▽ More
Advances in perception for self-driving cars have accelerated in recent years due to the availability of large-scale datasets, typically collected at specific locations and under nice weather conditions. Yet, to achieve the high safety requirement, these perceptual systems must operate robustly under a wide variety of weather conditions including snow and rain. In this paper, we present a new dataset to enable robust autonomous driving via a novel data collection process - data is repeatedly recorded along a 15 km route under diverse scene (urban, highway, rural, campus), weather (snow, rain, sun), time (day/night), and traffic conditions (pedestrians, cyclists and cars). The dataset includes images and point clouds from cameras and LiDAR sensors, along with high-precision GPS/INS to establish correspondence across routes. The dataset includes road and object annotations using amodal masks to capture partial occlusions and 3D bounding boxes. We demonstrate the uniqueness of this dataset by analyzing the performance of baselines in amodal segmentation of road and objects, depth estimation, and 3D object detection. The repeated routes opens new research directions in object discovery, continual learning, and anomaly detection. Link to Ithaca365: https://ithaca365.mae.cornell.edu/
△ Less
Submitted 1 August, 2022;
originally announced August 2022.
-
Diagnosing and Remedying Shot Sensitivity with Cosine Few-Shot Learners
Authors:
Davis Wertheimer,
Luming Tang,
Bharath Hariharan
Abstract:
Few-shot recognition involves training an image classifier to distinguish novel concepts at test time using few examples (shot). Existing approaches generally assume that the shot number at test time is known in advance. This is not realistic, and the performance of a popular and foundational method has been shown to suffer when train and test shots do not match. We conduct a systematic empirical…
▽ More
Few-shot recognition involves training an image classifier to distinguish novel concepts at test time using few examples (shot). Existing approaches generally assume that the shot number at test time is known in advance. This is not realistic, and the performance of a popular and foundational method has been shown to suffer when train and test shots do not match. We conduct a systematic empirical study of this phenomenon. In line with prior work, we find that shot sensitivity is broadly present across metric-based few-shot learners, but in contrast to prior work, larger neural architectures provide a degree of built-in robustness to varying test shot. More importantly, a simple, previously known but greatly overlooked class of approaches based on cosine distance consistently and greatly improves robustness to shot variation, by removing sensitivity to sample noise. We derive cosine alternatives to popular and recent few-shot classifiers, broadening their applicability to realistic settings. These cosine models consistently improve shot-robustness, outperform prior shot-robust state of the art, and provide competitive accuracy on a range of benchmarks and architectures, including notable gains in the very-low-shot regime.
△ Less
Submitted 7 July, 2022;
originally announced July 2022.
-
Activation Regression for Continuous Domain Generalization with Applications to Crop Classification
Authors:
Samar Khanna,
Bram Wallace,
Kavita Bala,
Bharath Hariharan
Abstract:
Geographic variance in satellite imagery impacts the ability of machine learning models to generalise to new regions. In this paper, we model geographic generalisation in medium resolution Landsat-8 satellite imagery as a continuous domain adaptation problem, demonstrating how models generalise better with appropriate domain knowledge. We develop a dataset spatially distributed across the entire c…
▽ More
Geographic variance in satellite imagery impacts the ability of machine learning models to generalise to new regions. In this paper, we model geographic generalisation in medium resolution Landsat-8 satellite imagery as a continuous domain adaptation problem, demonstrating how models generalise better with appropriate domain knowledge. We develop a dataset spatially distributed across the entire continental United States, providing macroscopic insight into the effects of geography on crop classification in multi-spectral and temporally distributed satellite imagery. Our method demonstrates improved generalisability from 1) passing geographically correlated climate variables along with the satellite data to a Transformer model and 2) regressing on the model features to reconstruct these domain variables. Combined, we provide a novel perspective on geographic generalisation in satellite imagery and a simple-yet-effective approach to leverage domain knowledge. Code is available at: \url{https://github.com/samar-khanna/cropmap}
△ Less
Submitted 14 April, 2022;
originally announced April 2022.
-
Learning to Detect Mobile Objects from LiDAR Scans Without Labels
Authors:
Yurong You,
Katie Z Luo,
Cheng Perng Phoo,
Wei-Lun Chao,
Wen Sun,
Bharath Hariharan,
Mark Campbell,
Kilian Q. Weinberger
Abstract:
Current 3D object detectors for autonomous driving are almost entirely trained on human-annotated data. Although of high quality, the generation of such data is laborious and costly, restricting them to a few specific locations and object types. This paper proposes an alternative approach entirely based on unlabeled data, which can be collected cheaply and in abundance almost everywhere on earth.…
▽ More
Current 3D object detectors for autonomous driving are almost entirely trained on human-annotated data. Although of high quality, the generation of such data is laborious and costly, restricting them to a few specific locations and object types. This paper proposes an alternative approach entirely based on unlabeled data, which can be collected cheaply and in abundance almost everywhere on earth. Our approach leverages several simple common sense heuristics to create an initial set of approximate seed labels. For example, relevant traffic participants are generally not persistent across multiple traversals of the same route, do not fly, and are never under ground. We demonstrate that these seed labels are highly effective to bootstrap a surprisingly accurate detector through repeated self-training without a single human annotated label.
△ Less
Submitted 29 March, 2022;
originally announced March 2022.
-
Visual Prompt Tuning
Authors:
Menglin Jia,
Luming Tang,
Bor-Chun Chen,
Claire Cardie,
Serge Belongie,
Bharath Hariharan,
Ser-Nam Lim
Abstract:
The current modus operandi in adapting pre-trained models involves updating all the backbone parameters, ie, full fine-tuning. This paper introduces Visual Prompt Tuning (VPT) as an efficient and effective alternative to full fine-tuning for large-scale Transformer models in vision. Taking inspiration from recent advances in efficiently tuning large language models, VPT introduces only a small amo…
▽ More
The current modus operandi in adapting pre-trained models involves updating all the backbone parameters, ie, full fine-tuning. This paper introduces Visual Prompt Tuning (VPT) as an efficient and effective alternative to full fine-tuning for large-scale Transformer models in vision. Taking inspiration from recent advances in efficiently tuning large language models, VPT introduces only a small amount (less than 1% of model parameters) of trainable parameters in the input space while kee** the model backbone frozen. Via extensive experiments on a wide variety of downstream recognition tasks, we show that VPT achieves significant performance gains compared to other parameter efficient tuning protocols. Most importantly, VPT even outperforms full fine-tuning in many cases across model capacities and training data scales, while reducing per-task storage cost.
△ Less
Submitted 20 July, 2022; v1 submitted 22 March, 2022;
originally announced March 2022.
-
Hindsight is 20/20: Leveraging Past Traversals to Aid 3D Perception
Authors:
Yurong You,
Katie Z Luo,
Xiangyu Chen,
Junan Chen,
Wei-Lun Chao,
Wen Sun,
Bharath Hariharan,
Mark Campbell,
Kilian Q. Weinberger
Abstract:
Self-driving cars must detect vehicles, pedestrians, and other traffic participants accurately to operate safely. Small, far-away, or highly occluded objects are particularly challenging because there is limited information in the LiDAR point clouds for detecting them. To address this challenge, we leverage valuable information from the past: in particular, data collected in past traversals of the…
▽ More
Self-driving cars must detect vehicles, pedestrians, and other traffic participants accurately to operate safely. Small, far-away, or highly occluded objects are particularly challenging because there is limited information in the LiDAR point clouds for detecting them. To address this challenge, we leverage valuable information from the past: in particular, data collected in past traversals of the same scene. We posit that these past data, which are typically discarded, provide rich contextual information for disambiguating the above-mentioned challenging cases. To this end, we propose a novel, end-to-end trainable Hindsight framework to extract this contextual information from past traversals and store it in an easy-to-query data structure, which can then be leveraged to aid future 3D object detection of the same scene. We show that this framework is compatible with most modern 3D detection architectures and can substantially improve their average precision on multiple autonomous driving datasets, most notably by more than 300% on the challenging cases.
△ Less
Submitted 21 March, 2022;
originally announced March 2022.
-
Unsupervised Semantic Segmentation by Distilling Feature Correspondences
Authors:
Mark Hamilton,
Zhoutong Zhang,
Bharath Hariharan,
Noah Snavely,
William T. Freeman
Abstract:
Unsupervised semantic segmentation aims to discover and localize semantically meaningful categories within image corpora without any form of annotation. To solve this task, algorithms must produce features for every pixel that are both semantically meaningful and compact enough to form distinct clusters. Unlike previous works which achieve this with a single end-to-end framework, we propose to sep…
▽ More
Unsupervised semantic segmentation aims to discover and localize semantically meaningful categories within image corpora without any form of annotation. To solve this task, algorithms must produce features for every pixel that are both semantically meaningful and compact enough to form distinct clusters. Unlike previous works which achieve this with a single end-to-end framework, we propose to separate feature learning from cluster compactification. Empirically, we show that current unsupervised feature learning frameworks already generate dense features whose correlations are semantically consistent. This observation motivates us to design STEGO ($\textbf{S}$elf-supervised $\textbf{T}$ransformer with $\textbf{E}$nergy-based $\textbf{G}$raph $\textbf{O}$ptimization), a novel framework that distills unsupervised features into high-quality discrete semantic labels. At the core of STEGO is a novel contrastive loss function that encourages features to form compact clusters while preserving their relationships across the corpora. STEGO yields a significant improvement over the prior state of the art, on both the CocoStuff ($\textbf{+14 mIoU}$) and Cityscapes ($\textbf{+9 mIoU}$) semantic segmentation challenges.
△ Less
Submitted 16 March, 2022;
originally announced March 2022.
-
Orientation-Discriminative Feature Representation for Decentralized Pedestrian Tracking
Authors:
Vikram Shree,
Carlos Diaz-Ruiz,
Chang Liu,
Bharath Hariharan,
Mark Campbell
Abstract:
This paper focuses on the problem of decentralized pedestrian tracking using a sensor network. Traditional works on pedestrian tracking usually use a centralized framework, which becomes less practical for robotic applications due to limited communication bandwidth. Our paper proposes a communication-efficient, orientation-discriminative feature representation to characterize pedestrian appearance…
▽ More
This paper focuses on the problem of decentralized pedestrian tracking using a sensor network. Traditional works on pedestrian tracking usually use a centralized framework, which becomes less practical for robotic applications due to limited communication bandwidth. Our paper proposes a communication-efficient, orientation-discriminative feature representation to characterize pedestrian appearance information, that can be shared among sensors. Building upon that representation, our work develops a cross-sensor track association approach to achieve decentralized tracking. Extensive evaluations are conducted on publicly available datasets and results show that our proposed approach leads to improved performance in multi-sensor tracking.
△ Less
Submitted 26 February, 2022;
originally announced February 2022.
-
Stay Positive: Non-Negative Image Synthesis for Augmented Reality
Authors:
Katie Luo,
Guandao Yang,
Wenqi Xian,
Harald Haraldsson,
Bharath Hariharan,
Serge Belongie
Abstract:
In applications such as optical see-through and projector augmented reality, producing images amounts to solving non-negative image generation, where one can only add light to an existing image. Most image generation methods, however, are ill-suited to this problem setting, as they make the assumption that one can assign arbitrary color to each pixel. In fact, naive application of existing methods…
▽ More
In applications such as optical see-through and projector augmented reality, producing images amounts to solving non-negative image generation, where one can only add light to an existing image. Most image generation methods, however, are ill-suited to this problem setting, as they make the assumption that one can assign arbitrary color to each pixel. In fact, naive application of existing methods fails even in simple domains such as MNIST digits, since one cannot create darker pixels by adding light. We know, however, that the human visual system can be fooled by optical illusions involving certain spatial configurations of brightness and contrast. Our key insight is that one can leverage this behavior to produce high quality images with negligible artifacts. For example, we can create the illusion of darker patches by brightening surrounding pixels. We propose a novel optimization procedure to produce images that satisfy both semantic and non-negativity constraints. Our approach can incorporate existing state-of-the-art methods, and exhibits strong performance in a variety of tasks including image-to-image translation and style transfer.
△ Less
Submitted 1 February, 2022;
originally announced February 2022.
-
Parametric analysis of the Lateral Distribution Function of Cherenkov light for Yakutsk EAS Array in the Energy Range 1-20 PeV
Authors:
Falah Al-Zubaidi,
A. A. Al-Rubaiee,
B. Hariharan
Abstract:
In this research, the simulation of lateral distribution function (LDF) of Cherenkov radiation was performed using CORSIKA software for two hadronic models QGSJET and GHEISHA. This simulation was performed for several elementary particles such as protons, iron nuclei, electrons and gamma quanta, in the range of energies 1-20 PeV for three zenith angles 0, 20 and 30 degrees. A parameterization of C…
▽ More
In this research, the simulation of lateral distribution function (LDF) of Cherenkov radiation was performed using CORSIKA software for two hadronic models QGSJET and GHEISHA. This simulation was performed for several elementary particles such as protons, iron nuclei, electrons and gamma quanta, in the range of energies 1-20 PeV for three zenith angles 0, 20 and 30 degrees. A parameterization of Cherenkov light LDF was performed for that simulated curves using Lorentzian function. The comparison between the obtained results for LDF of Cherenkov light with that measured with Yakutsk EAS array gave a good agreement within the distances of 100-1000 m from the shower axis.
△ Less
Submitted 16 December, 2021;
originally announced January 2022.
-
Field-Guide-Inspired Zero-Shot Learning
Authors:
Utkarsh Mall,
Bharath Hariharan,
Kavita Bala
Abstract:
Modern recognition systems require large amounts of supervision to achieve accuracy. Adapting to new domains requires significant data from experts, which is onerous and can become too expensive. Zero-shot learning requires an annotated set of attributes for a novel category. Annotating the full set of attributes for a novel category proves to be a tedious and expensive task in deployment. This is…
▽ More
Modern recognition systems require large amounts of supervision to achieve accuracy. Adapting to new domains requires significant data from experts, which is onerous and can become too expensive. Zero-shot learning requires an annotated set of attributes for a novel category. Annotating the full set of attributes for a novel category proves to be a tedious and expensive task in deployment. This is especially the case when the recognition domain is an expert domain. We introduce a new field-guide-inspired approach to zero-shot annotation where the learner model interactively asks for the most useful attributes that define a class. We evaluate our method on classification benchmarks with attribute annotations like CUB, SUN, and AWA2 and show that our model achieves the performance of a model with full annotations at the cost of a significantly fewer number of annotations. Since the time of experts is precious, decreasing annotation cost can be very valuable for real-world deployment.
△ Less
Submitted 24 August, 2021;
originally announced August 2021.
-
Extreme Rotation Estimation using Dense Correlation Volumes
Authors:
Ruo** Cai,
Bharath Hariharan,
Noah Snavely,
Hadar Averbuch-Elor
Abstract:
We present a technique for estimating the relative 3D rotation of an RGB image pair in an extreme setting, where the images have little or no overlap. We observe that, even when images do not overlap, there may be rich hidden cues as to their geometric relationship, such as light source directions, vanishing points, and symmetries present in the scene. We propose a network design that can automati…
▽ More
We present a technique for estimating the relative 3D rotation of an RGB image pair in an extreme setting, where the images have little or no overlap. We observe that, even when images do not overlap, there may be rich hidden cues as to their geometric relationship, such as light source directions, vanishing points, and symmetries present in the scene. We propose a network design that can automatically learn such implicit cues by comparing all pairs of points between the two input images. Our method therefore constructs dense feature correlation volumes and processes these to predict relative 3D rotations. Our predictions are formed over a fine-grained discretization of rotations, bypassing difficulties associated with regressing 3D rotations. We demonstrate our approach on a large variety of extreme RGB image pairs, including indoor and outdoor images captured under different lighting conditions and geographic locations. Our evaluation shows that our model can successfully estimate relative rotations among non-overlap** images without compromising performance over overlap** image pairs.
△ Less
Submitted 19 July, 2021; v1 submitted 27 April, 2021;
originally announced April 2021.
-
PiCIE: Unsupervised Semantic Segmentation using Invariance and Equivariance in Clustering
Authors:
Jang Hyun Cho,
Utkarsh Mall,
Kavita Bala,
Bharath Hariharan
Abstract:
We present a new framework for semantic segmentation without annotations via clustering. Off-the-shelf clustering methods are limited to curated, single-label, and object-centric images yet real-world data are dominantly uncurated, multi-label, and scene-centric. We extend clustering from images to pixels and assign separate cluster membership to different instances within each image. However, sol…
▽ More
We present a new framework for semantic segmentation without annotations via clustering. Off-the-shelf clustering methods are limited to curated, single-label, and object-centric images yet real-world data are dominantly uncurated, multi-label, and scene-centric. We extend clustering from images to pixels and assign separate cluster membership to different instances within each image. However, solely relying on pixel-wise feature similarity fails to learn high-level semantic concepts and overfits to low-level visual cues. We propose a method to incorporate geometric consistency as an inductive bias to learn invariance and equivariance for photometric and geometric variations. With our novel learning objective, our framework can learn high-level semantic concepts. Our method, PiCIE (Pixel-level feature Clustering using Invariance and Equivariance), is the first method capable of segmenting both things and stuff categories without any hyperparameter tuning or task-specific pre-processing. Our method largely outperforms existing baselines on COCO and Cityscapes with +17.5 Acc. and +4.5 mIoU. We show that PiCIE gives a better initialization for standard supervised training. The code is available at https://github.com/janghyuncho/PiCIE.
△ Less
Submitted 29 March, 2021;
originally announced March 2021.
-
Exploiting Playbacks in Unsupervised Domain Adaptation for 3D Object Detection
Authors:
Yurong You,
Carlos Andres Diaz-Ruiz,
Yan Wang,
Wei-Lun Chao,
Bharath Hariharan,
Mark Campbell,
Kilian Q Weinberger
Abstract:
Self-driving cars must detect other vehicles and pedestrians in 3D to plan safe routes and avoid collisions. State-of-the-art 3D object detectors, based on deep learning, have shown promising accuracy but are prone to over-fit to domain idiosyncrasies, making them fail in new environments -- a serious problem if autonomous vehicles are meant to operate freely. In this paper, we propose a novel lea…
▽ More
Self-driving cars must detect other vehicles and pedestrians in 3D to plan safe routes and avoid collisions. State-of-the-art 3D object detectors, based on deep learning, have shown promising accuracy but are prone to over-fit to domain idiosyncrasies, making them fail in new environments -- a serious problem if autonomous vehicles are meant to operate freely. In this paper, we propose a novel learning approach that drastically reduces this gap by fine-tuning the detector on pseudo-labels in the target domain, which our method generates while the vehicle is parked, based on replays of previously recorded driving sequences. In these replays, objects are tracked over time, and detections are interpolated and extrapolated -- crucially, leveraging future information to catch hard cases. We show, on five autonomous driving datasets, that fine-tuning the object detector on these pseudo-labels substantially reduces the domain gap to new driving environments, yielding drastic improvements in accuracy and detection reliability.
△ Less
Submitted 10 July, 2022; v1 submitted 25 March, 2021;
originally announced March 2021.
-
Few-Shot Classification with Feature Map Reconstruction Networks
Authors:
Davis Wertheimer,
Luming Tang,
Bharath Hariharan
Abstract:
In this paper we reformulate few-shot classification as a reconstruction problem in latent space. The ability of the network to reconstruct a query feature map from support features of a given class predicts membership of the query in that class. We introduce a novel mechanism for few-shot classification by regressing directly from support features to query features in closed form, without introdu…
▽ More
In this paper we reformulate few-shot classification as a reconstruction problem in latent space. The ability of the network to reconstruct a query feature map from support features of a given class predicts membership of the query in that class. We introduce a novel mechanism for few-shot classification by regressing directly from support features to query features in closed form, without introducing any new modules or large-scale learnable parameters. The resulting Feature Map Reconstruction Networks are both more performant and computationally efficient than previous approaches. We demonstrate consistent and substantial accuracy gains on four fine-grained benchmarks with varying neural architectures. Our model is also competitive on the non-fine-grained mini-ImageNet and tiered-ImageNet benchmarks with minimal bells and whistles.
△ Less
Submitted 27 April, 2021; v1 submitted 2 December, 2020;
originally announced December 2020.
-
Augmentation-Interpolative AutoEncoders for Unsupervised Few-Shot Image Generation
Authors:
Davis Wertheimer,
Omid Poursaeed,
Bharath Hariharan
Abstract:
We aim to build image generation models that generalize to new domains from few examples. To this end, we first investigate the generalization properties of classic image generators, and discover that autoencoders generalize extremely well to new domains, even when trained on highly constrained data. We leverage this insight to produce a robust, unsupervised few-shot image generation algorithm, an…
▽ More
We aim to build image generation models that generalize to new domains from few examples. To this end, we first investigate the generalization properties of classic image generators, and discover that autoencoders generalize extremely well to new domains, even when trained on highly constrained data. We leverage this insight to produce a robust, unsupervised few-shot image generation algorithm, and introduce a novel training procedure based on recovering an image from data augmentations. Our Augmentation-Interpolative AutoEncoders synthesize realistic images of novel objects from only a few reference images, and outperform both prior interpolative models and supervised few-shot image generators. Our procedure is simple and lightweight, generalizes broadly, and requires no category labels or other supervision during training.
△ Less
Submitted 25 November, 2020;
originally announced November 2020.
-
Self-training for Few-shot Transfer Across Extreme Task Differences
Authors:
Cheng Perng Phoo,
Bharath Hariharan
Abstract:
Most few-shot learning techniques are pre-trained on a large, labeled "base dataset". In problem domains where such large labeled datasets are not available for pre-training (e.g., X-ray, satellite images), one must resort to pre-training in a different "source" problem domain (e.g., ImageNet), which can be very different from the desired target task. Traditional few-shot and transfer learning tec…
▽ More
Most few-shot learning techniques are pre-trained on a large, labeled "base dataset". In problem domains where such large labeled datasets are not available for pre-training (e.g., X-ray, satellite images), one must resort to pre-training in a different "source" problem domain (e.g., ImageNet), which can be very different from the desired target task. Traditional few-shot and transfer learning techniques fail in the presence of such extreme differences between the source and target tasks. In this paper, we present a simple and effective solution to tackle this extreme domain gap: self-training a source domain representation on unlabeled data from the target domain. We show that this improves one-shot performance on the target domain by 2.9 points on average on the challenging BSCD-FSL benchmark consisting of datasets from multiple domains. Our code is available at https://github.com/cpphoo/STARTUP.
△ Less
Submitted 17 March, 2021; v1 submitted 15 October, 2020;
originally announced October 2020.
-
Learning Gradient Fields for Shape Generation
Authors:
Ruo** Cai,
Guandao Yang,
Hadar Averbuch-Elor,
Zekun Hao,
Serge Belongie,
Noah Snavely,
Bharath Hariharan
Abstract:
In this work, we propose a novel technique to generate shapes from point cloud data. A point cloud can be viewed as samples from a distribution of 3D points whose density is concentrated near the surface of the shape. Point cloud generation thus amounts to moving randomly sampled points to high-density areas. We generate point clouds by performing stochastic gradient ascent on an unnormalized prob…
▽ More
In this work, we propose a novel technique to generate shapes from point cloud data. A point cloud can be viewed as samples from a distribution of 3D points whose density is concentrated near the surface of the shape. Point cloud generation thus amounts to moving randomly sampled points to high-density areas. We generate point clouds by performing stochastic gradient ascent on an unnormalized probability density, thereby moving sampled points toward the high-likelihood regions. Our model directly predicts the gradient of the log density field and can be trained with a simple objective adapted from score-based generative models. We show that our method can reach state-of-the-art performance for point cloud auto-encoding and generation, while also allowing for extraction of a high-quality implicit surface. Code is available at https://github.com/Ruo**Cai/ShapeGF.
△ Less
Submitted 18 August, 2020; v1 submitted 14 August, 2020;
originally announced August 2020.
-
Wasserstein Distances for Stereo Disparity Estimation
Authors:
Divyansh Garg,
Yan Wang,
Bharath Hariharan,
Mark Campbell,
Kilian Q. Weinberger,
Wei-Lun Chao
Abstract:
Existing approaches to depth or disparity estimation output a distribution over a set of pre-defined discrete values. This leads to inaccurate results when the true depth or disparity does not match any of these values. The fact that this distribution is usually learned indirectly through a regression loss causes further problems in ambiguous regions around object boundaries. We address these issu…
▽ More
Existing approaches to depth or disparity estimation output a distribution over a set of pre-defined discrete values. This leads to inaccurate results when the true depth or disparity does not match any of these values. The fact that this distribution is usually learned indirectly through a regression loss causes further problems in ambiguous regions around object boundaries. We address these issues using a new neural network architecture that is capable of outputting arbitrary depth values, and a new loss function that is derived from the Wasserstein distance between the true and the predicted distributions. We validate our approach on a variety of tasks, including stereo disparity and depth estimation, and the downstream 3D object detection. Our approach drastically reduces the error in ambiguous regions, especially around object boundaries that greatly affect the localization of objects in 3D, achieving the state-of-the-art in 3D object detection for autonomous driving. Our code will be available at https://github.com/Div99/W-Stereo-Disp.
△ Less
Submitted 29 March, 2021; v1 submitted 6 July, 2020;
originally announced July 2020.
-
Train in Germany, Test in The USA: Making 3D Object Detectors Generalize
Authors:
Yan Wang,
Xiangyu Chen,
Yurong You,
Li Erran,
Bharath Hariharan,
Mark Campbell,
Kilian Q. Weinberger,
Wei-Lun Chao
Abstract:
In the domain of autonomous driving, deep learning has substantially improved the 3D object detection accuracy for LiDAR and stereo camera data alike. While deep networks are great at generalization, they are also notorious to over-fit to all kinds of spurious artifacts, such as brightness, car sizes and models, that may appear consistently throughout the data. In fact, most datasets for autonomou…
▽ More
In the domain of autonomous driving, deep learning has substantially improved the 3D object detection accuracy for LiDAR and stereo camera data alike. While deep networks are great at generalization, they are also notorious to over-fit to all kinds of spurious artifacts, such as brightness, car sizes and models, that may appear consistently throughout the data. In fact, most datasets for autonomous driving are collected within a narrow subset of cities within one country, typically under similar weather conditions. In this paper we consider the task of adapting 3D object detectors from one dataset to another. We observe that naively, this appears to be a very challenging task, resulting in drastic drops in accuracy levels. We provide extensive experiments to investigate the true adaptation challenges and arrive at a surprising conclusion: the primary adaptation hurdle to overcome are differences in car sizes across geographic areas. A simple correction based on the average car size yields a strong correction of the adaptation gap. Our proposed method is simple and easily incorporated into most 3D object detection frameworks. It provides a first baseline for 3D object detection adaptation across countries, and gives hope that the underlying problem may be more within grasp than one may have hoped to believe. Our code is available at https://github.com/cxy1997/3D_adapt_auto_driving.
△ Less
Submitted 16 May, 2020;
originally announced May 2020.
-
Energy sensitivity of the GRAPES-3 EAS array for primary cosmic ray protons
Authors:
B. Hariharan,
S. Ahmad,
M. Chakraborty,
A. Chandra,
S. R. Dugad,
S. K. Gupta,
Y. Hayashi,
H. Kojima,
S. S. R. Inbanathan,
P. Jagadeesan,
A. Jain,
P. Jain,
V. B. Jhansi,
S. Kawakami,
P. K. Mohanty,
S. D. Morris,
P. K. Nayak,
A. Oshima,
D. Pattanaik,
P. S. Rakshe,
K. Ramesh,
B. S. Rao,
L. V. Reddy,
S. Shibata,
F. Varsi
, et al. (1 additional authors not shown)
Abstract:
Low energy ground-based cosmic ray air shower experiments generally have energy threshold in the range of a few tens to a few hundreds of TeV. The shower observables are measured indirectly with an array of detectors. The atmospheric absorption of low energy secondaries limits their detection frequencies at the Earth's surface. However, due to selection effects, a tiny fraction of low energy showe…
▽ More
Low energy ground-based cosmic ray air shower experiments generally have energy threshold in the range of a few tens to a few hundreds of TeV. The shower observables are measured indirectly with an array of detectors. The atmospheric absorption of low energy secondaries limits their detection frequencies at the Earth's surface. However, due to selection effects, a tiny fraction of low energy showers, which are produced in the lower atmosphere can reach the observational level. But, due to less information of shower observables, the reconstruction of these showers are arduous. Hence, it is believed that direct measurements by experiments aboard on satellites and balloon flights are more reliable at low energies. Despite having very small efficiency ($\sim$0.1%) at low energies, the large acceptance ($\sim$5 m$^2$sr) of GRAPES-3 experiment allows observing primary cosmic rays down below to $\sim$1 TeV and opens up the possibility to measure primary energy spectrum spanning from a few TeV to beyond cosmic ray knee (up to 10$^{16}$ eV), covering five orders of magnitude. The GRAPES-3 energy threshold for primary protons through Monte Carlo simulations are calculated, which gives reasonably good agreement with data. Furthermore, the total efficiencies and acceptance are also calculated for protons primaries. The ability of GRAPES-3 experiment to cover such a broader energy range may provide a unique handle to bridge the energy spectrum between direct measurements at low energies and indirect measurements at ultra-high energies.
△ Less
Submitted 3 May, 2020;
originally announced May 2020.
-
Learning Feature Descriptors using Camera Pose Supervision
Authors:
Qianqian Wang,
Xiaowei Zhou,
Bharath Hariharan,
Noah Snavely
Abstract:
Recent research on learned visual descriptors has shown promising improvements in correspondence estimation, a key component of many 3D vision tasks. However, existing descriptor learning frameworks typically require ground-truth correspondences between feature points for training, which are challenging to acquire at scale. In this paper we propose a novel weakly-supervised framework that can lear…
▽ More
Recent research on learned visual descriptors has shown promising improvements in correspondence estimation, a key component of many 3D vision tasks. However, existing descriptor learning frameworks typically require ground-truth correspondences between feature points for training, which are challenging to acquire at scale. In this paper we propose a novel weakly-supervised framework that can learn feature descriptors solely from relative camera poses between images. To do so, we devise both a new loss function that exploits the epipolar constraint given by camera poses, and a new model architecture that makes the whole pipeline differentiable and efficient. Because we no longer need pixel-level ground-truth correspondences, our framework opens up the possibility of training on much larger and more diverse datasets for better and unbiased descriptors. We call the resulting descriptors CAmera Pose Supervised, or CAPS, descriptors. Though trained with weak supervision, CAPS descriptors outperform even prior fully-supervised descriptors and achieve state-of-the-art performance on a variety of geometric tasks. Project Page: https://qianqianwang68.github.io/CAPS/
△ Less
Submitted 29 January, 2024; v1 submitted 28 April, 2020;
originally announced April 2020.
-
Fashionpedia: Ontology, Segmentation, and an Attribute Localization Dataset
Authors:
Menglin Jia,
Mengyun Shi,
Mikhail Sirotenko,
Yin Cui,
Claire Cardie,
Bharath Hariharan,
Hartwig Adam,
Serge Belongie
Abstract:
In this work we explore the task of instance segmentation with attribute localization, which unifies instance segmentation (detect and segment each object instance) and fine-grained visual attribute categorization (recognize one or multiple attributes). The proposed task requires both localizing an object and describing its properties. To illustrate the various aspects of this task, we focus on th…
▽ More
In this work we explore the task of instance segmentation with attribute localization, which unifies instance segmentation (detect and segment each object instance) and fine-grained visual attribute categorization (recognize one or multiple attributes). The proposed task requires both localizing an object and describing its properties. To illustrate the various aspects of this task, we focus on the domain of fashion and introduce Fashionpedia as a step toward map** out the visual aspects of the fashion world. Fashionpedia consists of two parts: (1) an ontology built by fashion experts containing 27 main apparel categories, 19 apparel parts, 294 fine-grained attributes and their relationships; (2) a dataset with everyday and celebrity event fashion images annotated with segmentation masks and their associated per-mask fine-grained attributes, built upon the Fashionpedia ontology. In order to solve this challenging task, we propose a novel Attribute-Mask RCNN model to jointly perform instance segmentation and localized attribute recognition, and provide a novel evaluation metric for the task. We also demonstrate instance segmentation models pre-trained on Fashionpedia achieve better transfer learning performance on other fashion datasets than ImageNet pre-training. Fashionpedia is available at: https://fashionpedia.github.io/home/index.html.
△ Less
Submitted 18 July, 2020; v1 submitted 25 April, 2020;
originally announced April 2020.
-
Extending and Analyzing Self-Supervised Learning Across Domains
Authors:
Bram Wallace,
Bharath Hariharan
Abstract:
Self-supervised representation learning has achieved impressive results in recent years, with experiments primarily coming on ImageNet or other similarly large internet imagery datasets. There has been little to no work with these methods on other smaller domains, such as satellite, textural, or biological imagery. We experiment with several popular methods on an unprecedented variety of domains.…
▽ More
Self-supervised representation learning has achieved impressive results in recent years, with experiments primarily coming on ImageNet or other similarly large internet imagery datasets. There has been little to no work with these methods on other smaller domains, such as satellite, textural, or biological imagery. We experiment with several popular methods on an unprecedented variety of domains. We discover, among other findings, that Rotation is by far the most semantically meaningful task, with much of the performance of Jigsaw and Instance Discrimination being attributable to the nature of their induced distribution rather than semantic understanding. Additionally, there are several areas, such as fine-grain classification, where all tasks underperform. We quantitatively and qualitatively diagnose the reasons for these failures and successes via novel experiments studying pretext generalization, random labelings, and implicit dimensionality. Code and models are available at https://github.com/BramSW/Extending_SSRL_Across_Domains/.
△ Less
Submitted 17 August, 2020; v1 submitted 24 April, 2020;
originally announced April 2020.
-
End-to-End Pseudo-LiDAR for Image-Based 3D Object Detection
Authors:
Rui Qian,
Divyansh Garg,
Yan Wang,
Yurong You,
Serge Belongie,
Bharath Hariharan,
Mark Campbell,
Kilian Q. Weinberger,
Wei-Lun Chao
Abstract:
Reliable and accurate 3D object detection is a necessity for safe autonomous driving. Although LiDAR sensors can provide accurate 3D point cloud estimates of the environment, they are also prohibitively expensive for many settings. Recently, the introduction of pseudo-LiDAR (PL) has led to a drastic reduction in the accuracy gap between methods based on LiDAR sensors and those based on cheap stere…
▽ More
Reliable and accurate 3D object detection is a necessity for safe autonomous driving. Although LiDAR sensors can provide accurate 3D point cloud estimates of the environment, they are also prohibitively expensive for many settings. Recently, the introduction of pseudo-LiDAR (PL) has led to a drastic reduction in the accuracy gap between methods based on LiDAR sensors and those based on cheap stereo cameras. PL combines state-of-the-art deep neural networks for 3D depth estimation with those for 3D object detection by converting 2D depth map outputs to 3D point cloud inputs. However, so far these two networks have to be trained separately. In this paper, we introduce a new framework based on differentiable Change of Representation (CoR) modules that allow the entire PL pipeline to be trained end-to-end. The resulting framework is compatible with most state-of-the-art networks for both tasks and in combination with PointRCNN improves over PL consistently across all benchmarks -- yielding the highest entry on the KITTI image-based 3D object detection leaderboard at the time of submission. Our code will be made available at https://github.com/mileyan/pseudo-LiDAR_e2e.
△ Less
Submitted 14 May, 2020; v1 submitted 6 April, 2020;
originally announced April 2020.
-
Revisiting Pose-Normalization for Fine-Grained Few-Shot Recognition
Authors:
Luming Tang,
Davis Wertheimer,
Bharath Hariharan
Abstract:
Few-shot, fine-grained classification requires a model to learn subtle, fine-grained distinctions between different classes (e.g., birds) based on a few images alone. This requires a remarkable degree of invariance to pose, articulation and background. A solution is to use pose-normalized representations: first localize semantic parts in each image, and then describe images by characterizing the a…
▽ More
Few-shot, fine-grained classification requires a model to learn subtle, fine-grained distinctions between different classes (e.g., birds) based on a few images alone. This requires a remarkable degree of invariance to pose, articulation and background. A solution is to use pose-normalized representations: first localize semantic parts in each image, and then describe images by characterizing the appearance of each part. While such representations are out of favor for fully supervised classification, we show that they are extremely effective for few-shot fine-grained classification. With a minimal increase in model capacity, pose normalization improves accuracy between 10 and 20 percentage points for shallow and deep architectures, generalizes better to new domains, and is effective for multiple few-shot algorithms and network backbones. Code is available at https://github.com/Tsingularity/PoseNorm_Fewshot
△ Less
Submitted 1 April, 2020;
originally announced April 2020.
-
The angular resolution of GRAPES-3 EAS array after correction for the shower front curvature
Authors:
V. B. Jhansi,
S. Ahmad,
M. Chakraborty,
S. R. Dugad,
S. K. Gupta,
B. Hariharan,
Y. Hayashi,
P. Jagadeesan,
A. Jain,
P. Jain,
S. Kawakami,
H. Kojima,
S. Mahapatra,
P. K. Mohanty,
S. D. Morris,
P. K. Nayak,
A. Oshima,
D. Pattanaik,
P. S. Rakshe,
K. Ramesh,
B. S. Rao,
L. V. Reddy,
S. Shibata,
F. Varsi,
M. Zuberi
Abstract:
The angular resolution of an extensive air shower (EAS) array plays a critical role in determining its sensitivity for the detection of point $γ$-ray sources in the multi-TeV energy range. The GRAPES-3 an EAS array located at Ooty in India (11.4$^{\circ}$N, 76.7$^{\circ}$E, 2200 m altitude) is designed to study $γ$-rays in the TeV-PeV energy range. It comprises of a dense array of 400 plastic scin…
▽ More
The angular resolution of an extensive air shower (EAS) array plays a critical role in determining its sensitivity for the detection of point $γ$-ray sources in the multi-TeV energy range. The GRAPES-3 an EAS array located at Ooty in India (11.4$^{\circ}$N, 76.7$^{\circ}$E, 2200 m altitude) is designed to study $γ$-rays in the TeV-PeV energy range. It comprises of a dense array of 400 plastic scintillators deployed over an area of 25000 m$^2$ and a large area (560 m$^2$) muon telescope. A new statistical method allowed real time determination of the propagation delay of each detector in the GRAPES-3 array. The shape of shower front is known to be curved and here the details of a new method developed for accurate measurement of the shower front curvature is presented. These two developments have led to a sizable improvement in the angular resolution of GRAPES-3 array. It is shown that the curvature depends on the size and age of an EAS. By employing two different techniques, namely, the odd-even and the left-right methods, independent estimates of the angular resolution are obtained. The odd-even method estimates the best achievable resolution of the array. For obtaining the angular resolution, the left-right method is used after implementing the size and age dependent curvature corrections. A comparison of the angular resolution as a function of EAS energy by these two methods shows them be virtually indistinguishable. The angular resolution of GRAPES-3 array is 47$^{\prime}$ for energies E$>$5 TeV and improves to 17$^{\prime}$ at E$>$100 TeV and finally approaching 10$^{\prime}$ at E$>$500 TeV.
△ Less
Submitted 12 November, 2019;
originally announced November 2019.
-
LDLS: 3-D Object Segmentation Through Label Diffusion From 2-D Images
Authors:
Brian H. Wang,
Wei-Lun Chao,
Yan Wang,
Bharath Hariharan,
Kilian Q. Weinberger,
Mark Campbell
Abstract:
Object segmentation in three-dimensional (3-D) point clouds is a critical task for robots capable of 3-D perception. Despite the impressive performance of deep learning-based approaches on object segmentation in 2-D images, deep learning has not been applied nearly as successfully for 3-D point cloud segmentation. Deep networks generally require large amounts of labeled training data, which are re…
▽ More
Object segmentation in three-dimensional (3-D) point clouds is a critical task for robots capable of 3-D perception. Despite the impressive performance of deep learning-based approaches on object segmentation in 2-D images, deep learning has not been applied nearly as successfully for 3-D point cloud segmentation. Deep networks generally require large amounts of labeled training data, which are readily available for 2-D images but are difficult to produce for 3-D point clouds. In this letter, we present Label Diffusion Lidar Segmentation (LDLS), a novel approach for 3-D point cloud segmentation, which leverages 2-D segmentation of an RGB image from an aligned camera to avoid the need for training on annotated 3-D data. We obtain 2-D segmentation predictions by applying Mask-RCNN to the RGB image, and then link this image to a 3-D lidar point cloud by building a graph of connections among 3-D points and 2-D pixels. This graph then directs a semi-supervised label diffusion process, where the 2-D pixels act as source nodes that diffuse object label information through the 3-D point cloud, resulting in a complete 3-D point cloud segmentation. We conduct empirical studies on the KITTI benchmark dataset and on a mobile robot, demonstrating wide applicability and superior performance of LDLS compared with the previous state of the art in 3-D point cloud segmentation, without any need for either 3-D training data or fine tuning of the 2-D image segmentation model.
△ Less
Submitted 30 October, 2019;
originally announced October 2019.
-
When Does Self-supervision Improve Few-shot Learning?
Authors:
Jong-Chyi Su,
Subhransu Maji,
Bharath Hariharan
Abstract:
We investigate the role of self-supervised learning (SSL) in the context of few-shot learning. Although recent research has shown the benefits of SSL on large unlabeled datasets, its utility on small datasets is relatively unexplored. We find that SSL reduces the relative error rate of few-shot meta-learners by 4%-27%, even when the datasets are small and only utilizing images within the datasets.…
▽ More
We investigate the role of self-supervised learning (SSL) in the context of few-shot learning. Although recent research has shown the benefits of SSL on large unlabeled datasets, its utility on small datasets is relatively unexplored. We find that SSL reduces the relative error rate of few-shot meta-learners by 4%-27%, even when the datasets are small and only utilizing images within the datasets. The improvements are greater when the training set is smaller or the task is more challenging. Although the benefits of SSL may increase with larger training sets, we observe that SSL can hurt the performance when the distributions of images used for meta-learning and SSL are different. We conduct a systematic study by varying the degree of domain shift and analyzing the performance of several meta-learners on a multitude of domains. Based on this analysis we present a technique that automatically selects images for SSL from a large, generic pool of unlabeled images for a given dataset that provides further improvements.
△ Less
Submitted 30 July, 2020; v1 submitted 8 October, 2019;
originally announced October 2019.
-
On the Efficacy of Knowledge Distillation
Authors:
Jang Hyun Cho,
Bharath Hariharan
Abstract:
In this paper, we present a thorough evaluation of the efficacy of knowledge distillation and its dependence on student and teacher architectures. Starting with the observation that more accurate teachers often don't make good teachers, we attempt to tease apart the factors that affect knowledge distillation performance. We find crucially that larger models do not often make better teachers. We sh…
▽ More
In this paper, we present a thorough evaluation of the efficacy of knowledge distillation and its dependence on student and teacher architectures. Starting with the observation that more accurate teachers often don't make good teachers, we attempt to tease apart the factors that affect knowledge distillation performance. We find crucially that larger models do not often make better teachers. We show that this is a consequence of mismatched capacity, and that small students are unable to mimic large teachers. We find typical ways of circumventing this (such as performing a sequence of knowledge distillation steps) to be ineffective. Finally, we show that this effect can be mitigated by stop** the teacher's training early. Our results generalize across datasets and models.
△ Less
Submitted 3 October, 2019;
originally announced October 2019.
-
Few-Shot Generalization for Single-Image 3D Reconstruction via Priors
Authors:
Bram Wallace,
Bharath Hariharan
Abstract:
Recent work on single-view 3D reconstruction shows impressive results, but has been restricted to a few fixed categories where extensive training data is available. The problem of generalizing these models to new classes with limited training data is largely open. To address this problem, we present a new model architecture that reframes single-view 3D reconstruction as learnt, category agnostic r…
▽ More
Recent work on single-view 3D reconstruction shows impressive results, but has been restricted to a few fixed categories where extensive training data is available. The problem of generalizing these models to new classes with limited training data is largely open. To address this problem, we present a new model architecture that reframes single-view 3D reconstruction as learnt, category agnostic refinement of a provided, category-specific prior. The provided prior shape for a novel class can be obtained from as few as one 3D shape from this class. Our model can start reconstructing objects from the novel class using this prior without seeing any training image for this class and without any retraining. Our model outperforms category-agnostic baselines and remains competitive with more sophisticated baselines that finetune on the novel categories. Additionally, our network is capable of improving the reconstruction given multiple views despite not being trained on task of multi-view reconstruction.
△ Less
Submitted 3 September, 2019;
originally announced September 2019.