Learning multiplane images from single views with self-supervision
Authors:
Gustavo Sutter P. Carvalho,
Diogo C. Luvizon,
Antonio Joia Neto,
Andre G. C. Pacheco,
Otavio A. B. Penatti
Abstract:
Generating static novel views from an already captured image is a hard task in computer vision and graphics, in particular when the single input image has dynamic parts such as persons or moving objects. In this paper, we tackle this problem by proposing a new framework, called CycleMPI, that is capable of learning a multiplane image representation from single images through a cyclic training stra…
▽ More
Generating static novel views from an already captured image is a hard task in computer vision and graphics, in particular when the single input image has dynamic parts such as persons or moving objects. In this paper, we tackle this problem by proposing a new framework, called CycleMPI, that is capable of learning a multiplane image representation from single images through a cyclic training strategy for self-supervision. Our framework does not require stereo data for training, therefore it can be trained with massive visual data from the Internet, resulting in a better generalization capability even for very challenging cases. Although our method does not require stereo data for supervision, it reaches results on stereo datasets comparable to the state of the art in a zero-shot scenario. We evaluated our method on RealEstate10K and Mannequin Challenge datasets for view synthesis and presented qualitative results on Places II dataset.
△ Less
Submitted 19 October, 2021; v1 submitted 18 October, 2021;
originally announced October 2021.
Adaptive Multiplane Image Generation from a Single Internet Picture
Authors:
Diogo C. Luvizon,
Gustavo Sutter P. Carvalho,
Andreza A. dos Santos,
Jhonatas S. Conceicao,
Jose L. Flores-Campana,
Luis G. L. Decker,
Marcos R. Souza,
Helio Pedrini,
Antonio Joia,
Otavio A. B. Penatti
Abstract:
In the last few years, several works have tackled the problem of novel view synthesis from stereo images or even from a single picture. However, previous methods are computationally expensive, specially for high-resolution images. In this paper, we address the problem of generating a multiplane image (MPI) from a single high-resolution picture. We present the adaptive-MPI representation, which all…
▽ More
In the last few years, several works have tackled the problem of novel view synthesis from stereo images or even from a single picture. However, previous methods are computationally expensive, specially for high-resolution images. In this paper, we address the problem of generating a multiplane image (MPI) from a single high-resolution picture. We present the adaptive-MPI representation, which allows rendering novel views with low computational requirements. To this end, we propose an adaptive slicing algorithm that produces an MPI with a variable number of image planes. We present a new lightweight CNN for depth estimation, which is learned by knowledge distillation from a larger network. Occluded regions in the adaptive-MPI are inpainted also by a lightweight CNN. We show that our method is capable of producing high-quality predictions with one order of magnitude less parameters compared to previous approaches. The robustness of our method is evidenced on challenging pictures from the Internet.
△ Less
Submitted 26 November, 2020;
originally announced November 2020.
