Showing 1–25 of 25 results for author: Kopf, J

Modeling Ambient Scene Dynamics for Free-view Synthesis

Authors: Meng-Li Shih, Jia-Bin Huang, Changil Kim, Rajvi Shah, Johannes Kopf, Chen Gao

Abstract: We introduce a novel method for dynamic free-view synthesis of an ambient scenes from a monocular capture bringing a immersive quality to the viewing experience. Our method builds upon the recent advancements in 3D Gaussian Splatting (3DGS) that can faithfully reconstruct complex static scenes. Previous attempts to extend 3DGS to represent dynamics have been confined to bounded scenes or require m… ▽ More We introduce a novel method for dynamic free-view synthesis of an ambient scenes from a monocular capture bringing a immersive quality to the viewing experience. Our method builds upon the recent advancements in 3D Gaussian Splatting (3DGS) that can faithfully reconstruct complex static scenes. Previous attempts to extend 3DGS to represent dynamics have been confined to bounded scenes or require multi-camera captures, and often fail to generalize to unseen motions, limiting their practical application. Our approach overcomes these constraints by leveraging the periodicity of ambient motions to learn the motion trajectory model, coupled with careful regularization. We also propose important practical strategies to improve the visual quality of the baseline 3DGS static reconstructions and to improve memory efficiency critical for GPU-memory intensive learning. We demonstrate high-quality photorealistic novel view synthesis of several ambient natural scenes with intricate textures and fine structural elements. △ Less

Submitted 13 June, 2024; originally announced June 2024.

Taming Latent Diffusion Model for Neural Radiance Field Inpainting

Authors: Chieh Hubert Lin, Changil Kim, Jia-Bin Huang, Qinbo Li, Chih-Yao Ma, Johannes Kopf, Ming-Hsuan Yang, Hung-Yu Tseng

Abstract: Neural Radiance Field (NeRF) is a representation for 3D reconstruction from multi-view images. Despite some recent work showing preliminary success in editing a reconstructed NeRF with diffusion prior, they remain struggling to synthesize reasonable geometry in completely uncovered regions. One major reason is the high diversity of synthetic contents from the diffusion model, which hinders the rad… ▽ More Neural Radiance Field (NeRF) is a representation for 3D reconstruction from multi-view images. Despite some recent work showing preliminary success in editing a reconstructed NeRF with diffusion prior, they remain struggling to synthesize reasonable geometry in completely uncovered regions. One major reason is the high diversity of synthetic contents from the diffusion model, which hinders the radiance field from converging to a crisp and deterministic geometry. Moreover, applying latent diffusion models on real data often yields a textural shift incoherent to the image condition due to auto-encoding errors. These two problems are further reinforced with the use of pixel-distance losses. To address these issues, we propose tempering the diffusion model's stochasticity with per-scene customization and mitigating the textural shift with masked adversarial training. During the analyses, we also found the commonly used pixel and perceptual losses are harmful in the NeRF inpainting task. Through rigorous experiments, our framework yields state-of-the-art NeRF inpainting results on various real-world scenes. Project page: https://hubert0527.github.io/MALD-NeRF △ Less

Submitted 15 April, 2024; originally announced April 2024.

Comments: Project page: https://hubert0527.github.io/MALD-NeRF

IRIS: Inverse Rendering of Indoor Scenes from Low Dynamic Range Images

Authors: Zhi-Hao Lin, Jia-Bin Huang, Zhengqin Li, Zhao Dong, Christian Richardt, Tuotuo Li, Michael Zollhöfer, Johannes Kopf, Shenlong Wang, Changil Kim

Abstract: While numerous 3D reconstruction and novel-view synthesis methods allow for photorealistic rendering of a scene from multi-view images easily captured with consumer cameras, they bake illumination in their representations and fall short of supporting advanced applications like material editing, relighting, and virtual object insertion. The reconstruction of physically based material properties and… ▽ More While numerous 3D reconstruction and novel-view synthesis methods allow for photorealistic rendering of a scene from multi-view images easily captured with consumer cameras, they bake illumination in their representations and fall short of supporting advanced applications like material editing, relighting, and virtual object insertion. The reconstruction of physically based material properties and lighting via inverse rendering promises to enable such applications. However, most inverse rendering techniques require high dynamic range (HDR) images as input, a setting that is inaccessible to most users. We present a method that recovers the physically based material properties and spatially-varying HDR lighting of a scene from multi-view, low-dynamic-range (LDR) images. We model the LDR image formation process in our inverse rendering pipeline and propose a novel optimization strategy for material, lighting, and a camera response model. We evaluate our approach with synthetic and real scenes compared to the state-of-the-art inverse rendering methods that take either LDR or HDR input. Our method outperforms existing methods taking LDR images as input, and allows for highly realistic relighting and object insertion. △ Less

Submitted 23 January, 2024; originally announced January 2024.

Comments: Project Website: https://irisldr.github.io/

Single-Image 3D Human Digitization with Shape-Guided Diffusion

Authors: Badour AlBahar, Shunsuke Saito, Hung-Yu Tseng, Changil Kim, Johannes Kopf, Jia-Bin Huang

Abstract: We present an approach to generate a 360-degree view of a person with a consistent, high-resolution appearance from a single input image. NeRF and its variants typically require videos or images from different viewpoints. Most existing approaches taking monocular input either rely on ground-truth 3D scans for supervision or lack 3D consistency. While recent 3D generative models show promise of 3D… ▽ More We present an approach to generate a 360-degree view of a person with a consistent, high-resolution appearance from a single input image. NeRF and its variants typically require videos or images from different viewpoints. Most existing approaches taking monocular input either rely on ground-truth 3D scans for supervision or lack 3D consistency. While recent 3D generative models show promise of 3D consistent human digitization, these approaches do not generalize well to diverse clothing appearances, and the results lack photorealism. Unlike existing work, we utilize high-capacity 2D diffusion models pretrained for general image synthesis tasks as an appearance prior of clothed humans. To achieve better 3D consistency while retaining the input identity, we progressively synthesize multiple views of the human in the input image by inpainting missing regions with shape-guided diffusion conditioned on silhouette and surface normal. We then fuse these synthesized multi-view images via inverse rendering to obtain a fully textured high-resolution 3D mesh of the given person. Experiments show that our approach outperforms prior methods and achieves photorealistic 360-degree synthesis of a wide range of clothed humans with complex textures from a single image. △ Less

Submitted 15 November, 2023; originally announced November 2023.

Comments: SIGGRAPH Asia 2023. Project website: https://human-sgd.github.io/

Consistent View Synthesis with Pose-Guided Diffusion Models

Authors: Hung-Yu Tseng, Qinbo Li, Changil Kim, Suhib Alsisan, Jia-Bin Huang, Johannes Kopf

Abstract: Novel view synthesis from a single image has been a cornerstone problem for many Virtual Reality applications that provide immersive experiences. However, most existing techniques can only synthesize novel views within a limited range of camera motion or fail to generate consistent and high-quality novel views under significant camera movement. In this work, we propose a pose-guided diffusion mode… ▽ More Novel view synthesis from a single image has been a cornerstone problem for many Virtual Reality applications that provide immersive experiences. However, most existing techniques can only synthesize novel views within a limited range of camera motion or fail to generate consistent and high-quality novel views under significant camera movement. In this work, we propose a pose-guided diffusion model to generate a consistent long-term video of novel views from a single image. We design an attention layer that uses epipolar lines as constraints to facilitate the association between different viewpoints. Experimental results on synthetic and real-world datasets demonstrate the effectiveness of the proposed diffusion model against state-of-the-art transformer-based and GAN-based approaches. △ Less

Submitted 30 March, 2023; originally announced March 2023.

Comments: CVPR 2023. Project page: https://poseguided-diffusion.github.io/

Progressively Optimized Local Radiance Fields for Robust View Synthesis

Authors: Andreas Meuleman, Yu-Lun Liu, Chen Gao, Jia-Bin Huang, Changil Kim, Min H. Kim, Johannes Kopf

Abstract: We present an algorithm for reconstructing the radiance field of a large-scale scene from a single casually captured video. The task poses two core challenges. First, most existing radiance field reconstruction approaches rely on accurate pre-estimated camera poses from Structure-from-Motion algorithms, which frequently fail on in-the-wild videos. Second, using a single, global radiance field with… ▽ More We present an algorithm for reconstructing the radiance field of a large-scale scene from a single casually captured video. The task poses two core challenges. First, most existing radiance field reconstruction approaches rely on accurate pre-estimated camera poses from Structure-from-Motion algorithms, which frequently fail on in-the-wild videos. Second, using a single, global radiance field with finite representational capacity does not scale to longer trajectories in an unbounded scene. For handling unknown poses, we jointly estimate the camera poses with radiance field in a progressive manner. We show that progressive optimization significantly improves the robustness of the reconstruction. For handling large unbounded scenes, we dynamically allocate new local radiance fields trained with frames within a temporal window. This further improves robustness (e.g., performs well even under moderate pose drifts) and allows us to scale to large scenes. Our extensive evaluation on the Tanks and Temples dataset and our collected outdoor dataset, Static Hikes, show that our approach compares favorably with the state-of-the-art. △ Less

Submitted 24 March, 2023; originally announced March 2023.

Comments: Project page: https://localrf.github.io/

Journal ref: CVPR 2023

Robust Dynamic Radiance Fields

Authors: Yu-Lun Liu, Chen Gao, Andreas Meuleman, Hung-Yu Tseng, Ayush Saraf, Changil Kim, Yung-Yu Chuang, Johannes Kopf, Jia-Bin Huang

Abstract: Dynamic radiance field reconstruction methods aim to model the time-varying structure and appearance of a dynamic scene. Existing methods, however, assume that accurate camera poses can be reliably estimated by Structure from Motion (SfM) algorithms. These methods, thus, are unreliable as SfM algorithms often fail or produce erroneous poses on challenging videos with highly dynamic objects, poorly… ▽ More Dynamic radiance field reconstruction methods aim to model the time-varying structure and appearance of a dynamic scene. Existing methods, however, assume that accurate camera poses can be reliably estimated by Structure from Motion (SfM) algorithms. These methods, thus, are unreliable as SfM algorithms often fail or produce erroneous poses on challenging videos with highly dynamic objects, poorly textured surfaces, and rotating camera motion. We address this robustness issue by jointly estimating the static and dynamic radiance fields along with the camera parameters (poses and focal length). We demonstrate the robustness of our approach via extensive quantitative and qualitative experiments. Our results show favorable performance over the state-of-the-art dynamic view synthesis methods. △ Less

Submitted 21 March, 2023; v1 submitted 5 January, 2023; originally announced January 2023.

Comments: CVPR 2023. Project page: https://robust-dynrf.github.io/

HyperReel: High-Fidelity 6-DoF Video with Ray-Conditioned Sampling

Authors: Benjamin Attal, Jia-Bin Huang, Christian Richardt, Michael Zollhoefer, Johannes Kopf, Matthew O'Toole, Changil Kim

Abstract: Volumetric scene representations enable photorealistic view synthesis for static scenes and form the basis of several existing 6-DoF video techniques. However, the volume rendering procedures that drive these representations necessitate careful trade-offs in terms of quality, rendering speed, and memory efficiency. In particular, existing methods fail to simultaneously achieve real-time performanc… ▽ More Volumetric scene representations enable photorealistic view synthesis for static scenes and form the basis of several existing 6-DoF video techniques. However, the volume rendering procedures that drive these representations necessitate careful trade-offs in terms of quality, rendering speed, and memory efficiency. In particular, existing methods fail to simultaneously achieve real-time performance, small memory footprint, and high-quality rendering for challenging real-world scenes. To address these issues, we present HyperReel -- a novel 6-DoF video representation. The two core components of HyperReel are: (1) a ray-conditioned sample prediction network that enables high-fidelity, high frame rate rendering at high resolutions and (2) a compact and memory-efficient dynamic volume representation. Our 6-DoF video pipeline achieves the best performance compared to prior and contemporary approaches in terms of visual quality with small memory requirements, while also rendering at up to 18 frames-per-second at megapixel resolution without any custom CUDA code. △ Less

Submitted 29 May, 2023; v1 submitted 5 January, 2023; originally announced January 2023.

Comments: Project page: https://hyperreel.github.io/

Learning Neural Light Fields with Ray-Space Embedding Networks

Authors: Benjamin Attal, Jia-Bin Huang, Michael Zollhoefer, Johannes Kopf, Changil Kim

Abstract: Neural radiance fields (NeRFs) produce state-of-the-art view synthesis results. However, they are slow to render, requiring hundreds of network evaluations per pixel to approximate a volume rendering integral. Baking NeRFs into explicit data structures enables efficient rendering, but results in a large increase in memory footprint and, in many cases, a quality reduction. In this paper, we propose… ▽ More Neural radiance fields (NeRFs) produce state-of-the-art view synthesis results. However, they are slow to render, requiring hundreds of network evaluations per pixel to approximate a volume rendering integral. Baking NeRFs into explicit data structures enables efficient rendering, but results in a large increase in memory footprint and, in many cases, a quality reduction. In this paper, we propose a novel neural light field representation that, in contrast, is compact and directly predicts integrated radiance along rays. Our method supports rendering with a single network evaluation per pixel for small baseline light field datasets and can also be applied to larger baselines with only a few evaluations per pixel. At the core of our approach is a ray-space embedding network that maps the 4D ray-space manifold into an intermediate, interpolable latent space. Our method achieves state-of-the-art quality on dense forward-facing datasets such as the Stanford Light Field dataset. In addition, for forward-facing scenes with sparser inputs we achieve results that are competitive with NeRF-based approaches in terms of quality while providing a better speed/quality/memory trade-off with far fewer network evaluations. △ Less

Submitted 10 May, 2022; v1 submitted 2 December, 2021; originally announced December 2021.

Comments: CVPR 2022 camera ready revision. Major changes include: 1. Additional comparison to NeX on Stanford, RealFF, Shiny datasets 2. Experiment on 360 degree lego bulldozer scene in the appendix, using Pluecker parameterization 3. Moving student-teacher results to the appendix 4. Clarity edits -- in particular, making it clear that our Stanford evaluation *does not* use subdivision

Dynamic View Synthesis from Dynamic Monocular Video

Authors: Chen Gao, Ayush Saraf, Johannes Kopf, Jia-Bin Huang

Abstract: We present an algorithm for generating novel views at arbitrary viewpoints and any input time step given a monocular video of a dynamic scene. Our work builds upon recent advances in neural implicit representation and uses continuous and differentiable functions for modeling the time-varying structure and the appearance of the scene. We jointly train a time-invariant static NeRF and a time-varying… ▽ More We present an algorithm for generating novel views at arbitrary viewpoints and any input time step given a monocular video of a dynamic scene. Our work builds upon recent advances in neural implicit representation and uses continuous and differentiable functions for modeling the time-varying structure and the appearance of the scene. We jointly train a time-invariant static NeRF and a time-varying dynamic NeRF, and learn how to blend the results in an unsupervised manner. However, learning this implicit function from a single video is highly ill-posed (with infinitely many solutions that match the input video). To resolve the ambiguity, we introduce regularization losses to encourage a more physically plausible solution. We show extensive quantitative and qualitative results of dynamic view synthesis from casually captured videos. △ Less

Submitted 13 May, 2021; originally announced May 2021.

Comments: Project webpage: https://free-view-video.github.io/

The Dayside Ionopause of Mars: Solar Wind Interaction, Pressure Balance, and Comparisons with Venus

Authors: F. Chu, Z. Girazian, F. Duru, R. Ramstad, J. Halekas, D. A. Gurnett, Xin Cao, A. J. Kopf

Abstract: Due to the lower ionospheric thermal pressure and existence of the crustal magnetism at Mars, the Martian ionopause is expected to behave differently from the ionopause at Venus. We study the solar wind interaction and pressure balance at the ionopause of Mars using both in situ and remote sounding measurements from the MARSIS (Mars Advanced Radar for Subsurface and Ionosphere Sounding) instrument… ▽ More Due to the lower ionospheric thermal pressure and existence of the crustal magnetism at Mars, the Martian ionopause is expected to behave differently from the ionopause at Venus. We study the solar wind interaction and pressure balance at the ionopause of Mars using both in situ and remote sounding measurements from the MARSIS (Mars Advanced Radar for Subsurface and Ionosphere Sounding) instrument on the Mars Express orbiter. We show that the magnetic pressure usually dominates the thermal pressure to hold off the solar wind at the ionopause at Mars, with only 13% of the cases where the ionospheric thermal pressure plays a more important role in pressure balance. This percentage at Venus, however, is up to 65%. We also find that the ionopause altitude at Mars decreases as the normal component of the solar wind dynamic pressure increases, similar to the altitude variation of the ionopauses at Venus. Moreover, our results show that the ionopause thickness at Mars and Venus is mainly determined by the ion gyromotion and is equivalent to about 5 ion gyroradii. △ Less

Submitted 1 November, 2021; v1 submitted 23 March, 2021; originally announced March 2021.

Robust Consistent Video Depth Estimation

Authors: Johannes Kopf, Xuejian Rong, Jia-Bin Huang

Abstract: We present an algorithm for estimating consistent dense depth maps and camera poses from a monocular video. We integrate a learning-based depth prior, in the form of a convolutional neural network trained for single-image depth estimation, with geometric optimization, to estimate a smooth camera trajectory as well as detailed and stable depth reconstruction. Our algorithm combines two complementar… ▽ More We present an algorithm for estimating consistent dense depth maps and camera poses from a monocular video. We integrate a learning-based depth prior, in the form of a convolutional neural network trained for single-image depth estimation, with geometric optimization, to estimate a smooth camera trajectory as well as detailed and stable depth reconstruction. Our algorithm combines two complementary techniques: (1) flexible deformation-splines for low-frequency large-scale alignment and (2) geometry-aware depth filtering for high-frequency alignment of fine depth details. In contrast to prior approaches, our method does not require camera poses as input and achieves robust reconstruction for challenging hand-held cell phone captures containing a significant amount of noise, shake, motion blur, and rolling shutter deformations. Our method quantitatively outperforms state-of-the-arts on the Sintel benchmark for both depth and pose estimations and attains favorable qualitative results across diverse wild datasets. △ Less

Submitted 21 June, 2021; v1 submitted 10 December, 2020; originally announced December 2020.

Comments: Project website: https://robust-cvd.github.io/

Space-time Neural Irradiance Fields for Free-Viewpoint Video

Authors: Wenqi Xian, Jia-Bin Huang, Johannes Kopf, Changil Kim

Abstract: We present a method that learns a spatiotemporal neural irradiance field for dynamic scenes from a single video. Our learned representation enables free-viewpoint rendering of the input video. Our method builds upon recent advances in implicit representations. Learning a spatiotemporal irradiance field from a single video poses significant challenges because the video contains only one observation… ▽ More We present a method that learns a spatiotemporal neural irradiance field for dynamic scenes from a single video. Our learned representation enables free-viewpoint rendering of the input video. Our method builds upon recent advances in implicit representations. Learning a spatiotemporal irradiance field from a single video poses significant challenges because the video contains only one observation of the scene at any point in time. The 3D geometry of a scene can be legitimately represented in numerous ways since varying geometry (motion) can be explained with varying appearance and vice versa. We address this ambiguity by constraining the time-varying geometry of our dynamic scene representation using the scene depth estimated from video depth estimation methods, aggregating contents from individual frames into a single global representation. We provide an extensive quantitative evaluation and demonstrate compelling free-viewpoint rendering results. △ Less

Submitted 18 June, 2021; v1 submitted 25 November, 2020; originally announced November 2020.

Comments: Project website: https://video-nerf.github.io/

Flow-edge Guided Video Completion

Authors: Chen Gao, Ayush Saraf, Jia-Bin Huang, Johannes Kopf

Abstract: We present a new flow-based video completion algorithm. Previous flow completion methods are often unable to retain the sharpness of motion boundaries. Our method first extracts and completes motion edges, and then uses them to guide piecewise-smooth flow completion with sharp edges. Existing methods propagate colors among local flow connections between adjacent frames. However, not all missing re… ▽ More We present a new flow-based video completion algorithm. Previous flow completion methods are often unable to retain the sharpness of motion boundaries. Our method first extracts and completes motion edges, and then uses them to guide piecewise-smooth flow completion with sharp edges. Existing methods propagate colors among local flow connections between adjacent frames. However, not all missing regions in a video can be reached in this way because the motion boundaries form impenetrable barriers. Our method alleviates this problem by introducing non-local flow connections to temporally distant frames, enabling propagating video content over motion boundaries. We validate our approach on the DAVIS dataset. Both visual and quantitative results show that our method compares favorably against the state-of-the-art algorithms. △ Less

Submitted 3 September, 2020; originally announced September 2020.

Comments: ECCV 2020. Project: http://chengao.vision/FGVC/

One Shot 3D Photography

Authors: Johannes Kopf, Kevin Matzen, Suhib Alsisan, Ocean Quigley, Francis Ge, Yangming Chong, Josh Patterson, Jan-Michael Frahm, Shu Wu, Matthew Yu, Peizhao Zhang, Zijian He, Peter Vajda, Ayush Saraf, Michael Cohen

Abstract: 3D photography is a new medium that allows viewers to more fully experience a captured moment. In this work, we refer to a 3D photo as one that displays parallax induced by moving the viewpoint (as opposed to a stereo pair with a fixed viewpoint). 3D photos are static in time, like traditional photos, but are displayed with interactive parallax on mobile or desktop screens, as well as on Virtual R… ▽ More 3D photography is a new medium that allows viewers to more fully experience a captured moment. In this work, we refer to a 3D photo as one that displays parallax induced by moving the viewpoint (as opposed to a stereo pair with a fixed viewpoint). 3D photos are static in time, like traditional photos, but are displayed with interactive parallax on mobile or desktop screens, as well as on Virtual Reality devices, where viewing it also includes stereo. We present an end-to-end system for creating and viewing 3D photos, and the algorithmic and design choices therein. Our 3D photos are captured in a single shot and processed directly on a mobile device. The method starts by estimating depth from the 2D input image using a new monocular depth estimation network that is optimized for mobile devices. It performs competitively to the state-of-the-art, but has lower latency and peak memory consumption and uses an order of magnitude fewer parameters. The resulting depth is lifted to a layered depth image, and new geometry is synthesized in parallax regions. We synthesize color texture and structures in the parallax regions as well, using an inpainting network, also optimized for mobile devices, on the LDI directly. Finally, we convert the result into a mesh-based representation that can be efficiently transmitted and rendered even on low-end devices and over poor network connections. Altogether, the processing takes just a few seconds on a mobile device, and the result can be instantly viewed and shared. We perform extensive quantitative evaluation to validate our system and compare its new components against the current state-of-the-art. △ Less

Submitted 1 September, 2020; v1 submitted 27 August, 2020; originally announced August 2020.

Comments: Project page: https://facebookresearch.github.io/one_shot_3d_photography/ Code: https://github.com/facebookresearch/one_shot_3d_photography

Journal ref: ACM Transactions on Graphics (Proceedings of SIGGRAPH 2020), Volume 39, Number 4, 2020

Consistent Video Depth Estimation

Authors: Xuan Luo, Jia-Bin Huang, Richard Szeliski, Kevin Matzen, Johannes Kopf

Abstract: We present an algorithm for reconstructing dense, geometrically consistent depth for all pixels in a monocular video. We leverage a conventional structure-from-motion reconstruction to establish geometric constraints on pixels in the video. Unlike the ad-hoc priors in classical reconstruction, we use a learning-based prior, i.e., a convolutional neural network trained for single-image depth estima… ▽ More We present an algorithm for reconstructing dense, geometrically consistent depth for all pixels in a monocular video. We leverage a conventional structure-from-motion reconstruction to establish geometric constraints on pixels in the video. Unlike the ad-hoc priors in classical reconstruction, we use a learning-based prior, i.e., a convolutional neural network trained for single-image depth estimation. At test time, we fine-tune this network to satisfy the geometric constraints of a particular input video, while retaining its ability to synthesize plausible depth details in parts of the video that are less constrained. We show through quantitative validation that our method achieves higher accuracy and a higher degree of geometric consistency than previous monocular reconstruction methods. Visually, our results appear more stable. Our algorithm is able to handle challenging hand-held captured input videos with a moderate degree of dynamic motion. The improved quality of the reconstruction enables several applications, such as scene reconstruction and advanced video-based visual effects. △ Less

Submitted 26 August, 2020; v1 submitted 30 April, 2020; originally announced April 2020.

Comments: SIGGRAPH 2020. Video: https://www.youtube.com/watch?v=5Tia2oblJAg Project page: https://roxanneluo.github.io/Consistent-Video-Depth-Estimation/ Code: https://github.com/facebookresearch/consistent_depth

3D Photography using Context-aware Layered Depth Inpainting

Authors: Meng-Li Shih, Shih-Yang Su, Johannes Kopf, Jia-Bin Huang

Abstract: We propose a method for converting a single RGB-D input image into a 3D photo - a multi-layer representation for novel view synthesis that contains hallucinated color and depth structures in regions occluded in the original view. We use a Layered Depth Image with explicit pixel connectivity as underlying representation, and present a learning-based inpainting model that synthesizes new local color… ▽ More We propose a method for converting a single RGB-D input image into a 3D photo - a multi-layer representation for novel view synthesis that contains hallucinated color and depth structures in regions occluded in the original view. We use a Layered Depth Image with explicit pixel connectivity as underlying representation, and present a learning-based inpainting model that synthesizes new local color-and-depth content into the occluded region in a spatial context-aware manner. The resulting 3D photos can be efficiently rendered with motion parallax using standard graphics engines. We validate the effectiveness of our method on a wide range of challenging everyday scenes and show fewer artifacts compared with the state of the arts. △ Less

Submitted 10 June, 2020; v1 submitted 9 April, 2020; originally announced April 2020.

Comments: CVPR 2020. Project page: https://shihmengli.github.io/3D-Photo-Inpainting/ Code: https://github.com/vt-vl-lab/3d-photo-inpainting Demo: https://colab.research.google.com/drive/1706ToQrkIZshRSJSHvZ1RuCiM__YX3Bz

Mars' plasma system. Scientific potential of coordinated multi-point missions: "The next generation" (A White Paper submitted to ESA's Voyage 2050 Call)

Authors: Beatriz Sánchez-Cano, Mark Lester, David J. Andrews, Hermann Opgenoorth, Robert Lillis, François Leblanc, Christopher M. Fowler, Xiaohua Fang, Oleg Vaisberg, Majd Mayyasi, Mika Holmberg, **gnan Guo, Maria Hamrin, Christian Mazelle, Kerstin Peter, Martin Pätzold, Katerina Stergiopoulou, Charlotte Goetz, Vladimir Nikolaevich Ermakov, Sergei Shuvalov, James Wild, Pierre-Louis Blelly, Michael Mendillo, Cesar Bertucci, Marco Cartacci , et al. (5 additional authors not shown)

Abstract: The objective of this White Paper submitted to ESA's Voyage 2050 call is to get a more holistic knowledge of the dynamics of the Martian plasma system from its surface up to the undisturbed solar wind outside of the induced magnetosphere. This can only be achieved with coordinated multi-point observations with high temporal resolution as they have the scientific potential to track the whole dynami… ▽ More The objective of this White Paper submitted to ESA's Voyage 2050 call is to get a more holistic knowledge of the dynamics of the Martian plasma system from its surface up to the undisturbed solar wind outside of the induced magnetosphere. This can only be achieved with coordinated multi-point observations with high temporal resolution as they have the scientific potential to track the whole dynamics of the system (from small to large scales), and they constitute the next generation of Mars' exploration as it happened at Earth few decades ago. This White Paper discusses the key science questions that are still open at Mars and how they could be addressed with coordinated multipoint missions. The main science questions are: (i) How does solar wind driving impact on magnetospheric and ionospheric dynamics? (ii) What is the structure and nature of the tail of Mars' magnetosphere at all scales? (iii) How does the lower atmosphere couple to the upper atmosphere? (iv) Why should we have a permanent in-situ Space Weather monitor at Mars? Each science question is devoted to a specific plasma region, and includes several specific scientific objectives to study in the coming decades. In addition, two mission concepts are also proposed based on coordinated multi-point science from a constellation of orbiting and ground-based platforms, which focus on understanding and solving the current science gaps. △ Less

Submitted 15 August, 2019; originally announced August 2019.

Comments: White Paper submitted to ESA's Voyage 2050 Call, 27 pages, 8 Figures, 2 Tables

Variations in the Ionospheric Peak Altitude at Mars in Response to Dust Storms: 13 Years of Observations from the Mars Express Radar Sounder

Authors: Z. Girazian, Z. Luppen, D. D. Morgan, F. Chu, L. Montabone, E. M. B. Thiemann, D. A. Gurnett, J. Halekas, A. J. Kopf, F. Nemec

Abstract: Previous observations have shown that, during Martian dust storms, the peak of the ionosphere rises in altitude. Observational studies of this type, however, have been extremely limited. Using 13 years of ionospheric peak altitude data from the Mars Advanced Radar for Subsurface and Ionosphere Sounding (MARSIS) instrument on Mars Express, we study how the peak altitude responded to dust storms dur… ▽ More Previous observations have shown that, during Martian dust storms, the peak of the ionosphere rises in altitude. Observational studies of this type, however, have been extremely limited. Using 13 years of ionospheric peak altitude data from the Mars Advanced Radar for Subsurface and Ionosphere Sounding (MARSIS) instrument on Mars Express, we study how the peak altitude responded to dust storms during six different Mars Years (MY). The peak altitude increased $\sim$10-15 km during all six events, which include a local dust storm (MY 33), three regional regional dust storms (MYs 27, 29, and 32), and two global dust storms (MYs 28 and 34). The peak altitude's orbit-to-orbit variability was exceptionally large at the apexes of the MY 29 and MY 32 dust seasons, and dramatically increased during the MY 28 and MY 34 global dust storms. We conclude that dust storms significantly increase upper atmospheric variability, which suggests that they enhance dynamical processes that couple the lower and upper atmospheres, such as upward propagating gravity waves or atmospheric tides. △ Less

Submitted 19 August, 2019; v1 submitted 17 June, 2019; originally announced June 2019.

The Effects of Solar Wind Dynamic Pressure on the Structure of the Topside Ionosphere of Mars

Authors: Z. Girazian, J. Halekas, D. D. Morgan, A. J. Kopf, D. A. Gurnett, F. Chu

Abstract: We use Mars Atmosphere and Volatile EvolutioN observations of the upstream solar wind, and Mars Express observations of ionospheric electron densities and magnetic fields, to study how the topside ionosphere ($>$ 320 km) of Mars is affected by variations in solar wind dynamic pressure. We find that high solar wind dynamic pressures result in the topside ionosphere being depleted of plasma at all s… ▽ More We use Mars Atmosphere and Volatile EvolutioN observations of the upstream solar wind, and Mars Express observations of ionospheric electron densities and magnetic fields, to study how the topside ionosphere ($>$ 320 km) of Mars is affected by variations in solar wind dynamic pressure. We find that high solar wind dynamic pressures result in the topside ionosphere being depleted of plasma at all solar zenith angles, coincident with increased induced magnetic field strengths. The depletion of topside plasma in response to high solar wind dynamic pressures is observed in both weak and strong crustal magnetic field regions. Taken together, our results suggest that high solar wind dynamic pressures lead to ionospheric compression, increased ion escape, and reduced day-to-night plasma transport in the high-altitude nightside ionosphere. △ Less

Submitted 31 July, 2019; v1 submitted 28 May, 2019; originally announced May 2019.

The effects of crustal magnetic fields and solar EUV flux on ionopause formation at Mars

Authors: F. Chu, Z. Girazian, D. A. Gurnett, D. D. Morgan, J. Halekas, A. J. Kopf, E. M. B. Thiemann, F. Duru

Abstract: We study the ionopause of Mars using a database of 6,893 ionopause detections obtained over 11 years by the MARSIS (Mars Advanced Radar for Subsurface and Ionosphere Sounding) experiment. The ionopause, in this work, is defined as a steep density gradient that appears in MARSIS remote sounding ionograms as a horizontal line at frequencies below 0.4 MHz. We find that the ionopause is located on ave… ▽ More We study the ionopause of Mars using a database of 6,893 ionopause detections obtained over 11 years by the MARSIS (Mars Advanced Radar for Subsurface and Ionosphere Sounding) experiment. The ionopause, in this work, is defined as a steep density gradient that appears in MARSIS remote sounding ionograms as a horizontal line at frequencies below 0.4 MHz. We find that the ionopause is located on average at an altitude of $363 \pm 65$ km. We also find that the ionopause altitude has a weak dependence on solar zenith angle and varies with the solar extreme ultraviolet (EUV) flux on annual and solar cycle time scales. Furthermore, our results show that very few ionopauses are observed when the crustal field strength at 400 km is greater than 40 nT. The strong crustal fields act as mini-magnetospheres that alter the solar wind interaction and prevent the ionopause from forming. △ Less

Submitted 29 August, 2019; v1 submitted 25 March, 2019; originally announced March 2019.

Comments: 22 pages, 5 figures

Journal ref: Geophysical Research Letters, 46 (2019)

MARSIS observations of field-aligned irregularities and ducted radio propagation in the Martian ionosphere

Authors: David J. Andrews, Hermann J. Opgenoorth, Thomas B. Leyser, Stephan Buchert, Niklas J. T. Edberg, David D. Morgan, Donald A. Gurnett, Andrew J. Kopf, Katy Fallows, Paul Withers

Abstract: Knowledge of Mars's ionosphere has been significantly advanced in recent years by observations from Mars Express (MEX) and lately MAVEN. A topic of particular interest are the interactions between the planet's ionospheric plasma and its highly structured crustal magnetic fields, and how these lead to the redistribution of plasma and affect the propagation of radio waves in the system. In this pape… ▽ More Knowledge of Mars's ionosphere has been significantly advanced in recent years by observations from Mars Express (MEX) and lately MAVEN. A topic of particular interest are the interactions between the planet's ionospheric plasma and its highly structured crustal magnetic fields, and how these lead to the redistribution of plasma and affect the propagation of radio waves in the system. In this paper, we elucidate a possible relationship between two anomalous radar signatures previously reported in observations from the MARSIS instrument on MEX. Relatively uncommon observations of localized, extreme increases in the ionospheric peak density in regions of radial (cusp-like) magnetic fields and spread-echo radar signatures are shown to be coincident with ducting of the same radar pulses at higher altitudes on the same field lines. We suggest that these two observations are both caused by a high electric field (perpendicular to $\mathbf{B}$) having distinctly different effects in two altitude regimes. At lower altitudes, where ions are demagnetized and electrons magnetized, and recombination dominantes, a high electric field causes irregularities, plasma turbulence, electron heating, slower recombination and ultimately enhanced plasma densities. However, at higher altitudes, where both ions and electrons are magnetized and atomic oxygen ions cannot recombine directly, the high electric field instead causes frictional heating, a faster production of molecular ions by charge exchange, and so a density decrease. The latter enables ducting of radar pulses on closed field lines, in an analogous fashion to inter-hemispheric ducting in the Earth's ionosphere. △ Less

Submitted 15 August, 2018; originally announced August 2018.

Comments: 17 pages, 4 figures

Journal ref: Journal of Geophysical Research: Space Physics, 2018, 123

DeepMVS: Learning Multi-view Stereopsis

Authors: Po-Han Huang, Kevin Matzen, Johannes Kopf, Narendra Ahuja, Jia-Bin Huang

Abstract: We present DeepMVS, a deep convolutional neural network (ConvNet) for multi-view stereo reconstruction. Taking an arbitrary number of posed images as input, we first produce a set of plane-sweep volumes and use the proposed DeepMVS network to predict high-quality disparity maps. The key contributions that enable these results are (1) supervised pretraining on a photorealistic synthetic dataset, (2… ▽ More We present DeepMVS, a deep convolutional neural network (ConvNet) for multi-view stereo reconstruction. Taking an arbitrary number of posed images as input, we first produce a set of plane-sweep volumes and use the proposed DeepMVS network to predict high-quality disparity maps. The key contributions that enable these results are (1) supervised pretraining on a photorealistic synthetic dataset, (2) an effective method for aggregating information across a set of unordered images, and (3) integrating multi-layer feature activations from the pre-trained VGG-19 network. We validate the efficacy of DeepMVS using the ETH3D Benchmark. Our results show that DeepMVS compares favorably against state-of-the-art conventional MVS algorithms and other ConvNet based methods, particularly for near-textureless regions and thin structures. △ Less

Submitted 2 April, 2018; originally announced April 2018.

Comments: CVPR 2018. Project page: https://phuang17.github.io/DeepMVS/ Code: https://github.com/phuang17/DeepMVS

Co-segmentation for Space-Time Co-located Collections

Authors: Hadar Averbuch-Elor, Johannes Kopf, Tamir Hazan, Daniel Cohen-Or

Abstract: We present a co-segmentation technique for space-time co-located image collections. These prevalent collections capture various dynamic events, usually by multiple photographers, and may contain multiple co-occurring objects which are not necessarily part of the intended foreground object, resulting in ambiguities for traditional co-segmentation techniques. Thus, to disambiguate what the common fo… ▽ More We present a co-segmentation technique for space-time co-located image collections. These prevalent collections capture various dynamic events, usually by multiple photographers, and may contain multiple co-occurring objects which are not necessarily part of the intended foreground object, resulting in ambiguities for traditional co-segmentation techniques. Thus, to disambiguate what the common foreground object is, we introduce a weakly-supervised technique, where we assume only a small seed, given in the form of a single segmented image. We take a distributed approach, where local belief models are propagated and reinforced with similar images. Our technique progressively expands the foreground and background belief models across the entire collection. The technique exploits the power of the entire set of image without building a global model, and thus successfully overcomes large variability in appearance of the common foreground object. We demonstrate that our method outperforms previous co-segmentation techniques on challenging space-time co-located collections, including dense benchmark datasets which were adapted for our novel problem setting. △ Less

Submitted 31 January, 2017; originally announced January 2017.

Virtual Rephotography: Novel View Prediction Error for 3D Reconstruction

Authors: Michael Waechter, Mate Beljan, Simon Fuhrmann, Nils Moehrle, Johannes Kopf, Michael Goesele

Abstract: The ultimate goal of many image-based modeling systems is to render photo-realistic novel views of a scene without visible artifacts. Existing evaluation metrics and benchmarks focus mainly on the geometric accuracy of the reconstructed model, which is, however, a poor predictor of visual accuracy. Furthermore, using only geometric accuracy by itself does not allow evaluating systems that either l… ▽ More The ultimate goal of many image-based modeling systems is to render photo-realistic novel views of a scene without visible artifacts. Existing evaluation metrics and benchmarks focus mainly on the geometric accuracy of the reconstructed model, which is, however, a poor predictor of visual accuracy. Furthermore, using only geometric accuracy by itself does not allow evaluating systems that either lack a geometric scene representation or utilize coarse proxy geometry. Examples include light field or image-based rendering systems. We propose a unified evaluation approach based on novel view prediction error that is able to analyze the visual quality of any method that can render novel views from input images. One of the key advantages of this approach is that it does not require ground truth geometry. This dramatically simplifies the creation of test datasets and benchmarks. It also allows us to evaluate the quality of an unknown scene during the acquisition and reconstruction process, which is useful for acquisition planning. We evaluate our approach on a range of methods including standard geometry-plus-texture pipelines as well as image-based rendering techniques, compare it to existing geometry-based benchmarks, and demonstrate its utility for a range of use cases. △ Less

Submitted 26 January, 2016; originally announced January 2016.

Comments: 10 pages, 12 figures, paper was submitted to ACM Transactions on Graphics for review

ACM Class: I.3.7