CMC: Few-shot Novel View Synthesis via Cross-view Multiplane Consistency

Abstract

Neural Radiance Field (NeRF) has shown impressive results in novel view synthesis, particularly in Virtual Reality (VR) and Augmented Reality (AR), thanks to its ability to represent scenes continuously. However, when just a few input view images are available, NeRF tends to overfit the given views and thus make the estimated depths of pixels share almost the same value. Unlike previous methods that conduct regularization by introducing complex priors or additional supervisions, we propose a simple yet effective method that explicitly builds depth-aware consistency across input views to tackle this challenge. Our key insight is that by forcing the same spatial points to be sampled repeatedly in different input views, we are able to strengthen the interactions between views and therefore alleviate the overfitting problem. To achieve this, we build the neural networks on layered representations (i.e., multiplane images), and the sampling point can thus be resampled on multiple discrete planes. Furthermore, to regularize the unseen target views, we constrain the rendered colors and depths from different input views to be the same. Although simple, extensive experiments demonstrate that our proposed method can achieve better synthesis quality over state-of-the-art methods.

Neural Radiance FieldsFew-shot view synthesisMultiplane ImagesCross-view consistency

Introduction

As a fundamental task in computer vision and computer graphics, novel view synthesis aims at rendering novel view images from given several posed input view images [4, 9]. Recently, Neural Radiance Field (NeRF) [25] has gained increasing popularity due to its powerful ability in continuous scene representation and its superior performance of novel view synthesis.

Figure 1: Given a few input views (e.g., 3 input views), (a) NeRF tends to overfit to input views and results in a dramatic performance drop, where the estimated depths of pixels share almost the same value. (b) Our key insight is to ensure the same spatial points can be sampled repeatedly in different input views. (c) Our proposed method can achieve smooth depth estimation by introducing cross-view multiplane consistency, resulting in better synthesis quality.

However, the success of NeRF and its variants depends on the number of input views to a large extent [16]. As shown in Fig. 1(a), when just a few input views are given, NeRF tends to overfit input views, resulting in the estimated depths of pixels sharing almost the same value [16, 59]. In principle, this overfitting problem could be alleviated by incorporating priors of different scenes into the neural network [6, 8, 23, 41, 46, 47, 57]. However, these methods require expensive pre-training cost and the pre-trained scenes usually exist domain gap for the target scene [28].

More recently, remarkable progress has also been made toward alleviating the overfitting problem by introducing external supervisions [10, 33, 43], pseudo views [1, 7, 18, 43, 52], or physical priors [16, 17, 28]. For example, Jain et al. [16] introduced semantic consistency between various views to encourage realistic renderings. Niemeyer et al. [28] regularized the geometry and appearance of patches for each unseen view. Although effective, the aforementioned methods either ignore the consistency across multiple views [28, 17] or impose the cross-view consistency solely on the image level [16], thereby limiting the performance.

To tackle this challenge, we make an assumption: due to fewer input views, the sampling point in each ray would rarely be used to render other views, therefore the neural networks tend to memorize colors of each input view instead of learning the underlying geometry [58, 2]. To validate this assumption, we propose Cross-view Multiplane Consistency (CMC), in which we force the sampling points to remain identical when rendering different views, as demonstrated in Fig. 1(b). In this way, the sampling points are able to be rendered to different-view images, resulting in depth-aware consistency across views. More specifically, for each input view, we build individual layered representations (i.e.,, Multiplane Images) by regarding the input view as the reference view of the Multiplane Images (MPI) [62]. Therefore, based on the discrete multiplane representation, all sampling points are forced to be distributed on the same fixed planes.

Given the multiplane representation for each input view, we aim at imposing cross-view consistency on multiplanes during the optimization. We recognize this in two aspects: 1) for the input views whose ground-truth images are available, we optimize each MPI using a reconstruction loss that minimizes the difference between the rendered input view images and the ground-truth input view images. 2) for the unseen views that lack ground-truth images, we leverage the underlying consistency: the colors and depths that are rendered from different input views (i.e.,, different MPIs) should maintain the same. As a result, we achieve cross-view multiplanes consistency.

We verify our assumptions and proposals on the common LLFF [24] and Shiny [50] dataset, where the overfitting problem can be well overcame with a promising improvement in the qualities of synthetic novel views.

The main contributions of this paper can be summarized as follows:

• We propose to force the sampling points to be the same when rendering different views, which alleviates the overfitting problem of few-shot novel view synthesis.

• To achieve cross-view multiplane consistency, in addition to reconstruction loss for input views, we propose to impose appearance and depth consistency to the unseen views.

• We provide an explanation for the overfitting problem and then give the intuition behind our proposed CMC.

• Our proposed method achieves state-of-the-art performance on various widely adopted datasets.

1 Related Work

1.1 Novel View Synthesis

As a long-standing problem in computer vision and computer graphics, novel view synthesis has been studied for decades with methods based on image-based rendering [4, 5, 9, 37], light fields [20, 24, 40, 51], point clouds [19, 39, 49, 55] and learning-based representation [11, 12, 32, 63]. Recently people have witnessed an increasing popularity for Neural Radiance Field (NeRF) [25] due to its remarkable performance for novel view synthesis. Given several 2D input view images of a static scene, NeRF can render photorealistic novel view images through coordinate-based implicit neural representation. It has been extended to several different tasks, such as dynamic scenes representation [29, 30], fast training and rendering [13, 27, 50, 56], stylization [15, 26, 45], generalizable scenes representation [6, 23, 47, 53] etc. Though NeRF achieved great synthesis quality, it depends on dense input view images, which would be not suitable for many practical applications. As a result, in this paper, we focus our attention on view synthesis with sparse input views, e.g., few-shot novel view synthesis.

1.2 Few-shot NeRF

When only a few input view images with big disparities are available, NeRF easily overfits these input views, as shown in Fig. 1(a). Some generalizable neural fields [6, 8, 47, 57] could avoid this problem by using large-scale cross-scenes datasets to learn scenes priors, while the performance will degrade significantly when there is a large domain gap between the test scenes and the training dataset. [10, 33, 43] proposed to overcome the overfitting tendency of the few-shot setting in a per-scene optimization manner with additional supervision signals, such as sparse depth estimated by Structure-from-Motion [34] or pixel correspondence estimated by [42]. To increase the number of training views available, [1, 7, 18, 52] proposed to use depth-war** to generate novel view images as pseudo labels. [16, 17, 28] made use of physical priors to regularize the scene geometry without any additional supervision signals. Recently, FreeNeRF [54] mitigated the overfitting problem from the pespective of frequency, where a novel frequency annealing strategy on positional encoding was proposed. SimpleNeRF [38] instead leveraged augmented models for better and stable few-shot view synthesis. MixNeRF [36] modeled rays as mixtures of Laplacianssians, followed by FlipNeRF [35] which used flipped reflection rays as additional training sources.

Though these methods would achieve promising results, they either heavily rely on pre-trained neural networks that are usually expensive [33], or only take advantage of physical priors as regularization terms on seen/unseen views independently, without cross-view interactions [17]. Instead, in this paper, we propose to make full use of cross-view consistency to achieve the few-shot novel view synthesis.

1.3 Multiplane Images

MPI was first proposed by [62] to expand the small baselines of stereo images. Then [24] extended MPI to view synthesis by constructing local MPIs and blending different MPIs to render novel views. To achieve a fast generation of MPI, DeepView was proposed by [11] through the leverage of learned gradient descent. To model the time-dependent effects of scenes shot at different times, DeepMPI was introduced by [22] in an unsupervised manner. [14, 21, 44] further proposed to use MPI to realize single-view synthesis. Recently [50] has been proposed to model view-dependent effects and to realize real-time rendering. Then [61] proposed to take advantage of MPIs to make a 2D GAN 3D-aware. In this paper, we first apply MPI to few-shot view synthesis, where every input view is treated as the reference view respectively. To enhance the interactions across different views, we propose two new loss functions, i.e., the appearance and depth consistency loss, based on the fact that the rendered colors and depths of the target view by different MPIs should be the same.

2 Preliminaries

Our method is built upon Neural Radiance Field (NeRF) [25] and Multiplane Images (MPI) [62]. We elaborate on them in this section.

2.1 Neural Radiance Field

NeRF [25] has emerged as a powerful tool for continuous scene representation by encoding scene properties into a neural network $F_{\theta}$, which is usually parameterized by one Multilayer Perceptron (MLP). Input the 3D coordinate $\mathbf{x}=(x,y,z)$ of a spatial point and one viewing direction $\mathbf{d}=(d_{x},d_{y},d_{z})$, NeRF outputs the corresponding color $\mathbf{c}$ and volume density $\sigma$, which is denoted as:

$$\centering\mathbf{c},\sigma=F_{\theta}(\gamma(\mathbf{x}),\gamma(\mathbf{d})),\@add@centering$$ (1)

where $\gamma$ is the position encoding operation [25] that aim to recovering high-frequency detail textures.

Given several input view images and their camera parameters, a pixel can be rendered by casting a ray $\mathbf{r}(t)=\mathbf{o}+t\mathbf{d}$ from the camera origin $\mathbf{o}$ towards the pixel along direction $\mathbf{d}$. Specifically, assuming $t\in[t_{n},t_{f}]$, the estimated color $\mathbf{C}(\mathbf{r})$ of this pixel is formulated as follows:

$$\centering\mathbf{C}(\mathbf{r})=\int_{t_{n}}^{t_{f}}T(t)\sigma(\mathbf{r}(t))\mathbf{c}(\mathbf{r}(t),\mathbf{d})dt,\@add@centering$$ (2)

where $T(t)=\exp\left(-\int_{t_{n}}^{t}\sigma(\mathbf{r}(s))ds\right)$, $\sigma$ and $\mathbf{c}$ are obtained by Eq. 1. NeRF is optimized by minimizing the following loss function:

$$\centering\mathcal{L}_{\text{MSE}}=\frac{1}{|\mathcal{R}|}\sum_{\mathbf{r}\in\mathcal{R}}\|\mathbf{C}(\mathbf{r})-\mathbf{C}_{gt}\|_{2}^{2},\@add@centering$$ (3)

where $\mathcal{R}$ is a set of sampling rays, $\mathbf{C}(\mathbf{r})$ is obtained by Eq. 2 and $\mathbf{C}_{gt}$ represents the ground-truth color.

2.2 Multiplane Images

As a layered scene representation, MPI [62] is constructed by a set of frontop-parallel planes with respect to a reference view, where all planes are fixed at specific depths that are distributed equally in the depth space. Considering one MPI with $D$ planes $(\mathbf{c}_{i},\alpha_{i})_{i=1}^{D}$, the ${i}$-th plane at depth $z_{i}$ can be viewed as a $4$-channel RGBA image that contains the color $\mathbf{c}_{i}$ and visibility $\alpha_{i}$.

To render a target view based on the MPI of the reference view, each plane of the MPI is warped to the target view $(\mathbf{c}_{i}^{\prime},\alpha_{i}^{\prime})_{i=1}^{D}$ using inverse homography war**, followed by an alpha-composition operation [14, 21, 44, 62]. The rendered image $\mathbf{I}_{t}$ and depth map ${Z}_{t}$ of target view are denoted as follows:

$$\centering\mathbf{I}_{t}=\sum_{i=1}^{D}\left(\mathbf{c}_{i}^{\prime}\alpha_{i}^{\prime}\prod_{j=1}^{i-1}\left(1-\alpha_{j}^{\prime}\right)\right)\@add@centering$$ (4)

$$\centering{Z}_{t}=\sum_{i=1}^{D}\left(z_{i}\alpha_{i}^{\prime}\prod_{j=1}^{i-1}\left(1-\alpha_{j}^{\prime}\right)\right).\@add@centering$$ (5)

We build our model on MPI. Therefore, the sampling point can be resampled on multiple discrete planes.

3 Cross-view Multiplane Consistency

Motivation.

As shown in Fig. 1(a), NeRF suffers from significant performance degradation when the number of input views is reduced, which also leads to the estimated depths of pixels sharing almost the same value [16, 59]. To tackle this problem, we assume that one plausible reason is that the sampling point in each ray would rarely be used to render other views due to fewer input views. Therefore, it is easier for the neural networks to memorize each input view images [58, 2], rather than learning the underlying geometry. Motivated by this, our key insight is to explicitly build depth-aware consistency across different views.

Method Overview.

As shown in Fig. CMC: Few-shot Novel View Synthesis via Cross-view Multiplane Consistency, to ensure that the sampling points are the same when rendering different views, we build individual layered representation (i.e., Multiplane Images) $F_{\theta}^{i}$ for each input view $i$ by utilizing the input view $i$ as the reference view of the Multiplane Images (MPI) [62]. Therefore, all sampling points are distributed on the same fixed planes. Inspired by previous works [50, 21], each MPI is presented by a multilayer perceptron (MLP) $F_{\theta}^{i}$, which outputs the color and visibility for each plane.

To optimize $F_{\theta}^{i}$, for the input views, we directly minimize the difference between rendered images and the ground-truth ones through a reconstruction loss. While there is no ground-truth image for the unseen views, we introduce an intuition that the colors and depth rendered by different input views should be the same. Specifically, we minimize the difference in the estimated colors and depths that are obtained by different MPIs.

3.1 Multiplane Representation for Input Views

As shown in Fig. CMC: Few-shot Novel View Synthesis via Cross-view Multiplane Consistency, given several sparse input view images $\{\mathbf{I}_{in}^{i}\}_{i=0}^{N-1}\in\mathit{R}^{H\times W\times 3}$ and their corresponding camera extrinsics $[\mathbf{R}_{in}^{i}|\mathbf{t}_{in}^{i}]_{i=0}^{N-1}\in SE(3)$ of a static scene, our goal is to render novel view images photorealisticly, where $H$ and $W$ are the image height and width, $N$ is the number of input views available, $\mathbf{R}\in\mathit{R}^{3\times 3}$ and $\mathbf{t}\in\mathit{R}^{3\times 1}$ represent the rotation matrix and translation vector.

As described in motivation, in this paper we use MPIs to represent the scene. Different from most MPI-based methods that randomly choose one input view as the reference view and the left input views as the target views [14, 21, 44, 50, 62], we propose to treat every input view as the reference view respectively and construct their corresponding MPIs $\{\mathbf{M}_{i}\}_{i=0}^{N-1}$ (i.e., per-view MPI), for the purpose of building depth-aware consistency across different input views. We adopt MLPs to present the MPIs following previous works [50, 21].

Specifically, considering the camera parameter $[\mathbf{R}_{t}|\mathbf{t}_{t}]$ of one target view and the $i$-th MPI $\mathbf{M}_{i}$ corresponding to $\mathbf{I}_{in}^{i}$ that has $D$ planes. When a ray $\mathbf{r}$ is cast from the camera origin $\mathbf{o}$ of the target view through one pixel at its image plane whose coordinate is $(u_{t},v_{t})$ along direction $\mathbf{d}_{i}^{(u_{t},v_{t})}$, it will have $D$ intersections with the $D$ planes of $\mathbf{M}_{i}$, which are denoted as $\{\mathbf{x}_{i}^{k}=(u_{i}^{k},v_{i}^{k},z_{i}^{k})\}_{k=0}^{D-1}$, where $(u_{i}^{k},v_{i}^{k})$ is the pixel coordinate of the $k$-th intersection and $z_{i}^{k}$ represents the depth that plane $k$ is placed. The pixel coordinate of each intersection can be computed by the inverse homography war** operation [14, 21, 44, 62, 50], which is formulated as follows:

$$\centering\left[\begin{array}[]{c}u_{i}^{k}\\ v_{i}^{k}\\ 1\end{array}\right]\sim\mathbf{K}_{in}^{i}\left(\mathbf{R}^{\prime}-\frac{\mathbf{t}^{\prime}\mathbf{n}^{\top}}{z_{i}^{k}}\right)\mathbf{K}_{t}^{-1}\left[\begin{array}[]{c}u_{t}\\ v_{t}\\ 1\end{array}\right],\@add@centering$$ (6)

where $\mathbf{K}_{in}^{i}\in\mathit{R}^{3\times 3}$ and $\mathbf{K}_{t}\in\mathit{R}^{3\times 3}$ are the camera intrinsics for the input view $\mathbf{I}_{in}^{i}$ and target view respectively, $\mathbf{n}=[0,0,1]^{\top}$ is the normal vector of the $k$-th plane, $\mathbf{R}^{\prime}$ and $\mathbf{t}^{\prime}$ are the relative camera extrinsic from the target view to the input view, which is computed as follows:

$$\centering\begin{bmatrix}\mathbf{R}^{\prime}_{3\times 3}&\mathbf{t}^{\prime}_{3\times 1}\\ \mathbf{0}_{1\times 3}&1\\ \end{bmatrix}=\begin{bmatrix}\mathbf{R}_{t}&\mathbf{t}_{t}\\ \mathbf{0}_{1\times 3}&1\\ \end{bmatrix}^{-1}\begin{bmatrix}\mathbf{R}_{in}^{i}&\mathbf{t}_{in}^{i}\\ \mathbf{0}_{1\times 3}&1\\ \end{bmatrix}.\@add@centering$$ (7)

With the computed coordinate $\mathbf{x}_{i}^{k}$ of each intersection along the ray $\mathbf{r}$ whose direction is $\mathbf{d}_{i}^{(u_{t},v_{t})}$, both $\mathbf{x}_{i}^{k}$ and $\mathbf{d}_{i}^{(u_{t},v_{t})}$ are fed into the MLP $F_{\theta}^{i}$ to estimate its color $\mathbf{c}_{i}^{k}$ and visibility $\alpha_{i}^{k}$ as shown in Fig. CMC: Few-shot Novel View Synthesis via Cross-view Multiplane Consistency, which is denoted as:

$$\centering\mathbf{c}_{i}^{k},\alpha_{i}^{k}=F_{\theta}^{i}(\gamma(\mathbf{x}_{i}^{k}),\gamma(\mathbf{d}_{i}^{(u_{t},v_{t})})),\@add@centering$$ (8)

where $\gamma$ is the position encoding operation [25] that is formulated as follows:

$$\centering\gamma(\mathbf{x})=(\sin(2^{0}\mathbf{x}),\cos(2^{0}\mathbf{x}),\cdots,\sin(2^{L-1}\mathbf{x}),\cos(2^{L-1}\mathbf{x})),\@add@centering$$ (9)

$L$ is the hand-crafted hyperparameter. Then the color $\mathbf{C}_{i}(\mathbf{r})$ and depth $Z_{i}(\mathbf{r})$ of the pixel $(u_{t},v_{t})$ in the target view can be rendered based on volume rendering by the $i$-th MPI.

3.2 Cross-view Consistency on Multplanes

Reconstruction Loss for Input Views.

Given the rendered color $\mathbf{C}_{i}(\mathbf{r})$, if the target view is one of the input views, then the reconstruction loss (Eq. 3) that minimizes the difference from $\mathbf{C}_{i}(\mathbf{r})$ to the ground truth color $\mathbf{C}_{gt}$ is adopted, which is denoted as follows:

$$\centering\mathcal{L}_{\text{MSE}}=\frac{1}{|{N}|}\frac{1}{|\mathcal{R}|}\sum_{i=0}^{N-1}\sum_{\mathbf{r}\in\mathcal{R}}\|\mathbf{C}_{i}(\mathbf{r})-\mathbf{C}_{gt}\|_{2}^{2},\@add@centering$$ (10)

where $\mathcal{R}$ is a set of sampling rays. In Sec. 4.3, we verify that with our multiplane representation, the reconstruction loss alone can overcome the overfitting problem well.

Appearance and Depth Consistency Loss for Unseen Views.

The above reconstruction loss utilizes consistency across known input views by forcing the spatial points to be sampled on the same planes. To obtain depth-aware consistency across views, we propose the appearance and depth consistency loss across unseen novel views.

Specifically, when the target view is a novel view that has no ground truth color, it still can be rendered by any MPI and deserve to have the same color and depth map, as shown in Fig. CMC: Few-shot Novel View Synthesis via Cross-view Multiplane Consistency. Based on such an observation, we propose the following loss functions:

$$\centering\mathcal{L}_{\mathrm{ac}}=\frac{2}{|{N(N-1)}|}\frac{1}{|\mathcal{R}|}\sum_{i=0}^{N-1}\sum_{j=i+1}^{N-1}\sum_{\mathbf{r}\in\mathcal{R}}\|\mathbf{C}_{i}(\mathbf{r})-\mathbf{C}_{j}(\mathbf{r})\|_{2}^{2},\@add@centering$$ (11)

$$\centering\mathcal{L}_{\mathrm{dc}}=\frac{2}{|{N(N-1)}|}\frac{1}{|\mathcal{R}|}\sum_{i=0}^{N-1}\sum_{j=i+1}^{N-1}\sum_{\mathbf{r}\in\mathcal{R}}\|{Z}_{i}(\mathbf{r})-{Z}_{j}(\mathbf{r})\|_{2}^{2},\@add@centering$$ (12)

where $\mathbf{C}_{i}$/${Z}_{i}$ and $\mathbf{C}_{j}$/${Z}_{j}$ represents the rendered colors and depths by the $i$-th MPI and $j$-th MPI respectively.

As a result, the whole loss function of our proposed method can be expressed as follows:

$$\centering\mathcal{L}=\mathcal{L}_{\text{MSE}}+\lambda_{ac}\mathcal{L}_{\mathrm{ac}}+\lambda_{dc}\mathcal{L}_{\mathrm{dc}},\@add@centering$$ (13)

where $\lambda_{ac}$ and $\lambda_{dc}$ are hyperparametes that balance the weights of $\mathcal{L}_{\mathrm{ac}}$ and $\mathcal{L}_{\mathrm{dc}}$.

Table 1: Quantitative comparisons on 8 scenes of the Shiny dataset.

Method	NeRF [25]			DietNeRF [16]			InfoNeRF [17]			Ours
Scene	PSNR$\uparrow$	SSIM$\uparrow$	LPIPS$\downarrow$	PSNR$\uparrow$	SSIM$\uparrow$	LPIPS$\downarrow$	PSNR$\uparrow$	SSIM$\uparrow$	LPIPS$\downarrow$	PSNR$\uparrow$	SSIM$\uparrow$	LPIPS$\downarrow$
Cake	15.98	0.514	0.576	18.04	0.556	0.543	14.71	0.469	0.653	17.08	0.564	0.496 [t]
Crest	11.50	0.152	0.729	9.74	0.105	0.733	12.28	0.181	0.736	14.54	0.268	0.564
Food	12.65	0.296	0.657	10.30	0.190	0.736	13.25	0.328	0.679	16.00	0.425	0.502
Giants	12.39	0.218	0.730	12.54	0.216	0.733	6.32	0.010	0.776	13.42	0.299	0.651
Pasta	13.95	0.370	0.550	13.96	0.373	0.545	13.84	0.353	0.632	14.89	0.389	0.523
Room	21.19	0.710	0.454	20.01	0.669	0.483	18.99	0.578	0.638	22.59	0.750	0.378
Seasoning	12.27	0.358	0.684	12.05	0.347	0.682	12.62	0.384	0.684	13.05	0.447	0.605
Tools	15.04	0.580	0.500	8.35	0.276	0.717	10.89	0.358	0.65	16.23	0.598	0.411 [b]
Average	14.37	0.399	0.610	13.12	0.341	0.646	12.86	0.332	0.681	15.98	0.468	0.516

3.3 Weighted Rendering

To render a target view from multiple MPIs, based on the assumption that the closest MPI to the target view should have a greater impact on its rendering process, we adopt a weighted rendering strategy. Specifically, the final output $\mathbf{C}(\mathbf{r})$ is obtained by calculating a weighted average of the rendering colors from different MPIs, which is denoted as follows:

$$\centering\mathbf{C}(\mathbf{r})=\sum_{i=0}^{N-1}w_{i}\cdot\mathbf{C}_{i}(\mathbf{r}),\@add@centering$$ (14)

where $\mathbf{C}_{i}(\mathbf{r})$ is the color rendered by the $i$-th MPI, $w_{i}$ is the weight calculated according to the distance $\mu_{i}$ from the $i$-th MPI to the target view, which is formulated as follows:

$$\centering w_{i}=\frac{\mu_{i}}{\sum_{j=0}^{N-1}\mu_{j}},\ \mu_{i}=||\mathbf{o}_{t}-\mathbf{o}_{i}||_{2}^{2},\@add@centering$$ (15)

where $\mathbf{o}_{t}$ and $\mathbf{o}_{i}$ represent the camera origins of the target view and the $i$-th MPI respectively.

3.4 Analysis on Cross-view Multiplane Consistency

To demonstrate the effectiveness of our proposed method, we make an analysis of CMC in this section. To begin with, we propose an assumption for the overfitting problem of NeRF under the few-shot setting. Specifically, given sparse input views, as shown in Fig. 1(b), a fact is that it is quite difficult for rays of different views to have the same sampling points due to the random uniform sampling strategy of NeRF [25], which is denoted as follows:

$$\centering t_{i}\sim\mathcal{U}\left[t_{n}+\frac{i-1}{M}\left(t_{f}-t_{n}\right),t_{n}+\frac{i}{M}\left(t_{f}-t_{n}\right)\right],\@add@centering$$ (16)

where $t_{n}$ and $t_{f}$ are the near and far bounds, $M$ is the number of sampling points along the ray and $t_{i}$ is the $i$-th sampling points. As a result, our assumption is that the sampling points in each ray would only take part in the rendering process of pixels corresponding to this ray, while rarely being used to render other views. Thus, the optimization process of NeRF [25] (Eq. 2) can be viewed as solving the following equation for each ray independently:

$$\centering\mathbf{C}_{gt}=\sum_{i=1}^{M}T(\sigma_{i})f(\sigma_{i})\mathbf{c}_{i},\@add@centering$$ (17)

where $T$ and $f$ are both functions of $\sigma_{i}$, $\mathbf{C}_{gt}$ is the ground-truth pixel of ray $\mathbf{r}$, $\sigma_{i}$ and $\mathbf{c}_{i}$ are unknowns to be estimated. Obviously, we have $2M$ unknowns while only one equation, which means that infinite solutions exist for this problem. Considering the memorization nature of neural networks [58, 2] and Occam’s Razor [3, 31], NeRF tends to converge to the simplest way to represent known input views, thus Eq. 17 is assumed to solve the following sparse optimization problem:

$$\centering\begin{aligned} \mathbf{c}_{i}^{*},\sigma_{i}^{*}=\mathop{\arg\min}_{\mathbf{c}_{i},\sigma_{i}}\{||&\sum_{i=1}^{M}T(\sigma_{i})f(\sigma_{i})\mathbf{c}_{i}-\mathbf{C}_{gt}||_{2}^{2}\\ +&\sum_{i=1}^{M}||\mathbf{c}_{i}||_{0}+\sum_{i=1}^{M}||\sigma_{i}||_{0}\}\end{aligned},\@add@centering$$ (18)

whose solution is

$$\{\mathbf{c}_{i}^{*},\sigma_{i}^{*}\}=\left\{\begin{aligned} &\{\mathbf{C}_{gt},1\},&i=0\\ &\{\mathbf{0},0\},&i\geq 1\end{aligned}\right.,$$ (19)

which thus leads to the overfitting problem.

Based on the analysis above, to overcome such a problem, a direct way is to impose the same point to be sampled in rays of different views. Take two input views $I_{0}$ and $I_{1}$ as an example, whose camera origins are $\mathbf{o}_{0}$ and $\mathbf{o}_{1}$ respectively. For one sampling point $\mathbf{x}_{0}=\mathbf{o}_{0}+t_{0}\mathbf{d_{0}}$ along ray $\mathbf{d_{0}}$ of $I_{0}$, our goal is that $\mathbf{x}_{0}$ can also be sampled in $I_{1}$ along ray $\mathbf{d_{1}}$, thus guarantee that the same sampling point can take part in the rendering process of pixels in different views. Assuming that the sampling point in ray $\mathbf{d_{1}}$ is denoted as $\mathbf{x}_{1}=\mathbf{o}_{1}+t_{1}\mathbf{d_{1}}$, then the problem can be converted into the following formulation (i.e., find the optimal $t_{1}$ that can minimize the distances between $\mathbf{x}_{0}$ and $\mathbf{x}_{1}$):

$$t_{1}^{*}=\mathop{\arg\min}_{t_{1}} ||\mathbf{x}_{1}-\mathbf{x}_{0}||_{2}^{2},\,t_{1}\in[t_{n},t_{f}],$$ (20)

where $\mathbf{x}_{0}$, $\mathbf{o}_{1}$ and $\mathbf{d_{1}}$ is known. As a result, our goal is to find the optimal $t_{1}$ that can satisfy the following formulation:

$$\centering\begin{aligned} ||\mathbf{x}_{1}-\mathbf{x}_{0}||_{2}^{2}\leq\delta\end{aligned},\@add@centering$$ (21)

where $\delta\to 0$.

Assuming that $\mathbf{o}_{0}=\{o_{0}^{x},o_{0}^{y},o_{0}^{z}\}$, $\mathbf{d}_{0}=\{d_{0}^{x},d_{0}^{y},d_{0}^{z}\}$, $\mathbf{o}_{1}=\{o_{1}^{x},o_{1}^{y},o_{1}^{z}\}$ and $\mathbf{d}_{1}=\{d_{1}^{x},d_{1}^{y},d_{1}^{z}\}$, then Eq. 21 can be converted into the following formulation:

$$\centering\begin{aligned} &||\{o_{0}^{x}+t_{0}d_{0}^{x},o_{0}^{y}+t_{0}d_{0}^{y},o_{0}^{z}+t_{0}d_{0}^{z}\}-\\ &\{o_{1}^{x}+t_{1}d_{1}^{x},o_{1}^{y}+t_{1}d_{1}^{y},o_{1}^{z}+t_{1}d_{1}^{z}\}||_{2}^{2}\leq\delta\end{aligned}.\@add@centering$$ (22)

Simplifying the above formula, we obtain:

$$\centering\begin{aligned} t_{1}^{*}=\Phi(t_{0})\end{aligned},\@add@centering$$ (23)

where

$$\centering\begin{aligned} \Phi(u)=((o_{0}^{x}&+ud_{0}^{x}-o_{1}^{x})^{2}+(o_{0}^{y}+ud_{0}^{y}-o_{1}^{y})^{2}\\ &+(o_{0}^{z}+ud_{0}^{z}-o_{1}^{z})^{2})^{1/2}\end{aligned},\@add@centering$$ (24)

which means that when sampling points in view $I_{0}$ are known, then all sampling points in view $I_{1}$ should be deterministic. Fortunately, this is exactly the nature of multiplane images. When view $I_{0}$ is selected as the reference view to construct the MPI, all the sampling points of different views are deterministic and forced to be distributed on the same planes. As a result, consistency across different views can be well guaranteed. Experiments in Sec. 4.3 also demonstrate the effectiveness of our analysis.

4 Experiments

We make a comparison with various state-of-the-art methods for few-shot novel view synthesis quantitatively and qualitatively. We also present a detailed analysis of the necessity of adopting per-view multiplane images and the appearance/depth consistency loss. See supplementary materials for demonstrations of Eq. 18 and Eq. 20, ablation studies on the influence of different numbers of MPI planes, and more visualization results of novel view synthesis. We only evaluate our proposed method on extremely sparse input views, i.e., 3 input views, as it is the most common case.

Figure 2: Qualitative comparisons on the Shiny dataset, where our proposed method can achieve better novel view synthesis and accurate geometry estimation (i.e., the depth map).

Figure 3: Qualitative comparisons on the LLFF dataset. Our proposed method can avoid the overfitting problem, where better novel view synthesis and more continuous depth estimation can be achieved.

4.1 Implementation Details

Datasets.

We perform experiments on the LLFF dataset [24] and the Shiny [50] dataset to validate the effectiveness of our proposed method. Both of the two datasets contain 8 complex real-world scenes with big disparities, while the Shiny dataset is more complicated because it has more view-dependent effects such as reflections and refraction. We follow the experimental protocols provided by [28], where the resolution of both input views and target views are 378 $\times$ 504. To make a fair comparison, similar to previous methods, for each scene we choose every 8-th image as the held-out test set and then select 3 images evenly from the remaining images as the input views. Notably, following [28], in our experiment the sampled input views are distributed uniformly in the camera pose space, where the distances across different input views are almost the same. However, our proposed method can also be applied to scenarios where input views are randomly selected and exhibit greater spatial separation. This flexibility stems from our individual construction of MPIs for each input view, and the capacity of each MPI to render novel views consistently. By assuming that the results of rendered novel views by different MPIs remain the same, regions in the novel view overlap** with input views are effectively constrained, producing coherent and reasonable outcomes.

Training Details.

As we discussed above, we construct per-view MPI with 80 planes for each input view, where each MPI is modeled by one independent four or six-layer leakyrelu-MLP with 256 nodes per layer. We set $\gamma$=10 for the spatial coordinate $\mathbf{x}_{i}^{k}$ while no position-encoding for the direction vector $\mathbf{d}_{i}^{(u_{t},v_{t})}$. The initial learning rate is $5\times 10^{-3}$ and then gradually reduce to $1\times 10^{-4}$. At the beginning of the training process, we use only the reconstruction loss (Eq. 10) to train the network. After 15 epochs the whole loss function (Eq. 13) with appearance and depth consistency loss is used, where both $\lambda_{ac}$ and $\lambda_{dc}$ are set to 1. We train our models with the Adam optimizer with randomly sampled 1024 rays in a batch within 50 epochs by a single NVIDIA RTX 3090 GPU. It takes about 2 hours to train a scene and 10 seconds to render a target view.

Metrics.

We evaluate the quality of rendered novel view images with Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM) [48], and Learned Perceptual Image Patch Similarity (LPIPS) [60]. For easier comparison, we also report the average score by calculating the geometric mean of $10^{-\text{PSNR}/10}$, $\sqrt{1-\text{SSIM}}$ and LPIPS for the LLFF dataset similar to [28].

Table 2: Quantitative comparisons on the LLFF dataset. Our proposed method can achieve state-of-the-art performance. ft indicates the results fine-tuned on each scene individually.

Method	PSNR$\uparrow$	SSIM$\uparrow$	LPIPS$\downarrow$	Average$\downarrow$
NeRF [25]	13.34	0.373	0.451	0.255 [t]
NeX [50]	17.36	0.591	0.369	0.163 [t]
DietNeRF [16]	14.94	0.370	0.496	0.232
InfoNeRF [17]	14.37	0.349	0.457	0.238
PixelNeRF-ft [57]	16.17	0.438	0.438	0.217
SRF-ft [8]	17.07	0.436	0.529	0.203
MVSNeRF-ft [6]	17.88	0.584	0.327	0.157
GeCoNeRF [18]	18.55	0.578	0.340	0.150
RegNeRF [28]	19.08	0.587	0.336	0.146
MixNeRF [36]	19.27	0.629	0.336	0.134
FlipNeRF [35]	19.34	0.631	0.335	0.133 [b]
Ours	19.45	0.659	0.310	0.127

4.2 Comparisons with State-of-the-art Methods

4.2.1 Results on the Shiny Dataset

We first compare our proposed method with vanilla NeRF [25], DietNeRF [16] and InfoNeRF [17] on the challenging Shiny dataset proposed by [50] to demonstrate the effectiveness of CMC. We choose 8 real-world scenes from the official shiny and shiny-extended dataset that contain complex view-dependent effects. As shown in Fig. 2, NeRF and DietNeRF will overfit to input views, where the estimated geometry (i.e., the depth map) is quite poor. For InfoNeRF, though it can render a more reasonable depth map, it will fail in more complicated scenes such as "Crest". On account that DietNeRF only uses a high-level semantic loss on the image level to realize consistency across different views, it will generate repeated contents on the rendered novel view image. Differently, InfoNeRF takes advantage of ray entropy loss to regularize the seen/unseen views independently, where no cross-views interactions exist. As a result, for some occluded areas in the novel view that don’t appear in the input views, it is quite difficult for them to estimate reasonable contents. Instead, our proposed method, i.e., CMC, can render accurate depth maps and novel view images by virtue of a fully utilize of cross-view consistency. As demonstrated in Tab. 1, CMC can achieve state-of-the-art performance on all the metrics, which reflects the fact that introducing only physical priors would not be strong enough to deal with complex scenes under the few-shot setting, leverage of cross-view consistency will be helpful for obtaining a more accurate geometry estimation.

4.2.2 Results on the LLFF Dataset

Similar to many previous works, we also perform experiments on the common LLFF dataset against many state-of-the-art methods to demonstrate the superiority of our proposed method. Specifically, we compare our method with pretraining-based methods (i.e., PixelNeRF [57], SRF [8], MVSNeRF [6]), regularization-based methods (i.e., DietNeRF [16], InfoNeRF [17], RegNeRF [28]), pseudo views-based method (i.e., GeCoNeRF [18]) and NeRF [25].

As verified in Tab. 2, our method can still achieve state-of-the-art performance with a big improvement in SSIM. For qualitative comparisons, as shown in Fig. 3, for methods based on pre-trained network such as MVSNeRF, though they can avoid overfitting to input views to some extent, the rendered novel view images would contain unreasonable artifacts due to the domain gap between training dataset and test scenes. Moreover, for input views with quite big disparities, MVSNeRF still falls into overfitting and estimates wrong geometry, as demonstrated by the scene named "Horns". For regularization-based methods such as InfoNeRF, severe artifacts will exist in the generated novel view images. For RegNeRF, the method with the best performance for few-shot novel view synthesis at present, it can overcome the overfitting problem to a large extent by means of depth smoothing regularization and a well-designed sampling annealing strategy. However, RegNeRF still generates some unreasonable geometry and results in discontinuous depth estimation, as demonstrated by the TV and conference table in the scene named "Room". On the contrary, our proposed method can achieve not only photorealistic novel view synthesis but also quite accurate and continuous depth estimation, without any physical priors serving as the regularization term or any hand-crafted complex sampling strategy to avoid overfitting. In other words, our method can realize few-shot novel view synthesis elegantly with lower complexity, which promises many practical applications.

Figure 4: Qualitative comparisons of different choices of loss functions. (1) Single MPI with $\mathcal{L}_{\text{MSE}}$. (2) Per-view MPI with $\mathcal{L}_{\text{MSE}}$. (3) Per-view MPI with $\mathcal{L}_{\text{MSE}}+\mathcal{L}_{\text{dc}}^{\text{I}}$. (4) Per-view MPI with $\mathcal{L}_{\text{MSE}}+\mathcal{L}_{\text{dc}}^{\text{I}}+\mathcal{L}_{\text{ac}}$. (5) Per-view MPI with $\mathcal{L}_{\text{MSE}}+\mathcal{L}_{\text{dc}}^{\text{I}}+\mathcal{L}_{\text{ac}}+\mathcal{L}_{\text{dc}}$.

4.3 Ablation Studies

To verify the importance of constructing per-view MPI and the appearance/depth consistency loss, we perform ablation studies on the choices of loss functions. Specifically, we choose loss functions (Eq. 13) composed of different combinations of the reconstruction loss $\mathcal{L}_{\text{MSE}}$ (Eq. 10), the depth consistency loss on input views $\mathcal{L}_{\text{dc}}^{\text{I}}$ (Eq. 12), the appearance consistency loss on novel views $\mathcal{L}_{\text{ac}}$ (Eq. 11) and the depth consistency loss on novel views $\mathcal{L}_{\text{dc}}$ (Eq. 12).

Single MPI.

As shown in Fig. 4, for the setting of single MPI (Sing. MPI), i.e., only one random input view is selected as the reference view and thus only one MPI is constructed to render novel views, though some artifacts exist, the neural network can already avoid overfitting to input views. Actually, a single MPI is a variant of NeRF where only the sampling points in rays of different input views are imposed to be distributed on the same planes. However, such a slight change can achieve nearly 4dB PSNR improvement over NeRF as demonstrated in Tab. 2 and Tab. 3. This observation reflects the effectiveness and superiority of our method, where cross-view multiplane consistency can benefit a lot for accurate geometry recovery.

Per-view MPI.

To enhance interactions across different input views, we further propose per-view MPI by treating every input view as the reference view and then constructing their corresponding MPIs. As shown in Fig. 4 and Tab. 3, with only $\mathcal{L}_{\text{MSE}}$, per-view MPI can witness an increase in rendering quality and generate more accurate geometry estimation, which demonstrates the effectiveness of per-view MPI. Then, when we successively add $\mathcal{L}_{\text{dc}}^{\text{I}}$, $\mathcal{L}_{\text{ac}}$ and $\mathcal{L}_{\text{dc}}$ into the loss function, a continuous growth of performance can be observed, where more photorealistic novel view images and better depth estimation can be achieved.

Table 3: Ablation studies on the choices of different loss functions.

Loss	Sing. MPI	Per-view MPI
$\mathcal{L}_{\text{MSE}}$	✓	✓	✓	✓	✓	✓ [t]
$\mathcal{L}_{\text{dc}}^{\text{I}}$	✗	✗	✓	✗	✓	✓
$\mathcal{L}_{\text{ac}}$	✗	✗	✗	✓	✓	✓
$\mathcal{L}_{\text{dc}}$	✗	✗	✗	✗	✗	✓ [b]
PSNR$\uparrow$	17.56	18.33	18.69	19.24	19.27	19.45
SSIM$\uparrow$	0.597	0.618	0.634	0.656	0.656	0.659
LPIPS$\downarrow$	0.359	0.345	0.334	0.336	0.321	0.310
Average$\downarrow$	0.158	0.146	0.139	0.133	0.130	0.129 [b]

5 Conclusions

We present a brand-new technique for few-shot novel view synthesis by cross-view multiplane consistency (i.e., CMC). We propose to address the overfitting problem of few-shot view synthesis by forcing the sampling points to be the identical when rendering different views through multiplane images. This is based on the assumption that given sparse input view images, the sampling point in each ray would rarely be used to render other views and thus cause the neural networks to memorize input views rather than learn the underlying geometry. Then, to enhance interactions across different views, we propose to construct per-view MPI by viewing every input view as the reference view followed by leverage of appearance and depth consistency loss. We further provide an explanation for the overfitting problem and give the intuition behind CMC. To verify our assumption and method, we conduct experiments on a large amount of complex real-world scenes, where our proposed CMC can achieve state-of-the-art few-shot novel view synthesis, without any scene priors or complicated hand-crafted sampling strategy.

6 Limitations and future works

Limitations.

The main limitation of our proposed method is that CMC doesn’t perform well on surrounding scenes that contain big camera rotations, such as the Blender dataset [25]. This is because CMC is based on Multiplane Images (i.e., MPI), which is specially designed for forward-facing scenes while not suitable to represent surrounding scenes.

Future works

Our future works include extending CMC to surrounding scenes using methods such as multisphere representation, where sampling points are forced to be distributed on the same spheres. Moreover, we will try to use only one MLP instead of per-view MPI to represent the scene, which would decrease the training burden to a large extent.

7 Acknowledgement

This work was supported in part by NSFC under Grant 62371434, U1908209, 62021001.