Sparse Views, Near Light:
A Practical Paradigm for Uncalibrated Point-light Photometric Stereo

Abstract

Neural approaches have shown a significant progress on camera-based reconstruction. But they require either a fairly dense sampling of the viewing sphere, or pre-training on an existing dataset, thereby limiting their generalizability. In contrast, photometric stereo (PS) approaches have shown great potential for achieving high-quality reconstruction under sparse viewpoints. Yet, they are impractical because they typically require tedious laboratory conditions, are restricted to dark rooms, and often multi-staged, making them subject to accumulated errors. To address these shortcomings, we propose an end-to-end uncalibrated multi-view PS framework for reconstructing high-resolution shapes acquired from sparse viewpoints in a real-world environment. We relax the dark room assumption, and allow a combination of static ambient lighting and dynamic near LED lighting, thereby enabling easy data capture outside the lab. Experimental validation confirms that it outperforms existing baseline approaches in the regime of sparse viewpoints by a large margin. This allows to bring high-accuracy 3D reconstruction from the dark room to the real world, while maintaining a reasonable data capture complexity.

Abstract

In this supplementary material, we show further details about our framework. Specifically, we describe the network architecture with all its parameters and training specifications. Then we elaborate on the capturing process to retrieve the synthetic and real-world photometric images. After that, we show additional results on three viewpoints with a small camera baseline, as well as error maps of the reconstructions presented in the main paper. Furthermore, we show additional reconstruction results on both captured scans and the multiview diligent dataset, as well as some relighting results. We also analyze the effect of the ratio of point light intensity on the reconstruction quality, as well as the effect of the number of viewpoints and lights. Finally, we elaborate on the limitations of our approach.

1 Introduction

Multi-illumination data at viewpoint 1	Multi-illumination data at viewpoint 2	PS-NeRF [66]	Ours

Figure 1: We introduce the first framework for multi-view uncalibrated point-light photometric stereo. Given a set of PS images captured from different viewpoints (left), our method recovers high-fidelity $3$D reconstruction (right). The acquisition of uncalibrated point-light PS imagery captured under ambient lighting in a sparse multi-view setup does not only allow for easy data capture, but also leads to $3$D reconstructions of unprecedented detail. Here, with as few as two views we are able to reconstruct the squirrel’s $3$D geometry at higher precision than the state-of-the-art.

The challenge of $3$D reconstruction stands as a cornerstone in both computer vision and computer graphics. Despite notable progress in recovering an object’s shape from dense image viewpoints, predicting consistent geometry from sparse viewpoints remains a difficult task. Contemporary approaches employing neural data structures depend heavily on extensive training data to achieve generalization in the context of sparse views. Additionally, the presence of a wide baseline or textureless objects form significant obstacles. In contrast, photometric methodologies like photometric stereo (PS) excel in reconstructing geometry, even in textureless regions. This capability is attributed to the abundance of shading information derived from images acquired under diverse illumination. Yet, such approaches typically require a controlled laboratory setup to fulfill the necessary dark room and directional light assumptions. As a consequence, PS approaches become impractical beyond the confines of a laboratory.

To address these shortcomings we combine state-of-the-art volume rendering formulations with a sparse multi-view photometric stereo model. In particular, we advocate a physically realistic lighting model that combines ambient light and uncalibrated point-light illumination. Our approach facilitates simplified data acquisition, and we introduce a novel pipeline capable of reconstructing an object’s geometry from a sparse set of viewpoints, even if the object is completely textureless. Furthermore, since we assume a point-light model instead of distant directional lighting, we can infer absolute depth from a single viewpoint, allowing us to address the challenge of wide camera baselines.

In detail, our contributions are as follows:

• We introduce the first framework for multi-view uncalibrated point-light photometric stereo. It combines a state-of-the-art volume rendering formulation with a physically realistic model of ambient light and point lights.

• We eliminate the need for a dark room environment, dense capturing process, and distant lighting. Thereby we enhance the accessibility and simplify data acquisition for setups beyond traditional laboratory settings.

• We validate that it can be successfully used for accurate shape reconstruction of textureless objects in highly sparse scenarios with wide baselines.

• Despite the absence of pre-processing or vast training data, we outperform cutting-edge approaches that either rely only on static ambient illumination or PS imagery.

2 Related Work

In this section, we categorize pertinent works into two distinct groups. The first category focuses on neural $3$D reconstruction, emphasizing the use of images captured under static ambient illumination. Subsequently, the second category delves into photometric stereo (PS) approaches, which specifically involve methods assuming images captured under varying illumination conditions.

2.1 Neural 3D reconstruction

In our context, the domain of neural 3D reconstruction under static ambient light can be bifurcated into two distinct subcategories. The first comprises methodologies that presume a dense sampling of the viewing sphere [42, 69, 70, 58, 39, 43, 15, 74, 13, 5, 11]. Conversely, the second subcategory encompasses techniques adept at handling sparse camera viewpoints [67, 9, 17, 65, 73, 14, 59, 64, 68, 51, 71, 63, 35, 47, 23, 62, 72, 57, 7].

Given our emphasis on a sparse set of camera viewpoints, our focus will be directed toward the latter set of methodologies. The majority of sparse reconstruction approaches within this category hinge on training with a specific dataset [67, 9, 17, 65, 73, 14, 59, 64, 68, 51, 71, 63]. The reliance on extensive training data renders these techniques susceptible to errors when confronted with objects not represented in the training dataset, thereby introducing sensitivity to the domain gap between training and test data.

Efforts have been undertaken to mitigate the challenges associated with the domain gap. One approach involves fine-tuning pre-trained architectures during test time with sparse input data [35, 47, 23]. However, while fine-tuning enables fine-scale refinement of the predicted reconstruction, the broader issue of generalization to entirely different datasets persists as a challenge. Furthermore, it is crucial to note that all the previously mentioned sparse approaches presume small camera baselines and textured objects. To achieve a full reconstruction of an object from sparse views, it becomes imperative to develop texture-agnostic approaches capable of handling wide baselines.

Another set of approaches involves the utilization of pre-training to establish a geometric prior [62, 72, 57]. Yu et al. [72] leverages this geometric prior to predict a depth and normal map, albeit with a limitation in fine-scale details due to its coarse resolution. Notably, [62, 57] address this limitation. Wu et al. [62] relax the assumption on small baselines, although a reasonable overlap of sparse views remains necessary for consistent 3D reconstruction. Vora et al. [57] demonstrate promising results with much less overlap, employing only three viewpoints. However, the use of template shapes as a prior term introduces the risk of the reconstruction quality being affected by the domain gap curse.

In contrast, our approach eliminates the requirement for training data and operates effectively in wide baseline scenarios, independent of the object’s texture. This accomplishment can be attributed to the integration of PS principles, which we will elucidate in the subsequent discussion.

2.2 Photometric Stereo

The principle of PS [60] has a long and established history, renowned for yielding high-quality and finely detailed geometric estimates, due to its capturing process: multiple images under different illumination of a static object are captured from a single camera viewpoint.

In our case, PS approaches can thus be categorized into two main types. The first category pertains to single-view methods [55, 24, 44, 49, 32, 38, 50, 31, 34, 18], wherein data is acquired as described above. The second category involves multi-view methods [30, 78, 1, 10, 33, 22, 61, 45, 28, 27, 66], where the capturing process is repeated at multiple camera viewpoints.

Within the context of our task, we concentrate on the multi-view scenario. Many existing approaches in this domain assume knowledge of the light source for each image, earning them the designation of calibrated PS (CPS). In this case, frequently the light is parameterized as directional [30, 78, 1, 10], while approaches employing point-light illumination are more sparsely represented [30, 33, 10], owing to the intricate model and the much more complex calibration process [50]. In a broader context, the calibration of each image’s light source is impractical and tedious, underscoring the advantage of the uncalibrated case, which inherently facilitates a more straightforward and streamlined capturing process.

Efforts have been dedicated to exploring the uncalibrated PS (UPS) scenario, where there is no prior knowledge of the light source [22, 61, 45, 28, 27, 66]. Nevertheless, all these uncalibrated methods typically assume a distant and directional light model or permit a lighting model based on spherical harmonics [2, 4, 48, 19, 20, 53, 52], albeit at the expense of being restricted to Lambertian objects [61]. Consequently, the exploration of the uncalibrated point-light case remains uncharted territory, particularly in the multi-view scenario.

Despite the impractical assumption of calibrated light or the constraint of uncalibrated directional lighting, the aforementioned multi-view photometric stereo approaches demand a dark room and lack accommodation for ambient light [30, 78, 1, 10, 33, 22, 45, 28, 27, 66]. Additionally, they comprise multiple stages, rendering them prone to initialization and accumulation errors [30, 33, 22, 61, 45, 28, 27, 66]. Furthermore, these methods assume Lambertian objects [33, 61] and crucially rely on a dense sampling of camera viewpoints around the object [30, 33, 22, 61, 45].

The approach most closely related to ours is the sparse method PS-NeRF [66], which assumes uncalibrated directional lighting with imagery captured in a dark room. This method comprises multiple stages, where initially, a pre-trained network [8] is utilized to estimate the initial light, along with normal information for each image and camera view. Subsequently, this acquired information is employed to estimate the object’s mesh through the optimization of an occupancy field [43]. The outputs of both stages are then combined to derive a consistent normal map of the object. Although PS-NeRF demonstrates impressive results in a sparse scenario with five viewpoints, its susceptibility to error accumulation is a potential limitation, given the involvement of multiple stages.

Our proposed model addresses the aforementioned limitations comprehensively. Notably, we excel in handling sparse yet wide baseline scenarios, where existing state-of-the-art methods encounter challenges. Our investigation tackles the open challenge of multi-view uncalibrated point-light PS within an end-to-end framework, liberating us from the constraints associated with Lambertian objects. Furthermore, our approach facilitates easy data capture in the presence of ambient light, eliminating the need for a dark room.

3 Setting and image formation model

We consider an object under static illumination photographed from several different viewpoints. For every viewpoint, we are given a set of $N+1$ images $(I^{l})_{l=0,\dots,N}$. Image $I^{0}$ is captured only under static ambient illumination and called the ambient image. For images $I^{1}$ to $I^{N}$, the object is in addition to the ambient illumination also illuminated by a single achromatic point-light source with scalar light intensity $L_{0}\in\mathbb{R}^{+}$, placed at a different location $\mathbf{p}^{l}\in\mathbb{R}^{3}$ for each image and viewpoint. We use the common assumption in volume rendering [70, 62, 39, 58] that the image brightness $I^{l}$ is the same as the accumulated radiance of the volume rendering, see Eq. 4. Note that we do not add an index for the viewpoint to the image to not clutter notation, whenever we later evaluate an image at a pixel location, we assume it implicitly means a pixel at a specific viewpoint. In particular, a sum over all pixels is the sum over all pixels at every viewpoint location.

According to the linearity of the radiance, the total radiance for the image $l$ is given by a sum $\mathcal{L}^{l}=\mathcal{L}_{\text{amb}}+\mathcal{L}_{\text{pt}}^{l}$, where $\mathcal{L}_{\text{amb}}$ is the radiance due to the static illumination, the same for each image, and $\mathcal{L}_{\text{pt}}^{l}$ is the radiance due to the point-light source used for that particular image. We will first express the radiance due to the point-light source with respect to the material and lighting parameters, which allows to take into account the different illuminations of the images. Subsequently, we elaborate on how we deal with the ambient component.

3.1 Point-light illumination and material

The field $L(\mathbf{x},\mathbf{n},\mathbf{v})$ maps points $\mathbf{x}\in\mathbb{R}^{3}$, normal direction $\mathbf{n}(\mathbf{x})\in\mathbb{S}^{2}$ and viewing direction $\mathbf{v}\in\mathbb{S}^{2}$ to its radiance. It can be expressed in terms of geometry, material and lighting with the classical rendering equation [25]. Since for now we only consider a single achromatic point-light source with intensity $L_{0}$ at position $\mathbf{p}$, this equation simplifies to

$$L(\mathbf{x},\mathbf{n},\mathbf{v})=\dfrac{L_{0}}{\left\lVert\mathbf{p}-\mathbf{x}\right\rVert^{2}}f_{\text{r}}(\mathbf{x},\mathbf{n},\mathbf{v},\mathbf{l})\max(0,\mathbf{n}\cdot\mathbf{l}),$$ (1)

where $\mathbf{l}=\frac{\mathbf{p}-\mathbf{x}}{\left\lVert\mathbf{p}-\mathbf{x}\right\rVert}$.

As a model for the spatially varying reflectance $f_{\text{r}}$, we use the simplified Disney BRDF [26]. It is relatively simple yet expressive enough to resemble a wide variety of materials. It has already been successfully used in several other prior works [36, 75, 5]. The Disney BRDF requires three parameters to be available at every point $\mathbf{x}\in\mathbb{R}^{3}$: An RGB diffuse albedo $\rho\in\mathbb{R}^{3}_{\geq 0}$, a roughness $\alpha_{\text{r}}>0$ and a specular albedo $\alpha_{\text{s}}\in[0,1]$.

As in [5], all three BRDF parameters can be represented with MLPs. One MLP with network parameters $\gamma_{1}$ computes the diffuse parameter $\rho$, a second MLP with network parameters $\gamma_{2}$ computes the specular parameters $(\alpha_{\text{r}},\alpha_{\text{s}})$. To encompass all point-light and BRDF parameters, we concatenate them into the vectors $\phi=(\mathbf{p}^{1},\dots,\mathbf{p}^{M})$, which includes all light source positions (where $M$ is the product of $N$ with the number of viewpoints), and $\gamma=(\gamma_{1},\gamma_{2})$, representing the BRDF parameters. Both $\phi$ and $\gamma$ serve as the network parameters to be learned during optimization. However, note that one of our contributions is a novel strategy to deal with the diffuse albedo later on in Sec. 5.2, which dramatically improves performance in extremely sparse scenarios.

We assume that the intensity $L_{0}$ remains constant across all images and views. This assumption is reasonable, e.g., when using a smartphone’s flashlight or a sole LED, as shown in our experiments. A constant intensity introduces a scale ambiguity between the $f_{\text{r}}$ and $L_{0}$. We set $L_{0}\equiv 1$ and subsequently estimate a scaled version of the BRDF, i.e., the intensity is absorbed into the BRDF. This is feasible, since we can scale the Disney BRDF arbitrarily.

3.2 Ambient light

Given the provided ambient image, we adopt the approach from [77], which involves subtracting the ambient image from all other images. This process effectively eliminates ambient radiance from our considerations, simulating a traditional dark room scenario. A drawback is that in practice, the captured images contain additive noise, hence such a subtraction will double the variance of the noise distribution, resulting in noisier input images. Nevertheless, as can be seen in the results, our approach works successfully with this strategy even if we use a simple smartphone camera for the data capture.

However, this simplification is enabled by the fact that the point-light sources are near the object, making it easy to obtain a well exposed contribution from the point-light source. In methods assuming directional lighting, the practical simulation often involves using a point-light source placed far from the object, ensuring that the object’s diameter is significantly smaller than the distance between the point-light and the object. For example, for the DiLiGenT-MV dataset [30], the light sources are roughly 1m from the object, whereas for our setup, the distance is approximately 25cm. As a consequence, for a given static ambient illumination, the distant point-light will need to be $16\times$ brighter in the Diligent setup to obtain a similar signal to noise ratio. This increased brightness requirement escalates the hardware demands, especially if the highest quality is sought.

3.3 Other illumination effects

We do not model indirect lighting and cast-shadows explicitly. However, multiple previous works [3, 21, 36, 41, 76] and PS frameworks [50, 18, 30, 10] have demonstrated that satisfactory results can be achieved without taking these into account. In addition, we use the robust $L^{1}$-norm in our data term (7) to further increase robustness. Furthermore, we will expose a simple strategy in Sec. 5 to reduce the impact of cast-shadows, which inevitably occur when capturing a nonconvex object under changing illumination.

4 Volume rendering of the surface

For differentiable rendering we build upon VolSDF [70] because in contrast to NeRF [39] it allows us to decouple geometric representation and appearance. As discussed in [62, 71, 51], VolSDF exhibits poor reconstruction quality in the sparse viewpoint scenario due to the shape-radiance ambiguity. We eliminate this ambiguity thanks to photometric stereo, providing shading cues from different illuminations for each viewpoint. To this end, we employ the radiance field in Eq. 1 and apply it to the multi-view uncalibrated point-light PS setting. The integration of differentiable volume rendering with a physically realistic lighting model provides an efficient solution to the multi-view photometric stereo problem, enabling high-quality reconstruction even with sparse viewpoints, as demonstrated in the subsequent results.

Geometry model and associated density. The geometry of the scene is modeled with a signed distance function (SDF) $d:\Omega\to\mathbb{R}$ within the volume $\Omega\subset\mathbb{R}^{3}$. We follow [46, 69, 58, 70] and represent the SDF with an MLP $d(\mathbf{x};\theta)$ parametrized by $\theta$. Note that we assume positive values of the SDF $d$ inside the object, thus the normal vector is given by $\mathbf{n}=\nabla d/\left\lVert\nabla d\right\rVert$ on $\Omega$. To render the scene, we follow [70] and transform the SDF into a density $\sigma:\Omega\to\mathbb{R}_{\geq 0}$ by means of the transformation

$$\sigma(\mathbf{x})=\beta^{-1}\Psi_{\beta}(-d(\mathbf{x})),$$ (2)

where $\Psi_{\beta}$ denotes the cumulative distribution function of the Laplace distribution with zero mean. The positive scale parameter $\beta$ is learned during optimization. For simplicity, we assume it is included in the set of SDF parameters $\theta$.

Volume Rendering. Using the density (2), we can now set up a distribution of weights $w(t)$ along each ray $\mathbf{x}(t)=\mathbf{c}+t\mathbf{v},t\geq 0$ defined by the center of projection $\mathbf{c}$ of the camera and viewing direction $\mathbf{v}$,

$$w(t)=\sigma(\mathbf{x}(t))\,\exp\left(-\int_{0}^{t}\sigma(\mathbf{x}(s))\;\mathrm{d}s\right).$$ (3)

This is the derivative of the opacity function, which is monotonically increasing from zero to one, and thus $w$ is indeed a probability distribution [70]. In particular, the integral

$$\mathcal{L}_{p}=\int_{0}^{\infty}w(t)L(\mathbf{x}(t),\mathbf{n}(t),\mathbf{v})\;\mathrm{d}t,$$ (4)

to compute radiance $\mathcal{L}_{p}$ in the pixel $p\in\mathbb{R}^{2}$ corresponding to the ray $\mathbf{x}(t)$ is well-defined. Intuitively, we accumulate the radiance field (1) of the point-light source at locations where opacity increases, i.e. where we move towards the inside of the object. For an ideal sharp boundary, $w$ would be a Dirac distribution with its peak placed at the boundary.

As was done in [70] we approximate the integral (4) using the quadrature rule

$$\mathcal{L}_{p}\approx\sum^{m-1}_{i=1}(t_{i+1}-t_{i})w(t_{i})L(\mathbf{x}(t_{i}),\mathbf{n}(t_{i}),\mathbf{v}).$$ (5)

for numerical integration with a discrete set of samples $t_{1}<t_{2}<\dots<t_{m}$. Note that the integral (3) required for $w(t_{i})$ in (5) can be numerically computed using an analogous formula while iteratively accumulating the sum (5).

5 Training objective and optimization

In order to reconstruct the scene by inverse rendering, we train our neural networks from the available input images. We optimize for the geometry parameters $\theta$ of the SDF network and point-light and BRDF parameters $\phi$ and $\gamma$ of the appearance network, respectively.

5.1 Vanilla loss function with MLP for albedo

The overall loss consists of the sum of an inverse rendering loss $E_{\text{RGB}}(\theta,\phi,\gamma)$, an Eikonal loss $E_{\text{eik}}(\theta)$ and a mask loss $E_{\text{mask}}(\theta)$ for the SDF,

$$E(\theta,\phi,\gamma)=E_{\text{RGB}}(\theta,\phi,\gamma)+\lambda_{1}E_{\text{eik}}(\theta)+\lambda_{2}E_{\text{mask}}(\theta),$$ (6)

with respective weighting parameters $\lambda_{1},\lambda_{2}\geq 0$ given as hyperparameters. We will describe the different terms in detail in the following.

The rendering loss

$$E_{\text{RGB}}(\theta,\phi,\gamma)=\sum_{p}\sum_{l\in B_{\tau}(p)}\left\lVert I^{l}_{p}-\mathcal{L}^{l}_{p}(\theta,\phi,\gamma)\right\rVert_{1}$$ (7)

encourages that the rendered images resemble the input images. As a strategy to deal with cast-shadows, for each pixel $p$, we use only the $\tau$ brightest views $B_{\tau}(p)\subset\{1\ldots N\}$, where $\tau>0$ is a hyper-parameter. This reduces the impact of cast-shadows, as a pixel will have the smallest possible brightness when it lies in shadow. Such a strategy is necessary since despite the use of an $L^{1}$-norm to improve robustness against outliers, some regions with frequent cast-shadows would still exhibit small artefacts, as can be seen in Fig. 2. In order to avoid wasting meaningful information due to this heuristic, we set $\tau=N$ for the first half of the iterations, and then set it to $\tau=\lceil\frac{3}{4}N\rceil$ for the second half for improved handling of cast-shadows.

The Eikonal term

$$E_{\text{eik}}(\theta)=\sum_{\mathbf{x}}(\left\lVert\nabla_{\mathbf{x}}d(\mathbf{x};\theta)\right\rVert-1)^{2}$$ (8)

encourages $d(\mathbf{x};\theta)$ to approximate an SDF, as it penalizes deviations from the respective partial differential equation which every SDF satisfies, see e.g. [16].

Finally, we follow [5, 58] and employ the mask loss

$$E_{\text{mask}}(\theta)=\sum_{p}\text{BCE}(M_{p},W_{p}(\theta)),$$ (9)

to impose silhouette consistency by means of the binary cross entropy (BCE) between the given binary mask $M_{p}$ at the pixel $p$ and the sum $W_{p}(\theta)=\sum_{i=1}^{m-1}w(t_{i};\theta)$ of the weights at the sampling locations $t_{i}$ used in (5).

Input image	w/o truncation	w/ truncation

Figure 2: Effect of cast-shadows on the result. The region behind the ear exhibits frequent cast-shadows in the input images (left), which induce artefacts in the estimated mesh (middle). Our truncation strategy (Sec. 5.1) successfully eliminates these (right).

Input image	[62]	[10] (directional light)	[10] (point-light)	[66]	Ours

Figure 3: Full 3D reconstruction of one synthetic and two real objects from 6 viewpoints. While some of the baseline reconstructions (with directional or point-light models) exhibit texture-induced patterns, distortions or other artefacts, the proposed framework consistently yields artefact- and distortion-free reconstructions with finer geometric details. Additional results are shown in the supplementary.

5.2 Using the optimal diffuse albedo

In this section, we introduce a strategy to handle the diffuse albedo. This is inspired by [18], however in their single-view approach they are only able to handle the least-squares case, i.e. $L^{2}$, while we extend their formulation to objective functions of the form of an $L^{p}$-norm, with $p=1$ in our case, see Eq. 7. In lieu of employing an MLP for modeling the diffuse albedo, we opt for approximating an optimal solution considering the remaining parameters. This allows us to exclude the diffuse albedo from the optimization process. Our subsequent demonstration will underscore the significant benefits of this approach in the context of sparse views, leading to markedly improved final geometry. A key technical change is the domain of the diffuse parameter $\rho$. So far, it has been a vector field defined in world space $\Omega$, implemented with a neural network. However, in the following we only try to recover the projection of $\rho$ on the image planes. Thus, the input to $\rho$ will be a pixel, and the output is the diffuse albedo at the intersection between the pixel’s ray and the object surface. In particular, the set of BRDF parameters is reduced to $\gamma=\gamma_{2}$ and now only includes specular parameters.

Reformulation of the data term. We will now derive an expression for the data term which allows to solve for diffuse albedo $\rho$ when given all of the other parameters. Note that the SVBRDF can be decomposed into a diffuse part and a specular part [26],

$$f_{\text{r}}(\mathbf{x},\mathbf{n},\mathbf{v},\mathbf{l})=\dfrac{\rho(\mathbf{x})}{\pi}+f_{\text{s}}(\mathbf{x},\mathbf{n},\mathbf{v},\mathbf{l}),$$ (10)

where the specular part $f_{\text{s}}$ depends on both the roughness and specular albedo. If we denote the BRDF-independent factor of the radiance field (1) by

$$S^{l}(\mathbf{x},\mathbf{n})=\dfrac{L_{0}}{\left\lVert\mathbf{x}-\mathbf{p}^{l}\right\rVert^{2}}\max(0,\mathbf{n}\cdot\mathbf{l}),$$ (11)

this implies that the total radiance field is the sum of diffuse radiance $\frac{\rho}{\pi}S^{l}$ and specular radiance $L_{s}^{l}=f_{\text{s}}S^{l}$.

Inserting this into the volume rendering equation (4), we can thus re-arrange the data term (7) as

$$E_{\text{RGB}}(\theta,\phi,\gamma,\rho) =\sum_{p}\sum_{l\in S_{\tau}(p)}\left\lVert b^{l}_{p}(\theta,\phi,\gamma)-\rho_{p}a^{l}_{p}(\theta,\phi)\right\rVert_{1}$$ (12)

$$\text{with }a^{l}_{p}(\theta,\phi) =\frac{1}{\pi}\int_{0}^{\infty}w_{\theta}(t)S^{l}(\mathbf{x}(t),\mathbf{n}_{\theta}(t))\;\mathrm{d}t,$$

$$b^{l}_{p}(\theta,\phi,\gamma) =I^{l}_{p}-\int_{0}^{\infty}w_{\theta}(t)L_{s}^{l}(\mathbf{x}(t),\mathbf{n}_{\theta}(t),\mathbf{v})\;\mathrm{d}t.$$

When kee** $\theta$, $\phi$, and $\gamma$ fixed, this expression can be minimized pixel-wise for $\rho$ as a linear least absolute deviation problem, as we show next.

Alternate treatment of the diffuse albedo. It is now possible to completely exclude the diffuse albedo $\rho$ from the optimization and simply replace it by the optimal value

$$\rho^{*}_{\theta,\gamma}=\operatorname*{arg\,min}_{\rho}E_{\text{RGB}}(\theta,\phi,\gamma,\rho)$$ (13)

with respect to the other parameters. In our case, we are facing a linear least absolute deviation problem which can be solved individually for each pixel.

dog1
dog2
	Input image	[62]	[66]	Ours

Figure 4: Results for both dog1 and dog2 obtained with 5 viewpoints reveal a notable performance degradation in [62] and [66] when confronted with complex textures. This deterioration is attributed to a generalization failure of their learned priors. In contrast, our approach, which does not rely on any prior, provides a superior reconstruction that is unaffected by the complex texture.

We thus approximate the solution using a few iterations of iterative reweighted least squares [6], as it efficiently solves the problem while providing a differentiable expression. In order to simplify notation, we omit the dependency w.r.t. the parameters and denote only the channel-wise operations, where we iteratively compute

$$\rho^{*}_{p}\approx\rho^{(k)}_{p}=\dfrac{b_{p}^{\top}W^{(k)}_{p}a_{p}}{a_{p}^{\top}W^{(k)}_{p}a_{p}}$$ (14)

with diagonal matrices

$$W^{(k)}_{p}=\begin{cases}\text{diag}(\max(\epsilon,|b_{p}-\rho^{(k-1)}_{p}a_{p}|)^{-1})&\text{if }k\geq 1,\\[1.29167pt] I&\text{otherwise.}\end{cases}$$ (15)

Above, $k$ is the index of iteration, and $a_{p}$ and $b_{p}$ are the vectors whose components are given in Eq. 12. The small constant $\epsilon>0$ avoids numerical issues, and $I$ is an identity matrix of matching size. The gradient of $\rho^{(k)}_{p}$ with respect to $\theta$ and $\gamma$ is computed using automatic differentiation, however the gradient of $W^{(k)}_{p}$ is not back-propagated, as it makes the optimization much more difficult and degrades the quality of the results.

In the forthcoming evaluation section, we demonstrate that this formulation delivers exceptional performance in $3$D reconstruction from sparse viewpoints. Notably, it proves particularly advantageous in the most sparse scenarios, where only two viewpoints are available, showcasing its superiority over the previous MLP-based formulation.

6 Results

	$\downarrow$RMSE$\times 100$					$\downarrow$MAE
	[62]	[10]D	[10]P	[66]	Ours	[62]	[10]D	[10]P	[66]	Ours
dog1	2.1	8.9	13.9	2.1	0.4	22.0	39.9	39.2	18.8	04.6
dog2	2.2	8.4	13.4	2.1	0.4	23.2	29.6	40.3	19.5	04.8
girl1	0.9	6.1	02.3	1.5	0.3	17.2	41.8	46.5	23.8	09.6
girl2	1.5	5.2	04.8	1.5	0.3	23.9	40.0	47.3	23.4	10.2

Table 1: RMSE and MAE for full 3D reconstructions. RMSE is computed based on the vertex to mesh distance, and the MAE is computed using the angular error between the normals of a vertex and its closest point in the ground truth mesh. OurAlbedoNet leads to highly similar results as our main framework in this scenario.

squirrel
bird
dog2
	Viewpoint 1	Viewpoint 2	[66]	OurAlbedoNet	Ours

Figure 5: Reconstructions from two distant viewpoints. Severe artefacts can be noticed for both [66] and OurAlbedoNet, and more particularly [66] introduce significant distortions in the shapes, as a result of using normal maps which do not provide strong enough constraints on the absolute position of the points.

To substantiate the efficacy of our framework, we conduct evaluations on both synthetic and real-world datasets. We generate four synthetic scans by combining two distinct geometries with two different materials. The first material is white, rendering it textureless, while the second material exhibits a high degree of texture. This design enables us to quantify the influence of texture on the obtained results. The four synthetic scans are denoted as dog1, dog2, girl1, and girl2. Additionally, we acquire real-world data for six objects, namely, bird, squirrel, hawk, rooster, flamingo and pumpkin. We also consider the DiLiGenT-MV dataset [30].
Evaluation. We first evaluate full 3D reconstruction using only six viewpoints in total, except for dog1 and dog2 where five viewpoints are considered. We evaluate against both state-of-the-art sparse viewpoint reconstruction approaches assuming static illumination [35, 51, 71, 62] and photometric stereo approaches [10, 66]. Since [10] allows for directional and point-light illumination, we consider both cases in our evaluation. [10]D and [10]P refers to the directional and point-light version, respectively. Given the calibrated nature of their framework, we furnish the estimated lighting parameters using our approach. Additionally, we explore a more challenging scenario wherein only two viewpoints with a wide baseline are available. We therefore introduce an ablated version of our framework, wherein the diffuse albedo is modeled with an MLP. This adaptation allows to highlight the advantages of employing the optimal diffuse albedo introduced in Sec. 5.2.
Full 3D reconstruction. As anticipated, [35, 51, 71] produce degenerate meshes in our wide baseline setup. Results in the supplementary demonstrate that they perform adequately when the distance between cameras is sufficiently small, although our framework consistently outperforms them. Conversely, Tab. 1 and Fig. 3 illustrate that our approach surpasses relevant baselines [62, 10, 66] both quantitatively and qualitatively. Our method yields sharper details without visible artifacts. Furthermore, error maps (shown in the supplementary material) reveal that the geometry obtained with [66] exhibits global distortions. Additionally, Fig. 4 demonstrates that both [62, 66] perform poorly when confronted with complex textures, while our results remain unaffected. This suggests that their pre-trained priors failed to generalize for the given texture, possibly misassigning it as geometric patterns. This shows the advantage of approaches without relying on learned priors. The results for the DiLiGenT-MV dataset [30] are shown in the supplementary, demonstrating a high accuracy in the classical scenario of a dark room and distant light sources.
Two viewpoints. We denote our ablated framework, which uses a diffuse albedo network, as OurAlbedoNet. We exclude [62] from consideration as they require at least three viewpoints. Fig. 5 vividly illustrates the advantage of our framework in this challenging scenario, especially with the use of the optimal diffuse albedo. Notably, [66] and OurAlbedoNet exhibit pronounced artifacts and inaccuracies in geometric details in certain regions. Moreover, [66] results in severe distortions of the shape. We attribute this to their heavy reliance on per-view normal maps for reconstruction, which lacks absolute depth information about the shape. In contrast, our method directly estimates the shape from various point-light illuminations, coupled with a suitable model (Eq. 1). This approach provides relevant cues about the absolute position of the points, contributing to more accurate and high-quality reconstructions.
Limitations and future work. A dedicated section is available in the supplementary.

7 Conclusion

We introduced the first framework for multi-view uncalibrated point-light photometric stereo. It combines a state-of-the-art volume rendering approach with a physically realistic illumination model consisting of an ambient component and uncalibrated point-lights. As demonstrated in a variety of experiments, the proposed approach offers a practical paradigm to create highly accurate $3$D reconstructions from sparse and distant viewpoints, even outside a controlled dark room environment.
Acknowledgment This work was supported by the ERC Advanced Grant SIMULACRON, and the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy – EXC 2117 – 422037984.

Supplementary Material

Appendix A Network Details

A.1 Architecture

As mentioned in the main paper, we use two multilayer perceptrons (MLPs). The first one describes the geometry via an SDF, $d_{\theta}$, and the other one is used for the specular parameters of the material, $\alpha_{\gamma}$. The MLP of $d_{\theta}$ consists of $6$ layers of width $256$, with a skip connection at the $4$-th layer, while the MLP $\alpha_{\gamma}$ consist of $3$ layers of width $256$.
In order to compensate the spectral bias of MLPs [40], the input is encoded by positional encoding using $6$ frequencies for both $d_{\theta}$ and $\alpha_{\gamma}$. For the ablation OurAlbedoNet, a third MLP describing the BRDF’s diffuse albedo, $\rho_{\gamma_{1}}$, is considered. It consists of $4$ layers of width $512$, and the input is encoded by positional encoding using $12$ frequencies.

A.2 Parameters and Cost Function

Similarly to [5, 76, 69], we assume that the scene of interest lies within the unit sphere, which can be achieved by normalizing the camera positions appropriately. To approximate the Volume rendering integral (4) using (5), we use $m=98$ samples which are also used to approximate (3), all with the sampling strategy of [70].
We set the objective’s function trade-off parameters $\lambda_{1}=\lambda_{2}=0.1$. Furthermore, the terms of the objective function (7) and (8) consist of a batch size of $800$ (inside the silhouette) and $1000$, respectively. For the mask term (9), we use the same batch as (7) and add $900$ additional rays outside the silhouette whose rays still intersect with the unit sphere.
Finally, we always normalize each objective function’s summand with its corresponding batch size.

A.3 Training

Our networks are trained using the Adam optimizer [29] with a learning rate initialized with $5\text{e}-4$ and decayed exponentially during training to $5\text{e}-5$, except for the MLP $\alpha_{\gamma}$ whose learning rate is constantly equal to $1\text{e}-5$. The light positions $\phi$ are initialized with the camera position of their corresponding viewpoint, with a learning rate initialized with $1\text{e}-2$, and decayed exponentially with the same rate as the other networks. The remaining parameters are kept to Pytorch’s default.
We train for $800$ epochs, which lasts about 6 hours using a single NVIDIA Titan GTX GPU with $12$GB memory and $6$ viewpoints.

Appendix B Data Acquisition

In this section, we describe how we generated the datasets used in this paper.

B.1 Synthetic Data

The synthetic datasets dog1, dog2, girl1, girl2 were generated using Blender [12] and Matlab [37], where Blender [12] is used to render normal, depth and BRDF parameter maps for each viewpoint, and Matlab [37] is used to render images using equation (1) of the main paper. We used $20$ point light illuminations for each viewpoint, with a ratio of $70\%$ of point light intensity (thus $30\%$ of ambient light), and we also added a zero-mean Gaussian noise with a standard deviation $\sigma=0.02$.

B.2 Real World Data

In order to generate the real-world datasets squirrel, bird, hawk, rooster, flamingo and pumpkin, we used a Samsung Galaxy Note 8 and the application "CameraProfessional" to generate RAW images as well as the smartphone’s images in parallel. We use the RAW images for our algorithm, and we pre-processed those using Matlab [37] by following [56]. Since our approach assumes very precise camera parameters, and in order to facilitate calibration, we captured a higher amount of viewpoints and used COLMAP [54] to obtain both camera poses and intrinsics with the smartphone’s images.
We move a hand-held LED to obtain $20$ images with different point light illumination per viewpoint.

Appendix C Small camera baseline

As mentioned in the main paper, [35, 51, 71] lead to degenerate meshes when considering distant cameras, and are thus not suited to reconstruct full $3$D objects from sparse viewpoints. For a more fair comparison, we focus here on a different scenario, where only a part of the object is reconstructed from three viewpoints with a very small camera baseline. Fig. 1 clearly indicates that our approach allows for much more accurate and complete $3$D reconstruction than [35, 51, 71]. Note that all the meshes were obtained solely by using the official implementations.

girl1
girl2
dog1
dog2
	Input viewpoints	[35]	[51]	[71]	Ours

Figure 1: Results using three viewpoints with small camera baseline.

Appendix D Error maps

For a better appreciation of the quality of the full $3$D reconstructions shown in the main paper, we show both the vertex-to-mesh distance and angular error maps in Fig. 2 and Fig. 3 respectively. We can see that our approach performs much better than the baseline at both the coarse and fine levels. Hence, it not only produces visually more pleasant reconstructions as can be seen in the main paper, but also with much higher fidelity.

girl1 girl2 dog1 dog2 [62] [10] (directional light) [10] (point-light) [66] Ours

Figure 2: Vertex-to-mesh distance error maps. Errors are truncated for better visibility.

girl1 girl2 dog1 dog2 [62] [10] (directional light) [10] (point-light) [66] Ours

Figure 3: Angular error maps. Note that for girl1 and girl2, a region of the plate at the bottom has significant errors for all approaches. This is due to the fact that in the ground truth mesh, vertices are only on the edges at that region, thus the angular error is not accurate there.

Appendix E Additional results

We can see in Fig. 4 the reconstruction results of our real-world scans that were not shown in the main paper. In order to further assess the quality of our framework on diverse materials, we performed an evaluation on the DiLiGenT-MV dataset [30]. Despite being captured with distant light sources, thereby satisfying the directional lighting assumption used in the baseline, our framework still achieves the best results both quantitatively and qualitatively as can be seen respectively in Tab. 2 and Fig. 5. Finally, we also show some relighting results in Fig. 8, together with the optimal diffuse albedo. This shows the validity of the estimated material parameters which can be successfully used for relighting, and indicates a proper disentanglement of the scene in terms of shape and material.

Input image	[62]	[10] (directional light)	[10] (point-light)	[66]	Ours

Figure 4: Full 3D reconstruction of real objects from 6 viewpoints.

[10]	[66]	Ours	Ground truth

Figure 5: Results on the DiLiGenT-MV dataset [30] using 6 viewpoints

	$\downarrow$MAE			$\downarrow$RMSE$\times 100$
	[10]	[66]	ours	[10]	[66]	ours
bear	19.1	8.8	3.5	2.2	1.0	0.6
buddha	39.6	13.9	10.8	2.5	0.7	0.5
pot2	29.2	9.2	5.1	10.2	0.7	0.5
reading	34.7	11.9	7.1	10.4	1.1	0.9
cow	25.3	8.9	3.6	6.1	0.9	0.5

Table 2: MAE and RMSE for the DiLiGenT-MV dataset [30].

Appendix F Effect of the ratio of point light

We further analyze the effect of the ratio of point light intensity on the quality of the result. This allows us to know how much ambient light can be handled by our approach while still providing accurate reconstructions. We remind that the total radiance can be decomposed into the sum of the point light radiance and the ambient light radiance, and we obtain the point light images by subtracting the input images with the ambient image. As discussed in section (3.2) of the main paper, one key issue with this strategy is that decreasing the amount of point light intensity yields point light images with a worse signal-to-noise ratio, which will inevitably affect the quality of the result. Consequently, we evaluate our approach on dog2 using the same five viewpoints as in the main paper to obtain a full $3$D reconstruction, with a point light intensity ratio ranging from $10\%$ to $100\%$ (dark room). We also consider two levels of noise, with standard deviations $\sigma\in\{0.02,0.04\}$. As shown in Tab. 3 and Fig. 8, the quality of the result indeed improves as expected when increasing the amount of point light intensity. Moreover, for a given desired accuracy, a higher amount of point light is required for a noisier sensor, since this last yields the worst signal-to-noise ratio for the point light images. Nevertheless, even with a significant amount of noise, a reasonable result can be obtained starting from $40\%$ of point light intensity, and a high accuracy with $70\%$ and above. As mentioned in section (3.2) of the main paper, satisfying those requirements in practice is highly facilitated by the fact that near point lights are handled properly by our framework, in contrast to the majority of photometric stereo frameworks which require distant lights.

	$\downarrow$RMSE$\times 1000$										$\downarrow$MAE
	$10\%$	$20\%$	$30\%$	$40\%$	$50\%$	$60\%$	$70\%$	$80\%$	$90\%$	$100\%$	$10\%$	$20\%$	$30\%$	$40\%$	$50\%$	$60\%$	$70\%$	$80\%$	$90\%$	$100\%$
$\sigma=0.02$	10.3	5.6	4.3	3.9	3.3	3.4	3.2	3.2	2.9	2.6	12.7	7.8	6.3	5.7	5.2	5.0	4.7	4.8	4.6	4.4
$\sigma=0.04$	16.1	10.3	6.5	5.2	4.5	3.9	3.6	3.6	3.5	3.2	16.7	12.8	9.4	7.8	6.9	6.2	5.8	5.5	5.4	5.0

Table 3: RMSE and MAE for different ratios of point light intensity, and two different levels of noise. RMSE is computed based on the vertex-to-mesh distance, and the MAE is computed using the angular error between the normals of a vertex and its closest point in the ground truth mesh.

$\sigma=0.02$ $\sigma=0.04$ $20\%$ $40\%$ $70\%$ $100\%$

ground truth

Figure 6: Results on dog2 for different ratios of point light intensity. The first and second rows correspond to the results with Gaussian noise of standard deviation $\sigma=0.02$ and $0.04$ respectively.

Ground truth	Our rendering	Optimal albedo

Figure 7: Relighting results.

	5L	10L	20L
4V	7.7	6.1	5.5
6V	6.4	5.0	4.6
8V	6.0	4.6	4.3
12V	5.1	4.2	3.9

(a) dog2

	5L	10L	20L
4V	13.7	12.4	12.4
6V	12.1	10.7	10.6
8V	11.3	10.3	10.1
12V	10.1	9.4	9.4

(b) girl2

Figure 8: MAE for different number of Viewpoints / Lights.

Appendix G Effect of the number of viewpoints and lights

Fig. 8 shows the MAE for both dog2 and girl2 using different numbers of viewpoints and light sources. Four viewpoints and five light sources allow to obtain a decent full $3$D reconstruction, and six viewpoints and ten light sources are already enough for a high quality result.

Appendix H Limitations

The optimal diffuse albedo allows to obtain great $3$D reconstruction results in the most sparse scenarios. However, it is only defined for the viewpoints used for training, and is not multiview consistent, hindering novel view synthesis from arbitrary viewpoint. On the other hand, this issue is mitigated with our ablation OurAlbedoNet by using a neural diffuse albedo, at the cost of failing in some highly sparse scenarios. A straightforward solution would be to first use the optimal diffuse albedo strategy, then fix the geometry and specular parameters, and learn a neural diffuse albedo in a second stage. Successfully achieving multiview consistent diffuse albedo in the most sparse scenarios without relying on a second stage might increase the overall robustness, and is left as a future work. Moreover, we presume the availability of camera poses, acknowledging the challenge of pose estimation, particularly in the context of sparse viewpoints. A valuable extension of our work could be to address this assumption, e.g., based on [23]. Finally, our BRDF choice is limited to opaque, non-metallic objects. Expanding our framework beyond those materials represents an intriguing avenue for future exploration.