Neural Reflectance Field from Shading and Shadow under a Fixed Viewpoint
Anonymous Author s)
AffiliationAddress
Abstract
In this paper, we address the “dual problem” of multi-view scene reconstructionin which we utilize single-view images captured under different point lights tolearn a neural scene representation.
Different from existing single-view methodswhich can only recover a 2.5D scene representation (i.e., a normal / depth mapfor the visible surface), our method learns a neural reflectance field to representthe 3D geometry and BRDFs of a scene.
Instead of relying on multi-view photoconsistency, our method exploits two information-rich monocular cues, namelyshading and shadow, to infer scene geometry.
Experiments on multiple challengingdatasets show that our method is capable of recovering 3D geometry, includingboth visible and invisible parts, of a scene from single-view images.
Thanks to theneural reflectance field representation, our method is robust to depth discontinuities.
It supports applications like novel-view synthesis and relighting.
Our code andmodel will be made publicly available.
Introduction
3D reconstruction from images is a central and important problem in computer vision.
Multi-viewstereo methods, which capture a target scene from multiple viewpoints under a fixed lighting condition [11, 22, 43, 44], are the most widely adopted approach for scene reconstruction.
These methods,however, often assume surfaces with Lambertian reflectance and have difficulties in recovering highfrequency surface details.
An alternative approach to scene reconstruction is to utilize images captured from a fixed viewpointbut under different point light sources (see Fig.
1 (a)).
This setup is adopted by photometric stereo(PS) methods [14, 45, 54] where shading information is utilized to reconstruct surface details of nonLambertian objects.
Shadow is another cue that has been exploited for shape recovery by shape-fromshadow methods [10, 59, 64].
However, existing single-view methods typically adopt a single normalor depth map to represent the visible surface, making them incapable of describing back-facing andoccluded surfaces (see Fig.
1 (b)).
Besides, methods relying on surface normal representation struggleto deal with depth discontinuities [23].
It is desirable to obtain a more complete scene reconstruction(including both visible and invisible parts) from single-view images.
In this paper, we realize this byexploiting both shading and shadow cues to recover both visible and invisible parts of a scene.
Recently, neural scene representations have achieved significant progress in multi-view reconstructionand novel-view synthesis [33, 46, 60].
These methods model a continuous 3D space (i.e., the scene)with a multi-layer perception (MLP) network which maps 3D points to scene properties ( e .
g ., densityand color in NeRF [33]).
Despite its great success in multi-view scene modeling, neural scenerepresentation has been less explored in single-view scene modeling.
[Figure]
In this paper, built on top of the recent advances in neural scene representation, we propose tooptimize a neural field using images captured from a single viewpoint under different point lights.
Our method is fundamentally different from existing works [33, 46, 60] in that, instead of relying onmulti-view photo-consistency, we exploit monocular shading and shadow cues to optimize our neuralfield for scene reconstruction.
A straightforward idea would be to condition the color MLP of NeRF [33] also on the point lightdirections.
However, we find such a naïve solution fails to recover the scene geometry and appearance.
To make better use of the photometric stereo images, we explicitly model the surface geometry and
BRDFs with a reflectance field and adopt a physics-based rendering to obtain the 3D point color [2, 3].
The 2D pixel color of a sampled ray can then be computed with volume rendering.
Differentiableshadow computation is explicitly considered by tracing a ray from a 3D point to the point light positionto check the light visibility [68].
As evaluating the light visibility of all points sampled along a ray iscomputationally expensive [47], we accelerate the computation by only evaluating the light visibilityof the expected surface point, making online shadow computation possible during optimization.
To summarize, our contributions are:
• We address a novel problem of 3D neural reflectance field optimization from single-view images
captured under different point lights.
Different from existing neural scene representation methodsthat rely on multi-view photo-consistency, our method exploits monocular shading and shadowcues for neural field optimization.•
Our method jointly recovers the geometry and BRDFs of a scene, and adopts an efficient onlineshadow computation to fully exploit the information-rich shading and shadow cues.• Experiments on multiple challenging datasets show that our method can faithfully reconstruct acomplete scene geometry from single-view images.
Our method is robust to depth discontinuities.
It supports applications like novel-view synthesis and relighting.
Related Work
Photometric stereo (PS) PS methods can recover pixel-wise surface normals from images capturedunder different light directions [14, 54].
Traditional PS methods treat specular observations asoutliers [34, 55, 56] or fit sophisticated reflectance models [9, 17, 52] to handle non-Lambertiansurfaces.
Recent methods resort to deep learning technique to solve this problem.
Supervised learningmethods learn a mapping from image observations to surface normals using synthetic dataset withground-truth normals [7, 8, 16, 24, 30, 41, 70].
Self-supervised methods optimize the networkparameters using an image reconstruction loss [21, 49].
The above methods assume directionallightings.
For near-field PS problem, methods based on PDE [38] and deep learning [27, 29, 32, 42]have been proposed.
More recently, Li et al.
[23] proposes a coordinate-based MLP to represent the normal map of the visible surface assuming directional lights.
In contrast, our method represents ascene with a continuous volume and recovers the full 3D scene geometry under a near-field setup.
Shape from shadow Shadow has been exploited to estimate shape information [10].
Yu and Changoptimized a height map from shadow cues using a graph-based representation [64].
Shadowcuts [6]explicitly considers shadow in Lambertian photometric stereo.
Yamashita et al.
[59] introduced a1D shadow graph to accelerate the shadow computation.
Recently, DeepShadow [20] models thedepth map of a scene by an MLP and optimizes the model with a shadow reconstruction loss.
Thesemethods can only recover a height map of the visible surface.
Besides, they require the detectedshadow regions as input, but shadow detection is itself a non-trivial problem.
Neural scene representation Neural scene representations have been successfully applied in novelview synthesis and multi-view reconstruction [36, 46, 50, 57, 60].
The popular neural radiance field(NeRF) [33] represents a continuous space with an MLP, which regresses the volume density and RGBcolor of a 3D point from the point coordinates and view direction.
Attracted by the photo-realisticrendering produced by NeRF, many follow-up works are introduced to improve the reconstructedsurface quality [37, 53, 61], rendering speed [13, 28, 39], optimization speed [35, 48, 62], androbustness [1, 31, 65].
The above methods consider each 3D point as an emitter, making them not able to model the surfacematerials and lighting separately.
Inverse rendering methods have been proposed to jointly recovershape, materials, and lightings in a casual capture setup [3, 4, 66, 68, 69].
NeRV [47] explicitly modelsshadow and indirect illumination assuming a known environment map.
NRF [2] and IRON [67] adopta co-located camera-light setup to simplify the image formation model.
There are some attempts to reconstruct a radiance field from a single-view image in a data-drivenmanner (e.g., conditioning the MLP input with image features [12, 40, 63]), or utilizing depth image asshape prior [58].
However, due to the strong ambiguity, these methods struggle to achieve high-qualityreconstruction.
Compared with the above approaches, our method extends neural scene representationto reconstruct accurate shape and materials from single-view photometric stereo images.
Method
Given N images captured from a single viewpoint under different near point lights, our method targetsat recovering the geometry and materials for the scene.
Following existing near-field photometricstereo methods [32, 42], we assume a calibrated perspective camera and known point light positions.
Instead of representing the visible surface with a normal / depth map like others [23, 32, 42], weadopt a 3D neural field representation [3, 33, 37] to describe the 3D scene.
3.1 Neural Reflectance Field Representation
Our method is built on top of the recent neural radiance field (NeRF) [33].
Following UNISURF [37],we adopt an occupancy field instead of a density field to better represent the surface geometry.
UNISURF uses an MLP to map a 3D point x ∈ R 3 and a view direction d ∈ R 3 to occupancyo ( x ) ∈ R and color c ( x , d ) ∈ R 3 .
An image can be generated through volume rendering in whichthe color of each pixel (or ray r ) is calculated by
[Equation]
where x i denotes a 3D point sampled along the ray r
= o + t d , with o ∈ R 3 being the camera centerand d ∈ R 3 the ray direction specified by the pixel, and N V ∈ R is the number of samples per ray.
Given multi-view images, a radiance field can be optimized to reproduce the input images.
However,applying NeRF-based methods to single-view images is non-trivial.
A straightforward idea wouldbe to condition the color MLP of NeRF also on light directions, but our experiments show that sucha naïve solution fails to produce reasonable reconstruction due to the lack of constraints on scenegeometry.
To utilize shading information in photometric stereo images, we explicitly model the BRDFs ofthe scene and recover 3D point color with physics-based rendering [2, 3].
Observing that shadow
MLP
MLP
MLP x x pe ρ d ω o ∇ n
Renderer c d p l
L e s L i =
L e S
MLP x pe o
Accu-L v x s p l x s
L e d
Accu-V { x i } N L { x i }
N V { x i } N V { x i } N L
Accu-L 1 −
Shading
Shadow shared-weight provides strong cues for inferring the geometry of both visible and invisible surfaces in a scene, wecompute shadow in an online manner by tracing a ray from a surface point to the light position todetermine its light visibility.
Give a point light located at p l ∈ R 3 with emitted intensity L e ∈ R ,
Eq.
(1) can be rewritten as
[Equation]
where the 3D point color c ( x , d ) is replaced by the product of light visibility f v ( p l ; x ) and physicsbased rendered color f c ( d , p l , L e ; x ) , the details of which are given in the following subsections.
3.2 Physics-based Color Rendering
We consider non-Lambertian surfaces with spatially-varying BRDFs.
The rendering equation for asurface point x viewed from a direction d under a near point light ( p l , L e ) can be written as
[Equation]
L [Equation] where L int ( p l , L e ; x ) denotes the incident light (taking light falloff into account), w i ( p l ; x ) theincident light direction, and f m ( d , w i ( p l ; x ); x ) the BRDF value at x .
The normal at x can bederived from the gradient of the occupancy field as n ( x ) = ∇ o ( x ) / ∥∇ o ( x ) ∥ 2 [37].
Lighting model
Following previous works [32, 42], we adopt the inverse-square law for point lightattenuation where light intensity L int is proportional to the multiplicative inverse of the square of thedistance s ( i .
e ., L int ∝ 1 /s 2 ).
The incident light direction w i and light intensity L int at a point xare given by
[Equation]
BRDF model Similar to [3, 66, 68], we adopt a BRDF model represented by a combination ofdiffuse color
ρ d and specular reflectance ρ s , which is given by
[Equation]
Following [15, 23], we model the isotropic specular reflectance by a weighted combination of
Sphere Gaussian (SG) bases, which demonstrates better results in modeling specular effects than theparametric Microfacet model [19].
The specular component ρ s is hence written as ρ s = ω T D ( h , n ) ,where D ( h , n ) =
G ( h , n ; λ ) = (cid:104)
e λ 1 ( h T n − 1 ) , · · · , e λ k ( h T n − 1 ) (cid:105) T denotes the SG bases, withλ ∗ ∈ R + controls the specular sharpness.
The diffuse component ρ d
and SG weights ω are estimatedby two MLPs.
3.3 Online Shadow Computation
A 3D point x is shadowed if there is any occluders in its line of sight for the light position p l .
Itfollows that light visibility f v ( p l , x ) ∈ [0 , 1] for a 3D point x can be computed by accumulatingoccupancies along this line (see Fig. 2), i .
e .,
[Equation]
[Equation]
) where N L is the number of points sampled along the line.
[Equation]
[Figure]
However, calculating light visibilities for all N V pointssampled along the ray for a pixel is computationally expensive ( i .
e .
, O ( N V N L ) MLP queries for each pixel / ray(see Fig.
3 (a)).
To speed up shadow computation, previous methods either adopt an MLP to directly regresslight visibility of a point [47] to reduce the queries foreach ray to O ( N V ) (see Fig.
3 (b)), or pre-extracts thesurface points (assuming a fixed scene geometry) [68] toreduce the number of MLP queries to O ( N L ) .
Instead,we first locate the expected surface points x s along theray by root-finding [37] and calculate its light visibilityin an online manner.
Eq.
(2) can be reformulated for efficient color rendering as
[Equation]
3.4 Optimization
Different from shape-from-shadow methods [20, 51], our method does not require direct supervisionfor shadow rendering.
We rely on image reconstruction loss for optimization.
Volume rendering loss The first loss is the L1 reconstruction loss between the volume renderedimage C v ( i . e ., the computed C ( r ) in Eq. (7)) and the input image:
[Equation]
Surface rendering loss UNISURF [37] proposes to combine the volume rendering and surfacerendering by gradually shortening the sampling range in a ray to refine the surface region.
However,we empirically found that the model will start to degrade when the sampling interval is decreased asthere is no multi-view information to constrain the non-sampled regions.
We therefore propose toadopt a joint volume and surface rendering strategy.
We additionally compute the surface renderingcolor C s ( r ) using the expected surface point x s and calculate the L1 loss, i .
e .,
[Equation] [Equation]
Normal smoothness loss Similar to [37], we also include a regularization loss to promote smoothnessin surface normal ( ϵ is a small random perturbation):
[Equation]
Overall loss The overall loss function used for optimization is as follow with α set to 0.005:
[Equation]
Experiments
Due to the page limit, we include the implementation details in the supplementary materials.
Weadopt mean angular error (MAE) in degree for surface normal evaluation and L1 error in cm fordepth assessment.
PSNR is used to measure the quality of images rendered under novel view or novellighting. [Table]
4.1 Datasets
Our method targets at recovering the complete scene by exploiting both shading and shadow.
However,existing photometric stereo datasets are mostly interested in the object region and intentionallyremove the influence of the background ( e .
g ., cover the background with black cloth to avoid interreflections [45]), which makes the shadow and shading information invisible in the backgroundregions.
Therefore, such datasets [32, 45] are not suitable to evaluate the full potential of our method.
Instead, we evaluate our method on multiple synthetic datasets with complicated scene geometry andmaterials.
Specifically, we used 10 3D objects for data rendering, where 5 objects from DiLiGent-MV
Dataset [25] (namely, BEAR , BUDDHA , COW , POT2 , and READING ), 2 objects from the internet(namely, BUNNY and ARMADILLO ), and 3 objects from NeRF’s blender dataset [33] (namely,
LEGO , CHAIR , and HOTDOG ).
We rendered LEGO , CHAIR , and HOTDOG with Blender’s Cyclespathtracer, and the other 7 objects with Mitsuba [18].
During rendering, we created a scene by addinga horizontal and a vertical plane to model the desk and wall, and objects are placed on the horizontalplane.
Each scene was rendered under 128 uniformly sampled near point lights, and the renderedimages are in linear space with a resolution of 512 × 512 .
4.2 Comparisons with Existing Methods
To justify the effectiveness of our method, we compare it with three types of methods, namely, neuralfield methods, photometric stereo methods, and single-image shape estimation methods.
Neural radiance field methods We first verify the design of our method by comparing it with twosimple baselines ( i .
e ., adapting NeRF [33] and UNISURF [37] for this problem by conditioning thecolor MLP on light direction).
Table 1 and Fig.
4 show the normal estimation and relighting resultsin the training view.
Although the baseline methods can achieve reasonable rendering results in termsof PSNR, they fail to predict accurate cast-shadow and cannot reconstruct the geometry of the scene(with a large average MAE of 77.12/52.39).
In contrast, our method is able to accurately reconstructthe shape with an average MAE of 2.92, and achieves the best rendering results (average PSNR of39.50).
This result indicates that simply conditioning the color MLP on light direction does notprovide sufficient constraint to regularize the scene geometry.
Single-view shape estimation methods We then compare with three state-of-the-art single-viewnormal / depth estimation methods, including two near-field PS methods (QY18 [38] and HS20 [42])and one single-image shape estimation method (ZL18 [26]).
QY18 [38] and HS20 [42] considerexactly the same setup as our method (multiple images captured under near point lights), so the inputare the same as our method.
ZL18 [26] assumes an image captured under co-located flash light asinput, so we choose the image illuminated by a point light that is closest to the camera as its input.
Asthese methods are designed to estimate the shape in the object region and have difficulty in dealingwith the background, we only report the normal and depth estimation results on the object region for [Table] [Figure] the training view in Table 2.
We can see that our method achieves the best results for both normal anddepth estimation.
Moreover, as shown in Fig. 5, our method can faithfully reconstruct both visibleand invisible parts of the scene, which is not possible by methods that rely on the normal or depthrepresentation.
4.3 Method Analysis
We next conduct ablation study for different components of our method, and evaluate our method ondifferent setups to further analyze its behavior.
Joint shading and shadow modeling Our method exploits both shading and shadow informationfor scene reconstruction.
To analyze the effect of both components in reconstructing the scenegeometry, we trained two variant models where (a) “w/o shading” replaces the BRDF module with a4-layer MLP to directly predict RGB values with additional light location input; and (b) “w/o shadow”removes the shadow module and only output the shading.
Results are summarized in Table 3 and
Fig.
We evaluate the results for both trained view and novel views, and report MAE of normal mapsand PSNR of rendered images.
As the “w/o shading” model fails to estimate proper depth and therecovered surfaces totally deviate from the ground truth, we omit its results for novel views.
Withoutshadow information, the model may still predict proper surface normal for the trained view.
However,the model fails to predict the depth and shape of the object since there is no constraint on invisibleregions.
By exploiting both shading and shadow cues, our method can well reconstruct the full scene.
Joint volume and surface rendering We also analyze the effectiveness of the surface renderingloss L s by comparing our full model with the one without L s .
Results in Table 3 and Fig.
6 showthat surface rendering loss can effectively refine the surface normals of the object surface. [Table]
Ours 45˚ 0˚ [Figure] Effect of occlusion/discontinuity We further demonstrate the potential of our method in reconstructing the complete geometry of the scene, especially when there are occlusions or discontinuoussurfaces.
Figure 7 shows the reconstructions for two scenes with multiple objects occluding eachother.
It is very difficult to identify the shape of the invisible regions just from the single-view images.
However, by effectively leveraging the shadow information, our method successfully predicts theshape ( i .
e ., the occupancy field) of the invisible regions, which is not feasible for existing works.
We also investigate the performance of our method on surface with invisible concave shape.
Note thatthe shape of invisible regions are mainly constrained by the shadow (which indicates the occupancyalong the light path).
In Fig.
8, we show the reconstruction of READING object which is posed tomake the concave surface invisible.
From the results of novel views, we can see that our method canproperly recover the concave surface, though some invisible regions are not fully consistent with theground truth shape.
This result demonstrates that shadow provides strong cue for shape recovery.
Effect of light distributions To analyze the robustness of our method on different light distribution,we evaluate it on scenes illuminated by different number of lights or different ranges (see our supplementary material for visualization of light distributions), and the numerical results are summarizedin Table 4 and Table 5.
The model fails to reconstruct the scene with only 4 light inputs, but canreconstruct faithful shape of the scene with 8 light inputs.
With more lights used, the surface of theobject is further refined (see Fig. 9).
We can also observe that our method can still work for smallrange of light distributions to recover invisible regions.
Overall, our method is robust to differentnumber and different range of lights.
Results on real scenes To further demonstrate the practicality of our method, we evaluate on threereal scenes, which were captured using a fixed camera and a handheld cellphone flashlight.
Tocalibrate light positions, we utilized the state-of-the-art uncalibrated PS method [7] to provide a roughinitialization and jointly optimize the lightings during training.
Please refer to our supplementarymaterials for more details.
Sample inputs and results are shown in Fig.
Even with this casualcapturing setup and uncalibrated lights, our method achieves satisfactory results in normal predictionand full 3D shape reconstruction.
[Table] [Table] GT / Input [Table]
Broad
[Figure] [Figure]  Conclusion In this paper, we have introduced a method to optimize a neural reflectance field for a non-Lambertianscene from single-view images captured under different near point lights.
Our method jointly recoversthe geometry ( i .
e ., occupancy field) and BRDFs of the scene by fully utilizing the shading andshadow cues.
Interestingly, our results on scenes with complicated shapes and materials show that thecomplete scene geometry can be faithfully reconstructed just from single-view photometric images.
Moreover, comprehensive method analysis demonstrates that our method is robust to scenes withdifferent geometry, materials, light number, and light distributions.
Additionally, our method supportsapplications like novel-view synthesis and relighting.
Limitation First, as existing near-field PS methods [42], our method requires known light positions,which requires additional efforts for lighting calibration.
Second, as our method relies on shadowcues for invisible shape reconstruction, its performance may decreased if the scene background ishighly complicated as the background geometry will affect the appearance of the shadows.
Third,although the shape of the invisible parts can be well reconstructed, the reflectance of those regionsare not well constrained by shadow.
Last, our method ignores the inter-reflection effects in imageformation.
In the future, we will further extend our method to tackle these limitations.
In the future,we will further extend our method to tackle these limitations.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 5 Limitation.(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] We discuss
it in supplementary materials.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No] Code will be
released upon publication.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See supplementary materials for training details.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See supplementary materials.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] See supplementary materials.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]