Original: NeIF: Representing General Reﬂectance as Neural Intrinsics Fields for Uncalibrated Photometric StereoAbstract Uncalibrated photometric stereo (UPS) is challenging due to the inherent shapelight.
Revised: NeIF: Representing General Reflectance as Neural Intrinsics Fields for Uncalibrated Photometric Stereo

Original: Existing solutions alleviate this ambiguity by either explicitly associatingreﬂectance to light estimation or resolving light conditions in a supervised manner.

Original: This paper establishes an implicit relation between light clues and light estimationand solves UPS in an unsupervised manner.

Original: The key idea is to represent thereﬂectance as four neural intrinsics ﬁelds, i.e , specular, shadow, light, and position,based on which the neural light ﬁeld is implicitly associated to light clues ofspecular reﬂectance and cast shadow.

Original: The unsupervised training of neural intrinsicsﬁelds fully exploits all observed pixel values, which facilitates solving UPS.

Original: Ourmethod demonstrates a superior performance advantage over state-of-the-art UPSmethods on public datasets and promising results under the challenging setting ofsparse UPS.

Original: We will release our codes and pre-trained models once the paper wasaccepted.

Original: Introduction
Revised: Abstract

Original: Photometric stereo (PS) [47] aims at recovering the surface normal from several light-varying imagescaptured at a ﬁxed viewpoint.

Original: As compared with other stereo approaches ( e g. , multi-view stereo

Original: [38],active sensor-based solutions [55]), photometric stereo is excellent at recovering ﬁne-detailed surfacesand has been widely used for Hollywood movies [9], industrial quality inspection [46] and biometrics[52].

Original: Calibrating accurate lighting directions is crucial to the performance of photometric stereomethods [51].

Original: However, lighting calibration is often tedious, dramatically limiting the applicability inthe real-world.

Original: To this end, researchers develop uncalibrated photometric stereo (UPS) methods thatestimate surface normal with unknown lights.

Original: Uncalibrated photometric stereo suffers from General Bas-Relief (GBR) ambiguity [6] for an integrable surface.

Original: Early solutions address the ambiguity by explicitly associating reﬂectance to light, i.e ,adopting analytic reﬂectance models ( e.g. , Lambertian reﬂectance [4], [34], parametric specular reﬂection [18], specular spikes [53], inter-reﬂection [10]) or imposing priors from reﬂectance properties[3, 22, 21, 41].

Original: Thus, these methods could be less effective for unknown reﬂectance due to the strongreliance reﬂectance assumption.

Original: Besides, these methods ignore the clue of cast shadow and even failfor shadow regions due to the shadow-free surface assumption.

Original: Further, most of them require thelight intensity to be identical for robust estimation.

Original: Recently, deep learning-based approaches addressthe ambiguity by estimating light direction and intensity before recovering surface normal [12, 13, 25,37].

Original: They train a light estimation network using a large-scale amount of rendered data in a supervisedmanner.

Original: However, the training data bias [30] can hardly be eliminated and could produce unexpectedestimation for real-world data.

Original: Because rendered training data inevitably contain the domain gapfrom real ones and could scarcely cover all surfaces with different geometry and reﬂectance in thereal-world.

Original: Besides, such a two-step solution could bring accumulating errors for surface normal

Original: Neural Shadow Field

Original: P o s i t i on

Original: Light direction ( ) + × × = × × × · · · 𝒏 𝑖 𝑘 𝑖 𝑑 𝜌 𝑖 𝑑 𝑘 𝑖 𝑠 Neural Position Field

Original: P o s i t i on

Original: Neural Specular Field

Original: S

Original: u r f a c

Original: e

Original: N o r m a l

Original: Light direction

Original: Neural Light Field

Original: O b s e r v ed I m age s

Original: Light direction · · · 𝜌 𝑖𝑑 (𝒍 𝑗 ,𝑒 𝑗 ) 𝑠 𝑖𝑗 𝑘 𝑖𝑑 𝑘 𝑖𝑠 𝜌 𝑖𝑗𝑠 𝒏 𝑖 𝑚 𝑖𝑗 · · · · · · · · · · · · · · · · · · position-dependent light- dependent 𝜌 𝑖𝑑 ✓  𝑘 𝑖 𝑠 ✓  𝑘 𝑖𝑑 ✓  𝒏 𝑖 ✓  𝒍 𝒋  ✓ 𝑒 𝑗  ✓ 𝜌 𝑖𝑗𝑠 ✓ ✓ 𝑠 𝑖𝑗 ✓ ✓

Original: To this end, we propose NeIF , representing general reﬂectance as Ne ural I ntrinsics F ields foruncalibrated photometric stereo.

Original: Our method differs from previous methods in three aspects: 1)it fully considers clues of specular reﬂectance and cast shadow from each observed pixel for lightestimation so that it is expected to produce accurate estimation for both light conditions and surfacenormal; 2) it does not make explicit assumptions about the reﬂectance or light so that it could workwith general surface reﬂectance and

Original: ﬂexible

Original: light settings; it

Original: infers

Original: light and surface normal

Original: in

Original: anunsupervised manner so that it could be free from training data bias and achieve stable performance

Original: for data from difference sources

Original: Our key idea is to represent the general reﬂectance as four neural intrinsics ﬁelds ( i.e , specularreﬂectance, cast shadow, light, and position, see Fig. 1), implemented by four multi-layer perceptrons(MLPs).

Original: The connection of these four ﬁelds is based on the implicit relation (or dependence) of theseintrinsics so that no explicit assumption is imposed, e.g. , we take the estimated light as the input torecover specular reﬂectance and cast shadow instead of explicitly exploiting them for light estimation.

Original: These intrinsics ﬁelds are optimized to reconstruct each pixel value from observed images, whichfully exploit mutual information across different dimensions, as shown in Fig.

Original: The reconstructionerror backpropagates to the neural light ﬁeld through neural specular and shadow ﬁelds, so that cluesof specular and shadow can be implicitly and fully considered for light estimation.

Original: Our contributionsare summarized as

Original: • We represent general reﬂectance as four intrinsic neural ﬁelds to implicitly associate per-

Original: pixel reﬂectance to light, which fully considers clues of specular reﬂectance and cast shadow

Original: for light estimation.

Original: • We propose the NeIF, an uncalibrated photometric stereo method trained in an unsupervisedmanner, which works with general surface reﬂectance, ﬂexible light settings, and trainingdata bias.

Original: • We show that our method achieves a superior performance over uncalibrated and unsupervised methods.

Original: We also demonstrate its excellent generalization capacity todata from different sources and promising performance with the challenging setup of sparse uncali- brated photometric stereo.  Related Work
Revised: Uncalibrated photometric stereo (UPS) is challenging due to the inherent ambiguitybrought by unknown light.

Original: In this section, we mainly review the latest works in neural reﬂectance representation, and related

Original: 2.1 Neural Reﬂectance Representation

Original: Neural Radiance Fields (NeRFs)

Original: [31] focuses on the 3D geometries without explicitly modeling theinteraction between light and objects via photometric image formation model.

Original: Many subsequentworks explore its application in various computer vision problems, such as relighting [14], photometricstereo [26], and multi-view photometric stereo [24].

Original: These works either require known lightingconditions [ bi2022neural , 43, 58, 26] or adopt simple reﬂectance model [56].

Original: There are alsosome works [7, 57] considering general reﬂectance, but they require multi-view information anddoes not explicitly consider cast shadow effects.

Original: Kaya et al.

Original: [24] implemented NeRF in multiview photometric stereo, but their work also leverages multi-view information.

Original: Different from thesemethods, our method considers general reﬂectance, cast-shadow effects, and unknown light conditionsfor uncalibrated photometric stereo by taking images captured at a single viewpoint.

Original: 2.2 Unsupervised Calibrated Photometric Stereo

Original: Classical methods solved the calibrated photometric stereo problem without knowing the groundtruth surface normal.

Original: Therefore, we classify them as unsupervised methods.

Original: The least square basedalgorithm [47] provides the simplest solution, which assumes the object to be Lambertian.

Original: Thealgorithm’s performance is robust in a wide range of datasets, but its strong assumption on thereﬂectance model remains much space for improvement.

Original: The following works either regard thenon-Lambertian reﬂectance components as the outliers [5, 16, 33, 48, 23, 49] or apply analyticreﬂectance models including Torrance-Sparrow[18], the Ward model[16], a mixture of multiple Wardmodels [19], [1] etc. to consider the non-Lambertian effects.

Original: However, the performance of thosemethods can only deal with limited types of materials.

Original: There are also more advanced methods thatutilize the general reﬂetance features such as reciprocity, isotropy [2], and monotonicity [21].

Original: Thosemethods give a reliable estimation for objects with a broad range of materials.

Original: Inspired by them, ourapproach considers the general reﬂectance properties.

Original: With the progress of deep learning, many learning-based frameworks have been proposed for calibrated photometric stereo.

Original: Taniai et al.

Original: [44] propose the ﬁrst unsupervised learning-based photometricstereo method through a rendering equation.

Original: However, their reﬂectance model does not separatelyconsider shadow, specular highlights, and diffuse components.

Original: Different from these methods that need the known lights, we address the uncalibrated photometricstereo.

Original: We also train our method in an unsupervised manner.

Original: Because it is expected to be free fromtraining data bias and thus obtain a stable estimation across data from different sources.

Original: 2.3 Uncalibrated Photometric Stereo

Original: Previous works study the phenomenon where Lambertian objects under arbitrary directions cangenerate identical images when similar transformations are applied to both the object’s normal andthe light’s direction.

Original: Belhumeur et al.

Original: [6] reduce the dimension of the transformation to a threeparameters GBR transformation by considering integrability constraints.

Original: Based on that, extra clueson reﬂectance, such as half-vector symmetry [28], albedo clustering [42], specular spikes [53], orassumptions on light source distribution, such as ring light [60], symmetry light [32], or uniformdistributed light sources [54, 4, 42, 50, 34, 28], are used to resolve the GBR ambiguity.

Original: However,all these methods require the light intensity to be identical, which is inapplicable in the real-worlddatasets such as D

Original: I L I G EN T [39], A PPLE & G OURD [2], and L IGHT S TAGE D ATA G ALLERY

Original: Choet al.

Original: [15] put up a semi-calibrated method todeal with non-uniform light intensity, but they assumedthe light directions to be known. Quéau et al. [36] address the photometric stereo problem underinaccurate lighting calibration, while the accuracy could signiﬁcantly drop when non-Lambertiancomponents become dominant. Recently, many deep learning methods have been proposed for uncalibrated photometric stereo.
Revised: Existing solutions alleviate the ambiguity by eitherexplicitly associating reflectance to light conditions or resolving light conditions ina supervised manner.

Original: Chen et al. [12] propose a supervised uncalibrated framework, SDPS-Net, which can simultaneouslyestimate the lightconditions (intensity and direction) and the surface normal. They suggest treatinglight estimationas a classiﬁcation problem and separate the normal and light prediction to reduce thecomplexity. Their following work, GC-Net[13], improved the performance of SDPS-Net by addingshading as an extra channel to the input of the light estimation network.
Revised: This paper establishes an implicit relation between light cluesand light estimation and solves UPS in an unsupervised manner.

Original: However, as a commonproblem for all supervised methods, an over-ﬁtting problem may occur due to the training data bias[30].

Original: In contrast, unsupervised methods do not have such a concern.

Original: Another obvious beneﬁt is that there is no need to synthesize training sets for unsupervised network training.

Original: To utilize theadvantage of unsupervised methods, Kaya et al.

Original: [25] propose a compromised method that trains thelight estimation network in a supervised manner (similar to [12]), but estimate the surface normal inan unsupervised way (similar to [44].

Original: However, they still suffer from the training data bias duringlight estimation.

Original: Compared to previous works, our method neither makes a strict assumption onreﬂectance nor needs special light source distribution.

Original: Instead, we solve the uncalibrated photometricstereo problem in an unsupervised manner for general reﬂectance surfaces with unknown light sourceinformation.

Original: Method

Original: 3.1 Problem Formulation

Original: Given a set of observations I , ( I 0 , I 1 , ..., I f ) of a static surface, illuminated by f unknowndirectional illuminations distributing on the upper-hemisphere, uncalibrated photometric stereo aimsat recovering light directions L , ( l 0 , l 1 , ..., l f ) , light intensities E , ( e 0 , e 1 , ..., e f ) , and surfacenormal N , { n i | i 2 P } , P is the set of all positions on the surface.

Original: The solution is achieved by 137 solving the optimization problem,

Original: [Equation]

Original: [-where ¯ m ij 
-]
Revised: [-where ¯ m ij 
-]

Original: I j is the observed pixel intensity at position i , # P is the number of elements in P 1 , m ijis the corresponding rendered pixel intensity, D ( · , · ) is a metric describing their difference.

Original: Under anorthographic camera with linear radiometric response, m ij is formulated as (simpliﬁed in a per-pixelform),

Original: [Equation]

Original: where v = [0 , 0 , 1] is the view direction pointing toward the viewer, ⇢ ij describes the generalreﬂectance, max( n > i l j , 0) represents the attach shadow.

Original: Unknown light brings two ambiguities when solving Eq.

Original: (1), i.e , shape-light ambiguity, which isdenoted as an invertable matrix G 2 R 3 ⇥ 3 , and reﬂectance-light ambiguity, which is denoted as anon-zero scalar c j 2 R ,

Original: [Equation]

Original: 3.2 Neural Intrinsics Fields

Original: General reﬂectance decomposition.

Original: To exploit clues of specular reﬂectance and cast shadow forlight estimation, we decompose the general reﬂectance as the cast shadow term s ij multiplying thebidirectional reﬂectance term,

Original: [Equation]

Original: where subscript ‘ i ’ and ‘ j ’ indicate pixel position- (or normal-) and light-dependent factors, respectively, the cast shadow term s ij is either 0 or 1, ⇢ di , ⇢ sij represent the diffuse and specular reﬂectance,and k di and k si are coefﬁcients that balance out the effects of specular and diffuse reﬂectance 2 .

Original: Neural ﬁelds of specular reﬂectance and cast shadow.

Original: Previous methods identify specular featuresfrom speciﬁc pixels and associate an explicit relation to light for light estimation.

Original: However, thisscheme could fail for the surface where the specular features are invisible or the explicit relationis violated.

Original: Besides, leaving out the clue of cast shadow could obstruct producing competitiveperformance.

Original: In contrast, we leverage both clues of specular and shadow for light estimation, which is achieved bybuilding the neural specular ﬁeld and neural shadow ﬁeld and associating the ﬁelds to light conditions.

Original: These neural ﬁelds make the utmost of all observed pixels, which fully exploits the mutual informationacross different normal-light (specular) and depth-light (shadow).

Original: Two MLPs, namely SpecularNetand ShadowNet , implement these neural ﬁelds, respectively.

Original: Instead of estimating light conditions think from SpecularNet or ShadowNet, we take the light direction as the input of these networks to achievetheir implicit association to light conditions.

Original: Since the specular reﬂectance and cast shadow arenormal- and pixel position-dependent, we also feed the estimated surface normal and the positionalcode to them, respectively.

Original: SpecularNet and ShadowNet output ⇢ sij and s ij , respectively, as shown in

Original: Fig.

Original: Neural light ﬁeld. There is mutual information across different observed images, i.e , observedimages with similar appearances are expected to be illuminated by similar lights. To fully considerand exploit this mutual information, we build the  neural light ﬁeld by concatenating a CNN encoderto an MLP, namely LightNet . The encoder extracts a light code from each observed image 3 , andthe LightNet outputs the corresponding light conditions ( i.e , e j , l j ) from the  light code, as shown in
Revised: The key idea is torepresent the reflectance as four neural intrinsics fields, i.e. , position, light, specular,and shadow, based on which the neural light field is implicitly associated withlight clues of specular reflectance and cast shadow.

Original: Fig.

Original: Neural position ﬁeld.

Original: There is mutual information across different positions on a surface, i.e , theconsistency of shape and diffuse reﬂectance in the spatial domain.

Original: To fully consider and exploit thismutual information, we establish a neural position ﬁeld, namely PositionNet , implemented by an MLP.

Original: The neural position ﬁeld output position-dependent, light-independent factors 4 , i.e , n i , ⇢ di , k di , k si .

Original: The PositionNet takes the positional code as the input, as shown in Fig. 2.

Original: 3.3 Optimizing Neural Intrinsics Fields

Original: We adopt the reconstruction loss function with the ` 1 metric to optimze our NeIF,

Original: [Equation]

Original: [Equation]

Original: Silhouette constraint.

Original: L rec cannot resolve the shape-light ambiguity in an unsupervised manner dueto the inherently severe ill-posedness.

Original: Therefore, we introduce the silhouette constraint (similar tothose in [ 8758025 , 13]) to stabilize the training of PositionNet.

Original: To be speciﬁc, we use polynomialﬁtting with a moving window block to traverse and pre-compute the contour’s normal of the givenobjects, represented as ˆ N si , { ˆ n si k 2 R 2 ⇥ 1 , k 2 S } , S is the point set of the contour.

Original: We considerˆ N si could guide the prediction of the azimuth of boundary normal (at the same positions) and introduce the silhouette loss function,

Original: [Equation]

Original: where n k 2 R 1 ⇥ 3 represents the estimated surface normal from PositionNet, C ( · ) cuts off the 3rddimension of n k ( i.e , C ( n k ) 2 R 1 ⇥ 2 ), and Nor ( · ) is the vector normalization operation.

Original: Warm-up loss functions.

Original: To avoid local minimum and achieve faster convergence, we warm upthe NeIF in early-stage during training with three additional loss functions.

Original: We use the azimuth oflighting direction estimated by YS97 [54] to guide the training of LightNet,

Original: [Equation]

Original: where l az j are estimated light directions by [54].

Original: We adopt the gradient penalty [20] L gp to stabilizethe training of SpecularNet,

Original: [Equation]

Original: The intuition is from Blinn-Phong model, where the specular reﬂectance is monotonically increasingw.r.t the n > i h j , i.e , r n > i

Original: h j ⇢ sij > 0 .

Original: We also supervise the training of ShadowNet using pseudoshadow maps ˆ S j , j = 1 , 2 , ..., f ,

Original: [Equation]

Original: The pseudo shadow maps are obtained by binarizing the observed images, i.e , considering an observedpixel to be cast shadow if its intensity value is smaller than 0 .

Original: 2 ⇥ the mean intensity values of thisimage.

Original: After early-stage training, we discard these loss functions for a broad range of reﬂectance.

Original: NeIF training.

Original: We train NeIF with the warm-up loss function in ﬁrst 10 epochs,

Original: [Equation]

Original: 10) where  si = 5 ,  az = 0 .

Original: 1 ,  gp = ,  shadow

Original: =

Original: We then train NeIF untill 500 epochs or convergingwith the loss function,

Original: [Equation]

Original: [Equation]

Original: where L recShadow is the another shadow map supervison loss to train the ShadowNet.

Original: The loss functionis the same to L shadow .

Original: The only difference is the calculation of ˆ S j .

Original: For L recShadow , we calculate ˆ S jby rendering a depth map with the estimated l j .

Original: The depth map is reconstructed from the estimatedsurface normal map N by method [8] 5 .

Original: L recShadow is used to align outputs of ShadowNet to those of

Original: PositionNet.

Original: Implementation details.

Original: We generate the positional code from the coordinate (in the image plane)by the same method in [31].

Original: With the assumption of isotropic reﬂectance, we simplify the inputof SpecularNet from { n > i l j , n > i v , l > j v } to { n > i h j , h > j v } 6 [28] for easier model training.

Original: Similarto the Cook-Torrance reﬂectance model [17], we assume k di + k si = 1 to reduce the numbers ofunknowns.

Original: The CNN decoder with declining channels processes the down-sampled images with adimension of 256 ⇥ 256 .

Original: The LightNet takes ﬂatten features to predict l j , e j in two different branches.

Original: For the ShadowNet, we also generate the positional code for l j and concatenate it to the feature in9-th layer.

Original: The output of ShadowNet is either 0 or 1, which is realized by a step function with asimilar implementation in [35] 7 .

Original: Experiments

Original: Training details.

Original: Our main framework is implemented in PyTorch, while the pre-calculations(method [54] and silhouette ﬁtting ) are implemented in MATLAB.

Original: We use Adam as the optimizer k .

Original: Please refer to our supplementary material for more details

Original: [Table]

Original: with a learning rate ↵ = 5 ⇥ 10  4 to train our framework in 500 epochs for each scene separately, andthe warm-up stage takes up ten epochs.

Original: The last 100 epochs use a lower learning rate ↵ = 5 ⇥ 10  5for ﬁne-tunning.

Original: The batch size is 32 for lighting and 256 for spatially random sampling.

Original: After everyepoch, the depth map is reconstructed according to the predicted normal map by method [8].

Original: Eachscene takes from 2 hours to 6 hours on one RTX 2080Ti 12GB GPU, depending on the resolution ofthe objects.

Original: Evaluation metrics.

Original: We adopt the same metric in [12], the scale-invariant relative error, to measurethe accuracy of recovered light intensity as

Original: [Equation]

Original: where, ⌘ is calculated by solving arg min ⌘

Original: P

Original: [Equation]

Original: by least squares minimization.

Original: Themetric to measure the accuracy of the predicted light directions and surface normal is the widely usedmean angle error (MAE) in degree.

Original: 4.1 Evaluation on Public Datasets

Original: Since the proposed NeIF is an unsupervised uncalibrated photometric stereo method, we compare itsperformance with state-of-the-art uncalibrated and unsupervised photometric stereo methods.

Original: Threereal-world datasets, including the benchmark dataset D I L

Original: I G EN T [39], A PPLE & G OURD dataset [2],and the L IGHT S TAGE D ATA G ALLERY

Original: [9], are used for evaluation.

Original: Quantitative comparison for normal map.

Original: Table 1 lists relevant works for a comprehensive surfacenormal estimation comparison.

Original: As summarized in Table 1, our method is the only method thataddresses UPS without the supervision of N , L , or E .

Original: However, our method achieves the bestperformance and maintains a considerable advantage over other competitors[12, 13] ( i.e , 7.86 for

Original: NeIF v.s. 8.71 for [13]).

Original: Objects in D I

Original: L I G EN T differ from shapes and reﬂectance.

Original: The numbers ofour method are either best or competitive (less than 1  as compared with the best performing UPSmethod), which shows its good generalization capacity to general reﬂectance and various shapes.

Original: Because our NeIF fully considers mutual information by building up intrinsics ﬁelds and such implicitmodeling facilitates modeling general reﬂectance.

Original: Visual quality comparison for normal map.

Original: Fig.

Original: 3 illustrates the visual quality comparison in termsof recovered surface normal maps and corresponding error maps on D I L I G EN T.

Original: The comparisonis conducted with two state-of-the-art UPS methods[12, 13] and a state-of-the-art unsupervised PSmethod [44].

Original: Our method is less sensitive to spatially-varying albedo due to the per-pixel manner(red boxes in the left subﬁgure of Fig. 3).

Original: However, although the positional code considers the

Original: [Figure]

Original: global shape effect, this per-pixel manner is less effective in modeling complex shape information ascompared with the all-pixel one.

Original: It could fail for regions with cast shadow (or overexposure) undermost light directions (red boxes in the right subﬁgure of Fig. 3).

Original: We also show the visual qualityresults for A PPLE & G OURD dataset [2] and the L IGHT S TAGE D ATA G ALLERY [9] in Fig.

Original: Ourmethod could produce reliable estimation for most regions, thanks to the full exploitation of mutualinformation across different dimensions.

Original: Quantitative comparison for light directions and intensities.

Original: As can be observed in Table 2 and

Original: Table 3, our method achieves a superior performance advantage over unsupervised methods[34, 4,42, 50, 29, 28] while maintaining competitive performance as compared with supervised methods.

Original: Although the supervised methods pre-deﬁne the intensity range, they produce less accurate estimationas compared with our approach due to the possible overﬁtting of supervised learning.

Original: In contrast,our LightNet builds the neural light ﬁeld, which could fully consider the mutual information amongdifferent observed images, resulting in the accurate estimation of the scale-invariant light intensity.

Original: 4.2 Evaluation on Sparse Uncalibrated Photometric Stereo

Original: As compared with other per-pixel PS methods ( e.g. , [59, 27]), the proposed NeIF exploits much moreconstraints to estimate the normal at a point, i.e , # P ⇥ f v.s. f .

Original: Because it learns ﬁelds instead ofregressing intensity proﬁle [40] to surface normal.

Original: Therefore, we investigate the effectiveness of our

Original: [Table]

Original: [Table]

Original: method under the challenging setting of sparse UPS.

Original: We randomly select 16 images illuminatedby different lights from D

Original: I L I G ENT and test our method using these images.

Original: We repeat this process30 times, similar to [59].

Original: We compare the performance with that from the classical PS method

Original: PF14 [34], as deep learning-based approaches ( e.g. , [12, 13]) perform poorly without re-training.

Original: Due to the fewer reconstruction terms to train our NeIF

Original: [Table]

Original: we slightly increase the dimension of thepositional encoding module from 4 to 6 by increasing the frequency [45] to stabilize the training, i.e ,strengthening the role of positional code to reduce the variance of estimated intrinsics.

Original: Besides, wedrop L az during early stage warm-up because YS97 [54] fails for objects with centralized specularcomponents ( e.g. , B ALL ).

Original: We report the mean results over 30 trails in Table 1.

Original: As can be observed, our method achievespromising performance on light and normal estimation for sparse UPS.

Original: Conclusion

Original: This paper proposes NeIF for UPS.

Original: By representing the general reﬂectance as four neural intrinsicsﬁelds, it implicitly associates the light clues of specualr reﬂectance and cast shadow to light estimation,which faciliate solving UPS with general reﬂectance.

Original: The proposed NeIF could fully exploit mutualinformace from all observed pixel valuesso that it could produce stable estimation even forsparse UPS.The unsupervised training manner of NeIF is beneﬁcial to the generalization capacity to datafrom different sources. Limitations.
Revised: The unsupervised optimizationof neural intrinsics fields can be free from training data bias and fully exploitsall observed pixel values for UPS.

Original: Although our method produces promising results for light condition and surface normalestimation, it has several limitations.

Original: First, the estimated intrinsics such as shadow, specular anddiffuse reﬂectance, and their balancing coefﬁcients are less accurate, as shown in Fig.

Original: Second,we need to balance the role of positional code and estimated intrinsics when applying our methodfor sparse UPS.

Original: However, how to ﬁnd an optimal balancing strategy is unknown.

Original: Third, as shownin Fig. 3, our method is less effective on global effects, especially for regions with cast shadow (oroverexposure) under most light directions due to the per-pixel manner.

Original: Besides, the inference of ourmethod is inefﬁcient due to the unsupervised training manner and the per-pixel estimation, which isinapplicable for real-time applications.

Original: Checklist

Original: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .

Original: You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.

Original: For example:

Original: • Did you include the license to the code and datasets?

Original: [Yes] See Section ?? .•

Original: Did you include the license to the code and datasets?

Original: [No] The code and the data areproprietary.

Original: • Did you include the license to the code and datasets?

Original: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the

Original: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s

Original: contributions and scope?

Original: [Yes] See Line 9-13,

Original: (b) Did you describe the limitations of your work?

Original: [Yes] See Line 287-

Original: (c) Did you discuss any potential negative societal impacts of your work?

Original: [No]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?

Original: [Yes]

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] See

Original: Supplementary Material

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they

Original: were chosen)?

Original: [Yes]

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Original: ments multiple times)?

Original: [Yes]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes]

Original: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?

Original: [No]

Original: (b) Did you mention the license of the assets?

Original: [No]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?

Original: [No]

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: information or offensive content?

Original: [No]

Original: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?

Original: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?

Original: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?

Original: [N/A]

Original: NeIF: Representing General Reﬂectance as Neural Intrinsics Fields for Uncalibrated Photometric Stereo

Original: Abstract

Original: Uncalibrated photometric stereo (UPS) is challenging due to the inherent ambiguitybrought by unknown light.

Original: Existing solutions alleviate the ambiguity by eitherexplicitly associating reﬂectance to light conditions or resolving light conditions ina supervised manner.

Original: This paper establishes an implicit relation between light cluesand light estimation and solves UPS in an unsupervised manner.

Original: The key idea is torepresent the reﬂectance as four neural intrinsics ﬁelds, i.e. , position, light, specular,and shadow, based on which the neural light ﬁeld is implicitly associated withlight clues of specular reﬂectance and cast shadow.

Original: The unsupervised optimizationof neural intrinsics ﬁelds can be free from training data bias and fully exploitsall observed pixel values for UPS. Our method achieves a superior performanceadvantage over state-of-the-art UPS methods on public datasets and promisingresults under the challenging setting of sparse UPS.
Revised: Our method achieves a superior performanceadvantage over state-of-the-art UPS methods on public datasets and promisingresults under the challenging setting of sparse UPS.

Original: The code will be releasedsoon.
Revised: The code will be releasedsoon.

Original:  Introduction
Revised:  Introduction

Original: Photometric stereo (PS) [50] aims at recovering the surface normal from several light-varying imagescaptured at a ﬁxed viewpoint.
Revised: Photometric stereo (PS) [50] aims at recovering the surface normal from several light-varying imagescaptured at a fixed viewpoint.

Original: As compared with other approaches ( e g. , multi-view stereo [41], activesensor-based solutions [58]), photometric stereo is excellent at recovering ﬁne-detailed surfaces andhas been widely used for Hollywood movies [11], industrial quality inspection [49] and biometrics[55].
Revised: As compared with other approaches ( e g. , multi-view stereo [41], activesensor-based solutions [58]), photometric stereo is excellent at recovering fine-detailed surfaces andhas been widely used for Hollywood movies [11], industrial quality inspection [49] and biometrics[55].

Original: Calibrating accurate lighting directions is crucial to the performance of photometric stereomethods [54].
Revised: Calibrating accurate lighting directions is crucial to the performance of photometric stereomethods [54].

Original: However, lighting calibration is often tedious, dramatically restricting the applicabilityin the real-world.
Revised: However, lighting calibration is often tedious, dramatically restricting the applicabilityin the real-world.

Original: To this end, researchers develop uncalibrated photometric stereo (UPS) methodsthat estimate surface normal with unknown lights.
Revised: To this end, researchers develop uncalibrated photometric stereo (UPS) methodsthat estimate surface normal with unknown lights.

Original: Uncalibrated photometric stereo suffers from General Bas-Relief (GBR) ambiguity [6] for an integrable surface.
Revised: Uncalibrated photometric stereo suffers from General Bas-Relief (GBR) ambiguity [6] for an integrable surface.

Original: Early solutions address the ambiguity by explicitly associating reﬂectance to light,i.e. , adopting analytic reﬂectance models ( e.g. , Lambertian reﬂectance [4], [37], parametric specularreﬂection [20], specular spikes [56], inter-reﬂection [12]) or imposing priors from reﬂectance properties [3, 25, 24, 44].
Revised: Early solutions address the ambiguity by explicitly associating reflectance to light,i.e. , adopting analytic reflectance models ( e.g. , Lambertian reflectance [4], [37], parametric specularreflection [20], specular spikes [56], inter-reflection [12]) or imposing priors from reflectance properties [3, 25, 24, 44].

Original: Thus, due to the strong reliance on reﬂectance assumption, these methods can beless effective for unknown reﬂectance.
Revised: Thus, due to the strong reliance on reflectance assumption, these methods can beless effective for unknown reflectance.

Original: Besides, these methods ignore the clue of cast shadow andeven fail in shadow regions due to the shadow-free surface assumption.
Revised: Besides, these methods ignore the clue of cast shadow andeven fail in shadow regions due to the shadow-free surface assumption.

Original: Further, most of them requirethe light intensity to be identical for robust estimation.
Revised: Further, most of them requirethe light intensity to be identical for robust estimation.

Original: Recently, deep learning-based approachesaddress the ambiguity by estimating light direction and intensity before recovering surface normal [13,15, 27, 40].
Revised: Recently, deep learning-based approachesaddress the ambiguity by estimating light direction and intensity before recovering surface normal [13,15, 27, 40].

Original: They train a light estimation network using a large-scale amount of rendered data in asupervised manner.
Revised: They train a light estimation network using a large-scale amount of rendered data in asupervised manner.

Original: However, the training data bias [33] can hardly be eliminated and can produceunexpected estimation for real-world data.
Revised: However, the training data bias [33] can hardly be eliminated and can produceunexpected estimation for real-world data.

Original: Because rendered training data inevitably contains thedomain gap from the real ones and scarcely cover all surfaces with different geometry and reﬂectance in the real-world.
Revised: Because rendered training data inevitably contains thedomain gap from the real ones and scarcely cover all surfaces with different geometry and reflectancein the real-world.

Original: Besides, such a two-step solution can bring accumulating errors for surface normal
Revised: Besides, such a two-step solution can bring accumulating errors for surface normal

Original: Neural Shadow Field
Revised: Neural Shadow Field

Original: P o s i t i on
Revised: P o s i t i on

Original: Light direction ( ) + × × = × × × · · ·  ! ! " !" 
Revised: Light direction ( ) + × × = × × × · · · 𝒏 ! 𝑘 !" 𝜌 !" 𝑘 !

Original: # Neural Position Field
Revised: # Neural Position Field

Original: P o s i t i on
Revised: P o s i t i on

Original: Neural Specular Field
Revised: Neural Specular Field

Original: S
Revised: S

Original: u r f a c e no r m a l
Revised: u r f a c e no r m a l

Original: Light direction
Revised: Light direction

Original: Neural Light Field
Revised: Neural Light Field

Original: O b s e r v ed i m age s
Revised: O b s e r v ed i m age s

Original: Light direction · · · ! !"
Revised: Light direction · · · 𝜌 !"

Original: ( !$ !  !
Revised: (𝒍 # ,𝑒 # ) 𝑠 ! # 𝑘 !" 𝑘 !$ 𝜌 !

Original: #$ ) !
Revised: #$ 𝒏 ! 𝑚 !

Original: Position-Dependent
Revised: Position-Dependent

Original: Light- Dependent ! !"
Revised: Light- Dependent 𝜌 !"

Original: ü û " !
Revised: ü û 𝑘 !

Original: # ü û " ! " 
Revised: # ü û 𝑘 ! " ü û 𝒏 !

Original: ü û # ! ü û $ $ û ü % % û ü ! !
Revised: ü û 𝒍 𝒋 û ü 𝑒 % û ü 𝜌 !

Original: # ü ü & !
Revised: # ü ü 𝑠 !

Original:  ü ü 
Revised: % ü ü estimation. Further, all these methods assume the light intensity distributed in a pre-defined range( i.e. , [0 . 2 , 2] ), restricting their applicability.

Original: To this end, we propose NeIF , representing general reﬂectance as Ne ural I ntrinsics F ields foruncalibrated photometric stereo.
Revised: To this end, we propose NeIF , representing general reflectance as Ne ural I ntrinsics F ields foruncalibrated photometric stereo.

Original: Our method differs from previous methods in three aspects: 1)it fully considers clues of specular reﬂectance and cast shadow from each observed pixel for lightestimation so that it is expected to produce accurate estimation for both light conditions and surfacenormal; 2) it does not make explicit assumptions about the reﬂectance or light so that it works withgeneral surface reﬂectance and ﬂexible light settings; 3) it infers light and surface normal in anunsupervised manner so that it is free from training data bias and achieves stable performance fordata from different sources.
Revised: Our method differs from previous methods in three aspects: 1)it fully considers clues of specular reflectance and cast shadow from each observed pixel for lightestimation so that it is expected to produce accurate estimation for both light conditions and surfacenormal; 2) it does not make explicit assumptions about the reflectance or light so that it works withgeneral surface reflectance and flexible light settings; 3) it infers light and surface normal in anunsupervised manner so that it is free from training data bias and achieves stable performance fordata from different sources.

Original: Our key idea is to represent the general reﬂectance as four neural intrinsics ﬁelds ( i.e. , position,light, specular, and shadow, see Fig. 1), implemented by four multi-layer perceptrons (MLPs).
Revised: Our key idea is to represent the general reflectance as four neural intrinsics fields ( i.e. , position,light, specular, and shadow, see Fig. 1), implemented by four multi-layer perceptrons (MLPs).

Original: Thesefour ﬁelds are connected based on the implicit relation (or dependence) of these intrinsics so thatno explicit assumption is imposed, e.g. , we take the estimated light as the input to recover specularreﬂectance and cast shadow instead of explicitly exploiting them for light estimation.
Revised: Thesefour fields are connected based on the implicit relation (or dependence) of these intrinsics so thatno explicit assumption is imposed, e.g. , we take the estimated light as the input to recover specularreflectance and cast shadow instead of explicitly exploiting them for light estimation.

Original: These intrinsicsﬁelds are optimized to reconstruct each pixel value from observed images, which fully exploitmutual information across different dimensions, as shown in Fig.
Revised: These intrinsicsfields are optimized to reconstruct each pixel value from observed images, which fully exploitmutual information across different dimensions, as shown in Fig.

Original: The reconstruction error isbackpropagated to the neural light ﬁeld through neural specular and shadow ﬁelds so that clues ofspecular and shadow can be implicitly and fully considered for light estimation.
Revised: The reconstruction error isbackpropagated to the neural light field through neural specular and shadow fields so that clues ofspecular and shadow can be implicitly and fully considered for light estimation.

Original: Our contributionsare summarized as:
Revised: Our contributionsare summarized as:

Original: • We represent general reﬂectance as four intrinsic neural ﬁelds to implicitly associate per-
Revised: • We represent general reflectance as four intrinsic neural fields to implicitly associate per-

Original: pixel reﬂectance to light, which solves uncalibrated photometric stereo by fully considering
Revised: pixel reflectance to light, which solves uncalibrated photometric stereo by fully considering

Original: clues of specular reﬂectance and cast shadow for light estimation.
Revised: clues of specular reflectance and cast shadow for light estimation.

Original: • We propose the NeIF, an uncalibrated photometric stereo method trained in an unsupervised
Revised: • We propose the NeIF, an uncalibrated photometric stereo method trained in an unsupervised

Original: manner, which works with general surface reﬂectance and ﬂexible light settings, and is free from training data bias.
Revised: manner, which works with general surface reflectance and flexible light settings, and is freefrom training data bias.

Original: • We show that our method achieves superior performance over uncalibrated and unsupervisedmethods.
Revised: • We show that our method achieves superior performance over uncalibrated and unsupervisedmethods.

Original: We also demonstrate its excellent generalization capacity to data from different sources and promising performance with the challenging setup of sparse uncalibrated
Revised: We also demonstrate its excellent generalization capacity to data from differentsources and promising performance with the challenging setup of sparse uncalibrated

Original: photometric stereo.
Revised: photometric stereo.

Original:  Related Work
Revised:  Related Work

Original: This section mainly reviews the latest works in neural reﬂectance representation, and related workson unsupervised PS methods and UPS methods.
Revised: This section mainly reviews the latest works in neural reflectance representation, and related workson unsupervised PS methods and UPS methods.

Original: Readers may refer to [45] and [13] for a morecomprehensive summary.
Revised: Readers may refer to [45] and [13] for a morecomprehensive summary.

Original: 2.1 Neural Reﬂectance Representation
Revised: 2.1 Neural Reflectance Representation

Original: Neural Radiance Fields (NeRFs)
Revised: Neural Radiance Fields (NeRFs)

Original: [34] focus on the 3D geometries without explicitly modeling theinteraction between light and objects via the image formation model.
Revised: [34] focus on the 3D geometries without explicitly modeling theinteraction between light and objects via the image formation model.

Original: Many subsequent worksexplore its application in various computer vision problems, such as relighting [16, 46], photometricstereo [29], and multi-view stereo [7, 9, 60, 28, 61].
Revised: Many subsequent worksexplore its application in various computer vision problems, such as relighting [16, 46], photometricstereo [29], and multi-view stereo [7, 9, 60, 28, 61].

Original: These works require known lighting conditions [7,46, 61, 29], adopt simple reﬂectance model [59], or leverage multi-view information [7, 9, 60, 28, 61].
Revised: These works require known lighting conditions [7,46, 61, 29], adopt simple reflectance model [59], or leverage multi-view information [7, 9, 60, 28, 61].

Original: Different from these methods, our method considers general reﬂectance, cast-shadow effects, andunknown light conditions for uncalibrated photometric stereo by taking images captured at a singleviewpoint.
Revised: Different from these methods, our method considers general reflectance, cast-shadow effects, andunknown light conditions for uncalibrated photometric stereo by taking images captured at a singleviewpoint.

Original: 2.2 Unsupervised Photometric Stereo
Revised: 2.2 Unsupervised Photometric Stereo

Original: Classical methods solve the calibrated photometric stereo problem without knowing the groundtruth surface normal.
Revised: Classical methods solve the calibrated photometric stereo problem without knowing the groundtruth surface normal.

Original: Therefore, we classify them as unsupervised methods.
Revised: Therefore, we classify them as unsupervised methods.

Original: The least squarebased algorithm [50] provides the simplest solution, which assumes the object to be Lambertian.
Revised: The least squarebased algorithm [50] provides the simplest solution, which assumes the object to be Lambertian.

Original: It is generally served as a baseline method due to its stability, but its strong assumption on thereﬂectance model makes it fail for non-Lambertian surface.
Revised: It is generally served as a baseline method due to its stability, but its strong assumption on thereflectance model makes it fail for non-Lambertian surface.

Original: The following works either regard thenon-Lambertian reﬂectance components as the outliers [5, 18, 36, 51, 26, 52] or apply analyticreﬂectance models including Torrance-Sparrow [20], the Ward model [18], a mixture of multiple
Revised: The following works either regard thenon-Lambertian reflectance components as the outliers [5, 18, 36, 51, 26, 52] or apply analyticreflectance models including Torrance-Sparrow [20], the Ward model [18], a mixture of multiple

Original: Ward models [21], [1] etc. to consider the non-Lambertian effects.
Revised: Ward models [21], [1] etc. to consider the non-Lambertian effects.

Original: However, the performance ofthose methods can only deal with limited types of materials.
Revised: However, the performance ofthose methods can only deal with limited types of materials.

Original: There are also more advanced methodsthat utilize the general reﬂetance features such as reciprocity, isotropy [2], and monotonicity [24].
Revised: There are also more advanced methodsthat utilize the general refletance features such as reciprocity, isotropy [2], and monotonicity [24].

Original: Those methods give a reliable estimation for objects with a broad range of materials.
Revised: Those methods give a reliable estimation for objects with a broad range of materials.

Original: With the progress of deep learning, many learning-based frameworks have been proposed for calibrated photometric stereo.
Revised: With the progress of deep learning, many learning-based frameworks have been proposed for calibrated photometric stereo.

Original: Taniai et al.
Revised: Taniai et al.

Original: [47] proposed the ﬁrst unsupervised learning-based photometric stereo method through a rendering equation.
Revised: [47] proposed the first unsupervised learning-based photometric stereo method through a rendering equation.

Original: However, their reﬂectance model does not separatelyconsider shadow, specular highlights, and diffuse components.
Revised: However, their reflectance model does not separatelyconsider shadow, specular highlights, and diffuse components.

Original: We also train our method in an unsupervised manner.
Revised: We also train our method in an unsupervised manner.

Original: Different from previous methods, we addressthe challenging problem of UPS and separately model cast shadow, specular reﬂectance, and diffusereﬂectance.
Revised: Different from previous methods, we addressthe challenging problem of UPS and separately model cast shadow, specular reflectance, and diffusereflectance.

Original: 2.3 Uncalibrated Photometric Stereo
Revised: 2.3 Uncalibrated Photometric Stereo

Original: Previous works hold the Lambertian assumption and address the ambiguity brought by a 3 ⇥ 3transformation matrix.
Revised: Previous works hold the Lambertian assumption and address the ambiguity brought by a 3 × 3transformation matrix.

Original: Belhumeur et al.
Revised: Belhumeur et al.

Original: [6] reduce the dimension of the transformation to a threeparameters GBR transformation by considering integrability constraints.
Revised: [6] reduce the dimension of the transformation to a threeparameters GBR transformation by considering integrability constraints.

Original: Based on that, extra cluesfrom reﬂectance, such as half-vector symmetry [31], albedo clustering [42], specular spikes [56], orassumptions of light source distribution, such as ring light [63], symmetry light [35], or uniformdistributed light sources [57, 4, 42, 53, 37, 31], are used to resolve the GBR ambiguity.
Revised: Based on that, extra cluesfrom reflectance, such as half-vector symmetry [31], albedo clustering [42], specular spikes [56], orassumptions of light source distribution, such as ring light [63], symmetry light [35], or uniformdistributed light sources [57, 4, 42, 53, 37, 31], are used to resolve the GBR ambiguity.

Original: However,all these methods require the light intensity to be identical, which is inapplicable in the real-worlddatasets such as D
Revised: However,all these methods require the light intensity to be identical, which is inapplicable in the real-worlddatasets such as D

Original: I L I G EN T [45], A PPLE & G OURD [2], and L IGHT S TAGE D ATA G ALLERY
Revised: I L I G EN T [45], A PPLE & G OURD [2], and L IGHT S TAGE D ATA G ALLERY

Original: Cho et al.
Revised: Cho et al.

Original: [17] put up a semi-calibrated method to deal with non-uniform light intensity, but theyassume the light directions to be known.
Revised: [17] put up a semi-calibrated method to deal with non-uniform light intensity, but theyassume the light directions to be known.

Original: Quéau et al.
Revised: Quéau et al.

Original: [39] address the photometric stereo problemunder inaccurate lighting calibration, while the accuracy can signiﬁcantly drop when non-Lambertiancomponents become dominant.
Revised: [39] address the photometric stereo problemunder inaccurate lighting calibration, while the accuracy can significantly drop when non-Lambertiancomponents become dominant.

Original: Recently, many deep learning methods have been proposed for uncalibrated photometric stereo.
Revised: Recently, many deep learning methods have been proposed for uncalibrated photometric stereo.

Original: Chen et al.
Revised: Chen et al.

Original: [13] propose a supervised uncalibrated framework, SDPS-Net, which can simultaneouslyestimate the light conditions (intensity and direction) and the surface normal.
Revised: [13] propose a supervised uncalibrated framework, SDPS-Net, which can simultaneouslyestimate the light conditions (intensity and direction) and the surface normal.

Original: They suggest treatinglight estimation as a classiﬁcation problem and separating the normal and light prediction to reducethe complexity.
Revised: They suggest treatinglight estimation as a classification problem and separating the normal and light prediction to reducethe complexity.

Original: Their following work, GC-Net [15], improves the performance of SDPS-Net by adding shading as an extra channel to the input of the light estimation network.
Revised: Their following work, GC-Net [15], improves the performance of SDPS-Net by adding shading as an extra channel to the input of the light estimation network.

Original: However, as acommon problem for all supervised methods, an over-ﬁtting problem may occur due to the trainingdata bias [33].
Revised: However, as acommon problem for all supervised methods, an over-fitting problem may occur due to the trainingdata bias [33].

Original: In contrast, unsupervised methods do not have such a concern.
Revised: In contrast, unsupervised methods do not have such a concern.

Original: Another beneﬁt isthat there is no need to synthesize training sets for unsupervised network training.
Revised: Another benefit isthat there is no need to synthesize training sets for unsupervised network training.

Original: To utilize theadvantage of unsupervised methods, Kaya et al.
Revised: To utilize theadvantage of unsupervised methods, Kaya et al.

Original: [27] propose a compromised method that trains thelight estimation network in a supervised manner (similar to [13]), but estimate the surface normal inan unsupervised way (similar to [47]).
Revised: [27] propose a compromised method that trains thelight estimation network in a supervised manner (similar to [13]), but estimate the surface normal inan unsupervised way (similar to [47]).

Original: However, they still suffer from training data bias during lightestimation.
Revised: However, they still suffer from training data bias during lightestimation.

Original: Besides, all these methods make a strict assumption that the light intensity distributed in apre-deﬁned range ( i.e. , [0.2, 2]) and suffers from accumulating error due to their two-step frameworks.
Revised: Besides, all these methods make a strict assumption that the light intensity distributed in apre-defined range ( i.e. , [0.2, 2]) and suffers from accumulating error due to their two-step frameworks.

Original: In contrast, our method neither makes a strict assumption on reﬂectance nor needs special light sourcedistribution and jointly solves light conditions and surface normal in an unsupervised manner.
Revised: In contrast, our method neither makes a strict assumption on reflectance nor needs special light sourcedistribution and jointly solves light conditions and surface normal in an unsupervised manner.

Original:  Method
Revised:  Method

Original: 3.1 Problem Formulation
Revised: 3.1 Problem Formulation

Original: Given a set of observations I , ( I 0 , I 1 , ..., I f ) of a static surface, illuminated by f unknowndirectional illuminations distributing on the upper-hemisphere, uncalibrated photometric stereo aimsat recovering light directions L , ( l 0 , l 1 , ..., l f ) , light intensities E , ( e 0 , e 1 , ..., e f ) , and surfacenormal N , { n i | i 2 P } , P is the set of all positions on the surface.
Revised: Given a set of observations I ≜ ( I 0 , I 1 , ..., I f ) of a static surface, illuminated by f unknowndirectional illuminations distributing on the upper-hemisphere, uncalibrated photometric stereo aimsat recovering light directions L ≜ ( l 0 , l 1 , ..., l f ) , light intensities E ≜ ( e 0 , e 1 , ..., e f ) , and surfacenormal N ≜ { n i | i ∈ P } , P is the set of all positions on the surface.

Original: The solution is achieved by 133 solving the optimization problem,
Revised: The solution is achieved bysolving the optimization problem,

Original: [Equation]
Revised: [Equation]

Original: (1) where ¯ m ij [-
-]
Revised: (1) where ¯ m ij [-
-]∈

Original: I j is the observed pixel intensity at position i , # P is the number of elements in P 1 , m ijis the corresponding rendered pixel intensity, D ( · , · ) is a metric describing their difference.
Revised: I j is the observed pixel intensity at position i , # P is the number of elements in P 1 , m ijis the corresponding rendered pixel intensity, D ( · , · ) is a metric describing their difference.

Original: Under anorthographic camera with linear radiometric response, m ij is formulated as (simpliﬁed in a per-pixelform),
Revised: Under anorthographic camera with linear radiometric response, m ij is formulated as (simplified in a per-pixelform),

Original: [Equation]
Revised: [Equation]

Original: where v = [0 , 0 , 1] is the view direction pointing toward the viewer, ⇢ ij describes the generalreﬂectance, max( n > i l j , 0) represents the attach shadow.
Revised: where v = [0 , 0 , 1] is the view direction pointing toward the viewer, ρ ij describes the generalreflectance, max( n ⊤ i l j , 0) represents the attach shadow.

Original: Unknown light brings two ambiguities when solving Eq.
Revised: Unknown light brings two ambiguities when solving Eq.

Original: (1), i.e. , shape-light ambiguity, which isdenoted as an invertable matrix G 2 R 3 ⇥ 3 , and reﬂectance-light ambiguity, which is denoted as anon- scalar c j 2 R ,
Revised: (1), i.e. , shape-light ambiguity, which isdenoted as an invertable matrix G ∈ R 3 × 3 , and reflectance-light ambiguity, which is denoted as anon-zero scalar c j ∈ R ,

Original: [Equation]
Revised: [Equation]

Original: 3.2 Neural Intrinsics Fields
Revised: 3.2 Neural Intrinsics Fields

Original: General reﬂectance decomposition.
Revised: General reflectance decomposition.

Original: To exploit clues of specular reﬂectance and cast shadow forlight estimation, we decompose the general reﬂectance as the cast shadow term s ij multiplying thebidirectional reﬂectance term,
Revised: To exploit clues of specular reflectance and cast shadow forlight estimation, we decompose the general reflectance as the cast shadow term s ij multiplying thebidirectional reflectance term,

Original: [Equation]
Revised: [Equation]

Original: where subscript ‘ i ’ and ‘ j ’ indicate position- (or normal-) and light-dependent factors, respectively;the cast shadow term s ij is either 0 or 1; ⇢ di , ⇢ sij represent the diffuse and specular reﬂectance, and k diand k si are coefﬁcients that balance out the effects of specular and diffuse reﬂectance 2 .
Revised: where subscript ‘ i ’ and ‘ j ’ indicate position- (or normal-) and light-dependent factors, respectively;the cast shadow term s ij is either 0 or 1; ρ di , ρ sij represent the diffuse and specular reflectance, and k diand k si are coefficients that balance out the effects of specular and diffuse reflectance 2 .

Original: Neural ﬁelds of specular reﬂectance and cast shadow.
Revised: Neural fields of specular reflectance and cast shadow.

Original: Previous methods identify specular featuresfrom speciﬁc pixels and associate an explicit relation to light for light estimation.
Revised: Previous methods identify specular featuresfrom specific pixels and associate an explicit relation to light for light estimation.

Original: However, thisscheme fails for the surface where the specular features are invisible or the explicit relation is violated.
Revised: However, thisscheme fails for the surface where the specular features are invisible or the explicit relation is violated.

Original: Besides, leaving out the clue of cast shadow can obstruct producing competitive performance.
Revised: Besides, leaving out the clue of cast shadow can obstruct producing competitive performance.

Original: In contrast, we leverage both clues of specular and shadow for light estimation, which is achieved bybuilding the neural specular ﬁeld and neural shadow ﬁeld, and associating the ﬁelds to light conditions.
Revised: In contrast, we leverage both clues of specular and shadow for light estimation, which is achieved bybuilding the neural specular field and neural shadow field, and associating the fields to light conditions.

Original: [Figure]
Revised: [Figure]

Original: PositionNet
Revised: PositionNet

Original: PE
Revised: PE

Original: LightNet" #
Revised: LightNet

Revised: [Figure]

Original: SpecularNet
Revised: SpecularNet

Original: PE
Revised: PE

Original: [Figure]
Revised: [Figure]

Original: Positional Encode $ ! =& !
Revised: Positional Encode 𝒉 ! =

Original:  + ( ||& !
Revised: 𝒍 ! + 𝒗 ||𝒍 !

Original: + (|| MLPs CNN Encoder * "! = + ! , "!- "# . "
Revised: + 𝒗|| MLPs CNN Encoder 𝑚 "!

Original: # + - "$ . "!
Revised: = 𝑒 ! 𝑠 "! 𝑘 "# 𝜌 "# + 𝑘 "$ 𝜌 "!

Original: $ max 2 "% & !
Revised: $ max 𝒏 "% 𝒍 ! , (𝑥,𝑦) 𝒍 !

Original: ShadowNet
Revised: ShadowNet

Original: PE - "$ - "# . " 
Revised: PE 𝑘 "$ 𝑘 "# 𝜌 "# 𝒏 " 𝜌 "!

Original: $ 2 "% $ ! ( % $ ! & !
Revised: $ 𝒏 "% 𝒉 !

Original: + ! 
Revised: 𝒗 % 𝒉 ! 𝒍 ! 𝑒 ! , ( ) (

Original: [Figure]
Revised: [Figure]

Original:  These neural ﬁelds make the utmost of all observed pixels, and exploit the mutual information acrossdifferent normal-light (specular) and position-light (shadow) for light estimation.
Revised: 𝑠 "! These neural fields make the utmost of all observed pixels, and exploit the mutual information acrossdifferent normal-light (specular) and position-light (shadow) for light estimation.

Original: Two MLPs, namely
Revised: Two MLPs, namely

Original: SpecularNet and ShadowNet , implement these neural ﬁelds, respectively.
Revised: SpecularNet and ShadowNet , implement these neural fields, respectively.

Original: we take the estimatedlight direction as the input of these networks to achieve their implicit association to light conditions.
Revised: we take the estimatedlight direction as the input of these networks to achieve their implicit association to light conditions.

Original: Since the specular reﬂectance and cast shadow are normal- and position-dependent, we also feed theestimated surface normal and the positional code to them, respectively.
Revised: Since the specular reflectance and cast shadow are normal- and position-dependent, we also feed theestimated surface normal and the positional code to them, respectively.

Original: SpecularNet and ShadowNetoutput ⇢ sij and s ij , respectively, as shown in Fig.
Revised: SpecularNet and ShadowNetoutput ρ sij and s ij , respectively, as shown in Fig.

Original: Neural light ﬁeld.
Revised: Neural light field.

Original: There is mutual information across different observed images, i.e. , observedimages with similar appearances are expected to be illuminated by similar lights.
Revised: There is mutual information across different observed images, i.e. , observedimages with similar appearances are expected to be illuminated by similar lights.

Original: To fully considerand exploit this mutual information, we build the neural light ﬁeld by concatenating a CNN encoderto an MLP, namely LightNet .
Revised: To fully considerand exploit this mutual information, we build the neural light field by concatenating a CNN encoderto an MLP, namely LightNet .

Original: The encoder extracts a light code from each observed image 3 , and the
Revised: The encoder extracts a light code from each observed image 3 , and the

Original: LightNet infers the corresponding light conditions ( i.e. , e j , l j ) from the light code, as shown in Fig.
Revised: LightNet infers the corresponding light conditions ( i.e. , e j , l j ) from the light code, as shown in Fig.

Original: Neural position ﬁeld.
Revised: Neural position field.

Original: There is mutual information across different positions on a surface, i.e. , theconsistency of shape and diffuse reﬂectance in the spatial domain.
Revised: There is mutual information across different positions on a surface, i.e. , theconsistency of shape and diffuse reflectance in the spatial domain.

Original: To fully consider and exploit thismutual information, we establish a neural position ﬁeld, namely PositionNet , implemented by an MLP.
Revised: To fully consider and exploit thismutual information, we establish a neural position field, namely PositionNet , implemented by an MLP.

Original: The neural position ﬁeld outputs position-dependent, light-independent factors 4 , i.e. , n i , ⇢ di , k di , k si .
Revised: The neural position field outputs position-dependent, light-independent factors 4 , i.e. , n i , ρ di , k di , k si .

Original: The PositionNet takes the positional code as the input, as shown in Fig. 2.
Revised: The PositionNet takes the positional code as the input, as shown in Fig. 2.

Original: 3.3 Optimizing Neural Intrinsics Fields
Revised: 3.3 Optimizing Neural Intrinsics Fields

Original: We adopt the reconstruction loss function with the ` 1 metric to optimize our NeIF,
Revised: We adopt the reconstruction loss function with the ℓ 1 metric to optimize our NeIF,

Original: [Equation]
Revised: [Equation]

Original: Silhouette constraint.
Revised: Silhouette constraint.

Original: L rec cannot resolve the shape-light ambiguity in an unsupervised manner dueto the inherently severe ill-posedness.
Revised: L rec cannot resolve the shape-light ambiguity in an unsupervised manner dueto the inherently severe ill-posedness.

Original: Therefore, we introduce the silhouette constraint (similar tothose in [23, 15]) to stabilize the training of PositionNet.
Revised: Therefore, we introduce the silhouette constraint (similar tothose in [23, 15]) to stabilize the training of PositionNet.

Original: To be speciﬁc, we use polynomial ﬁttingwith a moving window block to traverse and pre-compute the contour’s normal of the given objects,represented as ˆ N si , { ˆ n si k 2 R 2 ⇥ 1 , k 2 S } , S is the point set of the contour.
Revised: To be specific, we use polynomial fittingwith a moving window block to traverse and pre-compute the contour’s normal of the given objects,represented as ˆ N si ≜ { ˆ n si k ∈ R 2 × 1 , k ∈ S } , S is the point set of the contour.

Original: We consider ˆ N si can guide the prediction of the azimuth of boundary normal (at the same positions) and introduce thesilhouette loss function,
Revised: We consider ˆ N si can guide the prediction of the azimuth of boundary normal (at the same positions) and introduce thesilhouette loss function,

Original: [Equation]
Revised: [Equation]

Original: where n k 2 R 1 ⇥ 3 represents the estimated surface normal at the positions of silhouette from
Revised: where n k ∈ R 1 × 3 represents the estimated surface normal at the positions of silhouette from

Original: PositionNet, C ( · ) cuts off the 3rd dimension of n k ( i.e. , C ( n k ) 2 R 1 ⇥ 2 ), and Nor ( · ) is the vectornormalization operation.
Revised: PositionNet, C ( · ) cuts off the 3rd dimension of n k ( i.e. , C ( n k ) ∈ R 1 × 2 ), and Nor ( · ) is the vectornormalization operation.

Original: Warm-up loss functions.
Revised: Warm-up loss functions.

Original: To avoid local minimum and achieve faster convergence, we warm upthe NeIF in early-stage during training with three additional loss functions.
Revised: To avoid local minimum and achieve faster convergence, we warm upthe NeIF in early-stage during training with three additional loss functions.

Original: We use the azimuth oflighting direction estimated by YS97 [57] to guide the training of LightNet,
Revised: We use the azimuth oflighting direction estimated by YS97 [57] to guide the training of LightNet,

Original: [Equation]
Revised: [Equation]

Original: where l az j are estimated light directions by [57].
Revised: where l az j are estimated light directions by [57].

Original: We adopt the gradient penalty [22] L gp to stabilizethe training of SpecularNet,
Revised: We adopt the gradient penalty [22] L gp to stabilizethe training of SpecularNet,

Original: [Equation]
Revised: [Equation]

Original: The intuition is from Blinn-Phong model [8], where the specular reﬂectance is monotonicallyincreasing w.r.t the n > i h j , i.e. , r n > i
Revised: The intuition is from Blinn-Phong model [8], where the specular reflectance is monotonicallyincreasing w.r.t the n ⊤ i h j , i.e. , ∇ n ⊤ i

Original: h j ⇢ sij > 0 .
Revised: h j ρ sij > 0 .

Original: We also supervise the training of ShadowNet usingpseudo shadow maps ˆ S j , j = 1 , 2 , ..., f ,
Revised: We also supervise the training of ShadowNet usingpseudo shadow maps ˆ S j , j = 1 , 2 , ..., f ,

Original: [Equation]
Revised: [Equation]

Original: The pseudo shadow maps are obtained by binarizing the observed images, i.e. , considering anobserved pixel to be cast shadow if its intensity value is smaller than 0.2 ⇥ the mean intensity value ofthis image.
Revised: The pseudo shadow maps are obtained by binarizing the observed images, i.e. , considering anobserved pixel to be cast shadow if its intensity value is smaller than 0.2 × the mean intensity value ofthis image.

Original: After early-stage training, we discard these loss functions for a broad range of reﬂectance.
Revised: After early-stage training, we discard these loss functions for a broad range of reflectance.

Original: NeIF training.
Revised: NeIF training.

Original: We train NeIF with the warm-up loss function in ﬁrst 10 epochs,
Revised: We train NeIF with the warm-up loss function in first 10 epochs,

Original: [Equation]
Revised: [Equation]

Original: where  si = 5 ,  az = 0 .
Revised: where λ si = 5 , λ az = 0 .

Original: 1 ,  gp = 10 ,  shadow = 10 .
Revised: 1 , λ gp = 10 , λ shadow = 10 .

Original: We then train NeIF until 500 epochs or convergingwith the loss function,
Revised: We then train NeIF until 500 epochs or convergingwith the loss function,

Original: [Equation][Equation]
Revised: [Equation]

Original: where L recShadow is the another shadow map supervision loss to train the ShadowNet.
Revised: where L recShadow is the another shadow map supervision loss to train the ShadowNet.

Original: The loss functionis the same to L shadow .
Revised: The loss functionis the same to L shadow .

Original: The only difference is the calculation of ˆ S j .
Revised: The only difference is the calculation of ˆ S j .

Original: For L recShadow , we calculate ˆ S jby rendering a depth map with the estimated l j .
Revised: For L recShadow , we calculate ˆ S jby rendering a depth map with the estimated l j .

Original: The depth map is reconstructed from the estimatedsurface normal map N by method [10].
Revised: The depth map is reconstructed from the estimatedsurface normal map N by method [10].

Original: L recShadow is used to align outputs of ShadowNet to those of
Revised: L recShadow is used to align outputs of ShadowNet to those of

Original: PositionNet.
Revised: PositionNet.

Original: Implementation details.
Revised: Implementation details.

Original: We generate the positional code from the coordinate (in the image plane)by the same method in [34].
Revised: We generate the positional code from the coordinate (in the image plane)by the same method in [34].

Original: With the assumption of isotropic reﬂectance, we simplify the input of
Revised: With the assumption of isotropic reflectance, we simplify the input of

Original: SpecularNet from { v > l j , v > n i , n > i l j } to { v > h j , n > i h j } 5 [31] for easier training.
Revised: SpecularNet from { v ⊤ l j , v ⊤ n i , n ⊤ i l j } to { v ⊤ h j , n ⊤ i h j } 5 [31] for easier training.

Original: Similar to the
Revised: Similar to the

Original: Cook-Torrance reﬂectance model [19], we assume k di + k si = 1 to reduce the number of unknowns.
Revised: Cook-Torrance reflectance model [19], we assume k di + k si = 1 to reduce the number of unknowns.

Original: The CNN decoder with declining channels processes the down-sampled images with a dimension of256 ⇥ 256 .
Revised: The CNN decoder with declining channels processes the down-sampled images with a dimension of256 × 256 .

Original: The LightNet takes ﬂatten features to predict l j , e j in two different branches.
Revised: The LightNet takes flatten features to predict l j , e j in two different branches.

Original: For the
Revised: For the

Original: ShadowNet, we also generate the positional code for l j and concatenate it to the feature in the 9thlayer.
Revised: ShadowNet, we also generate the positional code for l j and concatenate it to the feature in the 9thlayer.

Original: The output of ShadowNet is either 0 or 1, which is realized by a step function with a similarimplementation in [38].
Revised: The output of ShadowNet is either 0 or 1, which is realized by a step function with a similarimplementation in [38].

Original: benchmark [45].
Revised: v ∥ .

Original: This table summarizes comparison methods. ’
Revised: This table summarizes comparison methods. ’

Original: N.A.’ represents not applicable as calibrated PS is with known ` and e .
Revised: N.A.’ represents not applicable as calibrated PS is with known ℓ and e .

Original: ‘Semi’ indicates certain method leverage partial information of light.
Revised: ‘Semi’ indicates certain method leverage partial information of light.

Original: ‘ 3 ’ (or ‘ 7 ’) represents that certain methods (do not) adopt supervised learning for the estimation of surface normal n , light direction l , or light intensity e .
Revised: ‘ ✓ ’ (or ‘ ✗ ’) represents that certain methods (do not) adopt supervised learning for the estimation of surface normal n , light direction l , or light intensity e .

Original: ‘Identical’ means certain methods require the light intensity of different illuminating to be identical.
Revised: ‘Identical’ means certain methods require the light intensity of different illuminating to be identical

Original: [Table]
Revised: [Table]

Original:  Experiments
Revised:  Experiments

Original: Training details.
Revised: Training details.

Original: Our main framework is implemented in PyTorch, while the pre-calculations(method [57] and silhouette ﬁtting) are implemented in MATLAB.
Revised: Our main framework is implemented in PyTorch, while the pre-calculations(method [57] and silhouette fitting) are implemented in MATLAB.

Original: We use Adam as the optimizerwith a learning rate ↵ = 5 ⇥ 10  4 to train our framework in 500 epochs for each scene separately, andthe warm-up stage takes up 10 epochs.
Revised: We use Adam as the optimizerwith a learning rate α = 5 × 10 − 4 to train our framework in 500 epochs for each scene separately, andthe warm-up stage takes up 10 epochs.

Original: The last 100 epochs use a lower learning rate ↵ = 5 ⇥ 10  5for ﬁne-tunning.
Revised: The last 100 epochs use a lower learning rate α = 5 × 10 − 5for fine-tunning.

Original: The batch size is 32 for lighting and 256 for spatially random sampling.
Revised: The batch size is 32 for lighting and 256 for spatially random sampling.

Original: After everyepoch, the depth map is reconstructed according to the predicted normal map by method [10].
Revised: After everyepoch, the depth map is reconstructed according to the predicted normal map by method [10].

Original: Eachscene takes from 2 hours to 6 hours on one RTX 2080Ti 12GB GPU, depending on the resolution ofthe objects.
Revised: Eachscene takes from 2 hours to 6 hours on one RTX 2080Ti 12GB GPU, depending on the resolution ofthe objects.

Original: Evaluation metrics.
Revised: Evaluation metrics.

Original: We adopt the same metric in [13], the scale-invariant relative error, to measurethe accuracy of recovered light intensity as,
Revised: We adopt the same metric in [13], the scale-invariant relative error, to measurethe accuracy of recovered light intensity as,

Original: [Equation]
Revised: [Equation]

Original: where, ⌘ is calculated by solving arg min ⌘
Revised: where, η is calculated by solving arg min η (cid:80) fj

Original: P fj =
Revised: =1 (

Original: [Equation]
Revised: [Equation]

Original:  ) 2 by least squares minimization.
Revised: ˜ e j ) 2 by least squares minimization.

Original: Themetric to measure the accuracy of the predicted light directions and surface normal is the widely usedmean angle error (MAE) in degree.
Revised: Themetric to measure the accuracy of the predicted light directions and surface normal is the widely usedmean angle error (MAE) in degree.

Original: 4.1 Evaluation on Public Datasets
Revised: 4.1 Evaluation on Public Datasets

Original: Since the proposed NeIF is an unsupervised uncalibrated photometric stereo method, we compare itsperformance with state-of-the-art uncalibrated and unsupervised photometric stereo methods.
Revised: Since the proposed NeIF is an unsupervised uncalibrated photometric stereo method, we compare itsperformance with state-of-the-art uncalibrated and unsupervised photometric stereo methods.

Original: Threereal-world datasets, including the D I L
Revised: Threereal-world datasets, including the D I L

Original: I G EN T benchmark dataset [45], A PPLE & G OURD dataset [2],and L IGHT S TAGE D ATA G ALLERY dataset [11], are used for evaluation.
Revised: I G EN T benchmark dataset [45], A PPLE & G OURD dataset [2],and L IGHT S TAGE D ATA G ALLERY dataset [11], are used for evaluation.

Original: Quantitative comparison for normal map.
Revised: Quantitative comparison for normal map.

Original: Table 1 lists relevant works for a comprehensive surfacenormal estimation comparison.
Revised: Table 1 lists relevant works for a comprehensive surfacenormal estimation comparison.

Original: As summarized in Table 1, our method is the only method thataddresses UPS without the supervision of N , L , or E .
Revised: As summarized in Table 1, our method is the only method thataddresses UPS without the supervision of N , L , or E .

Original: However, our method achieves the bestperformance and maintains a considerable advantage over other competitors [13, 15] ( e.g. , 7.86 for
Revised: However, our method achieves the bestperformance and maintains a considerable advantage over other competitors [13, 15] ( e.g. , 7.86 for

Original: NeIF vs. 8.71 for [15]).
Revised: NeIF vs. 8.71 for [15]).

Original: Handling the objects in D I L I G EN T dataset [45] with different shapes andreﬂectance, numbers from our method are either best or competitive (about 1  as compared with thebest performing UPS method), which shows its good generalization capacity to general reﬂectanceand various shapes.
Revised: Handling the objects in D I L I G EN T dataset [45] with different shapes andreflectance, numbers from our method are either best or competitive (about 1 ◦ as compared with thebest performing UPS method), which shows its good generalization capacity to general reflectanceand various shapes.

Original: This is because our NeIF fully considers mutual information by building upintrinsics ﬁelds and the implicit modeling facilitates general reﬂectance modeling.
Revised: This is because our NeIF fully considers mutual information by building upintrinsics fields and the implicit modeling facilitates general reflectance modeling.

Original: Visual quality comparison for the normal map.
Revised: Visual quality comparison for the normal map.

Original: Fig.
Revised: Fig.

Original: 3 illustrates the visual quality comparison interms of recovered surface normal maps and corresponding error maps on D
Revised: 3 illustrates the visual quality comparison interms of recovered surface normal maps and corresponding error maps on D

Original: I L I G EN T dataset [45].
Revised: I L I G EN T dataset [45].

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: GO URD GO URD
Revised: GO URD GO URD

Revised: [Figure]

Original: S T A ND
Revised: S T A ND

Original: I N
Revised: I N

Original: [Figure][Figure]
Revised: [Figure]

Original: M

Original: [Figure]
Revised: [Figure]

Original: The comparison is conducted with two state-of-the-art UPS methods [13, 15] and a state-of-the-artunsupervised PS method [47].
Revised: The comparison is conducted with two state-of-the-art UPS methods [13, 15] and a state-of-the-artunsupervised PS method [47].

Original: Our method is less sensitive to spatially-varying albedo due to theper-pixel manner (red boxes in the left subﬁgure of Fig. 3).
Revised: Our method is less sensitive to spatially-varying albedo due to theper-pixel manner (red boxes in the left subfigure of Fig. 3).

Original: However, although the positional codeconsiders the global shape effect, this per-pixel manner is less effective in modeling complex shapeinformation as compared with the all-pixel one.
Revised: However, although the positional codeconsiders the global shape effect, this per-pixel manner is less effective in modeling complex shapeinformation as compared with the all-pixel one.

Original: It fails for regions with cast shadow (or overexposure)under most light directions (red boxes in the right subﬁgure of Fig. 3).
Revised: It fails for regions with cast shadow (or overexposure)under most light directions (red boxes in the right subfigure of Fig. 3).

Original: We also show the visualquality results for A PPLE & G OURD dataset [2] and the L IGHT S TAGE D ATA G ALLERY dataset [11]in Fig. 4.
Revised: We also show the visualquality results for A PPLE & G OURD dataset [2] and the L IGHT S TAGE D ATA G ALLERY dataset [11]in Fig. 4.

Original: Our method produces reliable estimation for most regions, thanks to the full exploitation ofmutual information across different dimensions.
Revised: Our method produces reliable estimation for most regions, thanks to the full exploitation ofmutual information across different dimensions.

Original: Quantitative comparison for light directions and intensities.
Revised: Quantitative comparison for light directions and intensities.

Original: As can be observed in Table 2and Table 3, our method achieves a superior performance advantage over unsupervised methods(YS97 [57] and PF14 [37]) while maintaining competitive performance as compared with supervisedmethods (CH19 [13] and CW20 [15]).
Revised: As can be observed in Table 2and Table 3, our method achieves a superior performance advantage over unsupervised methods(YS97 [57] and PF14 [37]) while maintaining competitive performance as compared with supervisedmethods (CH19 [13] and CW20 [15]).

Original: These supervised methods adopt two-step solutions andsuffer from accumulating error.
Revised: These supervised methods adopt two-step solutions andsuffer from accumulating error.

Original: Therefore, even though they achieve similar performance in terms oflight conditions accuracy, they are less effective on estimating surface normal as compared with ourmethod (see Table 1). 
Revised: Therefore, even though they achieve similar performance in terms oflight conditions accuracy, they are less effective on estimating surface normal as compared with ourmethod (see Table 1). [Table] B

Original: [Table] 
Revised: [Table] B UDDHA C AT C OW

Original: [Table]
Revised: G OBLET H ARVEST P OT P OT

Original:  4.2 Evaluation on Sparse Uncalibrated Photometric Stereo
Revised: R EADING [Table] [Table] 4.2 Evaluation on Sparse Uncalibrated Photometric Stereo

Original: As compared with other per-pixel PS methods ( e.g. , ZJ19 [62] and LL19 [30])
Revised: As compared with other per-pixel PS methods ( e

Original: [Table]
Revised: [Table]

Original: the proposed NeIFexploits much more constraints to estimate the normal at a point ( i
Revised: g. , ZJ19 [62] and LL19 [30])

Original: [Table]
Revised: [Table]

Original: e. , # P ⇥ f vs. f ).
Revised: the proposed NeIFexploits much more constraints to estimate the normal at a point ( i.e. , # P × f vs. f ).

Original: Because itlearns ﬁelds instead of regressing intensity proﬁle [43] to surface normal.
Revised: Because itlearns fields instead of regressing intensity profile [43] to surface normal.

Original: Therefore, we investigatethe effectiveness of our method under the challenging setting of sparse UPS.
Revised: Therefore, we investigatethe effectiveness of our method under the challenging setting of sparse UPS.

Original: We randomly select 10 or 16 images illuminated by different lights from D I L I G EN T dataset [45] andtest our method using these images.
Revised: We randomly select 10 or 16 images illuminated by different lights from D I L I G EN T dataset [45] andtest our method using these images.

Original: We repeat this process 30 times, similar to [62].
Revised: We repeat this process 30 times, similar to [62].

Original: We compare theperformance with that from the classical UPS method PF14 [37], a deep learning based per-pixelapproach ZJ19 [62], and an all-pixel approach CH18 [14].
Revised: We compare theperformance with that from the classical UPS method PF14 [37], a deep learning based per-pixelapproach ZJ19 [62], and an all-pixel approach CH18 [14].

Original: Due to the fewer reconstruction terms totrain our NeIF, we slightly increase the dimension of the positional encoding module from 4 to 6 thatincreases the frequency [48] to stabilize the training, i.e. , strengthening the role of positional code toreduce the variance of estimated intrinsics.
Revised: Due to the fewer reconstruction terms totrain our NeIF, we slightly increase the dimension of the positional encoding module from 4 to 6 thatincreases the frequency [48] to stabilize the training, i.e. , strengthening the role of positional code toreduce the variance of estimated intrinsics.

Original: Besides, we drop L az during early stage warm-up because
Revised: Besides, we drop L az during early stage warm-up because

Original: YS97 [57] fails for the sparse inputs.
Revised: YS97 [57] fails for the sparse inputs.

Original: We report the mean results over 30 trails in Table 4.
Revised: We report the mean results over 30 trails in Table 4.

Original: As can be observed, our method achievescompetitive performance on normal estimation as compared with state-of-the-art sparse PS methodswith known light ( i.e. , 11.62 for NeIF vs. 9.82 for ZJ19 [62] under 10 lights, and 10.38 for NeIF vs.9.00 for CH18 [14] under 16 lights).
Revised: As can be observed, our method achievescompetitive performance on normal estimation as compared with state-of-the-art sparse PS methodswith known light ( i.e. , 11.62 for NeIF vs. 9.82 for ZJ19 [62] under 10 lights, and 10.38 for NeIF vs.9.00 for CH18 [14] under 16 lights).

Original: Kindly note that we do not require any ground truth normal forsupervision.
Revised: Kindly note that we do not require any ground truth normal forsupervision.

Original: As compared with the unsupervised UPS method (PF14 [37]), we achieve a superiorperformance advantage.
Revised: As compared with the unsupervised UPS method (PF14 [37]), we achieve a superiorperformance advantage.

Original: The results validate the effectiveness of building up neural intrinsics ﬁeldsthat fully exploit the mutual information across different dimensions.
Revised: The results validate the effectiveness of building up neural intrinsics fieldsthat fully exploit the mutual information across different dimensions.

Original:  Conclusion
Revised:  Conclusion

Original: This paper proposes NeIF for UPS.
Revised: This paper proposes NeIF for UPS.

Original: By representing the general reﬂectance as four neural intrinsicsﬁelds, it implicitly imposes the light clues of specular reﬂectance and cast shadow for light estimation,which facilitates solving UPS with general reﬂectance.
Revised: By representing the general reflectance as four neural intrinsicsfields, it implicitly imposes the light clues of specular reflectance and cast shadow for light estimation,which facilitates solving UPS with general reflectance.

Original: The proposed NeIF can fully exploit mutualinformation from all observed pixel values so that it produces stable estimation for PS and sparse PS.
Revised: The proposed NeIF can fully exploit mutualinformation from all observed pixel values so that it produces stable estimation for PS and sparse PS.

Original: The unsupervised training manner of NeIF is beneﬁcial to the generalization capacity of data fromdifferent sources.
Revised: The unsupervised training manner of NeIF is beneficial to the generalization capacity of data fromdifferent sources.

Original: Limitations.
Revised: Limitations.

Original: Although our method produces promising results for light conditions and surfacenormal estimation, it has several limitations.
Revised: Although our method produces promising results for light conditions and surfacenormal estimation, it has several limitations.

Original: First, the estimated intrinsics, such as shadow, specularand diffuse reﬂectance, and their balancing coefﬁcients are less accurate for objects with complicatedgeometries, as shown in Fig.
Revised: First, the estimated intrinsics, such as shadow, specularand diffuse reflectance, and their balancing coefficients are less accurate for objects with complicatedgeometries, as shown in Fig.

Original: Second, we need to balance the role of positional code and estimatedintrinsics when applying our method for sparse UPS.
Revised: Second, we need to balance the role of positional code and estimatedintrinsics when applying our method for sparse UPS.

Original: However, how to ﬁnd an optimal balancingstrategy is unknown.
Revised: However, how to find an optimal balancingstrategy is unknown.

Original: Third, as shown in Fig.
Revised: Third, as shown in Fig.

Original: 3, our method is less effective on global effects,especially for regions with cast shadow (or overexposure) under most light directions due to our
Revised: 3, our method is less effective on global effects,especially for regions with cast shadow (or overexposure) under most light directions due to our

Original: [Table]
Revised: [Table]

Original: per-pixel manner.
Revised: per-pixel manner.

Original: Besides, the inference of our method is inefﬁcient due to the unsupervised trainingmanner and the per-pixel estimation, which is inapplicable for real-time applications.
Revised: Besides, the inference of our method is inefficient due to the unsupervised trainingmanner and the per-pixel estimation, which is inapplicable for real-time applications.