Original: NeIF: Representing General Reﬂectance as Neural Intrinsics Fields for Uncalibrated Photometric Stereo
Revised: NeIF: Representing General Reﬂectance as Neural Intrinsics Fields for Uncalibrated Photometric Stereo

Original: Abstract
Revised: Abstract

Original: Uncalibrated photometric stereo (UPS) is challenging due to the inherent shapelight.
Revised: Uncalibrated photometric stereo (UPS) is challenging due to the inherent ambiguitybrought by unknown light.

Original: Existing solutions alleviate this ambiguity by either explicitly associatingreﬂectance to light estimation or resolving light conditions in a supervised manner.
Revised: Existing solutions alleviate the ambiguity by eitherexplicitly associating reﬂectance to light conditions or resolving light conditions ina supervised manner.

Original: This paper establishes an implicit relation between light clues and light estimationand solves UPS in an unsupervised manner.
Revised: This paper establishes an implicit relation between light cluesand light estimation and solves UPS in an unsupervised manner.

Original: The key idea is to represent thereﬂectance as four neural intrinsics ﬁelds, i.e , specular, shadow, light, and position,based on which the neural light ﬁeld is implicitly associated to light clues ofspecular reﬂectance and cast shadow.
Revised: The key idea is torepresent the reﬂectance as four neural intrinsics ﬁelds, i.e. , position, light, specular,and shadow, based on which the neural light ﬁeld is implicitly associated withlight clues of specular reﬂectance and cast shadow.

Original: The unsupervised training of neural intrinsicsﬁelds fully exploits all observed pixel values, which facilitates solving UPS.
Revised: The unsupervised optimizationof neural intrinsics ﬁelds can be free from training data bias and fully exploitsall observed pixel values for UPS.

Original: Ourmethod demonstrates a superior performance advantage over state-of-the-art UPSmethods on public datasets and promising results under the challenging setting ofsparse UPS.
Revised: Our method achieves a superior performanceadvantage over state-of-the-art UPS methods on public datasets and promisingresults under the challenging setting of sparse UPS.

Original: We will release our codes and pre-trained models once the paper wasaccepted.
Revised: The code will be releasedsoon.

Original:  Introduction
Revised:  Introduction

Original: Photometric stereo (PS) [47] aims at recovering the surface normal from several light-varying imagescaptured at a ﬁxed viewpoint.
Revised: Photometric stereo (PS) [50] aims at recovering the surface normal from several light-varying imagescaptured at a ﬁxed viewpoint.

Original: As compared with otherstereo approaches ( e g. , multi-view stereo [38],active sensor-based solutions [55]), photometric stereo is excellent at recovering ﬁne-detailed surfacesand has been widely used for Hollywood movies [9], industrial quality inspection [46] and biometrics[52].
Revised: As compared with other approaches ( e g. , multi-view stereo [41], activesensor-based solutions [58]), photometric stereo is excellent at recovering ﬁne-detailed surfaces andhas been widely used for Hollywood movies [11], industrial quality inspection [49] and biometrics[55].

Original: Calibrating accurate lighting directions is crucial to the performance of photometric stereomethods [51].
Revised: Calibrating accurate lighting directions is crucial to the performance of photometric stereomethods [54].

Original: However, lighting calibration is often tedious, dramatically limiting theapplicability inthe real-world.
Revised: However, lighting calibration is often tedious, dramatically restricting the applicabilityin the real-world.

Original: To this end, researchers develop uncalibrated photometric stereo (UPS) methods thatestimate surface normal with unknown lights.
Revised: To this end, researchers develop uncalibrated photometric stereo (UPS) methodsthat estimate surface normal with unknown lights.

Original: Uncalibrated photometric stereo suffers from General Bas-Relief (GBR) ambiguity [6] for an integrable surface.
Revised: Uncalibrated photometric stereo suffers from General Bas-Relief (GBR) ambiguity [6] for an integrable surface.

Original: Early solutions address the ambiguity by explicitly associating reﬂectance to light, i.e ,adopting analytic reﬂectance models ( e.g. , Lambertian reﬂectance [4], [34], parametric specular reﬂection [18], specular spikes [53], inter-reﬂection [10]) or imposing priors from reﬂectance properties[3, 22, 21, 41].
Revised: Early solutions address the ambiguity by explicitly associating reﬂectance to light,i.e. , adopting analytic reﬂectance models ( e.g. , Lambertian reﬂectance [4], [37], parametric specularreﬂection [20], specular spikes [56], inter-reﬂection [12]) or imposing priors from reﬂectance properties [3, 25, 24, 44].

Original: Thus,  these methods could be less effective for unknown reﬂectance due to the strongreliance reﬂectance assumption.
Revised: Thus, due to the strong reliance on reﬂectance assumption, these methods can beless effective for unknown reﬂectance.

Original: Besides, these methods ignore the clue of cast shadow and even failfor shadow regions due to the shadow-free surface assumption.
Revised: Besides, these methods ignore the clue of cast shadow andeven fail in shadow regions due to the shadow-free surface assumption.

Original: Further, most of them require thelight intensity to be identical for robust estimation.
Revised: Further, most of them requirethe light intensity to be identical for robust estimation.

Original: Recently, deep learning-based approaches addressthe ambiguity by estimating light direction and intensity before recovering surface normal [12, 13, 25,37].
Revised: Recently, deep learning-based approachesaddress the ambiguity by estimating light direction and intensity before recovering surface normal [13,15, 27, 40].

Original: They train a light estimation network using a large-scale amount of rendered data in a supervisedmanner.
Revised: They train a light estimation network using a large-scale amount of rendered data in asupervised manner.

Original: However, the training data bias [30] can hardly be eliminated and could produce unexpectedestimation for real-world data.
Revised: However, the training data bias [33] can hardly be eliminated and can produceunexpected estimation for real-world data.

Original: Because rendered training data inevitably contain thedomain gapfrom real ones andcould scarcely cover all surfaces with different geometry and reﬂectance in thereal-world.
Revised: Because rendered training data inevitably contains thedomain gap from the real ones and scarcely cover all surfaces with different geometry and reﬂectance in the real-world.

Original: Besides, such a two-step solution could bring accumulating errors for surface normal
Revised: Besides, such a two-step solution can bring accumulating errors for surface normal

Original: Neural Shadow Field
Revised: Neural Shadow Field

Original: P o s i t i on
Revised: P o s i t i on

Original: Light direction ( ) + × × = × × × · · · 𝒏 𝑖 𝑘 𝑖 𝑑 𝜌 𝑖 𝑑 𝑘 𝑖 𝑠 Neural Position Field
Revised: Light direction ( ) + × × = × × × · · · ! ! " !" # Neural Position Field

Original: P o s i t i on
Revised: P o s i t i on

Original: Neural Specular Field
Revised: Neural Specular Field

Original: S
Revised: S

Original: u r f a c e N o r m a l
Revised: u r f a c e no r m a l

Original: Light direction
Revised: Light direction

Original: Neural Light Field
Revised: Neural Light Field

Original: O b s e r v ed I m age s
Revised: O b s e r v ed i m age s

Original: Light direction · · · 𝜌 𝑖𝑑 (𝒍 𝑗 ,𝑒 𝑗 ) 𝑠 𝑖𝑗 𝑘 𝑖𝑑 𝑘 𝑖𝑠 𝜌 𝑖𝑗𝑠 𝒏 𝑖 𝑚 𝑖𝑗 · · · · · · · · · · · · · · · · · · position-dependent light- dependent 𝜌 𝑖𝑑 ✓  𝑘 𝑖 𝑠 ✓  𝑘 𝑖𝑑 ✓  𝒏 𝑖 ✓  𝒍 𝒋  ✓ 𝑒 𝑗  ✓ 𝜌 𝑖𝑗𝑠 ✓ ✓ 𝑠 𝑖𝑗 ✓ ✓
Revised: Light direction · · · ! !" ( !$ ! ! #$ ) ! Position-Dependent

Original: To this end, we propose NeIF , representing general reﬂectance as Ne ural I ntrinsics F ields foruncalibrated photometric stereo. Our method differs from previous methods in three aspects: 1)it fully considers clues of specular reﬂectance and cast shadow from each observed pixel for lightestimation so that it is expected to produce accurate estimation for both light conditions and surfacenormal; 2) it does not make explicit assumptions about the reﬂectance or light so that it could workwith general surface reﬂectance and
Revised: Light- Dependent ! !" ü û " ! # ü û " ! "

Original: ﬂexible
Revised: ü û # !

Original: light settings; it
Revised: ü û $ $ û ü % % û ü ! ! # ü ü & !

Original: infers
Revised: ü ü

Original: light and surface normal in anunsupervised manner so that it could be free from training data bias and achieve stable performance
Revised: To this end, we propose NeIF , representing general reﬂectance as Ne ural I ntrinsics F ields foruncalibrated photometric stereo.

Original:  for  data  from difference sources
Revised: Our method differs from previous methods in three aspects: 1)it fully considers clues of specular reﬂectance and cast shadow from each observed pixel for lightestimation so that it is expected to produce accurate estimation for both light conditions and surfacenormal; 2) it does not make explicit assumptions about the reﬂectance or light so that it works withgeneral surface reﬂectance and ﬂexible light settings; 3) it infers light and surface normal in anunsupervised manner so that it is free from training data bias and achieves stable performance fordata from different sources.

Original: Our key idea is to represent the general reﬂectance as four neural intrinsics ﬁelds ( i.e , specularreﬂectance, cast shadow, light, and position, see Fig. 1), implemented by four multi-layer perceptrons(MLPs).
Revised: Our key idea is to represent the general reﬂectance as four neural intrinsics ﬁelds ( i.e. , position,light, specular, and shadow, see Fig. 1), implemented by four multi-layer perceptrons (MLPs).

Original: The connection of these four ﬁelds is based on the implicit relation (or dependence) of theseintrinsics so that no explicit assumption is imposed, e.g. , we take the estimated light as the input torecover specular reﬂectance and cast shadow instead of explicitly exploiting them for light estimation.
Revised: Thesefour ﬁelds are connected based on the implicit relation (or dependence) of these intrinsics so thatno explicit assumption is imposed, e.g. , we take the estimated light as the input to recover specularreﬂectance and cast shadow instead of explicitly exploiting them for light estimation.

Original: These intrinsics ﬁelds are optimized to reconstruct each pixel value from observed images, whichfully exploit mutual information across different dimensions, as shown in Fig.
Revised: These intrinsicsﬁelds are optimized to reconstruct each pixel value from observed images, which fully exploitmutual information across different dimensions, as shown in Fig.

Original: The reconstructionerror backpropagates to the neural light ﬁeld through neural specular and shadow ﬁelds, so that cluesof specular and shadow can be implicitly and fully considered for light estimation.
Revised: The reconstruction error isbackpropagated to the neural light ﬁeld through neural specular and shadow ﬁelds so that clues ofspecular and shadow can be implicitly and fully considered for light estimation.

Original: Our contributionsare summarized as
Revised: Our contributionsare summarized as:

Original: • We represent general reﬂectance as four intrinsic neural ﬁelds to implicitly associate per-
Revised: • We represent general reﬂectance as four intrinsic neural ﬁelds to implicitly associate per-

Original: pixel reﬂectance to light, which  fully considers clues of specular reﬂectance and cast shadow for light estimation.
Revised: pixel reﬂectance to light, which solves uncalibrated photometric stereo by fully considering clues of specular reﬂectance and cast shadow for light estimation.

Original: • We propose the NeIF, an uncalibrated photometric stereo method trained in an unsupervisedmanner, which works with general surface reﬂectance, ﬂexible light settings, and trainingdata bias.
Revised: • We propose the NeIF, an uncalibrated photometric stereo method trained in an unsupervised manner, which works with general surface reﬂectance and ﬂexible light settings, and is free from training data bias.

Original: • We show that our method achievesa superior performance over uncalibrated and unsupervised methods.
Revised: • We show that our method achieves superior performance over uncalibrated and unsupervisedmethods.

Original: We also demonstrate its excellent generalization capacity to data from 
Revised: We also demonstrate its excellent generalization capacity to data from different

Original: different sources and promising performance with the challenging setup of sparse uncali-
Revised: sources and promising performance with the challenging setup of sparse uncalibrated

Original: brated photometric stereo.
Revised: photometric stereo.

Original:  Related Work
Revised:  Related Work

Original: In this section, we mainly review the latest works in neural reﬂectance representation, and related 
Revised: This section mainly reviews the latest works in neural reﬂectance representation, and related workson unsupervised PS methods and UPS methods.

Revised: Readers may refer to [45] and [13] for a morecomprehensive summary.

Original: 2.1 Neural Reﬂectance Representation
Revised: 2.1 Neural Reﬂectance Representation

Original: Neural Radiance Fields (NeRFs)
Revised: Neural Radiance Fields (NeRFs)

Original: [31] focuses on the 3D geometries without explicitly modeling theinteraction between light and objects via photometric image formation model.
Revised: [34] focus on the 3D geometries without explicitly modeling theinteraction between light and objects via the image formation model.

Original: Many subsequentworks explore its application in various computer vision problems, such as relighting [14], photometricstereo [26], and multi-viewphotometric stereo [24].
Revised: Many subsequent worksexplore its application in various computer vision problems, such as relighting [16, 46], photometricstereo [29], and multi-view stereo [7, 9, 60, 28, 61].

Original: These workseither require known lightingconditions [ bi2022neural , 43, 58, 26] or adopt simple reﬂectance model [56]. There are alsosome works [7, 57] considering general reﬂectance, but they require multi-view information anddoes not explicitly consider cast shadow effects. Kaya et al. [24] implemented NeRF in multiview photometric stereo, but their work also leverages multi-view information.
Revised: These works require known lighting conditions [7,46, 61, 29], adopt simple reﬂectance model [59], or leverage multi-view information [7, 9, 60, 28, 61].

Original: Different from thesemethods, our method considers general reﬂectance, cast-shadow effects, and unknown light conditionsfor uncalibrated photometric stereo by taking images captured at a single viewpoint.
Revised: Different from these methods, our method considers general reﬂectance, cast-shadow effects, andunknown light conditions for uncalibrated photometric stereo by taking images captured at a singleviewpoint.

Original: 2.2 UnsupervisedCalibrated Photometric Stereo
Revised: 2.2 Unsupervised Photometric Stereo

Original: Classical methods solved the calibrated photometric stereo problem without knowing the groundtruth surface normal.
Revised: Classical methods solve the calibrated photometric stereo problem without knowing the groundtruth surface normal.

Original: Therefore, we classify them as unsupervised methods.
Revised: Therefore, we classify them as unsupervised methods.

Original: The least square basedalgorithm [47] provides the simplest solution, which assumes the object to be Lambertian.
Revised: The least squarebased algorithm [50] provides the simplest solution, which assumes the object to be Lambertian.

Original: Thealgorithm’s performance is robust in a wide range of datasets, but its strong assumption on thereﬂectance model remains much space for improvement.
Revised: It is generally served as a baseline method due to its stability, but its strong assumption on thereﬂectance model makes it fail for non-Lambertian surface.

Original: The following works either regard thenon-Lambertian reﬂectance components as the outliers [5, 16, 33, 48, 23, 49] or apply analyticreﬂectance models including Torrance-Sparrow[18], the Ward model[16], a mixture of multiple Wardmodels [19], [1] etc. to consider the non-Lambertian effects.
Revised: The following works either regard thenon-Lambertian reﬂectance components as the outliers [5, 18, 36, 51, 26, 52] or apply analyticreﬂectance models including Torrance-Sparrow [20], the Ward model [18], a mixture of multiple Ward models [21], [1] etc. to consider the non-Lambertian effects.

Original: However, the performance of thosemethods can only deal with limited types of materials.
Revised: However, the performance ofthose methods can only deal with limited types of materials.

Original: There are also more advanced methods thatutilize the general reﬂetance features such as reciprocity, isotropy [2], and monotonicity [21].
Revised: There are also more advanced methodsthat utilize the general reﬂetance features such as reciprocity, isotropy [2], and monotonicity [24].

Original: Thosemethods give a reliable estimation for objects with a broad range of materials.
Revised: Those methods give a reliable estimation for objects with a broad range of materials.

Original: Inspired by them, ourapproach considers the general reﬂectance properties.

Original: With the progress of deep learning, many learning-based frameworks have been proposed for calibrated photometric stereo.
Revised: With the progress of deep learning, many learning-based frameworks have been proposed for calibrated photometric stereo.

Original: Taniai et al.
Revised: Taniai et al.

Original: [44] propose the ﬁrst unsupervised learning-based photometricstereo method through a rendering equation.
Revised: [47] proposed the ﬁrst unsupervised learning-based photometric stereo method through a rendering equation.

Original: However, their reﬂectance model does not separatelyconsider shadow, specular highlights, and diffuse components.
Revised: However, their reﬂectance model does not separatelyconsider shadow, specular highlights, and diffuse components.

Original: Different from these methods that need the known lights, we address the uncalibrated photometricstereo.

Original: We also train our method in an unsupervised manner.
Revised: We also train our method in an unsupervised manner.

Original: Because it is expected to be free fromtraining data bias and thus obtain a stable estimation across data from different sources.
Revised: Different from previous methods, we addressthe challenging problem of UPS and separately model cast shadow, specular reﬂectance, and diffusereﬂectance.

Original: 2.3 Uncalibrated Photometric Stereo
Revised: 2.3 Uncalibrated Photometric Stereo

Original: Previous works study thephenomenon where Lambertian objects under arbitrary directions cangenerate identical images when similar transformations are applied to both the object’s normal andthe light’s direction.
Revised: Previous works hold the Lambertian assumption and address the ambiguity brought by a 3 ⇥ 3transformation matrix.

Original: Belhumeur et al.
Revised: Belhumeur et al.

Original: [6] reduce the dimension of the transformation to a threeparameters GBR transformation by considering integrability constraints.
Revised: [6] reduce the dimension of the transformation to a threeparameters GBR transformation by considering integrability constraints.

Original: Based on that, extra clueson reﬂectance, such as half-vector symmetry [28], albedo clustering [42], specular spikes [53], orassumptions on light source distribution, such as ring light [60], symmetry light [32], or uniformdistributed light sources [54, 4, 42, 50, 34, 28], are used to resolve the GBR ambiguity.
Revised: Based on that, extra cluesfrom reﬂectance, such as half-vector symmetry [31], albedo clustering [42], specular spikes [56], orassumptions of light source distribution, such as ring light [63], symmetry light [35], or uniformdistributed light sources [57, 4, 42, 53, 37, 31], are used to resolve the GBR ambiguity.

Original: However,all these methods require the light intensity to be identical, which is inapplicable in the real-worlddatasets such as D
Revised: However,all these methods require the light intensity to be identical, which is inapplicable in the real-worlddatasets such as D

Original: I L I G EN T [39], A PPLE & G OURD [2], and L IGHT S TAGE D ATA G ALLERY
Revised: I L I G EN T [45], A PPLE & G OURD [2], and L IGHT S TAGE D ATA G ALLERY

Original: Choet al.
Revised: Cho et al.

Original: [15] put up a semi-calibrated method to deal with non-uniform light intensity, but they assumedthe light directions to be known.
Revised: [17] put up a semi-calibrated method to deal with non-uniform light intensity, but theyassume the light directions to be known.

Original: Quéau et al.
Revised: Quéau et al.

Original: [36] address the photometric stereo problem underinaccurate lighting calibration, while the accuracy could signiﬁcantly drop when non-Lambertiancomponents become dominant.
Revised: [39] address the photometric stereo problemunder inaccurate lighting calibration, while the accuracy can signiﬁcantly drop when non-Lambertiancomponents become dominant.

Original: Recently, many deep learning methods have been proposed for uncalibrated photometric stereo.
Revised: Recently, many deep learning methods have been proposed for uncalibrated photometric stereo.

Original: Chen et al.
Revised: Chen et al.

Original: [12] propose a supervised uncalibrated framework, SDPS-Net, which can simultaneouslyestimate the light conditions (intensity and direction) and the surface normal.
Revised: [13] propose a supervised uncalibrated framework, SDPS-Net, which can simultaneouslyestimate the light conditions (intensity and direction) and the surface normal.

Original: They suggest treatinglight estimation as a classiﬁcation problem and separate the normal and light prediction to reduce thecomplexity.
Revised: They suggest treatinglight estimation as a classiﬁcation problem and separating the normal and light prediction to reducethe complexity.

Original: Their following work, GC-Net[13], improved the performance of SDPS-Net by addingshading as an extra channel to the input of the light estimation network.
Revised: Their following work, GC-Net [15], improves the performance of SDPS-Net by adding shading as an extra channel to the input of the light estimation network.

Original: However, as a commonproblem for all supervised methods, an over-ﬁtting problem may occur due to the training data bias[30].
Revised: However, as acommon problem for all supervised methods, an over-ﬁtting problem may occur due to the trainingdata bias [33].

Original: In contrast, unsupervised methods do not have such a concern.
Revised: In contrast, unsupervised methods do not have such a concern.

Original: Anotherobvious beneﬁt is that there is no need to synthesize training sets for unsupervised network training.
Revised: Another beneﬁt isthat there is no need to synthesize training sets for unsupervised network training.

Original: To utilize theadvantage of unsupervised methods, Kaya et al.
Revised: To utilize theadvantage of unsupervised methods, Kaya et al.

Original: [25] propose a compromised method that trains thelight estimation network in a supervised manner (similar to [12]), but estimate the surface normal inan unsupervised way (similar to [44].
Revised: [27] propose a compromised method that trains thelight estimation network in a supervised manner (similar to [13]), but estimate the surface normal inan unsupervised way (similar to [47]).

Original: However, they still suffer fromthe training data bias duringlight estimation.
Revised: However, they still suffer from training data bias during lightestimation.

Original: Compared to previous works, our method neither makes a strict assumption onreﬂectance nor needs special light source distribution. Instead, we solve the uncalibrated photometricstereo problem in an unsupervised manner for general reﬂectance surfaces with unknown light sourceinformation.
Revised: Besides, all these methods make a strict assumption that the light intensity distributed in apre-deﬁned range ( i.e. , [0.2, 2]) and suffers from accumulating error due to their two-step frameworks. In contrast, our method neither makes a strict assumption on reﬂectance nor needs special light sourcedistribution and jointly solves light conditions and surface normal in an unsupervised manner.

Original:  Method
Revised:  Method

Original: 3.1 Problem Formulation
Revised: 3.1 Problem Formulation

Original: Given a set of observations I , ( I 0 , I 1 , ..., I f ) of a static surface, illuminated by f unknowndirectional illuminations distributing on the upper-hemisphere, uncalibrated photometric stereo aimsat recovering light directions L , ( l 0 , l 1 , ..., l f ) , light intensities E , ( e 0 , e 1 , ..., e f ) , and surfacenormal N , { n i | i 2 P } , P is the set of all positions on the surface.
Revised: Given a set of observations I , ( I 0 , I 1 , ..., I f ) of a static surface, illuminated by f unknowndirectional illuminations distributing on the upper-hemisphere, uncalibrated photometric stereo aimsat recovering light directions L , ( l 0 , l 1 , ..., l f ) , light intensities E , ( e 0 , e 1 , ..., e f ) , and surfacenormal N , { n i | i 2 P } , P is the set of all positions on the surface.

Original: The solution is achieved by 137 solving the optimization problem,
Revised: The solution is achieved by 133 solving the optimization problem,

Original: [Equation]
Revised: [Equation]

Original:  where ¯ m ij 

Revised: (1) where ¯ m ij

Original: I j is the observed pixel intensity at position i , # P is the number of elements in P 1 , m ijis the corresponding rendered pixel intensity, D ( · , · ) is a metric describing their difference.
Revised: I j is the observed pixel intensity at position i , # P is the number of elements in P 1 , m ijis the corresponding rendered pixel intensity, D ( · , · ) is a metric describing their difference.

Original: Under anorthographic camera with linear radiometric response, m ij is formulated as (simpliﬁed in a per-pixelform),
Revised: Under anorthographic camera with linear radiometric response, m ij is formulated as (simpliﬁed in a per-pixelform),

Original: [Equation]
Revised: [Equation]

Original: where v = [0 , 0 , 1] is the view direction pointing toward the viewer, ⇢ ij describes the generalreﬂectance, max( n > i l j , 0) represents the attach shadow.
Revised: where v = [0 , 0 , 1] is the view direction pointing toward the viewer, ⇢ ij describes the generalreﬂectance, max( n > i l j , 0) represents the attach shadow.

Original: Unknown light brings two ambiguities when solving Eq.
Revised: Unknown light brings two ambiguities when solving Eq.

Original: (1), i.e , shape-light ambiguity, which isdenoted as an invertable matrix G 2 R 3 ⇥ 3 , and reﬂectance-light ambiguity, which is denoted as anon-zero scalar c j 2 R ,
Revised: (1), i.e. , shape-light ambiguity, which isdenoted as an invertable matrix G 2 R 3 ⇥ 3 , and reﬂectance-light ambiguity, which is denoted as anon- scalar c j 2 R ,

Original: [Equation]
Revised: [Equation]

Original: 3.2 Neural Intrinsics Fields
Revised: 3.2 Neural Intrinsics Fields

Original: General reﬂectance decomposition.
Revised: General reﬂectance decomposition.

Original: To exploit clues of specular reﬂectance and cast shadow forlight estimation, we decompose the general reﬂectance as the cast shadow term s ij multiplying thebidirectional reﬂectance term,
Revised: To exploit clues of specular reﬂectance and cast shadow forlight estimation, we decompose the general reﬂectance as the cast shadow term s ij multiplying thebidirectional reﬂectance term,

Original: [Equation]
Revised: [Equation]

Original: where subscript ‘ i ’ and ‘ j ’ indicatepixel position- (or normal-) and light-dependent factors, respectively, the cast shadow term s ij is either 0 or 1, ⇢ di , ⇢ sij represent the diffuse and specular reﬂectance,and k di and k  si are coefﬁcients that balance out the effects of specular and diffuse reﬂectance 2 .
Revised: where subscript ‘ i ’ and ‘ j ’ indicate position- (or normal-) and light-dependent factors, respectively;the cast shadow term s ij is either 0 or 1; ⇢ di , ⇢ sij represent the diffuse and specular reﬂectance, and k diand k si are coefﬁcients that balance out the effects of specular and diffuse reﬂectance 2 .

Original: Neural ﬁelds of specular reﬂectance and cast shadow.
Revised: Neural ﬁelds of specular reﬂectance and cast shadow.

Original: Previous methods identify specular featuresfrom speciﬁc pixels and associate an explicit relation to light for light estimation.
Revised: Previous methods identify specular featuresfrom speciﬁc pixels and associate an explicit relation to light for light estimation.

Original: However, thisscheme could fail for the surface where the specular features are invisible or the explicit relationis violated.
Revised: However, thisscheme fails for the surface where the specular features are invisible or the explicit relation is violated.

Original: Besides, leaving out the clue of cast shadow could obstruct producing competitiveperformance.
Revised: Besides, leaving out the clue of cast shadow can obstruct producing competitive performance.

Original: In contrast, we leverage both clues of specular and shadow for light estimation, which is achieved bybuilding the neural specular ﬁeld and neural shadow ﬁeld and associating the ﬁelds to light conditions. 
Revised: In contrast, we leverage both clues of specular and shadow for light estimation, which is achieved bybuilding the neural specular ﬁeld and neural shadow ﬁeld, and associating the ﬁelds to light conditions. [Figure] PositionNet PE

Original: These neural ﬁelds make the utmost of all observed pixels, which fully exploits the mutual informationacross different normal-light (specular) and depth-light (shadow).
Revised: LightNet " # SpecularNet PE [Figure]

Revised: Positional Encode $ ! = & !

Revised: + ( ||& !

Revised: + (|| MLPs CNN Encoder * "!

Revised: = + ! , "! - "# . "

Revised: # + - "$ . "!

Revised: $ max 2 "% & !

Original: Two MLPs, namely SpecularNetand ShadowNet , implement these neural ﬁelds, respectively.
Revised: ShadowNet PE - "$ - "# . " $ 2 "% $ ! ( % $ ! & !

Revised: + !

Revised: [Figure]

Revised: These neural ﬁelds make the utmost of all observed pixels, and exploit the mutual information acrossdifferent normal-light (specular) and position-light (shadow) for light estimation.

Revised: Two MLPs, namely

Original: Instead of estimating light conditions think from SpecularNet or ShadowNet, we take the light direction as the input of these networks to achievetheir implicit association to light conditions.
Revised: SpecularNet and ShadowNet , implement these neural ﬁelds, respectively. we take the estimatedlight direction as the input of these networks to achieve their implicit association to light conditions.

Original: Since the specular reﬂectance and cast shadow arenormal- andpixel position-dependent, we also feed the estimated surface normal and the positionalcode to them, respectively.
Revised: Since the specular reﬂectance and cast shadow are normal- and position-dependent, we also feed theestimated surface normal and the positional code to them, respectively.

Original: SpecularNet and ShadowNet output ⇢ sij and s ij , respectively, as shown in Fig.
Revised: SpecularNet and ShadowNetoutput ⇢ sij and s ij , respectively, as shown in Fig.

Original: Neural light ﬁeld.
Revised: Neural light ﬁeld.

Original: There is mutual information across different observed images, i.e , observedimages with similar appearances are expected to be illuminated by similar lights.
Revised: There is mutual information across different observed images, i.e. , observedimages with similar appearances are expected to be illuminated by similar lights.

Original: To fully considerand exploit this mutual information, we build the neural light ﬁeld by concatenating a CNN encoderto an MLP, namely LightNet .
Revised: To fully considerand exploit this mutual information, we build the neural light ﬁeld by concatenating a CNN encoderto an MLP, namely LightNet .

Original: The encoder extracts a light code from each observed image 3 , andthe LightNet outputs the corresponding light conditions ( i.e , e j , l j ) from the light code, as shown in Fig.
Revised: The encoder extracts a light code from each observed image 3 , and the LightNet infers the corresponding light conditions ( i.e. , e j , l j ) from the light code, as shown in Fig.

Original: Neural position ﬁeld.
Revised: Neural position ﬁeld.

Original: There is mutual information across different positions on a surface, i.e , theconsistency of shape and diffuse reﬂectance in the spatial domain.
Revised: There is mutual information across different positions on a surface, i.e. , theconsistency of shape and diffuse reﬂectance in the spatial domain.

Original: To fully consider and exploit thismutual information, we establish a neural position ﬁeld, namely PositionNet , implemented by an MLP.
Revised: To fully consider and exploit thismutual information, we establish a neural position ﬁeld, namely PositionNet , implemented by an MLP.

Original: The neural position ﬁeld output position-dependent, light-independent factors 4 , i.e , n i , ⇢ di , k di , k si .
Revised: The neural position ﬁeld outputs position-dependent, light-independent factors 4 , i.e. , n i , ⇢ di , k di , k si .

Original: The PositionNet takes the positional code as the input, as shown in Fig. 2.
Revised: The PositionNet takes the positional code as the input, as shown in Fig. 2.

Original: 3.3 Optimizing Neural Intrinsics Fields
Revised: 3.3 Optimizing Neural Intrinsics Fields

Original: We adopt the reconstruction loss function with the ` 1 metric to optimze our NeIF,
Revised: We adopt the reconstruction loss function with the ` 1 metric to optimize our NeIF,

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Silhouette constraint.
Revised: Silhouette constraint.

Original: L rec cannot resolve the shape-light ambiguity in an unsupervised manner dueto the inherently severe ill-posedness.
Revised: L rec cannot resolve the shape-light ambiguity in an unsupervised manner dueto the inherently severe ill-posedness.

Original: Therefore, we introduce the silhouette constraint (similar tothose in [ 8758025 , 13]) to stabilize the training of PositionNet.
Revised: Therefore, we introduce the silhouette constraint (similar tothose in [23, 15]) to stabilize the training of PositionNet.

Original: To be speciﬁc, we use polynomialﬁtting with a moving window block to traverse and pre-compute the contour’s normal of the givenobjects, represented as ˆ N si , { ˆ n si k 2 R 2 ⇥ 1 , k 2 S } , S is the point set of the contour.
Revised: To be speciﬁc, we use polynomial ﬁttingwith a moving window block to traverse and pre-compute the contour’s normal of the given objects,represented as ˆ N si , { ˆ n si k 2 R 2 ⇥ 1 , k 2 S } , S is the point set of the contour.

Original: We considerˆ N si could guide the prediction of the azimuth of boundary normal (at the same positions) and introduce the silhouette loss function,
Revised: We consider ˆ N si can guide the prediction of the azimuth of boundary normal (at the same positions) and introduce thesilhouette loss function,

Original: [Equation]
Revised: [Equation]

Original: where n k 2 R 1 ⇥ 3 represents the estimated surface normal  from PositionNet, C ( · ) cuts off the 3rddimension of n k ( i.e , C ( n k ) 2 R 1 ⇥ 2 ), and Nor ( · ) is the vector normalization operation.
Revised: where n k 2 R 1 ⇥ 3 represents the estimated surface normal at the positions of silhouette from PositionNet, C ( · ) cuts off the 3rd dimension of n k ( i.e. , C ( n k ) 2 R 1 ⇥ 2 ), and Nor ( · ) is the vectornormalization operation.

Original: Warm-up loss functions.
Revised: Warm-up loss functions.

Original: To avoid local minimum and achieve faster convergence, we warm upthe NeIF in early-stage during training with three additional loss functions.
Revised: To avoid local minimum and achieve faster convergence, we warm upthe NeIF in early-stage during training with three additional loss functions.

Original: We use the azimuth oflighting direction estimated by YS97 [54] to guide the training of LightNet,
Revised: We use the azimuth oflighting direction estimated by YS97 [57] to guide the training of LightNet,

Original: [Equation]
Revised: [Equation]

Original: where l az j are estimated light directions by [54].
Revised: where l az j are estimated light directions by [57].

Original: We adopt the gradient penalty [20] L gp to stabilizethe training of SpecularNet,
Revised: We adopt the gradient penalty [22] L gp to stabilizethe training of SpecularNet,

Original: [Equation]
Revised: [Equation]

Original: The intuition is from Blinn-Phong model, where the specular reﬂectance is monotonically increasingw.r.t the n > i h j , i.e , r n > i
Revised: The intuition is from Blinn-Phong model [8], where the specular reﬂectance is monotonicallyincreasing w.r.t the n > i h j , i.e. , r n > i

Original: h j ⇢ sij > 0 .
Revised: h j ⇢ sij > 0 .

Original: We also supervise the training of ShadowNet using pseudoshadow maps ˆ S j , j = 1 , 2 , ..., f ,
Revised: We also supervise the training of ShadowNet usingpseudo shadow maps ˆ S j , j = 1 , 2 , ..., f ,

Original: [Equation]
Revised: [Equation]

Original: The pseudo shadow maps are obtained by binarizing the observed images, i.e , considering an observedpixel to be cast shadow if its intensity value is smaller than 0 . 2 ⇥ the mean intensity values of thisimage.
Revised: The pseudo shadow maps are obtained by binarizing the observed images, i.e. , considering anobserved pixel to be cast shadow if its intensity value is smaller than 0.2 ⇥ the mean intensity value ofthis image.

Original: After early-stage training, we discard these loss functions for a broad range of reﬂectance.
Revised: After early-stage training, we discard these loss functions for a broad range of reﬂectance.

Original: NeIF training.
Revised: NeIF training.

Original: We train NeIF with the warm-up loss function in ﬁrst 10 epochs,
Revised: We train NeIF with the warm-up loss function in ﬁrst 10 epochs,

Original: [Equation]
Revised: [Equation]

Original: 10) where  si = 5 ,  az = 0 .
Revised: where  si = 5 ,  az = 0 .

Original: 1 ,  gp =  ,  shadow = 
Revised: 1 ,  gp = 10 ,  shadow = 10 .

Original: We then train NeIF untill 500 epochs or convergingwith the loss function,
Revised: We then train NeIF until 500 epochs or convergingwith the loss function,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where L recShadow is the another shadow map supervison loss to train the ShadowNet.
Revised: where L recShadow is the another shadow map supervision loss to train the ShadowNet.

Original: The loss functionis the same to L shadow .
Revised: The loss functionis the same to L shadow .

Original: The only difference is the calculation of ˆ S j .
Revised: The only difference is the calculation of ˆ S j .

Original: For L recShadow , we calculate ˆ S jby rendering a depth map with the estimated l j .
Revised: For L recShadow , we calculate ˆ S jby rendering a depth map with the estimated l j .

Original: The depth map is reconstructed from the estimatedsurface normal map N by method [8] 5 .
Revised: The depth map is reconstructed from the estimatedsurface normal map N by method [10].

Original: L recShadow is used to align outputs of ShadowNet to those of
Revised: L recShadow is used to align outputs of ShadowNet to those of

Original: PositionNet.
Revised: PositionNet.

Original: Implementation details.
Revised: Implementation details.

Original: We generate the positional code from the coordinate (in the image plane)by the same method in [31].
Revised: We generate the positional code from the coordinate (in the image plane)by the same method in [34].

Original: With the assumption of isotropic reﬂectance, we simplify the inputof SpecularNet from { n >i l j , n >  iv , l >  jv } to { n >i h j , h >  jv } 6 [28] for easiermodel training.
Revised: With the assumption of isotropic reﬂectance, we simplify the input of SpecularNet from { v > l j , v > n i , n > i l j } to { v > h j , n > i h j } 5 [31] for easier training.

Original: Similarto the Cook-Torrance reﬂectance model [17], we assume k di + k si = 1 to reduce the numbers ofunknowns.
Revised: Similar to the Cook-Torrance reﬂectance model [19], we assume k di + k si = 1 to reduce the number of unknowns.

Original: The CNN decoder with declining channels processes the down-sampled images with adimension of 256 ⇥ 256 .
Revised: The CNN decoder with declining channels processes the down-sampled images with a dimension of256 ⇥ 256 .

Original: The LightNet takes ﬂatten features to predict l j , e j in two different branches.
Revised: The LightNet takes ﬂatten features to predict l j , e j in two different branches.

Original: For the ShadowNet, we also generate the positional code for l j and concatenate it to the feature in9-th layer.
Revised: For the ShadowNet, we also generate the positional code for l j and concatenate it to the feature in the 9thlayer.

Original: The output of ShadowNet is either 0 or 1, which is realized by a step function with asimilar implementation in [35] 7 .
Revised: The output of ShadowNet is either 0 or 1, which is realized by a step function with a similarimplementation in [38]. benchmark [45].

Revised: This table summarizes comparison methods. ’

Revised: N.A.’ represents not applicable as calibrated PS is with known ` and e .

Revised: ‘Semi’ indicates certain method leverage partial information of light.

Original:   Experiments
Revised: ‘ 3 ’ (or ‘ 7 ’) represents that certain methods (do not) adopt supervised learning for the estimation of surface normal n , light direction l , or light intensity e . ‘Identical’ means certain methods require the light intensity of different illuminating to be identical. [Table]  Experiments

Original: Training details.
Revised: Training details.

Original: Our main framework is implemented in PyTorch, while the pre-calculations(method [54] and silhouette ﬁtting ) are implemented in MATLAB.
Revised: Our main framework is implemented in PyTorch, while the pre-calculations(method [57] and silhouette ﬁtting) are implemented in MATLAB.

Original: We use Adam as the optimizer k . Please refer to our supplementary material for more details [Table] with a learning rate ↵ = 5 ⇥ 10  4 to train our framework in 500 epochs for each scene separately, andthe warm-up stage takes up ten epochs.
Revised: We use Adam as the optimizerwith a learning rate ↵ = 5 ⇥ 10  4 to train our framework in 500 epochs for each scene separately, andthe warm-up stage takes up 10 epochs.

Original: The last 100 epochs use a lower learning rate ↵ = 5 ⇥ 10  5for ﬁne-tunning.
Revised: The last 100 epochs use a lower learning rate ↵ = 5 ⇥ 10  5for ﬁne-tunning.

Original: The batch size is 32 for lighting and 256 for spatially random sampling.
Revised: The batch size is 32 for lighting and 256 for spatially random sampling.

Original: After everyepoch, the depth map is reconstructed according to the predicted normal map by method [8].
Revised: After everyepoch, the depth map is reconstructed according to the predicted normal map by method [10].

Original: Eachscene takes from 2 hours to 6 hours on one RTX 2080Ti 12GB GPU, depending on the resolution ofthe objects.
Revised: Eachscene takes from 2 hours to 6 hours on one RTX 2080Ti 12GB GPU, depending on the resolution ofthe objects.

Original: Evaluation metrics.
Revised: Evaluation metrics.

Original: We adopt the same metric in [12], the scale-invariant relative error, to measurethe accuracy of recovered light intensity as
Revised: We adopt the same metric in [13], the scale-invariant relative error, to measurethe accuracy of recovered light intensity as,

Original: [Equation]
Revised: [Equation]

Original: where, ⌘ is calculated by solving arg min ⌘
Revised: where, ⌘ is calculated by solving arg min ⌘

Original: P 
Revised: P fj =

Original: [Equation]
Revised: [Equation]

Original:  by least squares minimization.
Revised: ) 2 by least squares minimization.

Original: Themetric to measure the accuracy of the predicted light directions and surface normal is the widely usedmean angle error (MAE) in degree.
Revised: Themetric to measure the accuracy of the predicted light directions and surface normal is the widely usedmean angle error (MAE) in degree.

Original: 4.1 Evaluation on Public Datasets
Revised: 4.1 Evaluation on Public Datasets

Original: Since the proposed NeIF is an unsupervised uncalibrated photometric stereo method, we compare itsperformance with state-of-the-art uncalibrated and unsupervised photometric stereo methods.
Revised: Since the proposed NeIF is an unsupervised uncalibrated photometric stereo method, we compare itsperformance with state-of-the-art uncalibrated and unsupervised photometric stereo methods.

Original: Threereal-world datasets, including thebenchmark dataset D I L
Revised: Threereal-world datasets, including the D I L

Original: I G EN T [39], A PPLE & G OURD dataset [2],andthe L IGHT S TAGE D ATA G ALLERY [9], are used for evaluation.
Revised: I G EN T benchmark dataset [45], A PPLE & G OURD dataset [2],and L IGHT S TAGE D ATA G ALLERY dataset [11], are used for evaluation.

Original: Quantitative comparison for normal map.
Revised: Quantitative comparison for normal map.

Original: Table 1 lists relevant works for a comprehensive surfacenormal estimation comparison.
Revised: Table 1 lists relevant works for a comprehensive surfacenormal estimation comparison.

Original: As summarized in Table 1, our method is the only method thataddresses UPS without the supervision of N , L , or E .
Revised: As summarized in Table 1, our method is the only method thataddresses UPS without the supervision of N , L , or E .

Original: However, our method achieves the bestperformance and maintains a considerable advantage over other competitors[12, 13] ( i.e , 7.86 for
Revised: However, our method achieves the bestperformance and maintains a considerable advantage over other competitors [13, 15] ( e.g. , 7.86 for

Original: NeIF v.s. 8.71 for [13]).
Revised: NeIF vs. 8.71 for [15]).

Original: Objects in D I L I G EN T differ from shapes and reﬂectance. The numbers ofour method are either best or competitive (less than 1  as compared with the best performing UPSmethod), which shows its good generalization capacity to general reﬂectance and various shapes.
Revised: Handling the objects in D I L I G EN T dataset [45] with different shapes andreﬂectance, numbers from our method are either best or competitive (about 1  as compared with thebest performing UPS method), which shows its good generalization capacity to general reﬂectanceand various shapes.

Original: Because our NeIF fully considers mutual information by building up intrinsics ﬁelds and such implicitmodeling facilitates modeling  general reﬂectance.
Revised: This is because our NeIF fully considers mutual information by building upintrinsics ﬁelds and the implicit modeling facilitates general reﬂectance modeling.

Original: Visual quality comparison for  normal map.
Revised: Visual quality comparison for the normal map.

Original: Fig.
Revised: Fig.

Original: 3 illustrates the visual quality comparison in termsof recovered surface normal maps and corresponding error maps on D I L I G EN T.
Revised: 3 illustrates the visual quality comparison interms of recovered surface normal maps and corresponding error maps on D I L I G EN T dataset [45].

Original: The comparisonis conducted with two state-of-the-art UPS methods[12, 13] and a state-of-the-art unsupervised PSmethod [44].
Revised: [Figure] [Figure] GO URD GO URD

Original: Our method is less sensitive to spatially-varying albedo due to the per-pixel manner(red boxes in the left subﬁgure of Fig. 3).
Revised: S T A ND I N [Figure] [Figure]

Original: However, although the positional code considers the
Revised: M

Original: [Figure]
Revised: [Figure]

Revised: The comparison is conducted with two state-of-the-art UPS methods [13, 15] and a state-of-the-artunsupervised PS method [47].

Revised: Our method is less sensitive to spatially-varying albedo due to theper-pixel manner (red boxes in the left subﬁgure of Fig. 3).

Original:  global shape effect, this per-pixel manner is less effective in modeling complex shape information ascompared with the all-pixel one.
Revised: However, although the positional codeconsiders the global shape effect, this per-pixel manner is less effective in modeling complex shapeinformation as compared with the all-pixel one.

Original: It could fail for regions with cast shadow (or overexposure) undermost light directions (red boxes in the right subﬁgure of Fig. 3).
Revised: It fails for regions with cast shadow (or overexposure)under most light directions (red boxes in the right subﬁgure of Fig. 3).

Original: We also show the visual qualityresults for A PPLE & G OURD dataset [2] and the L IGHT S TAGE D ATA G ALLERY [9] in Fig. 
Revised: We also show the visualquality results for A PPLE & G OURD dataset [2] and the L IGHT S TAGE D ATA G ALLERY dataset [11]in Fig. 4.

Original: Ourmethod could produce reliable estimation for most regions, thanks to the full exploitation of mutualinformation across different dimensions.
Revised: Our method produces reliable estimation for most regions, thanks to the full exploitation ofmutual information across different dimensions.

Original: Quantitative comparison for light directions and intensities.
Revised: Quantitative comparison for light directions and intensities.

Original: As can be observed in Table 2 and Table 3, our method achieves a superior performance advantage over unsupervised methods[34, 4,42, 50, 29, 28] while maintaining competitive performance as compared with supervised methods.
Revised: As can be observed in Table 2and Table 3, our method achieves a superior performance advantage over unsupervised methods(YS97 [57] and PF14 [37]) while maintaining competitive performance as compared with supervisedmethods (CH19 [13] and CW20 [15]).

Original: Although the supervised methods pre-deﬁne the intensity range, they produce less accurate estimationas compared with our approach due to the possible overﬁtting of supervised learning.
Revised: These supervised methods adopt two-step solutions andsuffer from accumulating error. Therefore, even though they achieve similar performance in terms oflight conditions accuracy, they are less effective on estimating surface normal as compared with ourmethod (see Table 1).

Original: In contrast,our LightNet builds the neural light ﬁeld, which could fully consider the mutual information amongdifferent observed images, resulting in the accurate estimation of the scale-invariant light intensity.
Revised: [Table] [Table]

Original: 4.2 Evaluation on Sparse Uncalibrated Photometric Stereo
Revised: 4.2 Evaluation on Sparse Uncalibrated Photometric Stereo

Original: As compared with other per-pixel PS methods ( e.g. , [59, 27]), the proposed NeIF exploits much moreconstraints to estimate the normal at a point, i.e , # P ⇥ f v.s. f .
Revised: As compared with other per-pixel PS methods ( e.g. , ZJ19 [62] and LL19 [30]) [Table] the proposed NeIFexploits much more constraints to estimate the normal at a point ( i [Table]

Revised: e. , # P ⇥ f vs. f ).

Original: Because it learns ﬁelds instead ofregressing intensity proﬁle [40] to surface normal.
Revised: Because itlearns ﬁelds instead of regressing intensity proﬁle [43] to surface normal.

Original: Therefore, we investigate the effectiveness of our[Table] [Table] method under the challenging setting of sparse UPS.
Revised: Therefore, we investigatethe effectiveness of our method under the challenging setting of sparse UPS.

Original: We randomly select  16 images illuminatedby different lights from D I L I G ENT and test our method using these images.
Revised: We randomly select 10 or 16 images illuminated by different lights from D I L I G EN T dataset [45] andtest our method using these images.

Original: We repeat this process30 times, similar to [59].
Revised: We repeat this process 30 times, similar to [62].

Original: We compare the performance with that from the classical PS method PF14 [34], as deep learning-based approaches ( e.g. , [12, 13]) perform poorly without re-training.
Revised: We compare theperformance with that from the classical UPS method PF14 [37], a deep learning based per-pixelapproach ZJ19 [62], and an all-pixel approach CH18 [14].

Original: Due to the fewer reconstruction terms to train our NeIF [Table] we slightly increase the dimension of thepositional encoding module from 4 to 6 by increasing the frequency [45] to stabilize the training, i.e ,strengthening the role of positional code to reduce the variance of estimated intrinsics.
Revised: Due to the fewer reconstruction terms totrain our NeIF, we slightly increase the dimension of the positional encoding module from 4 to 6 thatincreases the frequency [48] to stabilize the training, i.e. , strengthening the role of positional code toreduce the variance of estimated intrinsics.

Original: Besides, wedrop L az during early stage warm-up because YS97 [54] fails for objects with centralized specularcomponents ( e.g. , B ALL ).
Revised: Besides, we drop L az during early stage warm-up because YS97 [57] fails for the sparse inputs.

Original: We report the mean results over 30 trails in Table 1.
Revised: We report the mean results over 30 trails in Table 4.

Original: As can be observed, our method achievespromising performance onlight and normal estimation for sparse UPS.
Revised: As can be observed, our method achievescompetitive performance on normal estimation as compared with state-of-the-art sparse PS methodswith known light ( i.e. , 11.62 for NeIF vs. 9.82 for ZJ19 [62] under 10 lights, and 10.38 for NeIF vs.9.00 for CH18 [14] under 16 lights). Kindly note that we do not require any ground truth normal forsupervision. As compared with the unsupervised UPS method (PF14 [37]), we achieve a superiorperformance advantage. The results validate the effectiveness of building up neural intrinsics ﬁeldsthat fully exploit the mutual information across different dimensions.

Original:  Conclusion
Revised:  Conclusion

Original: This paper proposes NeIF for UPS.
Revised: This paper proposes NeIF for UPS.

Original: By representing the general reﬂectance as four neural intrinsicsﬁelds, it implicitly associates the light clues of specualr reﬂectance and cast shadow to light estimation,which faciliate solving UPS with general reﬂectance.
Revised: By representing the general reﬂectance as four neural intrinsicsﬁelds, it implicitly imposes the light clues of specular reﬂectance and cast shadow for light estimation,which facilitates solving UPS with general reﬂectance.

Original: The proposed NeIF could fully exploit mutualinformace from all observed pixel values so that it could produce stable estimationeven for  sparse UPS.
Revised: The proposed NeIF can fully exploit mutualinformation from all observed pixel values so that it produces stable estimation for PS and sparse PS.

Original: The unsupervised training manner of NeIF is beneﬁcial to the generalization capacity to datafrom different sources.
Revised: The unsupervised training manner of NeIF is beneﬁcial to the generalization capacity of data fromdifferent sources.

Original: Limitations.
Revised: Limitations.

Original: Although our method produces promising results for light condition and surface normalestimation, it has several limitations.
Revised: Although our method produces promising results for light conditions and surfacenormal estimation, it has several limitations.

Original: First, the estimated intrinsics such as shadow, specular anddiffuse reﬂectance, and their balancing coefﬁcients are less accurate, as shown in Fig.
Revised: First, the estimated intrinsics, such as shadow, specularand diffuse reﬂectance, and their balancing coefﬁcients are less accurate for objects with complicatedgeometries, as shown in Fig.

Original: Second,we need to balance the role of positional code and estimated intrinsics when applying our methodfor sparse UPS.
Revised: Second, we need to balance the role of positional code and estimatedintrinsics when applying our method for sparse UPS.

Original: However, how to ﬁnd an optimal balancing strategy is unknown.
Revised: However, how to ﬁnd an optimal balancingstrategy is unknown.

Original: Third, as shownin Fig.3, our method is less effective on global effects, especially for regions with cast shadow (oroverexposure) under most light directions due to the per-pixel manner. Besides, the inference of ourmethod is inefﬁcient due to the unsupervised training manner and the per-pixel estimation, which isinapplicable for real-time applications. Checklist The checklist follows the references.
Revised: Third, as shown in Fig.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .

Original: You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.

Original: For example:

Original: • Did you include the license to the code and datasets?

Original: [Yes] See Section ?? .•

Original: Did you include the license to the code and datasets?

Original: [No] The code and the data areproprietary.

Original: • Did you include the license to the code and datasets?

Original: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the

Original: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s

Original: contributions and scope?

Original: [Yes] See Line 9-13,

Original: (b) Did you describe the limitations of your work? [Yes] See Line 287- (c) Did you discuss any potential negative societal impacts of your work? [No]
Revised: 3, our method is less effective on global effects,especially for regions with cast shadow (or overexposure) under most light directions due to our

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them? [Yes]
Revised: [Table]

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] See

Original: Supplementary Material

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
Revised: per-pixel manner.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Original: ments multiple times)?

Original: [Yes]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes]

Original: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?

Original: [No]

Original: (b) Did you mention the license of the assets?

Original: [No]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?

Original: [No]

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: information or offensive content?

Original: [No]

Original: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?

Original: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?

Original: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] NeIF: Representing General Reﬂectance as Neural Intrinsics Fields for Uncalibrated Photometric Stereo
Revised: Besides, the inference of our method is inefﬁcient due to the unsupervised trainingmanner and the per-pixel estimation, which is inapplicable for real-time applications.

Original: Abstract

Original: Uncalibrated photometric stereo (UPS) is challenging due to the inherent ambiguity brought by unknown light.

Original: Existing solutions alleviate the ambiguity by either explicitly associating reﬂectance to light conditions or resolving light conditions in a supervised manner.

Original: This paper establishes an implicit relation between light clues and light estimation and solves UPS in an unsupervised manner.

Original: The key idea is to represent the reﬂectance as four neural intrinsics ﬁelds, i.e. , position, light, specular, and shadow, based on which the neural light ﬁeld is implicitly associated with light clues of specular reﬂectance and cast shadow.

Original: The unsupervised optimization of neural intrinsics ﬁelds can be free from training data bias and fully exploits all observed pixel values for UPS.

Original: Our method achieves a superior performance advantage over state-of-the-art UPS methods on public datasets and promising results under the challenging setting of sparse UPS.

Original: The code will be released soon.

Original:  Introduction
Revised:  Introduction

Original: Photometric stereo (PS) [50] aims at recovering the surface normal from several light-varying images captured at a ﬁxed viewpoint.
Revised: Photometric stereo (PS) [47] aims at recovering the surface normal from several light-varying imagescaptured at a ﬁxed viewpoint.

Original: As compared with other  approaches ( e g. , multi-view stereo [41], active sensor-based solutions [58]), photometric stereo is excellent at recovering ﬁne-detailed surfaces and has been widely used for Hollywood movies [11], industrial quality inspection [49] and biometrics [55].
Revised: As compared with other stereo approaches ( e g. , multi-view stereo [38],active sensor-based solutions [55]), photometric stereo is excellent at recovering ﬁne-detailed surfacesand has been widely used for Hollywood movies [9], industrial quality inspection [46] and biometrics[52].

Original: Calibrating accurate lighting directions is crucial to the performance of photometric stereo methods [54].
Revised: Calibrating accurate lighting directions is crucial to the performance of photometric stereomethods [51].

Original: However, lighting calibration is often tedious, dramatically restricting the applicability in the real-world.
Revised: However, lighting calibration is often tedious, dramatically limiting the applicability inthe real-world.

Original: To this end, researchers develop uncalibrated photometric stereo (UPS) methods that estimate surface normal with unknown lights.
Revised: To this end, researchers develop uncalibrated photometric stereo (UPS) methods thatestimate surface normal with unknown lights.

Original: Uncalibrated photometric stereo suffers from General Bas-Relief (GBR) ambiguity [6] for an integrable surface.
Revised: Uncalibrated photometric stereo suffers from General Bas-Relief (GBR) ambiguity [6] for an integrable surface.

Original: Early solutions address the ambiguity by explicitly associating reﬂectance to light, i.e. , adopting analytic reﬂectance models ( e.g. , Lambertian reﬂectance [4], [37], parametric specular reﬂection [20], specular spikes [56], inter-reﬂection [12]) or imposing priors from reﬂectance properties [3, 25, 24, 44].
Revised: Early solutions address the ambiguity by explicitly associating reﬂectance to light, i.e ,adopting analytic reﬂectance models ( e.g. , Lambertian reﬂectance [4], [34], parametric specular reﬂection [18], specular spikes [53], inter-reﬂection [10]) or imposing priors from reﬂectance properties[3, 22, 21, 41].

Original: Thus,due to the strong reliance on reﬂectance assumption, these methods can be less effective for unknown reﬂectance.
Revised: Thus, these methods could be less effective for unknown reﬂectance due to the strongreliance reﬂectance assumption.

Original: Besides, these methods ignore the clue of cast shadow and even fail in shadow regions due to the shadow-free surface assumption.
Revised: Besides, these methods ignore the clue of cast shadow and even failfor shadow regions due to the shadow-free surface assumption.

Original: Further, most of them require the light intensity to be identical for robust estimation.
Revised: Further, most of them require thelight intensity to be identical for robust estimation.

Original: Recently, deep learning-based approaches address the ambiguity by estimating light direction and intensity before recovering surface normal [13, 15, 27, 40].
Revised: Recently, deep learning-based approaches addressthe ambiguity by estimating light direction and intensity before recovering surface normal [12, 13, 25,37].

Original: They train a light estimation network using a large-scale amount of rendered data in a supervised manner.
Revised: They train a light estimation network using a large-scale amount of rendered data in a supervisedmanner.

Original: However, the training data bias [33] can hardly be eliminated and can produce unexpected estimation for real-world data.
Revised: However, the training data bias [30] can hardly be eliminated and could produce unexpectedestimation for real-world data.

Original: Because rendered training data inevitably contains the domain gap from the real ones and  scarcely cover all surfaces with different geometry and reﬂectance in the real-world.
Revised: Because rendered training data inevitably contain the domain gapfrom real ones and could scarcely cover all surfaces with different geometry and reﬂectance in thereal-world.

Original: Besides, such a two-step solution can bring accumulating errors for surface normal
Revised: Besides, such a two-step solution could bring accumulating errors for surface normal

Original: Neural Shadow Field
Revised: Neural Shadow Field

Original: P o s i t i on
Revised: P o s i t i on

Original: Light direction ( ) + × × = × × × · · · ! ! " !" # Neural Position Field
Revised: Light direction ( ) + × × = × × × · · · 𝒏 𝑖 𝑘 𝑖 𝑑 𝜌 𝑖 𝑑 𝑘 𝑖 𝑠 Neural Position Field

Original: P o s i t i on
Revised: P o s i t i on

Original: Neural Specular Field
Revised: Neural Specular Field

Original: S
Revised: S

Original: u r f a c e no r m a l
Revised: u r f a c e N o r m a l

Original: Light direction
Revised: Light direction

Original: Neural Light Field
Revised: Neural Light Field

Original: O b s e r v ed i m age s
Revised: O b s e r v ed I m age s

Original: Light direction · · · ! !" ( !$ ! ! #$ ) ! Position-Dependent
Revised: Light direction · · · 𝜌 𝑖𝑑 (𝒍 𝑗 ,𝑒 𝑗 ) 𝑠 𝑖𝑗 𝑘 𝑖𝑑 𝑘 𝑖𝑠 𝜌 𝑖𝑗𝑠 𝒏 𝑖 𝑚 𝑖𝑗 · · · · · · · · · · · · · · · · · · position-dependent light- dependent 𝜌 𝑖𝑑 ✓  𝑘 𝑖 𝑠 ✓  𝑘 𝑖𝑑 ✓  𝒏 𝑖 ✓  𝒍 𝒋  ✓ 𝑒 𝑗  ✓ 𝜌 𝑖𝑗𝑠 ✓ ✓ 𝑠 𝑖𝑗 ✓ ✓

Original: Light- Dependent ! !"

Original: ü û " !

Original: # ü û " ! "

Original: ü û # !

Original: ü û $ $ û ü % % û ü ! !

Original: # ü ü & ! ü ü To this end, we propose NeIF , representing general reﬂectance as Ne ural I ntrinsics F ields for uncalibrated photometric stereo.
Revised: To this end, we propose NeIF , representing general reﬂectance as Ne ural I ntrinsics F ields foruncalibrated photometric stereo.

Original: Our method differs from previous methods in three aspects: 1) it fully considers clues of specular reﬂectance and cast shadow from each observed pixel for light estimation so that it is expected to produce accurate estimation for both light conditions and surface normal; 2) it does not make explicit assumptions about the reﬂectance or light so that it works with general surface reﬂectance and ﬂexible light settings;3) it inferslight and surface normal in an unsupervised manner so that it is free from training data bias and achieves stable performance for data from different sources.
Revised: Our method differs from previous methods in three aspects: 1)it fully considers clues of specular reﬂectance and cast shadow from each observed pixel for lightestimation so that it is expected to produce accurate estimation for both light conditions and surfacenormal; 2) it does not make explicit assumptions about the reﬂectance or light so that it could workwith general surface reﬂectance and ﬂexible light settings; it infers

Revised: light and surface normal

Original:  Our key idea is to represent the general reﬂectance as four neural intrinsics ﬁelds ( i.e. , position, light,specular, and shadow, see Fig. 1), implemented by four multi-layer perceptrons (MLPs).
Revised: in anunsupervised manner so that it could be free from training data bias and achieve stable performance for data from difference sources Our key idea is to represent the general reﬂectance as four neural intrinsics ﬁelds ( i.e , specularreﬂectance, cast shadow, light, and position, see Fig. 1), implemented by four multi-layer perceptrons(MLPs).

Original: These four ﬁelds are connected based on the implicit relation (or dependence) of these intrinsics so that no explicit assumption is imposed, e.g. , we take the estimated light as the input to recover specular reﬂectance and cast shadow instead of explicitly exploiting them for light estimation.
Revised: The connection of these four ﬁelds is based on the implicit relation (or dependence) of theseintrinsics so that no explicit assumption is imposed, e.g. , we take the estimated light as the input torecover specular reﬂectance and cast shadow instead of explicitly exploiting them for light estimation.

Original: These intrinsics ﬁelds are optimized to reconstruct each pixel value from observed images, which fully exploit mutual information across different dimensions, as shown in Fig.
Revised: These intrinsics ﬁelds are optimized to reconstruct each pixel value from observed images, whichfully exploit mutual information across different dimensions, as shown in Fig.

Original: The reconstruction error is backpropagated to the neural light ﬁeld through neural specular and shadow ﬁelds so that clues of specular and shadow can be implicitly and fully considered for light estimation.
Revised: The reconstructionerror backpropagates to the neural light ﬁeld through neural specular and shadow ﬁelds, so that cluesof specular and shadow can be implicitly and fully considered for light estimation.

Original: Our contributions are summarized as:
Revised: Our contributionsare summarized as

Original: • We represent general reﬂectance as four intrinsic neural ﬁelds to implicitly associate perpixel reﬂectance to light, whichsolves uncalibrated photometric stereo by fully considering clues of specular reﬂectance and cast shadow for light estimation.
Revised: • We represent general reﬂectance as four intrinsic neural ﬁelds to implicitly associate per- pixel reﬂectance to light, which fully considers clues of specular reﬂectance and cast shadow for light estimation.

Original: • We propose the NeIF, an uncalibrated photometric stereo method trained in an unsupervised manner, which works with general surface reﬂectance and ﬂexible light settings, and is free from training data bias.
Revised: • We propose the NeIF, an uncalibrated photometric stereo method trained in an unsupervisedmanner, which works with general surface reﬂectance, ﬂexible light settings, and trainingdata bias.

Original: • We show that our method achieves  superior performance over uncalibrated and unsupervised methods.
Revised: • We show that our method achieves a superior performance over uncalibrated and unsupervised methods.

Original: We also demonstrate its excellent generalization capacity to data from different sources and promising performance with the challenging setup of sparse uncalibrated photometric stereo.
Revised: We also demonstrate its excellent generalization capacity to data from different sources and promising performance with the challenging setup of sparse uncali- brated photometric stereo.

Original:  Related Work
Revised:  Related Work

Original: This section mainly reviews the latest works in neural reﬂectance representation, and relatedworks on unsupervised PS methods and UPS methods.
Revised: In this section, we mainly review the latest works in neural reﬂectance representation, and related

Original: Readers may refer to [45] and [13] for a more comprehensive summary.

Original: 2.1 Neural Reﬂectance Representation
Revised: 2.1 Neural Reﬂectance Representation

Original: Neural Radiance Fields (NeRFs)
Revised: Neural Radiance Fields (NeRFs)

Original: [34] focus on the 3D geometries without explicitly modeling the interaction between light and objects via the image formation model.
Revised: [31] focuses on the 3D geometries without explicitly modeling theinteraction between light and objects via photometric image formation model.

Original: Many subsequent works explore its application in various computer vision problems, such as relighting [16, 46], photometric stereo [29], and multi-view  stereo [7, 9, 60, 28, 61].
Revised: Many subsequentworks explore its application in various computer vision problems, such as relighting [14], photometricstereo [26], and multi-view photometric stereo [24].

Original: These works  require known lighting conditions [7, 46, 61, 29], adopt simple reﬂectance model [59], or leverage multi-view information [7, 9, 60, 28, 61].
Revised: These works either require known lightingconditions [ bi2022neural , 43, 58, 26] or adopt simple reﬂectance model [56]. There are alsosome works [7, 57] considering general reﬂectance, but they require multi-view information anddoes not explicitly consider cast shadow effects. Kaya et al. [24] implemented NeRF in multiview photometric stereo, but their work also leverages multi-view information.

Original: Different from these methods, our method considers general reﬂectance, cast-shadow effects, and unknown light conditions for uncalibrated photometric stereo by taking images captured at a single viewpoint.
Revised: Different from thesemethods, our method considers general reﬂectance, cast-shadow effects, and unknown light conditionsfor uncalibrated photometric stereo by taking images captured at a single viewpoint.

Original: 2.2 Unsupervised  Photometric Stereo
Revised: 2.2 Unsupervised Calibrated Photometric Stereo

Original: Classical methods solve the calibrated photometric stereo problem without knowing the ground truth surface normal.
Revised: Classical methods solved the calibrated photometric stereo problem without knowing the groundtruth surface normal.

Original: Therefore, we classify them as unsupervised methods.
Revised: Therefore, we classify them as unsupervised methods.

Original: The least squarebased algorithm [50] provides the simplest solution, which assumes the object to be Lambertian.
Revised: The least square basedalgorithm [47] provides the simplest solution, which assumes the object to be Lambertian.

Original: It is generally served as a baseline method due to its stability, but its strong assumption on the reﬂectance model makes it fail for non-Lambertian surface.
Revised: Thealgorithm’s performance is robust in a wide range of datasets, but its strong assumption on thereﬂectance model remains much space for improvement.

Original: The following works either regard the non-Lambertian reﬂectance components as the outliers [5, 18, 36, 51, 26, 52] or apply analytic reﬂectance models including Torrance-Sparrow [20], the Ward model [18], a mixture of multiple Ward models [21], [1] etc. to consider the non-Lambertian effects.
Revised: The following works either regard thenon-Lambertian reﬂectance components as the outliers [5, 16, 33, 48, 23, 49] or apply analyticreﬂectance models including Torrance-Sparrow[18], the Ward model[16], a mixture of multiple Wardmodels [19], [1] etc. to consider the non-Lambertian effects.

Original: However, the performance of those methods can only deal with limited types of materials.
Revised: However, the performance of thosemethods can only deal with limited types of materials.

Original: There are also more advanced methods that utilize the general reﬂetance features such as reciprocity, isotropy [2], and monotonicity [24].
Revised: There are also more advanced methods thatutilize the general reﬂetance features such as reciprocity, isotropy [2], and monotonicity [21].

Original: Those methods give a reliable estimation for objects with a broad range of materials.
Revised: Thosemethods give a reliable estimation for objects with a broad range of materials.

Revised: Inspired by them, ourapproach considers the general reﬂectance properties.

Original: With the progress of deep learning, many learning-based frameworks have been proposed for calibrated photometric stereo.
Revised: With the progress of deep learning, many learning-based frameworks have been proposed for calibrated photometric stereo.

Original: Taniai et al.
Revised: Taniai et al.

Original: [47] proposed the ﬁrst unsupervised learning-based photometric stereo method through a rendering equation.
Revised: [44] propose the ﬁrst unsupervised learning-based photometricstereo method through a rendering equation.

Original: However, their reﬂectance model does not separately consider shadow, specular highlights, and diffuse components.
Revised: However, their reﬂectance model does not separatelyconsider shadow, specular highlights, and diffuse components.

Revised: Different from these methods that need the known lights, we address the uncalibrated photometricstereo.

Original: We also train our method in an unsupervised manner.
Revised: We also train our method in an unsupervised manner.

Original: Different from previous methods, we address the challenging problem of UPS and separately model cast shadow, specular reﬂectance, and diffuse reﬂectance.
Revised: Because it is expected to be free fromtraining data bias and thus obtain a stable estimation across data from different sources.

Original: 2.3 Uncalibrated Photometric Stereo
Revised: 2.3 Uncalibrated Photometric Stereo

Original: Previous works hold the  Lambertian assumption and address the ambiguity brought by a 3 ⇥transformation matrix.
Revised: Previous works study the phenomenon where Lambertian objects under arbitrary directions cangenerate identical images when similar transformations are applied to both the object’s normal andthe light’s direction.

Original: Belhumeur et al.
Revised: Belhumeur et al.

Original: [6] reduce the dimension of the transformation to a three parameters GBR transformation by considering integrability constraints.
Revised: [6] reduce the dimension of the transformation to a threeparameters GBR transformation by considering integrability constraints.

Original: Based on that, extra clues from reﬂectance, such as half-vector symmetry [31], albedo clustering [42], specular spikes [56], or assumptions of light source distribution, such as ring light [63], symmetry light [35], or uniformdistributed light sources [57, 4, 42, 53, 37, 31], are used to resolve the GBR ambiguity.
Revised: Based on that, extra clueson reﬂectance, such as half-vector symmetry [28], albedo clustering [42], specular spikes [53], orassumptions on light source distribution, such as ring light [60], symmetry light [32], or uniformdistributed light sources [54, 4, 42, 50, 34, 28], are used to resolve the GBR ambiguity.

Original: However, all these methods require the light intensity to be identical, which is inapplicable in the real-world datasets such as D
Revised: However,all these methods require the light intensity to be identical, which is inapplicable in the real-worlddatasets such as D

Original: I L I G EN T [45], A PPLE & G OURD [2], and L IGHT S TAGE D ATA G ALLERY
Revised: I L I G EN T [39], A PPLE & G OURD [2], and L IGHT S TAGE D ATA G ALLERY

Original: Cho et al.
Revised: Choet al.

Original: [17] put up a semi-calibrated method to deal with non-uniform light intensity, but they assume the light directions to be known.
Revised: [15] put up a semi-calibrated method to deal with non-uniform light intensity, but they assumedthe light directions to be known.

Original: Quéau et al.
Revised: Quéau et al.

Original: [39] address the photometric stereo problem under inaccurate lighting calibration, while the accuracy can signiﬁcantly drop when non-Lambertian components become dominant.
Revised: [36] address the photometric stereo problem underinaccurate lighting calibration, while the accuracy could signiﬁcantly drop when non-Lambertiancomponents become dominant.

Original: Recently, many deep learning methods have been proposed for uncalibrated photometric stereo.
Revised: Recently, many deep learning methods have been proposed for uncalibrated photometric stereo.

Original: Chen et al.
Revised: Chen et al.

Original: [13] propose a supervised uncalibrated framework, SDPS-Net, which can simultaneously estimate the light conditions (intensity and direction) and the surface normal.
Revised: [12] propose a supervised uncalibrated framework, SDPS-Net, which can simultaneouslyestimate the light conditions (intensity and direction) and the surface normal.

Original: They suggest treating light estimation as a classiﬁcation problem and separating the normal and light prediction to reduce the complexity.
Revised: They suggest treatinglight estimation as a classiﬁcation problem and separate the normal and light prediction to reduce thecomplexity.

Original: Their following work, GC-Net [15], improves the performance of SDPS-Net by adding shading as an extra channel to the input of the light estimation network.
Revised: Their following work, GC-Net[13], improved the performance of SDPS-Net by addingshading as an extra channel to the input of the light estimation network.

Original: However, as a common problem for all supervised methods, an over-ﬁtting problem may occur due to the training data bias [33].
Revised: However, as a commonproblem for all supervised methods, an over-ﬁtting problem may occur due to the training data bias[30].

Original: In contrast, unsupervised methods do not have such a concern.
Revised: In contrast, unsupervised methods do not have such a concern.

Original: Another  beneﬁt is that there is no need to synthesize training sets for unsupervised network training.
Revised: Another obvious beneﬁt is that there is no need to synthesize training sets for unsupervised network training.

Original: To utilize the advantage of unsupervised methods, Kaya et al.
Revised: To utilize theadvantage of unsupervised methods, Kaya et al.

Original: [27] propose a compromised method that trains the light estimation network in a supervised manner (similar to [13]), but estimate the surface normal in an unsupervised way (similar to [47]).
Revised: [25] propose a compromised method that trains thelight estimation network in a supervised manner (similar to [12]), but estimate the surface normal inan unsupervised way (similar to [44].

Original: However, they still suffer from  training data bias during light estimation.
Revised: However, they still suffer from the training data bias duringlight estimation.

Original: Besides, all these methods make a strict assumption that the light intensity distributed in a pre-deﬁned range ( i.e. , [0.2, 2]) and suffers from accumulating error due to their two-step frameworks. In contrast, our method neither makes a strict assumption on reﬂectance nor needs special light source distribution and jointly solves light conditions and surface normal in an unsupervised manner.
Revised: Compared to previous works, our method neither makes a strict assumption onreﬂectance nor needs special light source distribution. Instead, we solve the uncalibrated photometricstereo problem in an unsupervised manner for general reﬂectance surfaces with unknown light sourceinformation.

Original:  Method3.1 Problem Formulation
Revised:  Method

Revised: 3.1 Problem Formulation

Original: Given a set of observations I , ( I 0 , I 1 , ..., I f ) of a static surface, illuminated by f unknown directional illuminations distributing on the upper-hemisphere, uncalibrated photometric stereo aims at recovering light directions L , ( l 0 , l 1 , ..., l f ) , light intensities E , ( e 0 , e 1 , ..., e f ) , and surface normal N , { n i | i 2 P } , P is the set of all positions on the surface.
Revised: Given a set of observations I , ( I 0 , I 1 , ..., I f ) of a static surface, illuminated by f unknowndirectional illuminations distributing on the upper-hemisphere, uncalibrated photometric stereo aimsat recovering light directions L , ( l 0 , l 1 , ..., l f ) , light intensities E , ( e 0 , e 1 , ..., e f ) , and surfacenormal N , { n i | i 2 P } , P is the set of all positions on the surface.

Original: The solution is achieved by  solving the optimization problem,
Revised: The solution is achieved by 137 solving the optimization problem,

Original: [Equation]
Revised: [Equation]

Original: where ¯ m ij 

Revised: where ¯ m ij

Original: I j is the observed pixel intensity at position i , # P is the number of elements in P 1 , m ij is the corresponding rendered pixel intensity, D ( · , · ) is a metric describing their difference.
Revised: I j is the observed pixel intensity at position i , # P is the number of elements in P 1 , m ijis the corresponding rendered pixel intensity, D ( · , · ) is a metric describing their difference.

Original: Under an orthographic camera with linear radiometric response, m ij is formulated as (simpliﬁed in a per-pixel form),
Revised: Under anorthographic camera with linear radiometric response, m ij is formulated as (simpliﬁed in a per-pixelform),

Original: [Equation]
Revised: [Equation]

Original: where v = [0 , 0 , 1] is the view direction pointing toward the viewer, ⇢ ij describes the general reﬂectance, max( n > i l j , 0) represents the attach shadow.
Revised: where v = [0 , 0 , 1] is the view direction pointing toward the viewer, ⇢ ij describes the generalreﬂectance, max( n > i l j , 0) represents the attach shadow.

Original: Unknown light brings two ambiguities when solving Eq.
Revised: Unknown light brings two ambiguities when solving Eq.

Original: (1), i.e. , shape-light ambiguity, which is denoted as an invertable matrix G 2 R 3 ⇥ 3 , and reﬂectance-light ambiguity, which is denoted as a non-zero scalar c j [Equation] R ,
Revised: (1), i.e , shape-light ambiguity, which isdenoted as an invertable matrix G 2 R 3 ⇥ 3 , and reﬂectance-light ambiguity, which is denoted as anon-zero scalar c j 2 R ,

Original: [Equation]
Revised: [Equation]

Original: 3.2 Neural Intrinsics Fields
Revised: 3.2 Neural Intrinsics Fields

Original: General reﬂectance decomposition.
Revised: General reﬂectance decomposition.

Original: To exploit clues of specular reﬂectance and cast shadow for light estimation, we decompose the general reﬂectance as the cast shadow term s ij multiplying the bidirectional reﬂectance term,
Revised: To exploit clues of specular reﬂectance and cast shadow forlight estimation, we decompose the general reﬂectance as the cast shadow term s ij multiplying thebidirectional reﬂectance term,

Original: [Equation]
Revised: [Equation]

Original: where subscript ‘ i ’ and ‘ j ’ indicate  position- (or normal-) and light-dependent factors, respectively; the cast shadow term s ij is either 0 or 1; ⇢ di , ⇢ sij represent the diffuse and specular reﬂectance, and k di and k si are coefﬁcients that balance out the effects of specular and diffuse reﬂectance 2 .
Revised: where subscript ‘ i ’ and ‘ j ’ indicate pixel position- (or normal-) and light-dependent factors, respectively, the cast shadow term s ij is either 0 or 1, ⇢ di , ⇢ sij represent the diffuse and specular reﬂectance,and k di and k si are coefﬁcients that balance out the effects of specular and diffuse reﬂectance 2 .

Original: Neural ﬁelds of specular reﬂectance and cast shadow.
Revised: Neural ﬁelds of specular reﬂectance and cast shadow.

Original: Previous methods identify specular features from speciﬁc pixels and associate an explicit relation to light for light estimation.
Revised: Previous methods identify specular featuresfrom speciﬁc pixels and associate an explicit relation to light for light estimation.

Original: However, this scheme fails for the surface where the specular features are invisible or the explicit relation is violated.
Revised: However, thisscheme could fail for the surface where the specular features are invisible or the explicit relationis violated.

Original: Besides, leaving out the clue of cast shadow can obstruct producing competitive performance.
Revised: Besides, leaving out the clue of cast shadow could obstruct producing competitiveperformance.

Original: In contrast, we leverage both clues of specular and shadow for light estimation, which is achieved by building the neural specular ﬁeld and neural shadow ﬁeld, and associating the ﬁelds to light conditions.[Figure] SpecularNet PE
Revised: In contrast, we leverage both clues of specular and shadow for light estimation, which is achieved bybuilding the neural specular ﬁeld and neural shadow ﬁeld and associating the ﬁelds to light conditions.

Original: [Figure]

Original: Encode $ ! = & !

Original: + ( ||& !

Original: + (|| MLPs CNN Encoder * "!

Original: = + ! , "! - "# . "

Original: # + - "$ . "!

Original: $ max 2 "% & !

Original: ShadowNet

Original: PE - "$ - "# . "

Original: $ 2 "% $ !

Original: ( % $ ! & !

Original: + ! [Figure] [Figure] These neural ﬁelds make the utmost of all observed pixels, and exploit the mutual information across different normal-light (specular) and position-light (shadow) for light estimation.
Revised: These neural ﬁelds make the utmost of all observed pixels, which fully exploits the mutual informationacross different normal-light (specular) and depth-light (shadow).

Original: Two MLPs, namely SpecularNet and ShadowNet , implement these neural ﬁelds, respectively.
Revised: Two MLPs, namely SpecularNetand ShadowNet , implement these neural ﬁelds, respectively.

Original:  we take theestimated light direction as the input of these networks to achieve their implicit association to light conditions.
Revised: Instead of estimating light conditions think from SpecularNet or ShadowNet, we take the light direction as the input of these networks to achievetheir implicit association to light conditions.

Original: Since the specular reﬂectance and cast shadow are normal- and  position-dependent, we also feed the estimated surface normal and the positional code to them, respectively.
Revised: Since the specular reﬂectance and cast shadow arenormal- and pixel position-dependent, we also feed the estimated surface normal and the positionalcode to them, respectively.

Original: SpecularNet and ShadowNet output ⇢ sij and s ij , respectively, as shown in Fig.
Revised: SpecularNet and ShadowNet output ⇢ sij and s ij , respectively, as shown in Fig.

Original: Neural light ﬁeld.
Revised: Neural light ﬁeld.

Original: There is mutual information across different observed images, i.e. , observed images with similar appearances are expected to be illuminated by similar lights.
Revised: There is mutual information across different observed images, i.e , observedimages with similar appearances are expected to be illuminated by similar lights.

Original: To fully consider and exploit this mutual information, we build the neural light ﬁeld by concatenating a CNN encoder to an MLP, namely LightNet .
Revised: To fully considerand exploit this mutual information, we build the neural light ﬁeld by concatenating a CNN encoderto an MLP, namely LightNet .

Original: The encoder extracts a light code from each observed image 3 , and the LightNet infers the corresponding light conditions ( i.e. , e j , l j ) from the light code, as shown in Fig.
Revised: The encoder extracts a light code from each observed image 3 , andthe LightNet outputs the corresponding light conditions ( i.e , e j , l j ) from the light code, as shown in Fig.

Original: Neural position ﬁeld.
Revised: Neural position ﬁeld.

Original: There is mutual information across different positions on a surface, i.e. , the consistency of shape and diffuse reﬂectance in the spatial domain.
Revised: There is mutual information across different positions on a surface, i.e , theconsistency of shape and diffuse reﬂectance in the spatial domain.

Original: To fully consider and exploit this mutual information, we establish a neural position ﬁeld, namely PositionNet , implemented by an MLP.
Revised: To fully consider and exploit thismutual information, we establish a neural position ﬁeld, namely PositionNet , implemented by an MLP.

Original: The neural position ﬁeld outputs position-dependent, light-independent factors 4 , i.e. , n i , ⇢ di , k di , k si .
Revised: The neural position ﬁeld output position-dependent, light-independent factors 4 , i.e , n i , ⇢ di , k di , k si .

Original: The PositionNet takes the positional code as the input, as shown in Fig. 2.
Revised: The PositionNet takes the positional code as the input, as shown in Fig. 2.

Original: 3.3 Optimizing Neural Intrinsics Fields
Revised: 3.3 Optimizing Neural Intrinsics Fields

Original: We adopt the reconstruction loss function with the ` 1 metric to optimize our NeIF,
Revised: We adopt the reconstruction loss function with the ` 1 metric to optimze our NeIF,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Silhouette constraint.
Revised: Silhouette constraint.

Original: L rec cannot resolve the shape-light ambiguity in an unsupervised manner due to the inherently severe ill-posedness.
Revised: L rec cannot resolve the shape-light ambiguity in an unsupervised manner dueto the inherently severe ill-posedness.

Original: Therefore, we introduce the silhouette constraint (similar to those in [23, 15]) to stabilize the training of PositionNet.
Revised: Therefore, we introduce the silhouette constraint (similar tothose in [ 8758025 , 13]) to stabilize the training of PositionNet.

Original: To be speciﬁc, we use polynomial ﬁtting with a moving window block to traverse and pre-compute the contour’s normal of the given objects, represented as ˆ N si , { ˆ n si k 2 R 2 ⇥ 1 , k 2 S } , S is the point set of the contour.
Revised: To be speciﬁc, we use polynomialﬁtting with a moving window block to traverse and pre-compute the contour’s normal of the givenobjects, represented as ˆ N si , { ˆ n si k 2 R 2 ⇥ 1 , k 2 S } , S is the point set of the contour.

Original: We consider ˆ N si can guide the prediction of the azimuth of boundary normal (at the same positions) and introduce the silhouette loss function,
Revised: We considerˆ N si could guide the prediction of the azimuth of boundary normal (at the same positions) and introduce the silhouette loss function,

Original: [Equation]
Revised: [Equation]

Original: where n k 2 R 1 ⇥ 3 represents the estimated surface normalat the positions of silhouette from PositionNet, C ( · ) cuts off the 3rd dimension of n k ( i.e. , C ( n k ) 2 R 1 ⇥ 2 ), and Nor ( · ) is the vector normalization operation.
Revised: where n k 2 R 1 ⇥ 3 represents the estimated surface normal from PositionNet, C ( · ) cuts off the 3rddimension of n k ( i.e , C ( n k ) 2 R 1 ⇥ 2 ), and Nor ( · ) is the vector normalization operation.

Original: Warm-up loss functions.
Revised: Warm-up loss functions.

Original: To avoid local minimum and achieve faster convergence, we warm up the NeIF in early-stage during training with three additional loss functions.
Revised: To avoid local minimum and achieve faster convergence, we warm upthe NeIF in early-stage during training with three additional loss functions.

Original: We use the azimuth of lighting direction estimated by YS97 [57] to guide the training of LightNet,
Revised: We use the azimuth oflighting direction estimated by YS97 [54] to guide the training of LightNet,

Original: [Equation]
Revised: [Equation]

Original: where l az j are estimated light directions by [57].
Revised: where l az j are estimated light directions by [54].

Original: We adopt the gradient penalty [22] L gp to stabilize the training of SpecularNet,
Revised: We adopt the gradient penalty [20] L gp to stabilizethe training of SpecularNet,

Original: [Equation]
Revised: [Equation]

Original: The intuition is from Blinn-Phong model [8], where the specular reﬂectance is monotonically increasing w.r.t the n > i h j , i.e. , r n > i
Revised: The intuition is from Blinn-Phong model, where the specular reﬂectance is monotonically increasingw.r.t the n > i h j , i.e , r n > i

Original: h j ⇢ sij > 0 .
Revised: h j ⇢ sij > 0 .

Original: We also supervise the training of ShadowNet using pseudo shadow maps ˆ S j , j = 1 , 2 , ..., f ,
Revised: We also supervise the training of ShadowNet using pseudoshadow maps ˆ S j , j = 1 , 2 , ..., f ,

Original: [Equation]
Revised: [Equation]

Original: The pseudo shadow maps are obtained by binarizing the observed images, i.e. , considering an observed pixel to be cast shadow if its intensity value is smaller than 0.2 ⇥ the mean intensity value of this image.
Revised: The pseudo shadow maps are obtained by binarizing the observed images, i.e , considering an observedpixel to be cast shadow if its intensity value is smaller than 0 . 2 ⇥ the mean intensity values of thisimage.

Original: After early-stage training, we discard these loss functions for a broad range of reﬂectance.
Revised: After early-stage training, we discard these loss functions for a broad range of reﬂectance.

Original: NeIF training.
Revised: NeIF training.

Original: We train NeIF with the warm-up loss function in ﬁrst 10 epochs,
Revised: We train NeIF with the warm-up loss function in ﬁrst 10 epochs,

Original: [Equation]
Revised: [Equation]

Original:  where  si = 5 ,  az = 0 .
Revised: 10) where  si = 5 ,  az = 0 .

Original: 1 ,  gp =10 ,  shadow =10 .
Revised: 1 ,  gp = ,  shadow =

Original: We then train NeIF until 500 epochs or converging with the loss function,
Revised: We then train NeIF untill 500 epochs or convergingwith the loss function,

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: where L recShadow is the another shadow map supervision loss to train the ShadowNet.
Revised: where L recShadow is the another shadow map supervison loss to train the ShadowNet.

Original: The loss function is the same to L shadow .
Revised: The loss functionis the same to L shadow .

Original: The only difference is the calculation of ˆ S j .
Revised: The only difference is the calculation of ˆ S j .

Original: For L recShadow , we calculate ˆ S j by rendering a depth map with the estimated l j .
Revised: For L recShadow , we calculate ˆ S jby rendering a depth map with the estimated l j .

Original: The depth map is reconstructed from the estimated surface normal map N by method [10].
Revised: The depth map is reconstructed from the estimatedsurface normal map N by method [8] 5 .

Original: L recShadow is used to align outputs of ShadowNet to those of
Revised: L recShadow is used to align outputs of ShadowNet to those of

Original: PositionNet.
Revised: PositionNet.

Original: Implementation details.
Revised: Implementation details.

Original: We generate the positional code from the coordinate (in the image plane) by the same method in [34].
Revised: We generate the positional code from the coordinate (in the image plane)by the same method in [31].

Original: With the assumption of isotropic reﬂectance, we simplify the input of SpecularNet from { v >  l j ,v > n i , n > i  l  j  } to { v >  h j ,n > i h  j  } 5 [31] for easier  training.
Revised: With the assumption of isotropic reﬂectance, we simplify the inputof SpecularNet from { n > i l j , n > i v , l > j v } to { n > i h j , h > j v } 6 [28] for easier model training.

Original: Similar to the Cook-Torrance reﬂectance model [19], we assume k di + k si = 1 to reduce the number of unknowns.
Revised: Similarto the Cook-Torrance reﬂectance model [17], we assume k di + k si = 1 to reduce the numbers ofunknowns.

Original: The CNN decoder with declining channels processes the down-sampled images with a dimension of 256 ⇥ 256 .
Revised: The CNN decoder with declining channels processes the down-sampled images with adimension of 256 ⇥ 256 .

Original: The LightNet takes ﬂatten features to predict l j , e j in two different branches.
Revised: The LightNet takes ﬂatten features to predict l j , e j in two different branches.

Original: For the ShadowNet, we also generate the positional code for l j and concatenate it to the feature in the 9th layer.
Revised: For the ShadowNet, we also generate the positional code for l j and concatenate it to the feature in9-th layer.

Original: The output of ShadowNet is either 0 or 1, which is realized by a step function with a similar implementation in [38]. benchmark [45].
Revised: The output of ShadowNet is either 0 or 1, which is realized by a step function with asimilar implementation in [35] 7 .

Original: This table summarizes comparison methods. ’

Original: N.A.’ represents not applicable as calibrated PS is with known ` and e .

Original: ‘Semi’ indicates certain method leverage partial information of light.

Original: ‘ 3 ’ (or ‘ 7 ’) represents that certain methods (do not) adopt supervised learning for the estimation of surface normal n , light direction l , or light intensity e .

Original: ‘Identical’ means certain methods require the light intensity of different illuminating to be identical.

Original: [Table]

Original:  Experiments
Revised:  Experiments

Original: Training details.
Revised: Training details.

Original: Our main framework is implemented in PyTorch, while the pre-calculations (method [57] and silhouette ﬁtting) are implemented in MATLAB.
Revised: Our main framework is implemented in PyTorch, while the pre-calculations(method [54] and silhouette ﬁtting ) are implemented in MATLAB.

Original: We use Adam as the optimizer  with a learning rate ↵ = 5 ⇥ 10  4 to train our framework in 500 epochs for each scene separately, and the warm-up stage takes up 10 epochs.
Revised: We use Adam as the optimizer k . Please refer to our supplementary material for more details [Table] with a learning rate ↵ = 5 ⇥ 10  4 to train our framework in 500 epochs for each scene separately, andthe warm-up stage takes up ten epochs.

Original: The last 100 epochs use a lower learning rate ↵ = 5 ⇥ 10 for ﬁne-tunning.
Revised: The last 100 epochs use a lower learning rate ↵ = 5 ⇥ 10  5for ﬁne-tunning.

Original: The batch size is 32 for lighting and 256 for spatially random sampling.
Revised: The batch size is 32 for lighting and 256 for spatially random sampling.

Original: After every epoch, the depth map is reconstructed according to the predicted normal map by method [10].
Revised: After everyepoch, the depth map is reconstructed according to the predicted normal map by method [8].

Original: Each scene takes from 2 hours to 6 hours on one RTX 2080Ti 12GB GPU, depending on the resolution of the objects.
Revised: Eachscene takes from 2 hours to 6 hours on one RTX 2080Ti 12GB GPU, depending on the resolution ofthe objects.

Original: Evaluation metrics.
Revised: Evaluation metrics.

Original: We adopt the same metric in [13], the scale-invariant relative error, to measure the accuracy of recovered light intensity as,
Revised: We adopt the same metric in [12], the scale-invariant relative error, to measurethe accuracy of recovered light intensity as

Original: [Equation]
Revised: [Equation]

Original: where, ⌘ is calculated by solving arg min ⌘ 
Revised: where, ⌘ is calculated by solving arg min ⌘ P

Original: [Equation]
Revised: [Equation]

Original: by least squares minimization.
Revised: by least squares minimization.

Original: The metric to measure the accuracy of the predicted light directions and surface normal is the widely used mean angle error (MAE) in degree.
Revised: Themetric to measure the accuracy of the predicted light directions and surface normal is the widely usedmean angle error (MAE) in degree.

Original: 4.1 Evaluation on Public Datasets
Revised: 4.1 Evaluation on Public Datasets

Original: Since the proposed NeIF is an unsupervised uncalibrated photometric stereo method, we compare its performance with state-of-the-art uncalibrated and unsupervised photometric stereo methods.
Revised: Since the proposed NeIF is an unsupervised uncalibrated photometric stereo method, we compare itsperformance with state-of-the-art uncalibrated and unsupervised photometric stereo methods.

Original: Three real-world datasets, including the  D I L I G EN T benchmark dataset [45], A PPLE & G OURD dataset [2], and L IGHT S TAGE D ATA G ALLERY dataset [11], are used for evaluation.
Revised: Threereal-world datasets, including the benchmark dataset D I L I G EN T [39], A PPLE & G OURD dataset [2],and the L IGHT S TAGE D ATA G ALLERY [9], are used for evaluation.

Original: Quantitative comparison for normal map.
Revised: Quantitative comparison for normal map.

Original: Table 1 lists relevant works for a comprehensive surface normal estimation comparison.
Revised: Table 1 lists relevant works for a comprehensive surfacenormal estimation comparison.

Original: As summarized in Table 1, our method is the only method that addresses UPS without the supervision of N , L , or E .
Revised: As summarized in Table 1, our method is the only method thataddresses UPS without the supervision of N , L , or E .

Original: However, our method achieves the best performance and maintains a considerable advantage over other competitors [13, 15] ( e.g. , 7.86 for
Revised: However, our method achieves the bestperformance and maintains a considerable advantage over other competitors[12, 13] ( i.e , 7.86 for

Original: NeIF vs. 8.71 for [15]).
Revised: NeIF v.s. 8.71 for [13]).

Original: Handling the objects in D I L I G EN T dataset [45] with different shapes and reﬂectance, numbers from our method are either best or competitive (about 1  as compared with the best performing UPS method), which shows its good generalization capacity to general reﬂectance and various shapes.
Revised: Objects in D I L I G EN T differ from shapes and reﬂectance. The numbers ofour method are either best or competitive (less than 1  as compared with the best performing UPSmethod), which shows its good generalization capacity to general reﬂectance and various shapes.

Original: This is because our NeIF fully considers mutual information by building up intrinsics ﬁelds and the implicit modeling facilitates  general reﬂectance modeling.
Revised: Because our NeIF fully considers mutual information by building up intrinsics ﬁelds and such implicitmodeling facilitates modeling general reﬂectance.

Original: Visual quality comparison forthe normal map.
Revised: Visual quality comparison for normal map.

Original: Fig.
Revised: Fig.

Original: 3 illustrates the visual quality comparison in terms of recovered surface normal maps and corresponding error maps on D I L I G EN T dataset [45]. [Figure] [Figure]
Revised: 3 illustrates the visual quality comparison in termsof recovered surface normal maps and corresponding error maps on D I L I G EN T.

Original: GO URD

Original: GO URD

Original: S T A ND

Original: I N

Original: [Figure]

Original: K .

Original: [Figure] L M [Figure] The comparison is conducted with two state-of-the-art UPS methods [13, 15] and a state-of-the-art unsupervised PS method [47].
Revised: The comparisonis conducted with two state-of-the-art UPS methods[12, 13] and a state-of-the-art unsupervised PSmethod [44].

Original: Our method is less sensitive to spatially-varying albedo due to the per-pixel manner (red boxes in the left subﬁgure of Fig. 3).
Revised: Our method is less sensitive to spatially-varying albedo due to the per-pixel manner(red boxes in the left subﬁgure of Fig. 3).

Original: However, although the positional code considers the  global shape effect, this per-pixel manner is less effective in modeling complex shape information as compared with the all-pixel one.
Revised: However, although the positional code considers the [Figure] global shape effect, this per-pixel manner is less effective in modeling complex shape information ascompared with the all-pixel one.

Original: It fails for regions with cast shadow (or overexposure) under most light directions (red boxes in the right subﬁgure of Fig. 3).
Revised: It could fail for regions with cast shadow (or overexposure) undermost light directions (red boxes in the right subﬁgure of Fig. 3).

Original: We also show the visual quality results for A PPLE & G OURD dataset [2] and the L IGHT S TAGE D ATA G ALLERY dataset [11] in Fig.
Revised: We also show the visual qualityresults for A PPLE & G OURD dataset [2] and the L IGHT S TAGE D ATA G ALLERY [9] in Fig.

Original: Our method produces reliable estimation for most regions, thanks to the full exploitation of mutual information across different dimensions.
Revised: Ourmethod could produce reliable estimation for most regions, thanks to the full exploitation of mutualinformation across different dimensions.

Original: Quantitative comparison for light directions and intensities.
Revised: Quantitative comparison for light directions and intensities.

Original: As can be observed in Tableand Table 3, our method achieves a superior performance advantage over unsupervised methods (YS97 [57] and PF14 [37]) while maintaining competitive performance as compared with supervised methods (CH19 [13] and CW20 [15]).
Revised: As can be observed in Table 2 and Table 3, our method achieves a superior performance advantage over unsupervised methods[34, 4,42, 50, 29, 28] while maintaining competitive performance as compared with supervised methods.

Original: These supervised methods adopt two-step solutions and suffer from accumulating error. Therefore, even though they achieve similar performance in terms of light conditions accuracy, they are less effective on estimating surface normal as compared with our method (see Table 1).
Revised: Although the supervised methods pre-deﬁne the intensity range, they produce less accurate estimationas compared with our approach due to the possible overﬁtting of supervised learning.

Original: [Table] [Table]
Revised: In contrast,our LightNet builds the neural light ﬁeld, which could fully consider the mutual information amongdifferent observed images, resulting in the accurate estimation of the scale-invariant light intensity.

Original: 4.2 Evaluation on Sparse Uncalibrated Photometric Stereo
Revised: 4.2 Evaluation on Sparse Uncalibrated Photometric Stereo

Original: As compared with other per-pixel PS methods ( e.g. , ZJ19 [62] and LL19 [30]) [Table] the proposed NeIF exploits much more constraints to estimate the normal at a point ( i.e. , # P ⇥ f vs. f ).
Revised: As compared with other per-pixel PS methods ( e.g. , [59, 27]), the proposed NeIF exploits much moreconstraints to estimate the normal at a point, i.e , # P ⇥ f v.s. f .

Original: Because it learns ﬁelds instead of regressing intensity proﬁle [43] to surface normal.
Revised: Because it learns ﬁelds instead ofregressing intensity proﬁle [40] to surface normal.

Original: Therefore, we investigate the effectiveness of our  method under the challenging setting of sparse UPS.
Revised: Therefore, we investigate the effectiveness of our [Table] [Table] method under the challenging setting of sparse UPS.

Original: We randomly select10 or 16 images illuminated by different lights from D I L I G EN T dataset [45] and test our method using these images.
Revised: We randomly select 16 images illuminatedby different lights from D I L I G ENT and test our method using these images.

Original: We repeat this process 30 times, similar to [62].
Revised: We repeat this process30 times, similar to [59].

Original: We compare the performance with that from the classical UPS method PF14 [37], a deep learning based per-pixel approach ZJ19 [62], and an all-pixel approach CH18 [14].
Revised: We compare the performance with that from the classical PS method PF14 [34], as deep learning-based approaches ( e.g. , [12, 13]) perform poorly without re-training.

Original: Due to the fewer reconstruction terms to train our NeIF, we slightly increase the dimension of the positional encoding module from 4 to 6 that increases the frequency [48] to stabilize the training, i.e. , strengthening the role of positional code to reduce the variance of estimated intrinsics.
Revised: Due to the fewer reconstruction terms to train our NeIF [Table] we slightly increase the dimension of thepositional encoding module from 4 to 6 by increasing the frequency [45] to stabilize the training, i.e ,strengthening the role of positional code to reduce the variance of estimated intrinsics.

Original: Besides, we drop L az during early stage warm-up because YS97 [57] fails for the sparse inputs.
Revised: Besides, wedrop L az during early stage warm-up because YS97 [54] fails for objects with centralized specularcomponents ( e.g. , B ALL ).

Original: We report the mean results over 30 trails in Table 4.
Revised: We report the mean results over 30 trails in Table 1.

Original: As can be observed, our method achieves competitive performance onnormal estimation as compared with state-of-the-art sparse PS methods with known light( i.e. , 11.62 for NeIF vs. 9.82 for ZJ19 [62] under 10 lights, and10.38 for NeIF vs. 9.00 for CH18 [14] under 16 lights). Kindly note that we do not require any ground truth normal  for supervision. As compared with the unsupervised UPS method (PF14 [37]), we achieve a superior performance advantage.
Revised: As can be observed, our method achievespromising performance on light and normal estimation for sparse UPS.

Original: The results validate the effectiveness of building up neural intrinsics ﬁelds that fully exploit the mutual information across different dimensions.

Original:  Conclusion
Revised:  Conclusion

Original: This paper proposes NeIF for UPS.
Revised: This paper proposes NeIF for UPS.

Original: By representing the general reﬂectance as four neural intrinsics ﬁelds, it implicitly imposes the light clues of specular reﬂectance and cast shadow for light estimation, which facilitates solving UPS with general reﬂectance.
Revised: By representing the general reﬂectance as four neural intrinsicsﬁelds, it implicitly associates the light clues of specualr reﬂectance and cast shadow to light estimation,which faciliate solving UPS with general reﬂectance.

Original: The proposed NeIF can fully exploit mutual information from all observed pixel values so that it produces stable estimation  forPS and sparse PS.
Revised: The proposed NeIF could fully exploit mutualinformace from all observed pixel values so that it could produce stable estimation even for sparse UPS.

Original: The unsupervised training manner of NeIF is beneﬁcial to the generalization capacity of data from different sources.
Revised: The unsupervised training manner of NeIF is beneﬁcial to the generalization capacity to datafrom different sources.

Original: Limitations.
Revised: Limitations.

Original: Although our method produces promising results for light conditions and surface normal estimation, it has several limitations.
Revised: Although our method produces promising results for light condition and surface normalestimation, it has several limitations.

Original: First, the estimated intrinsics, such as shadow, specular and diffuse reﬂectance, and their balancing coefﬁcients are less accurate for objects with complicated geometries, as shown in Fig.
Revised: First, the estimated intrinsics such as shadow, specular anddiffuse reﬂectance, and their balancing coefﬁcients are less accurate, as shown in Fig.

Original: Second, we need to balance the role of positional code and estimated intrinsics when applying our method for sparse UPS.
Revised: Second,we need to balance the role of positional code and estimated intrinsics when applying our methodfor sparse UPS.

Original: However, how to ﬁnd an optimal balancing strategy is unknown.
Revised: However, how to ﬁnd an optimal balancing strategy is unknown.

Original: Third, as shown in Fig. 3, our method is less effective on global effects, especially for regions with cast shadow (or overexposure) under most light directions due to our
Revised: Third, as shownin Fig. 3, our method is less effective on global effects, especially for regions with cast shadow (oroverexposure) under most light directions due to the per-pixel manner.

Revised: Besides, the inference of ourmethod is inefﬁcient due to the unsupervised training manner and the per-pixel estimation, which isinapplicable for real-time applications.

Original: [Table]
Revised: Checklist

Revised: The checklist follows the references.

Revised: Please read the checklist guidelines carefully for information onhow to answer these questions.

Revised: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .

Revised: You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.

Original: per-pixel manner.
Revised: For example: • Did you include the license to the code and datasets? [Yes] See Section ?? .• Did you include the license to the code and datasets?

Revised: [No] The code and the data areproprietary.

Revised: • Did you include the license to the code and datasets?

Original: Besides, the inference of our method is inefﬁcient due to the unsupervised training manner and  the per-pixel estimation, which is inapplicable for real-time applications.
Revised: [N/A] Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit.

Revised: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.

Revised: For all authors...

Revised: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s

Revised: contributions and scope?

Revised: [Yes] See Line 9-13,

Revised: (b) Did you describe the limitations of your work?

Revised: [Yes] See Line 287-

Revised: (c) Did you discuss any potential negative societal impacts of your work?

Revised: [No]

Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Revised: them?

Revised: [Yes]

Revised: If you are including theoretical results...

Revised: (a) Did you state the full set of assumptions of all theoretical results?

Revised: [Yes]

Revised: (b) Did you include complete proofs of all theoretical results?

Revised: [N/A]

Revised: If you ran experiments...

Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Revised: [Yes] See

Revised: Supplementary Material

Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Revised: how they

Revised: were chosen)?

Revised: [Yes]

Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Revised: ments multiple times)?

Revised: [Yes]

Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Revised: [Yes]

Revised: If you are using existing assets (e

Revised: g., code, data, models) or curating/releasing new assets...

Revised: (a) If your work uses existing assets, did you cite the creators?

Revised: [No]

Revised: (b) Did you mention the license of the assets?

Revised: [No]

Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Revised: [No]

Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Revised: using/curating?

Revised: [No]

Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Revised: information or offensive content?

Revised: [No]

Revised: If you used crowdsourcing or conducted research with human subjects...

Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Revised: applicable?

Revised: [N/A]

Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Revised: Board (IRB) approvals, if applicable?

Revised: [N/A]

Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Revised: spent on participant compensation?

Revised: [N/A]