Original: Shape, Light & Material Decomposition from Images using Monte Carlo Rendering and Denoising
Revised: Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Recent advances in differentiable rendering have enabled high-quality reconstruction of 3D scenes from multi-view images.
Revised: Recent advances in differentiable rendering have enabled high-quality reconstruction of 3D scenes from multi-view images.

Original: Most methods rely on simple renderingalgorithms: pre-filtered direct lighting or learned representations of irradiance.
Revised: Most methods rely on simple rendering algorithms: pre-filtered direct lighting or learned representations of irradiance.

Original: Weshow that a more realistic shading model, incorporating ray tracing and Monte
Revised: We show that a more realistic shading model, incorporating ray tracing and Monte

Original: Carlo integration, substantially improves decomposition into shape, materials &lighting.
Revised: Carlo integration, substantially improves decomposition into shape, materials & lighting.

Original: Unfortunately, Monte Carlo integration provides estimates with significant noise, even at large sample counts, which makes gradient-based inverserendering very challenging.
Revised: Unfortunately, Monte Carlo integration provides estimates with significant noise, even at large sample counts, which makes gradient-based inverse rendering very challenging.

Original: To address this, we incorporate multiple importancesampling and denoising in a novel inverse rendering pipeline.
Revised: To address this, we incorporate multiple importance sampling and denoising in a novel inverse rendering pipeline.

Original: Thissubstantially improves convergence and enables gradient-based optimization at low sample counts.
Revised: This improves convergence and enables gradient-based optimization at low sample counts.

Original: We present an efficient method to jointly reconstruct geometry (explicit trianglemeshes), materials, and lighting, which substantially improves material and lightseparation compared to previous work.
Revised: We present an efficient method to jointly reconstruct geometry (explicit triangle meshes), materials, and lighting, which substantially improves material and light separation compared to previous work.

Original: We argue that denoising can become anintegral part of high quality inverse rendering pipelines.
Revised: We argue that denoising can become an integral part of high quality inverse rendering pipelines.

Original:  Introduction
Revised:  Introduction

Original: Differentiable rendering shows great promise for accurate multi-view 3D reconstruction from imageobservations.
Revised: Differentiable rendering shows great promise for accurate multi-view 3D reconstruction from image observations.

Original: NeRF [39] use differentiable volume rendering to create high quality view interpolationthrough neural, density-based light-fields.
Revised: NeRF [39] use differentiable volume rendering to create high quality view interpolation through neural, density-based light-fields.

Original: Surface-based methods apply signed distance fields [43,47, 67, 61, 71] or triangle meshes [41, 54] to capture high quality geometry.
Revised: Surface-based methods apply signed distance fields [43, 47, 68, 62, 73] or triangle meshes [41, 54] to capture high quality geometry.

Original: Recent work [6, 41, 73]further decompose these representations into geometry, material and environment light.
Revised: Recent work [6, 41, 75] further decompose these representations into geometry, material, and environment light.

Original: Most aforementioned methods rely on appearance baked into neural light fields or apply simpleshading models.
Revised: Most aforementioned methods rely on appearance baked into neural light fields or apply simple shading models.

Original: Typically, direct lighting without shadows is considered, combined with pre-filteredrepresentations of environment lighting [41, 71].
Revised: Typically, direct lighting without shadows is considered, combined with pre-filtered representations of environment lighting [41, 73].

Original: Some methods account for shadowing and indirectillumination [6, 73, 75] but often lock the geometry optimization when sampling the shadow term,or rely on learned representations of irradiance.
Revised: Some methods account for shadowing and indirect illumination [6, 75, 77] but often lock the geometry optimization when sampling the shadow term, or rely on learned representations of irradiance.

Original: While results are impressive, the deviations fromphysically-based shading models makes it harder for these methods to plausibly disentangle shape,material and lighting, as shown in Figure 1.
Revised: While results are impressive, the deviations from physically-based shading models makes it harder for these methods to plausibly disentangle shape, material and lighting, as shown in Figure 1.

Original: In theory, it is straightforward to replace the rendering engines of these 3D reconstruction methodswith photorealistic differentiable renderers [31, 36, 46, 45, 69, 70] and optimize in a setting withmore accurate light simulation, including global illumination effects.
Revised: In theory, it is straightforward to replace the rendering engines of these 3D reconstruction methods with photorealistic differentiable renderers [31, 36, 46, 45, 71, 72] and optimize in a setting with more accurate light simulation, including global illumination effects.

Original: In practice, however, the noisein multi-bounce Monte Carlo rendering makes gradient-based optimization challenging.
Revised: In practice, however, the noise in multi-bounce Monte Carlo rendering makes gradient-based optimization challenging.

Original: Very highsample counts are required, which result in intractable iteration times.
Revised: Very high sample counts are required, which result in intractable iteration times.

Original: In this paper, we bridge the gap between current multi-view 3D reconstruction and physicallybased differentiable rendering, and demonstrate high-quality reconstructions at competitive runtimeperformance.
Revised: In this paper, we bridge the gap between current multi-view 3D reconstruction and physicallybased differentiable rendering, and demonstrate high-quality reconstructions at competitive runtime performance.

Original: We attack the challenging case of extracting explicit triangle meshes , PBR materialsand environment lighting from a set of multi-view images, in a format directly compatible with
Revised: We attack the challenging case of extracting explicit triangle meshes , PBR materials

Original: [Figure]
Revised: [Figure]

Original:  current DCC tools and game engines.
Revised: and environment lighting from a set of multi-view images, in a format directly compatible with current DCC tools and game engines.

Original: For improved visual fidelity, we compute direct illuminationusing Monte Carlo integration with ray tracing, and add several techniques to combat the increasednoise levels.
Revised: For improved visual fidelity, we compute direct illumination using Monte Carlo integration with ray tracing, and add several techniques to combat the increased noise levels.

Original: By carefully trading variance for bias, we enable efficient gradient-based optimizationin a physically-based inverse rendering pipeline.
Revised: By carefully trading variance for bias, we enable efficient gradient-based optimization in a physically-based inverse rendering pipeline.

Original: Compared to previous 3D reconstruction methods,our formulation primarily improves material and light separation.
Revised: Compared to previous 3D reconstruction methods, our formulation primarily improves material and light separation.

Original: Concretely, we reduce variance by combining multiple importance sampling [57] and differentiabledenoisers.
Revised: Concretely, we reduce variance by combining multiple importance sampling [58] and differentiable denoisers.

Original: We evaluate both neural denoisers [2, 10, 22] and cross-bilateral filters [52] in our pipeline.
Revised: We evaluate both neural denoisers [2, 10, 22] and cross-bilateral filters [52] in our pipeline.

Original: Furthermore, we decompose the rendering equation into albedo, demodulated diffuse lighting andspecular lighting, which enable precise regularization to improve light and material separation.
Revised: Furthermore, we decompose the rendering equation into albedo, demodulated diffuse lighting and specular lighting, which enable precise regularization to improve light and material separation.

Original:  Previous Work
Revised:  Previous Work

Original: Neural methods for multi-view reconstruction These methods fall in two categories: implicitor explicit scene representations.
Revised: Neural methods for multi-view reconstruction These methods fall in two categories: implicit or explicit scene representations.

Original: NeRF [39] and follow-ups [38, 42, 49, 72, 63, 15, 40, 50, 68, 65],use volumetric representations and compute radiance by ray marching through a neurally encoded5D light field.
Revised: NeRF [39] and follow-ups [38, 42, 49, 74, 64, 15, 40, 50, 69, 66], use volumetric representations and compute radiance by ray marching through a neurally encoded 5D light field.

Original: While achieving impressive results on novel view synthesis, geometric quality suffersfrom the ambiguity of volume rendering [72].
Revised: While achieving impressive results on novel view synthesis, geometric quality suffers from the ambiguity of volume rendering [74].

Original: Surface-based rendering methods [43, 47, 67, 61]optimizing the underlying surface directly using implicit differentiation, or gradually morph from avolumetric representation into a surface representation.
Revised: Surface-based rendering methods [43, 47, 68, 62] optimizing the underlying surface directly using implicit differentiation, or gradually morph from a volumetric representation into a surface representation.

Original: Methods with explicit representation estimate3D meshes from images, where most approaches assume a given mesh topology [35, 11, 12], butrecent work also include topology optimization [34, 14, 54, 41].
Revised: Methods with explicit representation estimate 3D meshes from images, where most approaches assume a given mesh topology [35, 11, 12], but recent work also include topology optimization [34, 14, 54, 41].

Original: BRDF and lighting estimation To estimate surface radiometric properties from images, previouswork on BTF and SVBRDF estimation rely on special viewing configurations, lighting patterns orcomplex capturing setups [30, 16, 17, 18, 64, 5, 8, 53, 20].
Revised: BRDF and lighting estimation To estimate surface radiometric properties from images, previous work on BTF and SVBRDF estimation rely on special viewing configurations, lighting patterns or complex capturing setups [30, 16, 17, 18, 65, 5, 8, 53, 20].

Original: Recent methods exploit neural networksto predict BRDFs from images [13, 19, 32, 33, 44, 37].
Revised: Recent methods exploit neural networks to predict BRDFs from images [13, 19, 32, 33, 44, 37].

Original: Differentiable rendering methods [35, 11, 74,12, 21] learn to predict geometry, SVBRDF and, in some cases, lighting via photometric loss.
Revised: Differentiable rendering methods [35, 11, 76, 12, 21] learn to predict geometry, SVBRDF and, in some cases, lighting via photometric loss.

Original: Most related to our work are neural 3D reconstruction methods with intrinsic decomposition ofshape, materials, and lighting from images [6, 7, 41,71, 73, 75].
Revised: Most related to our work are neural 3D reconstruction methods with intrinsic decomposition of shape, materials, and lighting from images [6, 7, 41, 73, 75, 77].

Original: Illumination is represented usingmixtures of spherical Gaussians [6, 41, 71, 75], pre-filtered approximations [7, 41], or low resolutionenvironment maps [73].
Revised: Illumination is represented using [Figure] mixtures of spherical Gaussians [6, 41, 73, 77], pre-filtered approximations [7, 41], or low resolution environment maps [75].

Original: When the shadowing term is accounted for [73], optimization is split intotwo passes where geometry is locked before the shadow term is sampled.
Revised: When the shadowing term is accounted for [75], optimization is split into two passes where geometry is locked before the shadow term is sampled.

Original: Other approaches representindirect illumination with neural networks [62, 75].
Revised: Other approaches represent indirect illumination with neural networks [63, 77].

Original: Image denoisers Denoisers are essential tools in both real-time- and production renderers.
Revised: Image denoisers Denoisers are essential tools in both real-time- and production renderers.

Original: Traditionally, variants of cross-bilateral filters are used [76], which require scene-specific manualadjustments.
Revised: Traditionally, variants of cross-bilateral filters are used [78], which require scene-specific manual adjustments.

Original: More recently, neural denoisers [2, 10, 22] trained on large datasets have shown impressive quality without the need for manual tuning, and are now incorporated in most productionrenderers.
Revised: More recently, neural denoisers [2, 10, 22] trained on large datasets have shown impressive quality without the need for manual tuning, and are now incorporated in most production renderers.

Original: We directly incorporate differentiable versions of these denoisers in our pipeline.
Revised: We directly incorporate differentiable versions of these denoisers in our pipeline.

Original: We arecurrently unaware of image denoisers applied in differentiable rendering, but we see a lot of potentialfor denoisers in physically-based inverse rendering going forward.[Figure]
Revised: We are currently unaware of image denoisers applied in differentiable rendering, but we see a lot of potential for denoisers in physically-based inverse rendering going forward.

Original:  System
Revised:  System

Original: We target the challenging task of joint optimization of shape, material and environment lighting froma set of multi-view images with known foreground segmentation masks and camera poses.
Revised: We target the challenging task of joint optimization of shape, material and environment lighting from a set of multi-view images with known foreground segmentation masks and camera poses.

Original: Our goalis to use physically-based rendering techniques to improve the intrinsic decomposition of lightingand materials, producing assets that can be relit, edited, animated, or used in simulation.
Revised: Our goal is to use physically-based rendering techniques to improve the intrinsic decomposition of lighting and materials, producing assets that can be relit, edited, animated, or used in simulation.

Original: As a proofof-concept, we extend a recent approach, NVDIFFREC [41], which targets the same optimization task(shape, materials and environment lighting).
Revised: As a proofof-concept, we extend a recent approach, NVDIFFREC [41], which targets the same optimization task (shape, materials and environment lighting).

Original: Notably, they directly optimize a triangular 3D model,which has obvious benefits: it is easy to import and modify the reconstructed models in existing DCCtools, and  can exploit hardware-accelerated differentiable rasterization [28].
Revised: Notably, they directly optimize a triangular 3D model, which has obvious benefits: it is easy to import and modify the reconstructed models in existing DCC tools, and a triangular representation can exploit hardware-accelerated differentiable rasterization [28].

Original: In our setting, triangular3D models also means we can leverage hardware-accelerated ray-tracing for efficient shadow tests.
Revised: In our setting, triangular 3D models also means we can leverage hardware-accelerated ray-tracing for efficient shadow tests.

Original: NVDIFFREC reports competitive results on view interpolation, material reconstruction, and relighting,and we will use their pipeline as a baseline in our evaluations.
Revised: NVDIFFREC reports competitive results on view interpolation, material reconstruction, and relighting, and we will use their pipeline as a baseline in our evaluations.

Original: Our system is summarized in Figure 2.
Revised: Our system is summarized in Figure 2.

Original: A triangular mesh with arbitrary topology is optimized from aset of images through 2D supervision.
Revised: A triangular mesh with arbitrary topology is optimized from a set of images through 2D supervision.

Original: Geometry is represented by a signed distance field defined ona three-dimensional grid and reduced to a triangular surface mesh through deep marching tetrahedra(DMTet)
Revised: Geometry is represented by a signed distance field defined on a three-dimensional grid and reduced to a triangular surface mesh through deep marching tetrahedra (DMTet) [54].

Original: Next, the extracted surface mesh is rendered in a differentiable renderer, using thephysically-based (PBR) material model from Disney [9].
Revised: Next, the extracted surface mesh is rendered in a differentiable renderer, using the physically-based (PBR) material model from Disney [9].

Original: This material model combines a diffuse termwith an isotropic, specular GGX lobe [60].
Revised: This material model combines a diffuse term with an isotropic, specular GGX lobe [61].

Original: A tangent space normal map is also included to capturehigh frequency shading detail.
Revised: A tangent space normal map is also included to capture high frequency shading detail.

Original: Finally, the rendered image is evaluated against a reference imageusing a photometric loss.
Revised: Finally, the rendered image is evaluated against a reference image using a photometric loss.

Original: In contrast to NVDIFFREC , which uses a simple renderer with deferredshading and the split-sum approximation for direct lighting (without shadows), we instead leverage arenderer which evaluates the direct lighting integral using Monte Carlo integration and ray tracing(shadow rays).
Revised: In contrast to NVDIFFREC , which uses a simple renderer with deferred shading and the split-sum approximation for direct lighting (without shadows), we instead leverage a renderer which evaluates the direct lighting integral using Monte Carlo integration and ray tracing (shadow rays).

Original: We represent the scene lighting using a high dynamic range light probe stored as afloating point texture, typically at a resolution of 256 × 256 texels.
Revised: We represent the scene lighting using a high dynamic range light probe stored as a floating point texture, typically at a resolution of 256 × 256 texels.

Original: Finally, to combat the inherentvariance that comes with Monte Carlo integration, we leverage differentiable image denoising andmultiple importance sampling. 
Revised: Finally, to combat the inherent variance that comes with Monte Carlo integration, we leverage differentiable image denoising and multiple importance sampling. [Figure]

Original: Optimization task Let ϕ denote the optimization parameters (shape, spatially varying materialsand light probe).
Revised: Optimization task Let ϕ denote the optimization parameters (shape, spatially varying materials and light probe).

Original: For a given camera pose, c , our differentiable renderer produces an image I ϕ ( c ) .
Revised: For a given camera pose, c , our differentiable renderer produces an image I ϕ ( c ) .

Original: Given that we use Monte Carlo integration during rendering, this image inherently includes noise,and we apply a differentiable image denoiser, D θ , with parameters, θ , to reduce the variance,
Revised: Given that we use Monte Carlo integration during rendering, this image inherently includes noise, and we apply a differentiable image denoiser, D θ , with parameters, θ , to reduce the variance,

Original: I denoised ϕ
Revised: I denoised ϕ

Original: ( c ) =
Revised: ( c ) =

Original: D θ ( I ϕ ( c )) .
Revised: D θ ( I ϕ ( c )) .

Original: The reference image I ref ( c ) is a view from the same camera.
Revised: The reference image I ref ( c ) is a view from the same camera.

Original: Given aphotometric loss function L , we minimize the empirical risk
Revised: Given a photometric loss function L , we minimize the empirical risk

Original: [Equation]
Revised: [Equation]

Original: (1) using Adam [26] based on gradients w.r.t.
Revised: using Adam [26] based on gradients w.r.t.

Original: the optimization parameters, ∂L/∂ϕ , and ∂L/∂θ , whichare obtained through differentiable rendering.
Revised: the optimization parameters, ∂L/∂ϕ , and ∂L/∂θ , which are obtained through differentiable rendering.

Original: We use the same loss function as NVDIFFREC .
Revised: We use the same loss function as NVDIFFREC .

Original: Anexample of the optimization process is illustrated in Figure 3,[Figure]
Revised: An example of the optimization process is illustrated in Figure 3,

Original: 3.1 Direct Illumination
Revised: 3.1 Direct Illumination

Original: [Figure] The outgoing radiance L ( ω o ) in direction ω o can be expressed using the rendering equation [24] as: 
Revised: The outgoing radiance L ( ω o ) in direction ω o can be expressed using the rendering equation [24] as: [Equation]

Original: [Equation]
Revised: [Equation]

Original: This is an integral of the product of the incident radiance, L i ( ω i ) from direction ω
Revised: This is an integral of the product of the incident radiance, L i ( ω i ) from direction ω

Original: i and the BSDFf ( ω i , ω o ) .
Revised: i and the BSDF f ( ω i , ω o ) .

Original: The integration domain is the hemisphere Ω around the surface normal, n .
Revised: The integration domain is the hemisphere Ω around the surface normal, n .

Original: Spherical Harmonics (SH) [11] or Spherical Gaussians (SG) [6, 71] are often used as efficientapproximations of direct illumination, but only work well for low- to medium-frequency lighting.
Revised: Spherical Harmonics (SH) [11] or Spherical Gaussians (SG) [6, 73] are often used as efficient approximations of direct illumination, but only work well for low- to medium-frequency lighting.

Original: Incontrast, the split sum approximation [25, 41] captures all-frequency image based lighting, but doesnot incorporate shadows.
Revised: In contrast, the split sum approximation [25, 41] captures all-frequency image based lighting, but does not incorporate shadows.

Original: Our goal is all-frequency lighting including shadows, which we tackle byevaluating the rendering equation using Monte Carlo integration:
Revised: Our goal is all-frequency lighting including shadows, which we tackle by evaluating the rendering equation using Monte Carlo integration:

Original: [Equation]
Revised: [Equation]

Original: with samples drawn from some distribution p ( ω i ) .
Revised: with samples drawn from some distribution p ( ω i ) .

Original: Note that L i ( ω i ) includes a visibility test, whichcan be evaluated by tracing a shadow ray in direction ω i .
Revised: Note that L i ( ω i ) includes a visibility test, which can be evaluated by tracing a shadow ray in direction ω i .

Original: Unfortunately, the variance levels in Monte
Revised: Unfortunately, the variance levels in Monte

Original: Carlo integration with low number of samples makes gradient-based optimization hard, particularlywith complex lighting.
Revised: Carlo integration with low number of samples makes gradient-based optimization hard, particularly with complex lighting.

Original: In Section 4 we propose several variance reduction techniques, which enablean inverse rendering pipeline that efficiently reconstructs complex geometry, a wide range of lightingconditions and spatially-varying BSDFs.
Revised: In Section 4 we propose several variance reduction techniques, which enable an inverse rendering pipeline that efficiently reconstructs complex geometry, a wide range of lighting conditions and spatially-varying BSDFs.

Original: Shadow gradients
Revised: Shadow gradients

Original: In single view optimization [59], shape from shadows [56], or direct optimization of the position/direction of analytical light sources, shadow ray visibility gradients [36, 3] arehighly beneficial.
Revised: In single view optimization [60], shape from shadows [57], or direct optimization of the position/direction of analytical light sources, shadow ray visibility gradients [36, 3] are highly beneficial.

Original: However, in our multi-view setting (50+ views), similar to Loubet et al.
Revised: However, in our multi-view setting (50+ views), similar to Loubet et al.

Original: [36], weobserved that gradients of diffuse scattering are negligible compared to the gradients of primaryvisibility.
Revised: [36], we observed that gradients of diffuse scattering are negligible compared to the gradients of primary visibility.

Original: Hence, for performance reasons, in the experiment presented in this paper, the shadow rayvisibility gradients are detached when evaluating the hemisphere integral, and shape optimization isdriven by primary visibility gradients, obtained from nvdiffrast [28].
Revised: Hence, for performance reasons, in the experiment presented in this paper, the shadow ray visibility gradients are detached when evaluating the hemisphere integral, and shape optimization is driven by primary visibility gradients, obtained from nvdiffrast [28].

Original:  Variance Reduction
Revised:  Variance Reduction

Original: We evaluate direct illumination with high frequency environment map lighting combined with a widerange of materials (diffuse, dielectrics, and metals).
Revised: We evaluate direct illumination with high frequency environment map lighting combined with a wide range of materials (diffuse, dielectrics, and metals).

Original: Strong directional sunlight, highly specular,mirror-like materials, and the visibility component can all introduce significant levels of noise.
Revised: Strong directional sunlight, highly specular, mirror-like materials, and the visibility component can all introduce significant levels of noise.

Original: Toenable optimization in an inverse rendering setting at practical sample counts, we carefully sampleeach of these contributing factors to obtain a signal with low variance.
Revised: To enable optimization in an inverse rendering setting at practical sample counts, we carefully sample each of these contributing factors to obtain a signal with low variance.

Original: Below, we describe how wecombat noise by using multiple importance sampling and denoising.
Revised: Below, we describe how we combat noise by using multiple importance sampling and denoising.

Original: 4.1 Multiple Importance Sampling
Revised: 4.1 Multiple Importance Sampling

Original: We leverage multiple importance sampling [57] (MIS), a framework to weigh a set of different sampling techniques to reduce variance in Monte Carlo integration.
Revised: We leverage multiple importance sampling [58] (MIS), a framework to weigh a set of different sampling techniques to reduce variance in Monte Carlo integration.

Original: Given a set of sampling techniques,[Figure] [Equation] each with a sampling distribution p i , the Monte Carlo estimator for an integral (cid:82) Ω g ( x ) dx given by
Revised: Given a set of sampling techniques, each with a sampling distribution p i , the Monte Carlo estimator for an integral (cid:82) Ω g ( x ) dx given by

Original: MIS is
Revised: MIS is

Original: [Equation]
Revised: [Equation]

Original: The weighting functions w i ( x ) are chosen using the balance heuristic .
Revised: The weighting functions w i ( x ) are chosen using the balance heuristic .

Original: Please refer to Veach’sthesis [57] or the excellent PBRT book [48] for further details.
Revised: Please refer to Veach’s thesis [58] or the excellent PBRT book [48] for further details.

Original: In our case, we apply MIS with three sampling techniques: light importance sampling, p light ( ω ) ,using a piecewise-constant 2D distribution sampling technique [48], cosine sampling, p diffuse ( ω ) , forthe diffuse lobe, and GGX importance sampling [23], p specular ( ω ) , for the specular lobe.
Revised: In our case, we apply MIS with three sampling techniques: light importance sampling, p light ( ω ) , using a piecewise-constant 2D distribution sampling technique [48], cosine sampling, p diffuse ( ω ) , for the diffuse lobe, and GGX importance sampling [23], p specular ( ω ) , for the specular lobe.

Original: Unlike inforward rendering with known materials and lights, our material and light parameters are optimizationvariables.
Revised: Unlike in forward rendering with known materials and lights, our material and light parameters are optimization variables.

Original: Thus, the sampling distributions, p i , are recomputed in each optimization iteration.
Revised: Thus, the sampling distributions, p i , are recomputed in each optimization iteration.

Original: Following the taxonomy of differentiable Monte Carlo estimators of Zeltner et al.
Revised: Following the taxonomy of differentiable Monte Carlo estimators of Zeltner et al.

Original: [69], our importancesampling is detached , i.e., we do not back-propagate gradients to the scene parameters in the samplingstep, only in the material evaluation.
Revised: [71], our importance sampling is detached , i.e., we do not back-propagate gradients to the scene parameters in the sampling step, only in the material evaluation.

Original: Please refer to Zeltner et al.
Revised: Please refer to Zeltner et al.

Original: for a careful analysis of Monte
Revised: for a careful analysis of Monte

Original: Carlo estimators for differentiable light transport.
Revised: Carlo estimators for differentiable light transport.

Original: MIS is unbiased, but chaining multiple iterations of our algorithm exhibits bias , as the current,importance sampled, iteration dictates the sampling distributions used in the next iteration.
Revised: MIS is unbiased, but chaining multiple iterations of our algorithm exhibits bias , as the current, importance sampled, iteration dictates the sampling distributions used in the next iteration.

Original: Light probeintensities, for example, are optimized based on an importance sampled image, which are then usedto construct the sampling distribution for the subsequent pass.
Revised: Light probe intensities, for example, are optimized based on an importance sampled image, which are then used to construct the sampling distribution for the subsequent pass.

Original: For unbiased rendering, the samplingdistribution must be estimated using a second set of uncorrelated samples instead.
Revised: For unbiased rendering, the sampling distribution must be estimated using a second set of uncorrelated samples instead.

Original: Furthermore, weexplicitly re-use the random seed from the forward pass during gradient backpropagation to scattergradients to the exact same set of parameters that contributed to the forward rendering.
Revised: Furthermore, we explicitly re-use the random seed from the forward pass during gradient backpropagation to scatter gradients to the exact same set of parameters that contributed to the forward rendering.

Original: This approachis clearly biased [58], but we empirically note that this is very effective in reducing variance, and inour setting this variance-bias trade-off works in our favor.
Revised: This approach is clearly biased [59], but we empirically note that this is very effective in reducing variance, and in our setting this variance-bias trade-off works in our favor.

Original: 4.2 Denoising
Revised: 4.2 Denoising

Original: For differentiable rendering, the benefits of denoising are twofold.
Revised: For differentiable rendering, the benefits of denoising are twofold.

Original: First, it improves the imagequality of the rendering in the forward pass, reducing the optimization error and the gradient noiseintroduced in the image loss.
Revised: First, it improves the image quality of the rendering in the forward pass, reducing the optimization error and the gradient noise introduced in the image loss.

Original: Second, as gradients back-propagate through the denoiser’s spatialfilter kernel, gradient sharing between neighboring pixels is enabled.
Revised: Second, as gradients back-propagate through the denoiser’s spatial filter kernel, gradient sharing between neighboring pixels is enabled.

Original: To see this, let’s consider asimple denoiser, O =
Revised: To see this, let’s consider a simple denoiser, O =

Original: X ⊛ F , where the noisy rendered image X is filtered by a low-pass filter, F( ⊛ represents an image space 2D convolution).
Revised: X ⊛ F , where the noisy rendered image X is filtered by a low-pass filter, F ( ⊛ represents an image space 2D convolution).

Original: Given a loss gradient, ∂L∂O , the gradients propagatedback to the renderer are ∂L∂X
Revised: Given a loss gradient, ∂L∂O , the gradients propagated back to the renderer are ∂L∂X

Original: = ∂L∂O ⊛ F T , which applies the same low-pass filter in case the filter isrotationally symmetric  F = F T .
Revised: = ∂L∂O ⊛ F T , which applies the same low-pass filter in case the filter is rotationally symmetric ( F = F T ).

Original: In other words, the renderer sees filtered loss gradients in a localspatial footprint.
Revised: In other words, the renderer sees filtered loss gradients in a local spatial footprint.

Original: While denoisers inherently trade a reduction in variance for increased bias, weempirically note that denoising significantly helps convergence at lower sample counts, and help toreconstruct higher frequency environment lighting.
Revised: While denoisers inherently trade a reduction in variance for increased bias, we empirically note that denoising significantly helps convergence at lower sample counts, and help to reconstruct higher frequency environment lighting.

Original: We show an illustrative example in Figure 9.
Revised: We show an illustrative example in Figure 9.

Original: Following previous work in denoising for production rendering [2], Figure 4 shows how we separatelighting into diffuse, c d , and specular, c s terms.
Revised: Following previous work in denoising for production rendering [2], Figure 4 shows how we separate lighting into diffuse, c d , and specular, c s terms.

Original: This lets us denoise each term separately, creatingdenoised buffers, D θ ( c d ) , and D θ ( c s ) .
Revised: This lets us denoise each term separately, creating denoised buffers, D θ ( c d ) , and D θ ( c s ) .

Original: More importantly, we can use demodulated diffuse lighting,which means that the lighting term has not yet been multiplied by material diffuse albedo, k d .
Revised: More importantly, we can use demodulated diffuse lighting, which means that the lighting term has not yet been multiplied by material diffuse albedo, k d .

Original: Inforward rendering, this is important as it decorrelates the noisy lighting from material textures, thusselectively denoising the noisy Monte-Carlo estimates and avoiding to low-pass filter high-frequenttexture information.
Revised: In forward rendering, this is important as it decorrelates the noisy lighting from material textures, thus [Figure] [Figure] selectively denoising the noisy Monte-Carlo estimates and avoiding to low-pass filter high-frequent texture information.

Original: In inverse rendering, we can additionally use it to improve material and lightdecomposition by adding regularization on the lighting terms, as disucussed in Section 5.
Revised: In inverse rendering, we can additionally use it to improve material and light decomposition by adding regularization on the lighting terms, as disucussed in Section 5.

Original: We composethe final image as c =
Revised: We compose the final image as c =

Original: k d · D θ ( c d ) + D θ ( c s ) .
Revised: k d · D θ ( c d ) + D θ ( c s ) .

Original: We currently do not demodulate specular lightingbecause of the view dependent Fresnel term, but expect this to be improved in future work.
Revised: We currently do not demodulate specular lighting because of the view dependent Fresnel term, but expect this to be improved in future work.

Original: In this we used decorrelated samples

Original: Cross-bilateral filters Cross-bilateral filters are commonly used to remove noise in renderedimages [76].
Revised: Cross-bilateral filters Cross-bilateral filters are commonly used to remove noise in rendered images [78].

Original: To evaluate this family of denoisers in our inverse pipeline, we adapted Spatio-temporal
Revised: To evaluate this family of denoisers in our inverse pipeline, we adapted Spatio-temporal

Original: Variance-Guided Filtering [52] (SVGF), which is a popular cross-bilateral filter using surface normalsand per-pixel depth as edge-stopping guides.
Revised: Variance-Guided Filtering [52] (SVGF), which is a popular cross-bilateral filter using surface normals and per-pixel depth as edge-stopping guides.

Original: The denoiser is applied to demodulated diffuse lighting,to avoid smearing texture detail.
Revised: The denoiser is applied to demodulated diffuse lighting, to avoid smearing texture detail.

Original: We disabled the temporal component of SVGF, as we focus on singleframe rendering, and implemented the filter as a differentiable module to allow for loss gradients to propagate back to our  parameters.
Revised: We disabled the temporal component of SVGF, as we focus on single frame rendering, and implemented the filter as a differentiable module to allow for loss gradients to propagate back to our scene parameters.

Original: Neural denoisers As a representative neural denoiser, we deploy the Open Image Denoiser(OIDN) [1], which is a U-Net [51] pre-trained on a large corpus of rendered images.
Revised: Neural denoisers As a representative neural denoiser, we deploy the Open Image Denoiser (OIDN) [1], which is a U-Net [51] pre-trained on a large corpus of rendered images.

Original: The denoiser is applied to the rendered image before computing the image space loss.
Revised: The denoiser is applied to the rendered image before computing the image space loss.

Original: As the network modelis fully convolutional, it is trivially differentiable, and we can propagate gradients from the imagespace loss, through the denoiser back to the renderer.
Revised: As the network model is fully convolutional, it is trivially differentiable, and we can propagate gradients from the image space loss, through the denoiser back to the renderer.

Original: In Figure 5, we provide an overview on the effect of denoisers during 3D scene reconstruction.
Revised: In Figure 5, we provide an overview on the effect of denoisers during 3D scene reconstruction.

Original: We observe that denoising is especially helpful at low sample counts, where we obtain similarreconstruction results at 8 spp with denoising, compared to 32 spp without a denoiser.
Revised: We observe that denoising is especially helpful at low sample counts, where we obtain similar reconstruction results at 8 spp with denoising, compared to 32 spp without a denoiser.

Original: At highersample counts, however, the benefit from denoising diminishes, as variance in the Monte-Carloestimates decreases.
Revised: At higher sample counts, however, the benefit from denoising diminishes, as variance in the Monte-Carlo estimates decreases.

Original: Given that denoisers enable significantly faster iteration times, we consider it avaluable tool for saving computational resources when fine-tuning model parameters for subsequentruns with high sample counts.
Revised: Given that denoisers enable significantly faster iteration times, we consider it a valuable tool for saving computational resources when fine-tuning model parameters for subsequent runs with high sample counts.

Original: Additionally, we empirically found that using a denoiser duringreconstruction yields higher quality light probes, as can be seen in Figure 9, both at low and highsample counts.
Revised: Additionally, we empirically found that using a denoiser during reconstruction yields higher quality light probes, as can be seen in Figure 9, both at low and high sample counts.

Original: We can also jointly optimize the denoiser parameters, θ , with the 3D scene reconstruction task.i.e.
Revised: We can also jointly optimize the denoiser parameters, θ , with the 3D scene reconstruction task.

Original: , the denoiser is fine-tuned for the current scene.
Revised: i.e., the denoiser is fine-tuned for the current scene.

Original: Unfortunately, this approach has undesirableside-effects: Features tend to get baked into the denoiser network weights instead of the materialsor light probe.
Revised: Unfortunately, this approach has undesirable side-effects: Features tend to get baked into the denoiser network weights instead of the materials or light probe.

Original: This is especially apparent with the OIDN [1] denoiser, which produced color shiftsdue to lack of regularization on the output.
Revised: This is especially apparent with the OIDN [1] denoiser, which produced color shifts due to lack of regularization on the output.

Original: We got notably better results with the hierarchical kernelprediction architecture from Hasselgren et al.
Revised: We got notably better results with the hierarchical kernel prediction architecture from Hasselgren et al.

Original: [22], which is more constrained.
Revised: [22], which is more constrained.

Original: However, the resultsstill lagged behind denoisers with locked weights.
Revised: However, the results still lagged behind denoisers with locked weights.

Original: We refer to the supplemental material for details.
Revised: We refer to the supplemental material for details.

Original:  Priors
Revised:  Priors

Original: In our setting: 3D reconstruction from multi-view images with constant lighting, regularization isessential in order to disentangle lighting and materials.
Revised: In our setting: 3D reconstruction from multi-view images with constant lighting, regularization is essential in order to disentangle lighting and materials.

Original: Following previous work [73, 41], we applysmoothness priors for albedo, specular, and normal map textures.
Revised: Following previous work [75, 41], we apply smoothness priors for albedo, specular, and normal map textures.

Original: Taking the albedo as an example, ifk d ( x ) denotes the diffuse albedo at world space position, x , and ϵ is a small random displacementvector, we define the smoothness prior for albedo as:
Revised: Taking the albedo as an example, if k d ( x ) denotes the diffuse albedo at world space position, x , and ϵ is a small random displacement vector, we define the smoothness prior for albedo as:

Original: [Equation]
Revised: [Equation]

Original: where x surf are the world space positions at the primary hit point on the object.
Revised: where x surf are the world space positions at the primary hit point on the object.

Original: We note that thesmoothness prior is not sufficient to disentangle material parameters and light, especially for sceneswith high frequency lighting and sharp shadows.Optimization tends to bake shadows into the albedo [Table] [Figure]
Revised: We note that the smoothness prior is not sufficient to disentangle material parameters and light, especially for scenes with high frequency lighting and sharp shadows.

Original: NVD

Original: I

Original: [Figure] [Table] [Figure] texture (easy) rather than reconstruct a high intensity, small area in the environment map (hard).
Revised: Optimization tends to bake shadows into the albedo texture (easy) rather than reconstruct a high intensity, small area in the environment map (hard).

Original: Toenable high quality relighting, we explicitly want to enforce shading detail represented by lighting,and only bake remaining details into the material textures.
Revised: To enable high quality relighting, we explicitly want to enforce shading detail represented by lighting, and only bake remaining details into the material textures.

Original: We propose a novel regularizer term thatis surprisingly effective.
Revised: We propose a novel regularizer term that is surprisingly effective.

Original: We compute a monochrome image loss between the demodulated lightingterms and the reference image:
Revised: We compute a monochrome image loss between the demodulated lighting terms and the reference image:

Original: [Equation]
Revised: [Equation]

Original: Here, Y ( x ) =
Revised: Here, Y ( x ) =

Original: ( x r + x g + x b ) / 3 is a simple luminance operator, and V ( x ) = max ( x r , x g , x b ) isthe HSV value component.
Revised: ( x r + x g + x b ) / 3 is a simple luminance operator, and V ( x ) = max ( x r , x g , x b ) is the HSV value component.

Original: The rationale for using HSV-value for the reference image is that themax operation approximates demodulation, e.g., a red and white pixel have identical values.
Revised: The rationale for using HSV-value for the reference image is that the max operation approximates demodulation, e.g., a red and white pixel have identical values.

Original: Weassume that the demodulated lighting is mostly monochrome, in which case Y ( x ) ∼ V ( x ) , andgiven that we need to propagate gradients to c d and c s , Y avoids discontinuities.
Revised: We assume that the demodulated lighting is mostly monochrome, in which case Y ( x ) ∼ V ( x ) , and given that we need to propagate gradients to c d and c s , Y avoids discontinuities.

Original: This regularizer islimited by our inability to demodulate the reference image.
Revised: This regularizer is limited by our inability to demodulate the reference image.

Original: The HSV-value ignores chrominance,but we cannot separate a shadow from a darker material.
Revised: The HSV-value ignores chrominance, but we cannot separate a shadow from a darker material.

Original: This has not been a problem in our tests,but could interfere with optimization if the regularizer is given too much weight.
Revised: This has not been a problem in our tests, but could interfere with optimization if the regularizer is given too much weight.

Original: Please refer to thesupplemental materials for complete regularizer details.
Revised: Please refer to the supplemental materials for complete regularizer details.

Original:  Experiments
Revised:  Experiments

Original: In our experiments, we use NVDIFFREC [41] as a baseline, and refer to their work for thoroughcomparisons against related work.
Revised: In our experiments, we use NVDIFFREC [41] as a baseline, and refer to their work for thorough comparisons against related work.

Original: We focus the evaluation on the quality of material and lightseparation.
Revised: We focus the evaluation on the quality of material and light separation.

Revised: At test time, all view interpolation results are generated without denoising at 2k spp.

Revised: All relighting results are rendered in Blender Cycles at 64 spp with denoising [1].

Original: Table 1 shows a quantitative comparison with NVDIFFREC and NeRFactor [73] onthe NeRFactor relighting setup.
Revised: Table 1 shows a quantitative comparison with NVDIFFREC and NeRFactor [75] on the NeRFactor relighting setup.

Original: Note that there is an indeterminate scale factor between materialreflectance (e.g., albedo) and the light intensity.
Revised: Note that there is an indeterminate scale factor between material reflectance (e.g., albedo) and the [Table] R e f

Original: This is accounted for by scaling each image tomatch the average luminance of the reference for the corresponding scene.
Revised: NVD I FF R E C O u r

Revised: Drums

Revised: Hotdog

Revised: Lego

Revised: Ficus

Revised: Chair

Original: The same methodology
Revised: Hotdog Lego [Figure] [Figure]

Revised: light intensity.

Revised: This is accounted for by each image to match the average luminance of the reference for the corresponding scene.

Original: Our recon.
Revised: The same methodology is applied for all algorithms in our comparisons. We outperform previous work, providing better material reconstruction. Figure 6 shows visual examples. We additionally perform relighting on the synthetic NeRF dataset, which is substantially more challenging than the NeRFactor variant, due to high frequency lighting and global illumination effects.

Original: [Figure]
Revised: NVDIFFREC produces severe artifacts, as exemplified by the Hotdog scene in Figure 6. Tableshows a significant increase in image quality for our approach. The visual examples show that our

Original: F a m il y
Revised: F a m il y

Original: O u r
Revised: O u r

Original: NVD I FF R
Revised: NVD I FF R

Original: E C
Revised: E C

Original: C h a r ac t e r
Revised: C h a r ac t e r

Original: O u r
Revised: O u r

Original: NVD I FF R
Revised: NVD I FF R

Original: E C
Revised: E C

Original: G o l d
Revised: G o l d

Original: C a p e
Revised: C a p e

Original: O u r
Revised: O u r

Original: NVD I FF R
Revised: NVD I FF R

Original: E C
Revised: E C

Original: Ref
Revised: Ref

Original: Our
Revised: Our

Original: Geometry k d k orm
Revised: Geometry k d k orm

Original: Normal
Revised: Normal

Original: Probeis applied for all algorithms in our comparisons. We outperform previous work, providing bettermaterial reconstruction.
Revised: Probe

Original: Figure 6 shows visual examples. We additionally perform relighting on the synthetic NeRF dataset, which is substantially morechallenging than the NeRFactor variant, due to high frequency lighting and global illumination effects. NVDIFFREC produces severe artifacts, as exemplified by the Hotdog scene in Figure 6. Table 1shows a significant increase in image quality for our approach.
Revised: PSNR

Original: The visual examples show that ourresults are plausible, though not without artifacts.
Revised: [Figure] results are plausible, though not without artifacts.

Original: Finally, we constructed a novel synthetic datasetwith three scenes with highly complex geometry to stress-test the system.
Revised: Finally, we constructed a novel synthetic dataset with three scenes with highly complex geometry to stress-test the system.

Original: Each scene contains 200training views and 200 novel views for evaluation.
Revised: Each scene containstraining views and 200 novel views for evaluation.

Original: The three scenes are shown in Figures 1 , 3 ,and 9.
Revised: The three scenes are shown in Figures 1 , 3 ,and 9.

Original: Quantitatively we outperform previous work by a larger margin, and Figure 1 shows very littleshading in the albedo textures.
Revised: Quantitatively we outperform previous work by a larger margin, and Figure 1 shows very little shading in the albedo textures.

Original: In Figures 7 and 8, we apply our method to datasets with real photos.
Revised: In Figures 7 and 8, we apply our method to datasets with real photos.

Original: These sets are more difficult,due to inaccuracies in foreground segmentation masks and camera poses.
Revised: These sets are more difficult, due to inaccuracies in foreground segmentation masks and camera poses.

Original: We extract triangle meshesthat can be trivially edited in 3D modeling software, and in Figure 7 we use Blender to perform sceneediting, material editing, and relighting.
Revised: We extract triangle meshes that can be trivially edited in 3D modeling software, and in Figure 7 we use Blender to perform scene editing, material editing, and relighting.

Original: Note that the results look plausible with the inserted objectbeing properly shaded and casting shadows on the statue, material editing looks good with the learnedenvironment lighting and relighting interacts properly with the learned materials.
Revised: Note that the results look plausible with the inserted object being properly shaded and casting shadows on the statue, material editing looks good with the learned environment lighting and relighting interacts properly with the learned materials.

Original: Figure 8 shows abreakdown of geometry, material parameters and environment light.
Revised: Figure 8 shows a breakdown of geometry, material parameters and environment light.

Original: To study the impact of denoising, we optimized the Porsche scene both with and without denoisingenabled.
Revised: To study the impact of denoising, we optimized the Porsche scene w/ and w/o denoising.

Original: As shown in Figure 9, denoising improves both visual quality and environment lightingdetail at equal sample counts.
Revised: As shown in Figure 9, denoising improves both visual quality and environment lighting detail at equal sample counts.

Original: The noise levels varies throughout the scene, and we note that denoising 
Revised: The noise levels varies throughout the scene, and we note that denoising is particularly helpful in regions with complex lighting or occlusion, such as the specular highlight and cockpit.

Original: [Table] NeRFactor synthetic Nerf synthetic Our synthetic
Revised: Neural light-fields, e.g., Mip-NeRF [4] excel at view interpolation.

Original: Real-world
Revised: We enforce material/light separation through additional regularization, which slightly degrades view interpolation results, as

Original: [Table]
Revised: [Table]

Original: is particularly helpful in regions with complex lighting or occlusion, such as the specular highlightand cockpit. View interpolation quality typically degrades with improved material and light separation, and neurallight-fields, e.g, Mip-NeRF [4] typically performs best. While not our target, we include viewinterpolation results in Table 2.
Revised: shown in Table 2.

Original: Ourview interpolation scores are slightly below NVDIFFREC , but, asshown above,we provide considerably better material and lighting separation.
Revised: Our scores are slightly below NVDIFFREC , but, as shown above, considerably better material and lighting separation.

Original: Compute resources Tracing rays to evaluate the direct illumination is considerably more expensivethan pre-filtered environment light approaches.
Revised: Compute resources Tracing rays to evaluate the direct illumination is considerably more expensive than pre-filtered environment light approaches.

Original: We leverage hardware-accelerated ray intersections,but note that our implementation is far from fully optimized.
Revised: We leverage hardware-accelerated ray intersections, but note that our implementation is far from fully optimized.

Original: Our method scales linearly with samplecount, which gives us a simple way to trade quality for performance.
Revised: Our method scales linearly with sample count, which gives us a simple way to trade quality for performance.

Original: With a batch size 8 at a renderingresolution of 512 × 512, we get the following iteration times on a single NVIDIA A6000 GPU.
Revised: With a batch size 8 at a rendering resolution of 512 × 512, we get the following iteration times on a single NVIDIA A6000 GPU.

Original: NVDIFFREC Our 2 spp Our 8 spp Our 32 spp Our 128 spp Iteration time 340 ms 310 ms 330 ms 490 ms 1050 ms
Revised: [Table]

Original: Our denoising strategies enable optimization at low sample counts.
Revised: Our denoising strategies enable optimization at low sample counts.

Original: Unless otherwise mentioned, forthe results presented in the paper, we use high quality settings of 128+ rays per pixel with 5000 × 2iterations (second pass with fixed topology and 2D textures), which takes ∼ 10 hours (A6000).
Revised: Unless otherwise mentioned, for the results presented in the paper, we use high quality settings of 128+ rays per pixel with 5000 ×iterations (second pass with fixed topology and 2D textures), which takes ∼ 4 hours (A6000).

Original:  Conclusions
Revised:  Conclusions

Original: Our restriction to direct illumination shows up in scenes with global illumination.
Revised: Our restriction to direct illumination shows up in scenes with global illumination.

Original: Similarly, we do nothandle specular chains (geometry seen through a glass window).
Revised: Similarly, we do not handle specular chains (geometry seen through a glass window).

Original: For this, we need to integrate multibounce path tracing, which is a clear avenue for future work, but comes with additional challenges inincreased noise-levels, visibility gradients through specular chains, and drastically increased iterationtimes.
Revised: For this, we need to integrate multibounce path tracing, which is a clear avenue for future work, but comes with additional challenges in increased noise-levels, visibility gradients through specular chains, and drastically increased iteration times.

Original: Our renderer is intentionally biased to improve optimization times, but unbiased renderingcould expect to generate better results for very high sample counts.
Revised: Our renderer is intentionally biased to improve optimization times, but unbiased rendering could expect to generate better results for very high sample counts.

Original: Other limitations include lack ofefficient regularization of material specular parameters and reliance on a foreground segmentationmask.
Revised: Other limitations include lack of efficient regularization of material specular parameters and reliance on a foreground segmentation mask.

Original: Our approach is computationally intense, requiring a high-end GPU for optimization runs.
Revised: Our approach is computationally intense, requiring a high-end GPU for optimization runs.

Original: To summarize, we have shown that differentiable Monte-Carlo rendering combined with variancereduction techniques is practical and applicable to multi-view 3D object reconstruction of explicittriangular 3D models.
Revised: To summarize we have shown that differentiable Monte-Carlo rendering combined with variancereduction techniques is practical and applicable to multi-view 3D object reconstruction of explicit triangular 3D models.

Original: Our physically-based renderer clearly improves material and light reconstruction over previous work.
Revised: Our physically-based renderer clearly improves material and light reconstruction over previous work.

Original: By leveraging hardware accelerated ray-tracing and differentiable imagedenoisers, we remain competitive to previous work in terms of optimization time.
Revised: By leveraging hardware accelerated ray-tracing and differentiable image denoisers, we remain competitive to previous work in terms of optimization time

Original: Checklist1.
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes](b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] See Section 7(c)
Revised: [Yes] See Section(c)

Original: Did you discuss any potential negative societal impacts of your work?
Revised: Did you discuss any potential negative societal impacts of your work?

Original: [No] There isno negative societal impacts particular to this work.
Revised: [No] There is no negative societal impacts particular to this work.

Original: While it shares the problem ofdeepfakes with all other multi-view 3D object reconstruction methods, we do not thinkit warrants a discussion in particular to this paper.
Revised: While it shares the problem of deepfakes with all other multi-view 3D object reconstruction methods, we do not think it warrants a discussion in particular to this paper.

Original: d ) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]2.
Revised: [Yes] 2.

Original: If you are including theoretical results a) Did you state the full set of assumptions of all theoretical results
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or as a URL)?[No] Code is
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: proprietary.

Original: We intend to work towards a release.

Original:  (b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
Revised: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] While it is impossible to detail everything in a large system, we describe the significant updates from NVDIFFREC [41].
Revised: [Yes] While it is impossible to detail everything in a large system, we describe the significant updates from NVDIFFREC [41].

Original: Details about regularization and optimization scheduling is included in supplemental work due to limited space.
Revised: Details about regularization and optimization scheduling is included in supplemental work due to limited space.

Original: (c) Did you report error bars (e g., with respect to the random seed after running ex- periments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [No] While interesting, the optimization time of 3D reconstruction algorithms (our and previous work) is too significant for such detailed analysis.
Revised: [No] While interesting, the optimization time of 3D reconstruction algorithms (our and previous work) is too significant for such detailed analysis.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] Optimization performance / compute resources are described in Section 64.
Revised: [Yes] Optimization performance / compute resources are described in Section 6. 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] The synthetic NeRFactor dataset does not specify a license.
Revised: [Yes] The synthetic NeRFactor dataset does not specify a license.

Original: It is tightly based on the synthetic NeRF dataset (CC-BY-3.0), andbased on the No additional restrictions clause we assume CC-BY-3.0 for NeRFactoras well.
Revised: It is tightly based on the synthetic NeRF dataset (CC-BY-3.0), and based on the No additional restrictions clause we assume CC-BY-3.0 for NeRFactor as well.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No] Our dataset is currently proprietary
Revised: [No] (d) Did you discuss whether

Original: but we hope to be able to release it. (d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 
Revised: [N/A] 5.

Original: No identifiable/offensive data

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]