Original: D 2 NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video
Revised: D 2 NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Given a monocular video, segmenting and decoupling dynamic objects while recovering the static environment is a widely studied problem in machine intelligence.
Revised: Given a monocular video, segmenting and decoupling dynamic objects while recovering the static environment is a widely studied problem in machine intelligence.

Original: Existing solutions usually approach this problem in the image domain, limitingtheir performance and understanding of the environment.
Revised: Existing solutions usually approach this problem in the image domain, limitingtheir performance and understanding of the environment.

Original: We introduce Decoupled
Revised: We introduce Decoupled

Original: Dynamic Neural Radiance Field (D 2 NeRF), a self-supervised approach that takesa monocular video and learns a 3D scene representation which decouples movingobjects, including their shadows, from the static background.
Revised: Dynamic Neural Radiance Field (D 2 NeRF), a self-supervised approach that takesa monocular video and learns a 3D scene representation which decouples movingobjects, including their shadows, from the static background.

Original: Our method represents the moving objects and the static background by two separate neural radiancefields with only one allowing for temporal changes.
Revised: Our method represents the moving objects and the static background by two separate neural radiancefields with only one allowing for temporal changes.

Original: A naive implementation ofthis approach leads to the dynamic component taking over the static one as therepresentation of the former is inherently more general and prone to overfitting.
Revised: A naive implementation ofthis approach leads to the dynamic component taking over the static one as therepresentation of the former is inherently more general and prone to overfitting.

Original: To this end, we propose a novel loss to promote correct separation of phenomena.
Revised: To this end, we propose a novel loss to promote correct separation of phenomena.

Original: We further propose a shadow field network to detect and decouple dynamicallymoving shadows.
Revised: We further propose a shadow field network to detect and decouple dynamicallymoving shadows.

Original: We introduce a new dataset containing various dynamic objects and shadows and demonstrate that our method can achieve better performance thanstate-of-the-art approaches in decoupling dynamic and static 3D objects, occlusionand shadow removal, and image segmentation for moving objects.
Revised: We introduce a new dataset containing various dynamic objects and shadows and demonstrate that our method can achieve better performance thanstate-of-the-art approaches in decoupling dynamic and static 3D objects, occlusionand shadow removal, and image segmentation for moving objects.

Original:  Introduction
Revised:  Introduction

Original: Reasoning about motion is a fundamental task in machine vision which facilitates intelligent interactions with the 3D environment for applications such as robotics and autonomous driving.
Revised: Reasoning about motion is a fundamental task in machine vision which facilitates intelligent interactions with the 3D environment for applications such as robotics and autonomous driving.

Original: Givena monocular RGB video captured from a moving casual camera, we consider the problem of disentangling the camera from object motion and simultaneously recovering a 3D model of the staticenvironment.
Revised: Givena monocular RGB video captured from a moving casual camera, we consider the problem of disentangling the camera from object motion and simultaneously recovering a 3D model of the staticenvironment.

Original: While decomposition of scenes in the image domain has been addressed in the literature, the useof 2D priors and inpainting technique lacks 3D understanding, leading to sub-optimal results.
Revised: While decomposition of scenes in the image domain has been addressed in the literature, the useof 2D priors and inpainting technique lacks 3D understanding, leading to sub-optimal results.

Original: Weapproach this problem in 3D, aiming to reconstruct a decoupled 3D scene representation that allowsfor synthesizing the dynamic and static objects separately in a free-view and time-varying fashion.
Revised: Weapproach this problem in 3D, aiming to reconstruct a decoupled 3D scene representation that allowsfor synthesizing the dynamic and static objects separately in a free-view and time-varying fashion.

Original: Compared to the task of static scene reconstruction [29], modeling a scene with time-dependenteffects is a severely ill-posed problem.
Revised: Compared to the task of static scene reconstruction [32], modeling a scene with time-dependenteffects is a severely ill-posed problem.

Original: Existing works seek for robust solutions by incorporatingadditional supervision such as multi-view capture [23], optical flow [9], or depth [55], but they treatevery part of the scene as time-dependent, leading to a poor reconstruction of background details dueto limited network capacity.
Revised: Existing works seek for robust solutions by incorporatingadditional supervision such as multi-view capture [25], optical flow [10], or depth [60], but they treatevery part of the scene as time-dependent, leading to a poor reconstruction of background details dueto limited network capacity.

Original: In this paper, we adapt Neural Radiance Fields [29] (NeRF) and its extension HyperNeRF [36] totime-varying scenes by decoupling the dynamic and static components of the scene into separateradiance fields.
Revised: In this paper, we adapt Neural Radiance Fields [32] (NeRF) and its extension HyperNeRF [40] totime-varying scenes by decoupling the dynamic and static components of the scene into separateradiance fields.

Original: Previous techniques that decouple dynamic and static scenes either rely on pre-trained object detection/segmentation modules [18, 24, 11, 22], or are limited to a single rigid object [63]or semi-static objects [50].
Revised: Previous techniques that decouple dynamic and static scenes either rely on pre-trained object detection/segmentation modules [19, 26, 12, 23], or are limited to a single rigid object [68]or semi-static objects [54].

Original: Our method learns dynamic and static components separately in aself-supervised fashion, using a novel skewed-entropy loss to encourage a clean separation of staticand dynamic objects.
Revised: Our method learns dynamic and static components separately in aself-supervised fashion, using a novel skewed-entropy loss to encourage a clean separation of staticand dynamic objects.

Original: A crucial issue in creating a clean separation is properly handling shadows , as dynamic objects castshadows that cause the radiance of the shadow receiver to vary with time.
Revised: A crucial issue in creating a clean separation is properly handling shadows , as dynamic objects castshadows that cause the radiance of the shadow receiver to vary with time.

Original: When the shadow receiveris part of the static component, this time-varying change in radiance cannot be directly modeled.
Revised: When the shadow receiveris part of the static component, this time-varying change in radiance cannot be directly modeled.

Original: Oursolution is to relax the static component with a time-varying shadow field that modulates the radiance,allowing the shadows cast by moving objects to be captured while constraining density and color tobe static.
Revised: Oursolution is to relax the static component with a time-varying shadow field that modulates the radiance,allowing the shadows cast by moving objects to be captured while constraining density and color tobe static.

Original: Our method enables 3D scene decoupling and reconstruction from a monocular video captured fromcasual equipment such as a mobile phone, and can be readily extended to multi-view videos.
Revised: Our method enables 3D scene decoupling and reconstruction from a monocular video captured fromcasual equipment such as a mobile phone, and can be readily extended to multi-view videos.

Original: Byseparately modeling the time-varying and time-independent targets in the video, our method canremove the dynamic occluders and their shadows, and synthesize a clean background from novelviews.
Revised: Byseparately modeling the time-varying and time-independent targets in the video, our method canremove the dynamic occluders and their shadows, and synthesize a clean background from novelviews.

Original: We demonstrate the effectiveness of our method in two aspects : (i) the quality of novel view synthesisof the decoupled static background for monocular videos where the dynamic objects and shadowsheavily occlude the scene, and (ii) the correctness of segmentation of dynamic objects and shadowson 2D images.
Revised: We demonstrate the effectiveness of our method in two aspects : (i) the quality of novel view synthesisof the decoupled static background for monocular videos where the dynamic objects and shadowsheavily occlude the scene, and (ii) the correctness of segmentation of dynamic objects and shadowson 2D images.

Original: We introduce a new dataset with rigid and non-rigid dynamic objects, rapid camera motion andvarious moving shadows in both the synthetic and real-world settings to evaluate these two aspects,and show that our method achieves better performance than state-of-the-art approaches.
Revised: We introduce a new dataset with rigid and non-rigid dynamic objects, rapid camera motion andvarious moving shadows in both the synthetic and real-world settings to evaluate these two aspects,and show that our method achieves better performance than state-of-the-art approaches.

Original:  Related Work
Revised:  Related Work

Original: As our method learns a decoupled neural 3D representation of the dynamic and static scenes, westart this section with a review of scene representations, and then focus on methods for object motiondecoupling.
Revised: As our method learns a decoupled neural 3D representation of the dynamic and static scenes, westart this section with a review of scene representations, and then focus on methods for object motiondecoupling.

Original: We also review prior works for 2D segmentation of moving objects.
Revised: We also review prior works for 2D segmentation of moving objects.

Original: Scene Representations A 3D scene representation is a data structure that encodes the geometry and appearance of a 3D scene, upon which many algorithms and applications are developed.
Revised: Scene Representations A 3D scene representation is a data structure that encodes the geometry and appearance of a 3D scene, upon which many algorithms and applications are developed.

Original: Recently, there has been a surge of methods that combine deep learning methods with traditional3D representations: point clouds [16, 37], meshes [7, 30], voxels [6, 54, 10, 1, 51, 3], implicit surfaces [34, 58, 14, 13, 47], and light fields [46, 48, 2, 44].
Revised: Recently, there has been a surge of methods that combine deep learning methods with traditional3D representations: point clouds [17, 41, 11, 1], meshes [7, 33, 3, 55], voxels [6, 59], implicitsurfaces [38, 63, 15, 14, 51], light fields [50, 52, 2, 48] and volumetric fields [28, 32].

Original: Among neural representations, NeRF [29]has attracted substantial attention due to its photo-realistic performance in novel view synthesisfor scenes with complex geometry, lighting, and materials.
Revised: Among neuralrepresentations, NeRF [32] has attracted substantial attention due to its photo-realistic performancein novel view synthesis for scenes with complex geometry, lighting, and materials.

Original: Via differentiable volume renderingand inputs of multiple views of the scene, NeRF applies an MLP to learn a 5D radiance field ofthe scene modeling the spatially and view-dependent radiance.
Revised: Via differentiablevolume rendering and inputs of multiple views of the scene, NeRF applies an MLP to learn a 5Dradiance field of the scene modeling the spatially and view-dependent radiance.

Original: Various extensions of NeRF havebeen developed to improve its performance and generality such as training with only one or fewviews [62, 17, 21, 32, 41], allowing for input images with inconsistent lighting and object locations [28, 64], learning large-scale scenes with street or satellite views [43, 56], speeding up renderingto reduce training and inference time [26, 8, 40, 42, 61, 12, 31, 25, 49, 53, 60], and capture ofdynamic effects within the scene [23, 39, 9, 55, 35, 36, 20].
Revised: Various extensionsof NeRF have been developed to improve its performance and generality such as training with onlyone or few views [67, 18, 22, 35, 45], allowing for input images with inconsistent lighting and objectlocations [31, 69], learning large-scale scenes with street or satellite views [47, 61], speeding uprendering to reduce training and inference time [29, 8, 44, 46, 66, 13, 34, 27, 53, 58, 65], capture ofdynamic effects within the scene [25, 43, 10, 60, 39, 40, 21, 36], and decomposing self-occludedshadows to recover correct albedo [9].

Original: We further extend NeRF to decoupledynamic from static effects.
Revised: We further extend NeRF to decouple dynamic from staticeffects.

Original: Motion Decoupling Prior works to acquire a decoupled 3D representation of dynamic and staticscenes can be divided into either supervised or self-supervised approaches.
Revised: Motion Decoupling Prior works to acquire a decoupled 3D representation of dynamic and staticscenes can be divided into either supervised or self-supervised approaches.

Original: Among the supervisedapproaches, STNeRF [18] learns individual NeRFs with deformation fields for each human ina dynamic scene through pre-trained human segmentation networks.
Revised: Among the supervisedapproaches, STNeRF [19] learns individual NeRFs with deformation fields for each human ina dynamic scene through pre-trained human segmentation networks.

Original: Similarly, NSFF [24] and
Revised: Similarly, NSFF [26] and

Original: DynNeRF [11] rely on pre-trained semantic and motion segmentation methods to obtain masks formoving objects in a monocular video, and explicitly guide the training of separate NeRF networksto decouple the scene based on motion.
Revised: DynNeRF [12] rely on pre-trained semantic and motion segmentation methods to obtain masks formoving objects in a monocular video, and explicitly guide the training of separate NeRF networksto decouple the scene based on motion.

Original: Among the self-supervised approaches, SIMONe [19]incorporates a transformer encoder and variational autodecoder to simultaneously recover novelviews, object segmentation masks and dynamic object trajectories, but they do not allow for asynthesis of dynamic or static objects alone.
Revised: Among the self-supervised approaches, SIMONe [20]incorporates a transformer encoder and variational autodecoder to simultaneously recover novelviews, object segmentation masks and dynamic object trajectories, but they do not allow for asynthesis of dynamic or static objects alone.

Original: NeRF-W [28] employs per-frame embeddings to modelnon-photometric consistent effects in unconstrained photo collections, but their design was not intended for a clean separation between moving objects and the static scene.
Revised: NeRF-W [31] employs per-frame embeddings to model non-photometric consistent effects in unconstrained photo collections, but their design was notintended for a clean separation between moving objects and the static scene.

Original: STaR [63] reconstructsand decomposes a rigid dynamic object and the static background simultaneously by optimizing two
Revised: STaR [68] reconstructsand decomposes a rigid dynamic object and the static background simultaneously by optimizing two

Original: NeRFs and a set of time-varying object poses in a self-supervised way, but it is only suitable forscenes with a single rigid dynamic object and requires multi-view videos.
Revised: NeRFs and a set of time-varying object poses in a self-supervised way, but it is only suitable forscenes with a single rigid dynamic object and requires multi-view videos.

Original: Conversely, our approachworks with more complex scenes involving multiple non-rigid and topologically varying objects,and our method can be directly applied from monocular video.
Revised: Conversely, our approachworks with more complex scenes involving multiple non-rigid and topologically varying objects,and our method can be directly applied from monocular video.

Original: NeuralDiff [50] incorporates three
Revised: NeuralDiff [54] incorporates three

Original: NeRF-based streamlines to decompose background, object and actor from an egocentric video,and it is the most similar to ours within the literature.
Revised: NeRF-based streamlines to decompose background, object and actor from an egocentric video,and it is the most similar to ours within the literature.

Original: However, its use of a naive time-varying
Revised: However, its use of a naive time-varying

Original: NeRF architecture leads to blurry results and therefore heavily limits its performance on both scenedecomposition and reconstruction.
Revised: NeRF architecture leads to blurry results and therefore heavily limits its performance on both scenedecomposition and reconstruction.

Original: Image Segmentation of Moving Objects Orthogonal to the reconstruction and disentanglement in3D, there have also been extensive researches in self-supervised and template-free segmentation at theimage level (i.e. 2D).
Revised: Image Segmentation of Moving Objects Orthogonal to the reconstruction and disentanglement in3D, there have also been extensive researches in self-supervised and template-free segmentation at theimage level (i.e. 2D).

Original: The majority relies on motion-clues to segment objects with different opticalflow patterns [4, 59, 33, 57].
Revised: The majority relies on motion-clues to segment objects with different opticalflow patterns [4, 64, 37, 62].

Original: Some techniques incorporate a transformer style slot-based attentionscheme to learn consistent object segmentation over a sequence of optical flow images [59], whileothers learn alpha-matting from a single video with smooth camera movement and homographicbackground and extend the segmentation target to correlated effects such as shadow or reflectance [27].
Revised: Some techniques incorporate a transformer style slot-based attentionscheme to learn consistent object segmentation over a sequence of optical flow images [64], whileothers learn alpha-matting from a single video with smooth camera movement and homographicbackground and extend the segmentation target to correlated effects such as shadow or reflectance [30].

Original: Those approaches come with obvious shortcomings, as they focus on image level segmentation andincorporate no 3D understanding, they cannot handle large scale camera motion, complicated staticbackground and cannot recover 3D geometry, or perform novel view synthesis.
Revised: Those approaches come with obvious shortcomings, as they focus on image level segmentation andincorporate no 3D understanding, they cannot handle large scale camera motion, complicated staticbackground and cannot recover 3D geometry, or perform novel view synthesis.

Original:  Method
Revised:  Method

Original:  Given a monocular video captured from a freely moving hand-held camera, our method reconstructsa neural scene representation that decouples moving objects from the static environment, assuming aconstant illumination and known camera poses (e.g. calibrated with COLMAP [45]).
Revised: [Figure] Given a monocular video captured from a freely moving hand-held camera, our method reconstructsa neural scene representation that decouples moving objects from the static environment, assuming aconstant illumination and known camera poses (e.g. calibrated with COLMAP [49]).

Original: As illustrated in
Revised: As illustrated in

Original: Figure 1, our method achieves this by learning separate radiance fields for static and dynamic portionsof the scene, and doing so in a fully self-supervised fashion.
Revised: Figure 1, our method achieves this by learning separate radiance fields for static and dynamic portionsof the scene, and doing so in a fully self-supervised fashion.

Original: We describe our architecture (Section 3.1),detail of our self-supervised losses (Section 3.2), and describe how, while shadows are not explicitlymodeled by NeRFs, a simple technique for their effective removal is attainable (Section 3.3).
Revised: We describe our architecture (Section 3.1),detail of our self-supervised losses (Section 3.2), and describe how, while shadows are not explicitlymodeled by NeRFs, a simple technique for their effective removal is attainable (Section 3.3).

Original: 3.1 Composite Neural Radiance Field
Revised: 3.1 Composite Neural Radiance Field

Original: The static component builds upon NeRF [29], which represents the scene as continuous spatialdependent density σ and spatial-view-dependent radiance c using an multi-layer perceptron F S :
Revised: The static component builds upon NeRF [32], which represents the scene as continuous spatialdependent density σ and spatial-view-dependent radiance c using an multi-layer perceptron F S :

Original: [Equation]
Revised: [Equation]

Original: where x ∈ R 3 is the spatial coordinate, and d ∈ R 3 , ∥ d ∥ = 1 is the view direction.
Revised: where x ∈ R 3 is the spatial coordinate, and d ∈ R 3 , ∥ d ∥ = 1 is the view direction.

Original: To model thedynamic component of a scene, we adapt HyperNeRF [36], which accurately captures scenes withnon-rigid motion as well as topological changes by introducing additional degree of freedom andnetwork capacity.
Revised: To model thedynamic component of a scene, we adapt HyperNeRF [40], which accurately captures scenes withnon-rigid motion as well as topological changes by introducing additional degree of freedom andnetwork capacity.

Original: For convenience, we denote it as a neural function F D :
Revised: For convenience, we denote it as a neural function F D :

Original: [Equation]
Revised: [Equation]

Original: where τ i ∈
Revised: where τ i ∈

Original: R m is the per-frame time latent code.
Revised: R m is the per-frame time latent code.

Original: Given a camera ray r = o + t d originatingfrom o and with direction d , the two models are then composited to calculate the color ˆ C of thecamera ray by integrating the radiance according to volumetric rendering within a pre-defined depthrange [ t n , t f ] :
Revised: Given a camera ray r = o + t d originatingfrom o and with direction d , the two models are then composited to calculate the color ˆ C of thecamera ray by integrating the radiance according to volumetric rendering within a pre-defined depthrange [ t n , t f ] :

Original: [Equation]
Revised: [Equation]

Original: where we simplify our notation as σ ( t ) ≡ σ ( r ( t )) and c ( t ) ≡ c ( r ( t ) , d ) .
Revised: where we simplify our notation as σ ( t ) ≡ σ ( r ( t )) and c ( t ) ≡ c ( r ( t ) , d ) .

Original: Note that, with such anadditive decomposition, samples from either fields are capable of terminating the camera ray andoccluding the other.
Revised: Note that, with such anadditive decomposition, samples from either fields are capable of terminating the camera ray andoccluding the other.

Original: 3.2 Supervision Losses
Revised: 3.2 Supervision Losses

Original: To find the parameters of the static (Eq. 1) and dynamic (Eq. 2) NeRF networks, a photometric loss isapplied to ensure that the output image sequences of the composite NeRF (Eq. 3) align with the inputvideo frames:
Revised: To find the parameters of the static (Eq. 1) and dynamic (Eq. 2) NeRF networks, a photometric loss isapplied to ensure that the output image sequences of the composite NeRF (Eq. 3) align with the inputvideo frames:

Original: [Equation]
Revised: [Equation]

Original: where C ( r , τ i ) indicates the true color of camera ray r obtained from the i -th input video frame.
Revised: where C ( r , τ i ) indicates the true color of camera ray r obtained from the i -th input video frame.

Original: However, note the dynamic component can naturally take over the static counterpart by incorrectlyassigning occupancy of static objects to dynamic NeRF, and the photometric loss alone also does notguarantee a correct separation.
Revised: However, note the dynamic component can naturally take over the static counterpart by incorrectlyassigning occupancy of static objects to dynamic NeRF, and the photometric loss alone also does notguarantee a correct separation.

Original: In what follows, we design a collection of regularizers that promotesuch decoupling in a self-supervised fashion.
Revised: In what follows, we design a collection of regularizers that promotesuch decoupling in a self-supervised fashion.

Original: Dynamic vs. Static Factorization
Revised: Dynamic vs. Static Factorization

Original: As physical objects cannot co-exist at the same spatial location,a physically realistic solution should have any position in space either occupied by a the static sceneor by a dynamic object, but not both .
Revised: As physical objects cannot co-exist at the same spatial location,a physically realistic solution should have any position in space either occupied by a the static sceneor by a dynamic object, but not both .

Original: To enforce this behavior we denote the spatial ratio of dynamicvs.
Revised: To enforce this behavior we denote the spatial ratio of dynamicvs.

Original: static density as:
Revised: static density as:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: and then penalize its deviation from a categorical { 0 , 1 } distribution via a binary entropy loss [63]:
Revised: and then penalize its deviation from a categorical { 0 , 1 } distribution via a binary entropy loss [68]:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: However, due to the strong expressive power of the dynamic networks (Eq. 2), optimizing theloss (Eq. 7) leads to the technique modeling parts of the scene as dynamic, regardless of whether theyare dynamic or static; see Figure 2 (right).
Revised: However, due to the strong expressive power of the dynamic networks (Eq. 2), optimizing theloss (Eq. 7) leads to the technique modeling parts of the scene as dynamic, regardless of whether theyare dynamic or static; see Figure 2 (right).

Original: To overcome this issue, we propose a skewed entropy lossto bias our loss to slightly favor static explanations of the scene with skewness hyper-parameter k ,that, as illustrated in Figure 2 (left, k> 1 ), attains the desired behavior:
Revised: To overcome this issue, we propose a skewed entropy lossto bias our loss to slightly favor static explanations of the scene with skewness hyper-parameter k ,that, as illustrated in Figure 2 (left, k> 1 ), attains the desired behavior:

Original: [Equation]
Revised: [Equation]

Original: By encouraging a more concentrated density distribution along each camera ray in static component, the recovered background contains less view-dependent artifacts.
Revised: By encouraging a more concentrated density distribution along each camera ray in static component, the recovered background contains less view-dependent artifacts.

Original: Shadow
Revised: Shadow

Original: – When shadows occur frequently in the input data, the average shadow gets integrated in the static component, and the dynamic component incorrectly learns the differential with respect to this average and appears as a brighter surface.
Revised: – When shadows occur frequently in the input data, the average shadow gets integrated in the static component, and the dynamic component incorrectly learns the differential with respect to this average and appears as a brighter surface.

Original: This can be avoided by a more direct modeling of shadow effects as a dynamic darkening of static regions (i.e. the shadow field).
Revised: This can be avoided by a more direct modeling of shadow effects as a dynamic darkening of static regions (i.e. the shadow field).

Original: Ray Regularization Choosing a large value of skewness k causes the appearance of fuzzyfloaters (low-density particles) in the static portion of the scene; see Figure 2 (right, k =10 ).
Revised: Ray Regularization Choosing a large value of skewness k causes the appearance of fuzzyfloaters (low-density particles) in the static portion of the scene; see Figure 2 (right, k =10 ).

Original: As it canbe intuitively understood from Figure 2 (left), this is caused by the small gradients of H b ( x k ) as xapproaches zero.
Revised: As it canbe intuitively understood from Figure 2 (left), this is caused by the small gradients of H b ( x k ) as xapproaches zero.

Original: To mitigate this effect, and reduce fuzziness in the reconstruction, we penalize themaximum of w along each camera ray:
Revised: To mitigate this effect, and reduce fuzziness in the reconstruction, we penalize themaximum of w along each camera ray:

Original: [Equation]
Revised: [Equation]

Original: Such loss can be intuitively interpreted as constraining the dynamic component to occupy as fewpixels as possible while keeping minimal impact on the overall loss for all samples.
Revised: Such loss can be intuitively interpreted as constraining the dynamic component to occupy as fewpixels as possible while keeping minimal impact on the overall loss for all samples.

Original: Note that L r onlyremoves density floaters that sit along camera rays that do not intersect with any dynamic objects.
Revised: Note that L r onlyremoves density floaters that sit along camera rays that do not intersect with any dynamic objects.

Original: Static Regularization We empirically found that static component may abuse the camera poseas the hint for the current time frame and learn dynamic effects as sparse clouds that lead to highfrequency appearance changes; see Figure 3.
Revised: Static Regularization We empirically found that static component may abuse the camera poseas the hint for the current time frame and learn dynamic effects as sparse clouds that lead to highfrequency appearance changes; see Figure 3.

Original: The ambiguity comes from the fact that we are usingmonocular casual videos where the camera almost never visits the exact same position twice duringthe capture.
Revised: The ambiguity comes from the fact that we are usingmonocular casual videos where the camera almost never visits the exact same position twice duringthe capture.

Original: That is, there exists a one-to-one mapping between camera pose and time variable.
Revised: That is, there exists a one-to-one mapping between camera pose and time variable.

Original: Wesolve this issue by imposing a prior on the distribution of density along a ray, penalizing densitydistributions that would cause cloud-like artifacts [21, 41]:
Revised: Wesolve this issue by imposing a prior on the distribution of density along a ray, penalizing densitydistributions that would cause cloud-like artifacts [22, 45]:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Revised: =

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 3.3 Shadow Fields
Revised: 3.3 Shadow Fields

Original: Neural radiance fields cannot faithfully model standalone shadows without significant changes toits architecture necessary to modeling materials and illumination; see NeRFactor [66].
Revised: Neural radiance fields cannot faithfully model standalone shadows without significant changes toits architecture necessary to modeling materials and illumination; see NeRFactor [71].

Original: In simple cases where shadows of the dynamic objects move rapidly, they could alternatively be learned bythe dynamic radiance field as semi-transparent layers on top of the static surface.
Revised: In simple cases where shadows of the dynamic objects move rapidly, they could alternatively be learned by thedynamic radiance field as semi-transparent layers on top of the static surface.

Original: However, thistends to fail for shadows that do not move much, or that are highly correlated with the camera view.
Revised: However, this tendsto fail for shadows that do not move much, or that are highly correlated with the camera motion.

Original: As shadows are texture-less, understanding their movement is ambiguous, and representing themas a semi-transparent layer causes difficulties in the optimization; see Figure 4.
Revised: As shadows are texture-less, understanding their movement is ambiguous, and representing themas a semi-transparent layer causes difficulties in the optimization; see Figure 4.

Original: To overcome thisissue, and under the assumption of a direct illumination model (i.e. negligible global illuminationeffects), we make the observation that a cast shadow can be represented as a pointwise reduction inthe radiance of the the static scene, and incorporate this within Eq.
Revised: To overcome thisissue, and under the assumption of a direct illumination model (i.e. negligible global illuminationeffects), we make the observation that a cast shadow can be represented as a pointwise reduction inthe radiance of the the static scene, and incorporate this within Eq.

Original: 3 as:
Revised: 3 as:

Original: [Equation]
Revised: [Equation]

Original: where ρ ( x , τ i ) is a shadow ratio that scales-down the radiance of the static scene to incorporate theshadow.
Revised: where ρ ( x , τ i ) is a shadow ratio that scales-down the radiance of the static scene to incorporate theshadow.

Original: To avoid the shadow-ratio from over-explaining dark regions of the scene, we penalize itsaverage squared magnitude along a ray:
Revised: To avoid the shadow-ratio from over-explaining dark regions of the scene, we penalize itsaverage squared magnitude along a ray:

Original: [Equation]
Revised: [Equation]

Original: Finally, note that shadows cast from dynamic objects onto other dynamic objects are already expressedfrom the radiance term of the dynamic branch, and do not need explicit modeling.
Revised: Finally, note that shadows cast from dynamic objects onto other dynamic objects are already expressedfrom the radiance term of the dynamic branch, and do not need explicit modeling.

Original:  Experiments
Revised:  Experiments

Original: 4.1 Implementation details
Revised: 4.1 Implementation details

Original: Our method is easily reproducible, as we intend to release code and datasets upon publication tofacilitate future research.
Revised: Our method is easily reproducible, as we intend to release code and datasets upon publication tofacilitate future research.

Original: We adopt the HyperNeRF [36] architecture as the dynamic component,which has a NeRF MLP network of 8 layers, each with 256 channels, and our static NeRF componenthas the same architecture.
Revised: We adopt the HyperNeRF [40] architecture as the dynamic component,which has a NeRF MLP network of 8 layers, each with 256 channels, and our static NeRF componenthas the same architecture.

Original: Similar to NeRF [29], we apply a hierarchical volume sampling with64 coarse and 64 fine samples.
Revised: Similar to NeRF [32], we apply a hierarchical volume sampling with64 coarse and 64 fine samples.

Original: The optimization takes 100 k iterations with batch size 1024 and anexponentially decayed learning rate from 10 − 3 to 10 − 5 .
Revised: The optimization takes 100 k iterations with batch size 1024 and anexponentially decayed learning rate from 10 − 3 to 10 − 5 .

Original: This training procedure spans approximatelytwo hours on four NVIDIA A100-SXM-80GB GPUs.
Revised: This training procedure spans approximatelytwo hours on four NVIDIA A100-SXM-80GB GPUs.

Original: The overall loss of our method is:
Revised: The overall loss of our method is:

Original: [Equation]
Revised: [Equation]

Original: where λ s , λ r , λ σ S , λ ρ are the weights of the regularization terms respectively.
Revised: where λ s , λ r , λ σ S , λ ρ are the weights of the regularization terms respectively.

Original: For scenes with amixture of dynamic objects and shadows, we apply shadow decay and set λ ρ =0 .
Revised: For scenes with amixture of dynamic objects and shadows, we apply shadow decay and set λ ρ =0 .

Original: We set λ ρ =0 .
Revised: We set λ ρ =0 .

Original: 001for scenes featuring view-correlated dynamic shadows only.
Revised: 001for scenes featuring view-correlated dynamic shadows only.

Original: We experimentally found that the optimalchoice of the hyperparameters, especially λ b , λ r and the skewness k , are strongly influenced by thelevel of object motion, camera motion, and video length.
Revised: We experimentally found that the optimalchoice of the hyperparameters, especially λ b , λ r and the skewness k , are strongly influenced by thelevel of object motion, camera motion, and video length.

Original: Therefore, we performed a grid search onour synthetic and held-out real-world scenes, and some scenes from DAVIS [38], to establish a set ofhyperparameters applicable to a variety of scenarios; details about hyperparameters can be foundin the supplementary.
Revised: Therefore, we performed a grid search onour synthetic and held-out real-world scenes, and some scenes from DAVIS [42], to establish a set ofhyperparameters applicable to a variety of scenarios; details about hyperparameters can be foundin the supplementary.

Original: We do not apply shadow field for evaluations on our synthetic scenes, as weempirically found that shadow field is not needed to learn correct shadows.
Revised: We do not apply shadow field for evaluations on our synthetic scenes, as weempirically found that shadow field is not needed to learn correct shadows.

Original: We also disable the viewdirection input for synthetic scenes as they do not contain strong view-dependent effects.
Revised: We also disable the viewdirection input for synthetic scenes as they do not contain strong view-dependent effects.

Original: 4.2 Evaluation
Revised: 4.2 Evaluation

Original: We demonstrate the performance of our method both quantitatively and qualitatively on three tasks.
Revised: We demonstrate the performance of our method both quantitatively and qualitatively on three tasks.

Original: We focus our attention to our main objective of decoupling and removing dynamic objects, includingtheir shadows, with a 3D reconstruction of the static environment.
Revised: We focus our attention to our main objective of decoupling and removing dynamic objects, includingtheir shadows, with a 3D reconstruction of the static environment.

Original: We only include a summary ofour results for novel view synthesis (in Figure 5) and video segmentation (in Figure 6), and refer thereader to the supplementary for a more in-depth discussion.
Revised: We only include a summary ofour results for novel view synthesis (in Figure 5) and video segmentation (in Figure 6), and refer thereader to the Supplementary E and D for a more in-depth discussion.

Original: We strongly encourage  the readers towatch our supplementary video to better appreciate our results.
Revised: We strongly encourage thereaders to watch videos on the project page (https://d2nerf.github.io/) to better appreciate our results.

Original: 4.3 Datasets
Revised: 4.3 Datasets

Original: In addition to the data obtained from HyperNeRF [36] and Nerfies [35], we acquire more complexdatasets in the real-world, as well as design a synthetic dataset to enable quantitative comparisons.
Revised: In addition to the data obtained from HyperNeRF [40] and Nerfies [39], we acquire more complexdatasets in the real-world, as well as design a synthetic dataset to enable quantitative comparisons.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Synthetic dataset We generate a synthetic dataset with ground-truth masks for moving objectsand their shadows with Kubric [15].
Revised: Synthetic dataset We generate a synthetic dataset with ground-truth masks for moving objectsand their shadows with Kubric [16].

Original: This dataset consists of five scenes containing one or multipledynamic objects from ShapeNet [5] with rigid or non-rigid motion, and the corresponding Kubricworker script is provided in our supplementary material.
Revised: This dataset consists of five scenes containing one or multipledynamic objects from ShapeNet [5] with rigid or non-rigid motion, and the corresponding Kubricworker script is provided in our supplementary material.

Original: We move the virtual camera over 10keyframes randomly sampled from azimuth [2 , 2 + π/ 4] and altitude [1 , 1 . 2] to generate a 200-framevideo sequence for training.
Revised: We move the virtual camera over 10keyframes randomly sampled from azimuth [2 , 2 + π/ 4] and altitude [1 , 1 . 2] to generate a 200-framevideo sequence for training.

Original: We also rotate the virtual camera around the center of all keyframesto generate 100 validation views with only the static background being visible.
Revised: We also rotate the virtual camera around the center of all keyframesto generate 100 validation views with only the static background being visible.

Original: We additionallygenerate masks for both the dynamic objects and their shadows, allowing us to quantitatively studythe performance of our algorithm.
Revised: We additionallygenerate masks for both the dynamic objects and their shadows, allowing us to quantitatively studythe performance of our algorithm.

Original: Note that shadows are usually absent in existing moving-objectssegmentation benchmarks.
Revised: Note that shadows are usually absent in existing moving-objectssegmentation benchmarks.

Original: Real-world dataset We also capture ten video sequences of real scenes to showcase our performance.
Revised: Real-world dataset We also capture ten video sequences of real scenes to showcase our performance.

Original: Compared to HyperNeRF’s, our dataset contains more challenging scenarios with rapidmotion and non-trivial dynamic shadows.
Revised: Compared to HyperNeRF’s, our dataset contains more challenging scenarios with rapidmotion and non-trivial dynamic shadows.

Original: Note however that we cannot perform quantitative analysisfor these datasets due to the absence of ground-truth views of a static scene or ground-truth masks.
Revised: Note however that we cannot perform quantitative analysisfor these datasets due to the absence of ground-truth views of a static scene or ground-truth masks.

Original: Five of real scenes are captured with a similar setting as Nerfies, where we use a dual-hold phone rigand synchronize the capture based on audio.
Revised: Five of real scenes are captured with a similar setting as Nerfies, where we use a dual-hold phonerig and synchronize the capture based on audio.

Original: We use the images captured by one of the two phones asvalidation views for novel-view synthesis, which are discussed in the supplementary material. Todemonstrate the ability of fully self-supervised scene decoupling, we do not apply any masks whenregistering real-world images using COLMAP [45].
Revised: We use the images captured by one of the twophones as validation views for novel-view synthesis, which are discussed in the Supplementary E. Todemonstrate the ability of fully self-supervised scene decoupling, we do not apply any masks whenregistering real-world images using COLMAP [49].

Original: 4.4 Scene Decoupling –
Revised: 4.4 Scene Decoupling –

Original: Table1, Figure 

Revised: Table Figure

Original: We report the evaluation of our method on its ability to decouple dynamic objects and their shadows,while recovering the static background.
Revised: We report the evaluation of our method on its ability to decouple dynamic objects and their shadows, while recovering the static background.

Original: We evaluate our performance against NeRF-W [28] and
Revised: We evaluate our performance against NeRF-W [31],

Original:  NeuralDiff [50].
Revised: NSFF [26] and NeuralDiff [54].

Original: ForNeuralDiff, we disabled the actor component (as our input videos are notegocentric) and only use the transient component. For NeRF-W, we used only transient embeddingand disabled the appearance embedding that models variable lighting for evaluation on syntheticscenes, as they have constant illumination. 
Revised: For NeRF-W, we used only transient embedding and disabled theappearance embedding that models variable lighting for evaluation on synthetic scenes, as they haveconstant illumination. For NSFF, we disabled the hard mining initialization via 2D masks input.

Original:  In the evaluation, we used each method to synthesize the static background from multiple validationviews with the moving objects and their shadows removed; see Figure 7 for qualitative results on realdata, as well as our supplementary for qualitative results on synthetic data.
Revised: For NeuralDiff, we disabled the actor component (as our input videos are not egocentric) and only usethe transient component. In the evaluation, we used each method to synthesize the static background from multiple validationviews with the moving objects and their shadows removed; see Figure 7 for qualitative results on realdata, as well as Supplementary C for qualitative results on synthetic data.

Original: We compare the resultswith the ground truth on the synthetic data and report LPIPS [65], Multi-Scale SSIM [52], and PSNRas the metrics for novel view synthesis of the decoupled static background; see Table 1.
Revised: We compare the resultswith the ground truth on the synthetic data and report LPIPS [70], Multi-Scale SSIM [57], and PSNRas the metrics for novel view synthesis of the decoupled static background; see Table 1.

Revised: Note thatalthough NSFF similarly learns separate dynamic and static NeRFs, it is not designed to maximizethe performance of scene decoupling and does not incorporate regularization on the dynamic NeRF.

Revised: Therefore, it is extremely unstable and can fail completely by learning everything as dynamic, leavingstatic component as an empty scene.

Original: 4.5 Ablations –
Revised: 4.5 Ablations –

Original: Table 2, Figure 8, Figure 

Revised: Table 2, Figure 8, Figure

Original: We quantitatively ablate our method on our synthetic dataset; see Table 2: where "skew" meansskewness is applied in the binary entropy regularization, and " L r " stands for the density ratio rayregularization.
Revised: We quantitatively ablate our method on our synthetic dataset; see Table 2: where "skew" meansskewness is applied in the binary entropy regularization, and " L r " stands for the density ratio rayregularization.

Original: We also qualitatively ablate it on a real scene; see Figure 8.
Revised: We also qualitatively ablate it on a real scene; see Figure 8.

Original: We also qualitativelyillustrate the ablation on shadow field network, which is necessary for decoupling shadows withlarge area, slow or repetitive motion, or shadows that are highly correlated with the camera view;see Figure 9.[Table]
Revised: We also qualitatively [Table] illustrate on shadow field network, which is necessary for decoupling shadows withlarge area, slow or repetitive motion, or shadows that are highly correlated with the camera view;see Figure 9.

Original:  Conclusions
Revised:  Conclusions

Original: We presented D 2 NeRF, a method for self-supervised 3D scene decoupling and reconstruction fromcasual monocular videos.
Revised: We presented D 2 NeRF, a method for self-supervised 3D scene decoupling and reconstruction fromcasual monocular videos.

Original: Our method decouples occluders and correlated shadows, recovers cleanbackground representations and enables high quality novel views synthesis.
Revised: Our method decouples occluders and correlated shadows, recovers cleanbackground representations, and enables high quality novel views synthesis.

Original: Our novel skewedentropy regularizer is critical to separate dynamic from static components of the scene, while ourshadow-field allows for the removal of dynamic shadows without having to explicitly model theinteraction between light and geometry.
Revised: Our novel skewedentropy regularizer is critical to separate dynamic from static components of the scene, while ourshadow-field allows for the removal of dynamic shadows without having to explicitly model theinteraction between light and geometry.

Original: We demonstrate superior results for multiple tasks on existingdatasets, as well as on two datasets that we introduce alongside our technique.
Revised: We demonstrate superior results for multiple tasks on existingdatasets, as well as on two datasets that we introduce alongside our technique.

Original: Limitations Similar to many NeRF-based methods, our approach relies on accurate camera registration to achieve success decoupling and reconstruction.
Revised: Limitations Similar to many NeRF-based methods, our approach relies on accurate camera registration to achieve success decoupling and reconstruction.

Original: Our approach also suffers from highfrequency view-dependent radiance change, such as those caused by the presence of reflective surfacewithin the scene. Due to the monocular moving camera setting, those effects could be misinterpretedas dynamic effects, resulting in incorrect decoupling. Removing texture-less target that repeatedly
Revised: Our approach also suffers from high

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Our  is able to remove large area of shadows even if they are strongly correlated with the view direction (e.g., shadow cast by camera or the photographer)
Revised: Our method is able to remove large area of shadows even if they are strongly correlated with the view direction (e.g., shadow cast by camera or the photographer).

Original: Note that the appearance embedding from NeRF-W [28] is not sufficient to remove shadow that is present throughout the capture; see the supplementary for additional qualitative results stays around the same position for the majority of the time during the video, our method is unable to fully decouple and remove the texture-less shadow. 
Revised: Note that the appearance embedding from NeRF-W [31] is not sufficient to remove shadow that is present throughout the capture; see the our website for additional qualitative results. hand stays around the same position for the majority of the time during the video, our method is unable to fully decouple and remove the texture-less shadow. frequency view-dependent radiance change, such as those caused by the presence of reflective surfacewithin the scene.

Revised: Due to the monocular moving camera setting, those effects could be misinterpretedas dynamic effects, resulting in incorrect decoupling.

Original: moves within a very small range is also difficult, as the motion clues are extremely ambiguous in thiscase; see Figure 10.
Revised: Removing texture-less target that repeatedlymoves within a very small range is also difficult, as the motion clues are extremely ambiguous in thiscase; see Figure 10.

Original: Checklist
Revised: Supplementary

Original: The checklist follows the references. Please read the checklist guidelines carefully for information onhow to answer these questions. For each question, change the default [TODO] to [Yes] , [No] , or[N/A] . You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
Revised: Material

Original: For example: • Did you include the license to the code and datasets? [Yes] See Section.•
Revised: All coda and data, as well as additional video results can be found on our project page: d2nerf.github.io.

Original: Did you include the license to the code and datasets? [No] The code and the data areproprietary.• Did you include the license to the code and datasets?
Revised: We also include a static copy of the code and website as zip files in the supplementary.

Original: [N/A] Please do not modify the questions and only use the provided macros for your answers. Note that the 
Revised: As we do not have large TPUs available for training, we incorporate a light-weighted HyperNeRF[40] as the dynamic component to reduce training time.

Original: Checklist section does not count towards the page limit. In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
Revised: Compared to the original hyperparametersdescribed in the paper, we reduce the number of samples per ray (64 vs. 128 in [40]), batch size(1024 vs. 6144 in [40]), and number of iterations (100k vs. 250k in [40]).

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: We also do not applythe background regularization, as it requires a set of known background 3D points, which rely onaccurate dynamic masks during COLMAP registration.

Original: contributions and scope? [Yes](b) Did you describe the limitations of your work?
Revised: We experimentally established a set of hyperparameters applicable for a variety of scenes.

Original: [Yes] See Section 5(c) Did you discuss any potential negative societal impacts of your work? [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: In total,we used five sets of hyperparameters for the evaluation on the real-world dataset, and four on thesynthetic dataset.

Original: [Yes]2. If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: To ensure the various scenes are fully separated into different components, weincrease λ s during training, where → indicates the value is linearly increased and ⇒ indicates it isexponentially increased. - entry for λ ρ indicates that the shadow field is not applied.

Original: [N/A]
Revised: [Table]

Original: (b) Did you include complete proofs of all theoretical results? [N/A] If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: C Scene Decoupling – Figure 11, Figure 13, Figure

Original: mental results (either in the supplemental material or as a URL)? [Yes] See supplemen-
Revised: We demonstrate additional qualitative results on scene decoupling task on both real-world andsynthetic scenes; see Figure 11, 12, 13, and

Original: tary (b) Did you specify all the training details (e g., data splits, hyperparameters,
Revised: As our method learns a density distribution of the dynamic objects in the scene, we can further producean alpha mask of the objects.

Original: how they were chosen)? [Yes] See Section 4.
Revised: We therefore also evaluate the correctness of object segmentation atthe image level.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No] Repeated evaluations not applicable due to the limitation in amount of computational resources required for training
Revised: Existing benchmarks on video segmentation [42, 56, 24] either contain too fewvideo frames for reliable SfM reconstruction, or do not have correct ground truth masks for all ofthe dynamic objects and effects in the scene.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)? [Yes] See Section 4.
Revised: Similarly, the dataset from NeuralDiff [54] focuses onegocentric videos and the over-exaggerated difference between frames in the videos is not suitablefor HyperNeRF which we use as the dynamic component.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: Hence, to highlight the ability of ourmethod to decouple dynamic objects and shadows from video sequences with large viewpoint shifts, we evaluate on our synthetic dataset.

Original: [Yes] See Section 4.
Revised: In addition to the NeRF-like baselines, we also compare with

Original: (b) Did you mention the license of the assets?
Revised: Motion Grouping (MG) [64], a motion-based 2D image segmentation method.

Original: [Yes] See Section 4. (c) Did you include any new assets either in thesupplemental material or as a URL?
Revised: We fine-tuned the

Original: [Yes]
Revised: [Table]

Original: See supplementary
Revised: pre-trained MG model on each scene for 5k iterations.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating? [Yes] Consent obtained from license (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: For other NeRF-based methods, we used thesame settings as in Section 4.4 and produced the alpha masks as the normalized radiance weightsof the time-varying component, and then applied a threshold of 0.1 to obtain the binary masks.

Original: information or offensive content?
Revised: See Table 4, Figure 15 for the results.

Original: [N/A] Only personally identifiable information would
Revised: Although the aim of our method is not to improve the quality of time-varying scene reconstruction,as a by-product, we find that by introducing a static component to fully utilize the network capacity,our method achieves more robust reconstruction for both the dynamic objects and background. Wetherefore also evaluate our method on the ability to synthesize the whole scene from novel views.

Original: be the "curls" dataset, which is obtained from Nerfies [35] and used under their license. If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: We compare several approaches for dynamic scene reconstruction, including NeuralDiff [54] and abaseline version of HyperNeRF [40], which has the same architecture as our dynamic component. Since our method uses an additional static component and naturally has more network parametersand capacity, we also compare with a fair version of HyperNeRF with roughly the same number oftotal parameters by extending the NeRF MLP width to 375.

Original: [N/A]
Revised: Unlike novel view experiments in [40],we do not interleave between two cameras, but use only the right camera as training view and the left where the dynamic object moves rapidly or training/validation views differ largely, HyperNeRF [40] fails to reconstruct a reasonable shape for the dynamic object, while ours might potentially predict a shifted object pose, but can still render the view with high fidelity. We additionally show the decoupled dynamic object from our method. The quality is slightly degraded compared to the decoupling results in Figure 7 and 11 as we are rendering from the more challenging novel views.

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: camera as validation view. This presents greater challenges to all the methods.

Original: Board (IRB) approvals, if applicable?
Revised: See Table 5, Figure 16for the results.

Original: [N/A]
Revised: [Table]

Original: (c) Did you include the estimated hourly wage paid to participants and  the total amountspent on participant compensation?
Revised: The shadow field network represents the density-less shadows in a more physically realistic way, andresolves the ambiguity in their motion. However, the aim of our method is to achieve decouplingof dynamic occluders from the static environment, and we do not over-extend to consider furtherdecoupling between objects and shadows, which would require more priors related to environmentallighting conditions and background texture. We empirically found that shadow field is not necessary for scenes with strong and fast-movingshadows, where they can be directly learned by the dynamic component as thin layers on top of thestatic geometry; see Figure 17.

Original: [N/A]
Revised: On the other hand, there exists unsolvable ambiguity between theshadow and dynamic object, especially for which with a similar or darker color to the background,and hence could be potentially explained as a moving shadow instead of an actual 3D shape dueto our monocular camera setting; see Figure 18. As the later case causes failed dynamic geometryreconstruction, leading to a severe decrease in novel view synthesis performance, we deliberatelychoose a large value of λ ρ to suppress the shadow field for scenes containing a mixture of dynamicobjects and shadows. Although this potentially favors the former case and causes more shadows to beinterpreted as thin-layers, we found that such setting has minimal impact on performance of bothscene decoupling and reconstruction, and is still sufficient to achieve correct shadow decoupling, asthe shadow field resolves the ambiguity in shadow in very early stage of the training. though such representation could potentially represent more than just shadows, it still tends to exclude unnecessary texture from static background and learns only the darkening effects.