Original: D 2 NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video
Revised: D 2 NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video

Original: Abstract
Revised: Abstract

Original: Given a monocular video, segmenting and decoupling dynamic objects while recovering the static environment is a widely studied problem in machine intelligence.
Revised: Given a monocular video, segmenting and decoupling dynamic objects while recovering the static environment is a widely studied problem in machine intelligence.

Original: Existing solutions usually approach this problem in the image domain, limiting their performance and understanding of the environment.
Revised: Existing solutions usually approach this problem in the image domain, limiting their performance and understanding of the environment.

Original: We introduce Decoupled
Revised: We introduce Decoupled

Original: Dynamic Neural Radiance Field (D 2 NeRF), a self-supervised approach that takes a monocular video and learns a 3D scene representation which decouples moving objects, including their shadows, from the static background.
Revised: Dynamic Neural Radiance Field (D 2 NeRF), a self-supervised approach that takes a monocular video and learns a 3D scene representation which decouples moving objects, including their shadows, from the static background.

Original: Our method represents the moving objects and the static background by two separate neural radiance fields with only one allowing for temporal changes.
Revised: Our method represents the moving objects and the static background by two separate neural radiance fields with only one allowing for temporal changes.

Original: A naive implementation of this approach leads to the dynamic component taking over the static one as the representation of the former is inherently more general and prone to overfitting.
Revised: A naive implementation of this approach leads to the dynamic component taking over the static one as the representation of the former is inherently more general and prone to overfitting.

Original: To this end, we propose a novel loss to promote correct separation of phenomena.
Revised: To this end, we propose a novel loss to promote correct separation of phenomena.

Original: We further propose a shadow field network to detect and decouple dynamically moving shadows.
Revised: We further propose a shadow field network to detect and decouple dynamically moving shadows.

Original: We introduce a new dataset containing various dynamic objects and shadows and demonstrate that our method can achieve better performance than state-of-the-art approaches in decoupling dynamic and static 3D objects, occlusion and shadow removal, and image segmentation for moving objects.
Revised: We introduce a new dataset containing various dynamic objects and shadows and demonstrate that our method can achieve better performance than state-of-the-art approaches in decoupling dynamic and static 3D objects, occlusion and shadow removal, and image segmentation for moving objects.

Original: Project page: d2nerf.github.io
Revised: Project page: d2nerf.github.io

Original:  Introduction
Revised:  Introduction

Original: Reasoning about motion is a fundamental task in machine vision which facilitates intelligent interactions with the 3D environment for applications such as robotics and autonomous driving.
Revised: Reasoning about motion is a fundamental task in machine vision which facilitates intelligent interactions with the 3D environment for applications such as robotics and autonomous driving.

Original: Given a monocular RGB video captured from a moving casual camera we consider the problem of disentangling the camera from object motion and simultaneously recovering a 3D model of the static environment.
Revised: Given a monocular RGB video captured from a moving casual camera we consider the problem of disentangling the camera from object motion and simultaneously recovering a 3D model of the static environment.

Original: While decomposition of scenes in the image domain has been addressed in the literature the use of 2D priors and inpainting technique lacks 3D understanding, leading to sub-optimal results.
Revised: While decomposition of scenes in the image domain has been addressed in the literature the use of 2D priors and inpainting technique lacks 3D understanding, leading to sub-optimal results.

Original: We approach this problem in 3D aiming to reconstruct a decoupled 3D scene representation that allows for synthesizing the dynamic and static objects separately in a free-view and time-varying fashion.
Revised: We approach this problem in 3D aiming to reconstruct a decoupled 3D scene representation that allows for synthesizing the dynamic and static objects separately in a free-view and time-varying fashion.

Original: Compared to the task of static scene reconstruction [32], modeling a scene with time-dependent effects is a severely ill-posed problem.
Revised: Compared to the task of static scene reconstruction [32], modeling a scene with time-dependent effects is a severely ill-posed problem.

Original: Existing works seek for robust solutions by incorporating additional supervision such as multi-view capture [25], optical flow [10], or depth [60], but they treat every part of the scene as time-dependent, leading to a poor reconstruction of background details due to limited network capacity.
Revised: Existing works seek for robust solutions by incorporating additional supervision such as multi-view capture [25], optical flow [10], or depth [60], but they treat every part of the scene as time-dependent, leading to a poor reconstruction of background details due to limited network capacity.

Original: In this paper, we adapt Neural Radiance Fields [32] (NeRF) and its extension HyperNeRF
Revised: In this paper, we adapt Neural Radiance Fields [32] (NeRF) and its extension HyperNeRF

Original: [40] to time-varying scenes by decoupling the dynamic and static components of the scene into separate radiance fields.
Revised: [40] to time-varying scenes by decoupling the dynamic and static components of the scene into separate radiance fields.

Original: Previous techniques that decouple dynamic and static scenes either rely on pre-trained object detection/segmentation modules [19, 26, 12, 23], or are limited to a single rigid object [68] or semi-static objects [54].
Revised: Previous techniques that decouple dynamic and static scenes either rely on pre-trained object detection/segmentation modules [19, 26, 12, 23], or are limited to a single rigid object [68] or semi-static objects [54].

Original: Our method learns dynamic and static components separately in a self-supervised fashion, using a novel skewed-entropy loss to encourage a clean separation of static and dynamic objects.
Revised: Our method learns dynamic and static components separately in a self-supervised fashion, using a novel skewed-entropy loss to encourage a clean separation of static and dynamic objects.

Original: A crucial issue in creating a clean separation is properly handling shadows , as dynamic objects cast shadows that cause the radiance of the shadow receiver to vary with time.
Revised: A crucial issue in creating a clean separation is properly handling shadows , as dynamic objects cast shadows that cause the radiance of the shadow receiver to vary with time.

Original: When the shadow receiver is part of the static component, this time-varying change in radiance cannot be directly modeled.
Revised: When the shadow receiver is part of the static component, this time-varying change in radiance cannot be directly modeled.

Original: Our solution is to relax the static component with a time-varying shadow field that modulates the radiance, allowing the shadows cast by moving objects to be captured while constraining density and color to be static.
Revised: Our solution is to relax the static component with a time-varying shadow field that modulates the radiance, allowing the shadows cast by moving objects to be captured while constraining density and color to be static.

Original: Our method enables 3D scene decoupling and reconstruction from a monocular video captured from casual equipment such as a mobile phone, and can be readily extended to multi-view videos.
Revised: Our method enables 3D scene decoupling and reconstruction from a monocular video captured from casual equipment such as a mobile phone, and can be readily extended to multi-view videos.

Original: By separately modeling the time-varying and time-independent targets in the video, our method can remove the dynamic occluders and their shadows, and synthesize a clean background from novel views.
Revised: By separately modeling the time-varying and time-independent targets in the video, our method can remove the dynamic occluders and their shadows, and synthesize a clean background from novel views.

Original: We demonstrate the effectiveness of our method in two aspects : (i) the quality of novel view synthesis of the decoupled static background for monocular videos where the dynamic objects and shadows heavily occlude the scene, and (ii) the correctness of segmentation of dynamic objects and shadows on 2D images.
Revised: We demonstrate the effectiveness of our method in two aspects : (i) the quality of novel view synthesis of the decoupled static background for monocular videos where the dynamic objects and shadows heavily occlude the scene, and (ii) the correctness of segmentation of dynamic objects and shadows on 2D images.

Original: We introduce a new dataset with rigid and non-rigid dynamic objects, rapid camera motion and various moving shadows in both the synthetic and real-world settings to evaluate these two aspects, and show that our method achieves better performance than state-of-the-art approaches.
Revised: We introduce a new dataset with rigid and non-rigid dynamic objects, rapid camera motion and various moving shadows in both the synthetic and real-world settings to evaluate these two aspects, and show that our method achieves better performance than state-of-the-art approaches.

Original:  Related Work
Revised:  Related Work

Original: As our method learns a decoupled neural 3D representation of the dynamic and static scenes, we start this section with a review of scene representations, and then focus on methods for object motion decoupling.
Revised: As our method learns a decoupled neural 3D representation of the dynamic and static scenes, we start this section with a review of scene representations, and then focus on methods for object motion decoupling.

Original: We also review prior works for 2D segmentation of moving objects.
Revised: We also review prior works for 2D segmentation of moving objects.

Original: Scene Representations A 3D scene representation is a data structure that encodes the geometry and appearance of a 3D scene, upon which many algorithms and applications are developed.
Revised: Scene Representations A 3D scene representation is a data structure that encodes the geometry and appearance of a 3D scene, upon which many algorithms and applications are developed.

Original: Recently, there has been a surge of methods that combine deep learning methods with traditional 3D representations: point clouds [17, 41, 11, 1], meshes [7, 33, 3, 55], voxels [6, 59], implicit surfaces [38, 63, 15, 14, 51], light fields [50, 52, 2, 48] and volumetric fields [28, 32].
Revised: Recently, there has been a surge of methods that combine deep learning methods with traditional 3D representations: point clouds [17, 41, 11, 1], meshes [7, 33, 3, 55], voxels [6, 59], implicit surfaces [38, 63, 15, 14, 51], light fields [50, 52, 2, 48] and volumetric fields [28, 32].

Original: Among neural representations, NeRF [32] has attracted substantial attention due to its photo-realistic performance in novel view synthesis for scenes with complex geometry, lighting, and materials.
Revised: Among neural representations, NeRF [32] has attracted substantial attention due to its photo-realistic performance in novel view synthesis for scenes with complex geometry, lighting, and materials.

Original: Via differentiable volume rendering and inputs of multiple views of the scene, NeRF applies an MLP to learn a 5D radiance field of the scene modeling the spatially and view-dependent radiance.
Revised: Via differentiable volume rendering and inputs of multiple views of the scene, NeRF applies an MLP to learn a 5D radiance field of the scene modeling the spatially and view-dependent radiance.

Original: Various extensions of NeRF have been developed to improve its performance and generality such as training with only one or few views [67, 18, 22, 35, 45], allowing for input images with inconsistent lighting and object locations [31, 69], learning large-scale scenes with street or satellite views [47, 61], speeding up rendering to reduce training and inference time [29, 8, 44, 46, 66, 13, 34, 27, 53, 58, 65], capture of dynamic effects within the scene [25, 43, 10, 60, 39, 40, 21, 36], and decomposing self-occluded shadows to recover correct albedo [9].
Revised: Various extensions of NeRF have been developed to improve its performance and generality such as training with only one or few views [67, 18, 22, 35, 45], allowing for input images with inconsistent lighting and object locations [31, 69], learning large-scale scenes with street or satellite views [47, 61], speeding up rendering to reduce training and inference time [29, 8, 44, 46, 66, 13, 34, 27, 53, 58, 65], capture of dynamic effects within the scene [25, 43, 10, 60, 39, 40, 21, 36], and decomposing self-occluded shadows to recover correct albedo [9].

Original: We further extend NeRF to decouple dynamic from static effects.
Revised: We further extend NeRF to decouple dynamic from static effects.

Original: Motion Decoupling Prior works to acquire a decoupled 3D representation of dynamic and static scenes can be divided into either supervised or self-supervised approaches.
Revised: Motion Decoupling Prior works to acquire a decoupled 3D representation of dynamic and static scenes can be divided into either supervised or self-supervised approaches.

Original: Among the supervised approaches, STNeRF [19] learns individual NeRFs with deformation fields for each human in a dynamic scene through pre-trained human segmentation networks.
Revised: Among the supervised approaches, STNeRF [19] learns individual NeRFs with deformation fields for each human in a dynamic scene through pre-trained human segmentation networks.

Original: Similarly, NSFF [26] and
Revised: Similarly, NSFF [26] and

Original: DynNeRF [12] rely on pre-trained semantic and motion segmentation methods to obtain masks for moving objects in a monocular video, and explicitly guide the training of separate NeRF networks to decouple the scene based on motion.
Revised: DynNeRF [12] rely on pre-trained semantic and motion segmentation methods to obtain masks for moving objects in a monocular video, and explicitly guide the training of separate NeRF networks to decouple the scene based on motion.

Original: Among the self-supervised approaches, SIMONe [20] incorporates a transformer encoder and variational autodecoder to simultaneously recover novel views, object segmentation masks and dynamic object trajectories, but they do not allow for a synthesis of dynamic or static objects alone.
Revised: Among the self-supervised approaches, SIMONe [20] incorporates a transformer encoder and variational autodecoder to simultaneously recover novel views, object segmentation masks and dynamic object trajectories, but they do not allow for a synthesis of dynamic or static objects alone.

Original: NeRF-W [31] employs per-frame embeddings to model non-photometric consistent effects in unconstrained photo collections, but their design was not intended for a clean separation between moving objects and the static scene.
Revised: NeRF-W [31] employs per-frame embeddings to model non-photometric consistent effects in unconstrained photo collections, but their design was not intended for a clean separation between moving objects and the static scene.

Original: STaR [68] reconstructs and decomposes a rigid dynamic object and the static background simultaneously by optimizing two
Revised: STaR [68] reconstructs and decomposes a rigid dynamic object and the static background simultaneously by optimizing two

Original: NeRFs and a set of time-varying object poses in a self-supervised way, but it is only suitable for scenes with a single rigid dynamic object and requires multi-view videos.
Revised: NeRFs and a set of time-varying object poses in a self-supervised way, but it is only suitable for scenes with a single rigid dynamic object and requires multi-view videos.

Original: Conversely, our approach works with more complex scenes involving multiple non-rigid and topologically varying objects, and our method can be directly applied from monocular video.
Revised: Conversely, our approach works with more complex scenes involving multiple non-rigid and topologically varying objects, and our method can be directly applied from monocular video.

Original: NeuralDiff [54] incorporates three
Revised: NeuralDiff [54] incorporates three

Original: NeRF-based streamlines to decompose background, object and actor from an egocentric video, and it is the most similar to ours within the literature.
Revised: NeRF-based streamlines to decompose background, object and actor from an egocentric video, and it is the most similar to ours within the literature.

Original: However, its use of a naive time-varying
Revised: However, its use of a naive time-varying

Original: NeRF architecture leads to blurry results and therefore heavily limits its performance on both scene decomposition and reconstruction.
Revised: NeRF architecture leads to blurry results and therefore heavily limits its performance on both scene decomposition and reconstruction.

Original: Image Segmentation of Moving Objects Orthogonal to the reconstruction and disentanglement in 3D, there have also been extensive researches in self-supervised and template-free segmentation at the image level (i.e. 2D).
Revised: Image Segmentation of Moving Objects Orthogonal to the reconstruction and disentanglement in 3D, there have also been extensive researches in self-supervised and template-free segmentation at the image level (i.e. 2D).

Original: The majority relies on motion-clues to segment objects with different optical flow patterns [4, 64, 37, 62].
Revised: The majority relies on motion-clues to segment objects with different optical flow patterns [4, 64, 37, 62].

Original: Some techniques incorporate a transformer style slot-based attention scheme to learn consistent object segmentation over a sequence of optical flow images [64], while others learn alpha-matting from a single video with smooth camera movement and homographic background and extend the segmentation target to correlated effects such as shadow or reflectance [30].
Revised: Some techniques incorporate a transformer style slot-based attention scheme to learn consistent object segmentation over a sequence of optical flow images [64], while others learn alpha-matting from a single video with smooth camera movement and homographic background and extend the segmentation target to correlated effects such as shadow or reflectance [30].

Original: Those approaches come with obvious shortcomings, as they focus on image level segmentation and incorporate no 3D understanding, they cannot handle large scale camera motion, complicated static background and cannot recover 3D geometry, or perform novel view synthesis.
Revised: Those approaches come with obvious shortcomings, as they focus on image level segmentation and incorporate no 3D understanding, they cannot handle large scale camera motion, complicated static background and cannot recover 3D geometry, or perform novel view synthesis.

Original:  Method
Revised:  Method

Original: Given a monocular video captured from a freely moving hand-held camera, our method reconstructs a neural scene representation that decouples moving objects from the static environment, assuming a constant illumination and known camera poses (e.g. calibrated with COLMAP [49]).
Revised: Given a monocular video captured from a freely moving hand-held camera, our method reconstructs a neural scene representation that decouples moving objects from the static environment, assuming a constant illumination and known camera poses (e.g. calibrated with COLMAP [49]).

Original: As illustrated in
Revised: As illustrated in

Original: Figure 1, our method achieves this by learning separate radiance fields for static and dynamic portions of the scene, and doing so in a fully self-supervised fashion.
Revised: Figure 1, our method achieves this by learning separate radiance fields for static and dynamic portions of the scene, and doing so in a fully self-supervised fashion.

Original: We describe our architecture (Section 3.1), detail of our self-supervised losses (Section 3.2), and describe how, while shadows are not explicitly modeled by NeRFs, a simple technique for their effective removal is attainable (Section 3.3).
Revised: We describe our architecture (Section 3.1), detail of our self-supervised losses (Section 3.2), and describe how, while shadows are not explicitly modeled by NeRFs, a simple technique for their effective removal is attainable (Section 3.3).

Original: 3.1 Composite Neural Radiance Field
Revised: 3.1 Composite Neural Radiance Field

Original: The static component builds upon NeRF [32], which represents the scene as continuous spatialdependent density σ and spatial-view-dependent radiance c using an multi-layer perceptron F S :
Revised: The static component builds upon NeRF [32], which represents the scene as continuous spatialdependent density σ and spatial-view-dependent radiance c using an multi-layer perceptron F S :

Original: [Equation]
Revised: [Equation]

Original: where x ∈ R 3 is the spatial coordinate, and d ∈ R 3 , ∥ d ∥ = 1 is the view direction.
Revised: where x ∈ R 3 is the spatial coordinate, and d ∈ R 3 , ∥ d ∥ = 1 is the view direction.

Original: To model the dynamic component of a scene, we adapt HyperNeRF [40], which accurately captures scenes with non-rigid motion as well as topological changes by introducing additional degree of freedom and network capacity.
Revised: To model the dynamic component of a scene, we adapt HyperNeRF [40], which accurately captures scenes with non-rigid motion as well as topological changes by introducing additional degree of freedom and network capacity.

Original: For convenience, we denote it as a neural function F D :
Revised: For convenience, we denote it as a neural function F D :

Original: [Equation]
Revised: [Equation]

Original: where τ i ∈
Revised: where τ i ∈

Original: R m is the per-frame time latent code.
Revised: R m is the per-frame time latent code.

Original: Given a camera ray r = o + t d originating from o and with direction d , the two models are then composited to calculate the color ˆ C of the camera ray by integrating the radiance according to volumetric rendering within a pre-defined depth range [ t n , t f ] :
Revised: Given a camera ray r = o + t d originating from o and with direction d , the two models are then composited to calculate the color ˆ C of the camera ray by integrating the radiance according to volumetric rendering within a pre-defined depth range [ t n , t f ] :

Original: [Equation]
Revised: [Equation]

Original: where we simplify our notation as σ ( t ) ≡ σ ( r ( t )) and c ( t ) ≡ c ( r ( t ) , d ) .
Revised: where we simplify our notation as σ ( t ) ≡ σ ( r ( t )) and c ( t ) ≡ c ( r ( t ) , d ) .

Original: Note that, with such an additive decomposition, samples from either fields are capable of terminating the camera ray and occluding the other.
Revised: Note that, with such an additive decomposition, samples from either fields are capable of terminating the camera ray and occluding the other.

Original: 3.2 Supervision Losses
Revised: 3.2 Supervision Losses

Original: To find the parameters of the static (Eq. 1) and dynamic (Eq. 2) NeRF networks, a photometric loss is applied to ensure that the output image sequences of the composite NeRF (Eq. 3) align with the input video frames:
Revised: To find the parameters of the static (Eq. 1) and dynamic (Eq. 2) NeRF networks, a photometric loss is applied to ensure that the output image sequences of the composite NeRF (Eq. 3) align with the input video frames:

Original: [Equation]
Revised: [Equation]

Original: where C ( r , τ i ) indicates the true color of camera ray r obtained from the i -th input video frame.
Revised: where C ( r , τ i ) indicates the true color of camera ray r obtained from the i -th input video frame.

Original: However, note the dynamic component can naturally take over the static counterpart by incorrectly assigning occupancy of static objects to dynamic NeRF, and the photometric loss alone also does not guarantee a correct separation.
Revised: However, note the dynamic component can naturally take over the static counterpart by incorrectly assigning occupancy of static objects to dynamic NeRF, and the photometric loss alone also does not guarantee a correct separation.

Original: In what follows, we design a collection of regularizers that promote such decoupling in a self-supervised fashion.
Revised: In what follows, we design a collection of regularizers that promote such decoupling in a self-supervised fashion.

Original: Dynamic vs. Static Factorization
Revised: Dynamic vs. Static Factorization

Original: As physical objects cannot co-exist at the same spatial location, a physically realistic solution should have any position in space either occupied by a the static scene or by a dynamic object, but not both .
Revised: As physical objects cannot co-exist at the same spatial location, a physically realistic solution should have any position in space either occupied by a the static scene or by a dynamic object, but not both .

Original: To enforce this behavior we denote the spatial ratio of dynamic vs. static density as:
Revised: To enforce this behavior we denote the spatial ratio of dynamic vs. static density as:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: and then penalize its deviation from a categorical { 0 , 1 } distribution via a binary entropy loss [68]:
Revised: and then penalize its deviation from a categorical { 0 , 1 } distribution via a binary entropy loss [68]:

Original: [Equation]
Revised: [Equation]

Original: By encouraging a more concentrated density distribution along each camera ray in static component, the recovered background contains less view-dependent artifacts.
Revised: By encouraging a more concentrated density distribution along each camera ray in static component, the recovered background contains less view-dependent artifacts.

Original: Shadow ambiguity – When shadows occur frequently in the input data, the average shadow gets integrated in the static component, and the dynamic component incorrectly learns the differential with respect to this average and appears as a brighter surface.
Revised: Shadow ambiguity – When shadows occur frequently in the input data, the average shadow gets integrated in the static component, and the dynamic component incorrectly learns the differential with respect to this average and appears as a brighter surface.

Original: This can be avoided by a more direct modeling of shadow effects as a dynamic darkening of static regions (i.e. the shadow field).
Revised: This can be avoided by a more direct modeling of shadow effects as a dynamic darkening of static regions (i.e. the shadow field).

Original: However, due to the strong expressive power of the dynamic networks (Eq. 2), optimizing the loss (Eq. 7) leads to the technique modeling parts of the scene as dynamic, regardless of whether they are dynamic or static; see Figure 2 (right).
Revised: However, due to the strong expressive power of the dynamic networks (Eq. 2), optimizing the loss (Eq. 7) leads to the technique modeling parts of the scene as dynamic, regardless of whether they are dynamic or static; see Figure 2 (right).

Original: To overcome this issue, we propose a skewed entropy loss to bias our loss to slightly favor static explanations of the scene with skewness hyper-parameter k , that, as illustrated in Figure 2 (left, k> 1 ), attains the desired behavior:
Revised: To overcome this issue, we propose a skewed entropy loss to bias our loss to slightly favor static explanations of the scene with skewness hyper-parameter k , that, as illustrated in Figure 2 (left, k> 1 ), attains the desired behavior:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Ray Regularization Choosing a large value of skewness k causes the appearance of fuzzy floaters (low-density particles) in the static portion of the scene; see Figure 2 (right, k =10 ).
Revised: Ray Regularization Choosing a large value of skewness k causes the appearance of fuzzy floaters (low-density particles) in the static portion of the scene; see Figure 2 (right, k =10 ).

Original: As it can be intuitively understood from Figure 2 (left), this is caused by the small gradients of H b ( x k ) as x approaches zero.
Revised: As it can be intuitively understood from Figure 2 (left), this is caused by the small gradients of H b ( x k ) as x approaches zero.

Original: To mitigate this effect, and reduce fuzziness in the reconstruction, we penalize the maximum of w along each camera ray:
Revised: To mitigate this effect, and reduce fuzziness in the reconstruction, we penalize the maximum of w along each camera ray:

Original: [Equation]
Revised: [Equation]

Original: Such loss can be intuitively interpreted as constraining the dynamic component to occupy as few pixels as possible while keeping minimal impact on the overall loss for all samples.
Revised: Such loss can be intuitively interpreted as constraining the dynamic component to occupy as few pixels as possible while keeping minimal impact on the overall loss for all samples.

Original: Note that L r only removes density floaters that sit along camera rays that do not intersect with any dynamic objects.
Revised: Note that L r only removes density floaters that sit along camera rays that do not intersect with any dynamic objects.

Original: Static Regularization We empirically found that static component may abuse the camera pose as the hint for the current time frame and learn dynamic effects as sparse clouds that lead to highfrequency appearance changes; see Figure 3.
Revised: Static Regularization We empirically found that static component may abuse the camera pose as the hint for the current time frame and learn dynamic effects as sparse clouds that lead to highfrequency appearance changes; see Figure 3.

Original: The ambiguity comes from the fact that we are using monocular casual videos where the camera almost never visits the exact same position twice during the capture.
Revised: The ambiguity comes from the fact that we are using monocular casual videos where the camera almost never visits the exact same position twice during the capture.

Original: That is, there exists a one-to-one mapping between camera pose and time variable.
Revised: That is, there exists a one-to-one mapping between camera pose and time variable.

Original: We solve this issue by imposing a prior on the distribution of density along a ray, penalizing density distributions that would cause cloud-like artifacts [22, 45]:
Revised: We solve this issue by imposing a prior on the distribution of density along a ray, penalizing density distributions that would cause cloud-like artifacts [22, 45]:

Original: [Equation]
Revised: [Equation]

Original: 3.3 Shadow Fields
Revised: 3.3 Shadow Fields

Original: Neural radiance fields cannot faithfully model standalone shadows without significant changes to its architecture necessary to modeling materials and illumination; see NeRFactor [71].
Revised: Neural radiance fields cannot faithfully model standalone shadows without significant changes to its architecture necessary to modeling materials and illumination; see NeRFactor [71].

Original: In simple cases where shadows of the dynamic objects move rapidly, they could alternatively be learned by the dynamic radiance field as semi-transparent layers on top of the static surface.
Revised: In simple cases where shadows of the dynamic objects move rapidly, they could alternatively be learned by the dynamic radiance field as semi-transparent layers on top of the static surface.

Original: However, this tends to fail for shadows that do not move much, or that are highly correlated with the camera motion.
Revised: However, this tends to fail for shadows that do not move much, or that are highly correlated with the camera motion.

Original: As shadows are texture-less, understanding their movement is ambiguous, and representing them as a semi-transparent layer causes difficulties in the optimization; see Figure 4.
Revised: As shadows are texture-less, understanding their movement is ambiguous, and representing them as a semi-transparent layer causes difficulties in the optimization; see Figure 4.

Original: To overcome this issue, and under the assumption of a direct illumination model (i.e. negligible global illumination effects), we make the observation that a cast shadow can be represented as a pointwise reduction in the radiance of the the static scene, and incorporate this within Eq.
Revised: To overcome this issue, and under the assumption of a direct illumination model (i.e. negligible global illumination effects), we make the observation that a cast shadow can be represented as a pointwise reduction in the radiance of the the static scene, and incorporate this within Eq.

Original: 3 as:
Revised: 3 as:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (13) where ρ ( x , τ i ) is a shadow ratio that scales-down the radiance of the static scene to incorporate the shadow.
Revised: (13) where ρ ( x , τ i ) is a shadow ratio that scales-down the radiance of the static scene to incorporate the shadow.

Original: To avoid the shadow-ratio from over-explaining dark regions of the scene, we penalize its average squared magnitude along a ray:
Revised: To avoid the shadow-ratio from over-explaining dark regions of the scene, we penalize its average squared magnitude along a ray:

Original: [Equation]
Revised: [Equation]

Original: Finally, note that shadows cast from dynamic objects onto other dynamic objects are already expressed from the radiance term of the dynamic branch, and do not need explicit modeling.
Revised: Finally, note that shadows cast from dynamic objects onto other dynamic objects are already expressed from the radiance term of the dynamic branch, and do not need explicit modeling.

Original:  Experiments 4.1 Implementation details
Revised:  Experiments 4.1 Implementation details

Original: Our method is easily reproducible, as we intend to release code and datasets upon publication to facilitate future research.
Revised: Our method is easily reproducible, as we intend to release code and datasets upon publication to facilitate future research.

Original: We adopt the HyperNeRF [40] architecture as the dynamic component, which has a NeRF MLP network of 8 layers, each with 256 channels, and our static NeRF component has the same architecture.
Revised: We adopt the HyperNeRF [40] architecture as the dynamic component, which has a NeRF MLP network of 8 layers, each with 256 channels, and our static NeRF component has the same architecture.

Original: Similar to NeRF [32], we apply a hierarchical volume sampling with 64 coarse and 64 fine samples.
Revised: Similar to NeRF [32], we apply a hierarchical volume sampling with 64 coarse and 64 fine samples.

Original: The optimization takes 100 k iterations with batch size 1024 and an exponentially decayed learning rate from 10 − 3 to 10 − 5 .
Revised: The optimization takes 100 k iterations with batch size 1024 and an exponentially decayed learning rate from 10 − 3 to 10 − 5 .

Original: This training procedure spans approximately two hours on four NVIDIA A100-SXM-80GB GPUs.
Revised: This training procedure spans approximately two hours on four NVIDIA A100-SXM-80GB GPUs.

Original: The overall loss of our method is:
Revised: The overall loss of our method is:

Original: [Equation]
Revised: [Equation]

Original: where λ s , λ r , λ σ S , λ ρ are the weights of the regularization terms respectively.
Revised: where λ s , λ r , λ σ S , λ ρ are the weights of the regularization terms respectively.

Original: For scenes with a mixture of dynamic objects and shadows, we apply shadow decay and set λ ρ =0 .
Revised: For scenes with a mixture of dynamic objects and shadows, we apply shadow decay and set λ ρ =0 .

Original: We set λ ρ =0 .for
Revised: We set λ ρ =0 .for

Original: scenes featuring view-correlated dynamic shadows only.
Revised: scenes featuring view-correlated dynamic shadows only.

Original: We experimentally found that the optimal choice of the hyperparameters, especially λ b , λ r and the skewness k , are strongly influenced by the level of object motion, camera motion, and video length.
Revised: We experimentally found that the optimal choice of the hyperparameters, especially λ b , λ r and the skewness k , are strongly influenced by the level of object motion, camera motion, and video length.

Original: Therefore, we performed a grid search on our synthetic and held-out real-world scenes, and some scenes from DAVIS [42], to establish a set of hyperparameters applicable to a variety of scenarios; details about hyperparameters can be found in the supplementary.
Revised: Therefore, we performed a grid search on our synthetic and held-out real-world scenes, and some scenes from DAVIS [42], to establish a set of hyperparameters applicable to a variety of scenarios; details about hyperparameters can be found in the supplementary.

Original: We do not apply shadow field for evaluations on our synthetic scenes, as we empirically found that shadow field is not needed to learn correct shadows.
Revised: We do not apply shadow field for evaluations on our synthetic scenes, as we empirically found that shadow field is not needed to learn correct shadows.

Original: We also disable the view direction input for synthetic scenes as they do not contain strong view-dependent effects.
Revised: We also disable the view direction input for synthetic scenes as they do not contain strong view-dependent effects.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 4.2 Evaluation
Revised: 4.2 Evaluation

Original: We demonstrate the performance of our method both quantitatively and qualitatively on three tasks.
Revised: We demonstrate the performance of our method both quantitatively and qualitatively on three tasks.

Original: We focus our attention to our main objective of decoupling and removing dynamic objects, including their shadows, with a 3D reconstruction of the static environment.
Revised: We focus our attention to our main objective of decoupling and removing dynamic objects, including their shadows, with a 3D reconstruction of the static environment.

Original: We only include a summary of our results for novel view synthesis (in Figure 5) and video segmentation (in Figure 6), and refer the reader to the Supplementary E and D for a more in-depth discussion.
Revised: We only include a summary of our results for novel view synthesis (in Figure 5) and video segmentation (in Figure 6), and refer the reader to the Supplementary E and D for a more in-depth discussion.

Original: We strongly encourage the readers to watch videos on the project page (https://d2nerf.github.io/) to better appreciate our results.
Revised: We strongly encourage the readers to watch videos on the project page (https://d2nerf.github.io/) to better appreciate our results.

Original: 4.3 Datasets
Revised: 4.3 Datasets

Original: In addition to the data obtained from HyperNeRF [40] and Nerfies [39], we acquire more complex datasets in the real-world, as well as design a synthetic dataset to enable quantitative comparisons.
Revised: In addition to the data obtained from HyperNeRF [40] and Nerfies [39], we acquire more complex datasets in the real-world, as well as design a synthetic dataset to enable quantitative comparisons.

Original: Synthetic dataset We generate a synthetic dataset with ground-truth masks for moving objects and their shadows with Kubric [16].
Revised: Synthetic dataset We generate a synthetic dataset with ground-truth masks for moving objects and their shadows with Kubric [16].

Original: This dataset consists of five scenes containing one or multiple dynamic objects from ShapeNet [5] with rigid or non-rigid motion, and the corresponding Kubric worker script is provided in our supplementary material.
Revised: This dataset consists of five scenes containing one or multiple dynamic objects from ShapeNet [5] with rigid or non-rigid motion, and the corresponding Kubric worker script is provided in our supplementary material.

Original: We move the virtual camera overkeyframes randomly sampled from azimuth [2 , 2 + π/ 4] and altitude [1 , 1 . 2] to generate a 200-frame video sequence for training.
Revised: We move the virtual camera overkeyframes randomly sampled from azimuth [2 , 2 + π/ 4] and altitude [1 , 1 . 2] to generate a 200-frame video sequence for training.

Original: We also rotate the virtual camera around the center of all keyframes to generate 100 validation views with only the static background being visible.
Revised: We also rotate the virtual camera around the center of all keyframes to generate 100 validation views with only the static background being visible.

Original: We additionally generate masks for both the dynamic objects and their shadows, allowing us to quantitatively study the performance of our algorithm.
Revised: We additionally generate masks for both the dynamic objects and their shadows, allowing us to quantitatively study the performance of our algorithm.

Original: Note that shadows are usually absent in existing moving-objects segmentation benchmarks.
Revised: Note that shadows are usually absent in existing moving-objects segmentation benchmarks.

Original: Real-world dataset We also capture ten video sequences of real scenes to showcase our performance.
Revised: Real-world dataset We also capture ten video sequences of real scenes to showcase our performance.

Original: Compared to HyperNeRF’s, our dataset contains more challenging scenarios with rapid motion and non-trivial dynamic shadows.
Revised: Compared to HyperNeRF’s, our dataset contains more challenging scenarios with rapid motion and non-trivial dynamic shadows.

Original: Note however that we cannot perform quantitative analysis for these datasets due to the absence of ground-truth views of a static scene or ground-truth masks.
Revised: Note however that we cannot perform quantitative analysis for these datasets due to the absence of ground-truth views of a static scene or ground-truth masks.

Original: Five of real scenes are captured with a similar setting as Nerfies, where we use a dual-hold phone rig and synchronize the capture based on audio.
Revised: Five of real scenes are captured with a similar setting as Nerfies, where we use a dual-hold phone rig and synchronize the capture based on audio.

Original: We use the images captured by one of the two phones as validation views for novel-view synthesis, which are discussed in the Supplementary E.
Revised: We use the images captured by one of the two phones as validation views for novel-view synthesis, which are discussed in the Supplementary E.

Original: To demonstrate the ability of fully self-supervised scene decoupling, we do not apply any masks when registering real-world images using COLMAP [49].
Revised: To demonstrate the ability of fully self-supervised scene decoupling, we do not apply any masks when registering real-world images using COLMAP [49].

Original: 4.4 Scene Decoupling – Table 1, Figure
Revised: 4.4 Scene Decoupling – Table 1, Figure

Original: We report the evaluation of our method on its ability to decouple dynamic objects and their shadows, while recovering the static background.
Revised: We report the evaluation of our method on its ability to decouple dynamic objects and their shadows, while recovering the static background.

Original: We evaluate our performance against NeRF-W [31],
Revised: We evaluate our performance against NeRF-W [31],

Original: NSFF [26] and NeuralDiff [54].
Revised: NSFF [26] and NeuralDiff [54].

Original: For NeRF-W, we used only transient embedding and disabled the appearance embedding that models variable lighting for evaluation on synthetic scenes, as they have constant illumination.
Revised: For NeRF-W, we used only transient embedding and disabled the appearance embedding that models variable lighting for evaluation on synthetic scenes, as they have constant illumination.

Original: For NSFF, we disabled the hard mining initialization via 2D masks input.
Revised: For NSFF, we disabled the hard mining initialization via 2D masks input.

Original: For
Revised: For

Original: NeuralDiff, we disabled the actor component (as our input videos are not egocentric) and only use the transient component.
Revised: NeuralDiff, we disabled the actor component (as our input videos are not egocentric) and only use the transient component.

Original: [Table]
Revised: [Table]

Original: In the evaluation, we used each method to synthesize the static background from multiple validation views with the moving objects and their shadows removed; see Figure 7 for qualitative results on real data, as well as Supplementary C for qualitative results on synthetic data.
Revised: In the evaluation, we used each method to synthesize the static background from multiple validation views with the moving objects and their shadows removed; see Figure 7 for qualitative results on real data, as well as Supplementary C for qualitative results on synthetic data.

Original: We compare the results with the ground truth on the synthetic data and report LPIPS [70], Multi-Scale SSIM [57], and PSNR as the metrics for novel view synthesis of the decoupled static background; see Table 1.
Revised: We compare the results with the ground truth on the synthetic data and report LPIPS [70], Multi-Scale SSIM [57], and PSNR as the metrics for novel view synthesis of the decoupled static background; see Table 1.

Original: Note that although NSFF similarly learns separate dynamic and static NeRFs, it is not designed to maximize the performance of scene decoupling and does not incorporate regularization on the dynamic NeRF.
Revised: Note that although NSFF similarly learns separate dynamic and static NeRFs, it is not designed to maximize the performance of scene decoupling and does not incorporate regularization on the dynamic NeRF.

Original: Therefore, it is extremely unstable and can fail completely by learning everything as dynamic, leaving static component as an empty scene.
Revised: Therefore, it is extremely unstable and can fail completely by learning everything as dynamic, leaving static component as an empty scene.

Original: 4.5 Ablations – Table 2, Figure 8, Figure
Revised: 4.5 Ablations – Table 2, Figure 8, Figure

Original: We quantitatively ablate our method on our synthetic dataset; see Table 2: where "skew" means skewness is applied in the binary entropy regularization, and " L r " stands for the density ratio ray regularization.
Revised: We quantitatively ablate our method on our synthetic dataset; see Table 2: where "skew" means skewness is applied in the binary entropy regularization, and " L r " stands for the density ratio ray regularization.

Original: We also qualitatively ablate it on a real scene; see Figure 8.
Revised: We also qualitatively ablate it on a real scene; see Figure 8.

Original: We also qualitatively illustrate the ablation on shadow field network, which is necessary for decoupling shadows with
Revised: We also qualitatively illustrate the ablation on shadow field network, which is necessary for decoupling shadows with

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: are strongly shadow camera or photographer)
Revised: are strongly shadow camera or photographer)

Original: Note that the appearance embedding from NeRF-W [31] is not sufficient to remove shadow that is present throughout the capture see the our website for additional qualitative results large area, slow or repetitive motion, or shadows that are highly correlated with the camera view; see Figure 9.
Revised: Note that the appearance embedding from NeRF-W [31] is not sufficient to remove shadow that is present throughout the capture see the our website for additional qualitative results large area, slow or repetitive motion, or shadows that are highly correlated with the camera view; see Figure 9.

Original:  Conclusions
Revised:  Conclusions

Original: We presented D 2 NeRF, a method for self-supervised 3D scene decoupling and reconstruction from casual monocular videos.
Revised: We presented D 2 NeRF, a method for self-supervised 3D scene decoupling and reconstruction from casual monocular videos.

Original: Our method decouples occluders and correlated shadows, recovers clean background representations, and enables high quality novel views synthesis.
Revised: Our method decouples occluders and correlated shadows, recovers clean background representations, and enables high quality novel views synthesis.

Original: Our novel skewed entropy regularizer is critical to separate dynamic from static components of the scene, while our shadow-field allows for the removal of dynamic shadows without having to explicitly model the interaction between light and geometry.
Revised: Our novel skewed entropy regularizer is critical to separate dynamic from static components of the scene, while our shadow-field allows for the removal of dynamic shadows without having to explicitly model the interaction between light and geometry.

Original: We demonstrate superior results for multiple tasks on existing datasets, as well as on two datasets that we introduce alongside our technique.
Revised: We demonstrate superior results for multiple tasks on existing datasets, as well as on two datasets that we introduce alongside our technique.

Original: Limitations Similar to many NeRF-based methods, our approach relies on accurate camera registration to achieve success decoupling and reconstruction.
Revised: Limitations Similar to many NeRF-based methods, our approach relies on accurate camera registration to achieve success decoupling and reconstruction.

Original: Our approach also suffers from high frequency view-dependent radiance change, such as those caused by the presence of reflective surface within the scene.
Revised: Our approach also suffers from high frequency view-dependent radiance change, such as those caused by the presence of reflective surface within the scene.

Original: Due to the monocular moving camera setting, those effects could be misinterpreted as dynamic effects, resulting in incorrect decoupling.
Revised: Due to the monocular moving camera setting, those effects could be misinterpreted as dynamic effects, resulting in incorrect decoupling.

Original: Removing texture-less target that repeatedly moves within a very small range is also difficult, as the motion clues are extremely ambiguous in this case; see Figure 10.
Revised: Removing texture-less target that repeatedly moves within a very small range is also difficult, as the motion clues are extremely ambiguous in this case; see Figure 10.

Original: We also note that the skewed binary entropy is not the only loss that suits the purpose of separating two NeRF components while suppressing one of them.
Revised: We also note that the skewed binary entropy is not the only loss that suits the purpose of separating two NeRF components while suppressing one of them.

Original: Many losses such as the beta distribution from [28] and Laplacian distribution from [45] can behave similarly by incorporating an exponential coefficient.
Revised: Many losses such as the beta distribution from [28] and Laplacian distribution from [45] can behave similarly by incorporating an exponential coefficient.

Original: The main difference between our skewed entropy loss and beta or Laplacian distributions is that the former has a vanishing gradient when approaching 0, while the gradients of the latter become infinity.
Revised: The main difference between our skewed entropy loss and beta or Laplacian distributions is that the former has a vanishing gradient when approaching 0, while the gradients of the latter become infinity.

Original: Choosing an appropriate form of loss could reduce the complexity of hyperparameters without affecting the decoupling accuracy.
Revised: Choosing an appropriate form of loss could reduce the complexity of hyperparameters without affecting the decoupling accuracy.

Original: Acknowledgements
Revised: Acknowledgements

Original: This work was supported by a UKRI Future Leaders Fellowship [grant number G104084].
Revised: This work was supported by a UKRI Future Leaders Fellowship [grant number G104084].

Original: We thank
Revised: We thank

Original: Ziwen Cui and Ningding Wang for their help with data collection.
Revised: Ziwen Cui and Ningding Wang for their help with data collection.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information on how to answer these questions.
Revised: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Section.
Revised: [Yes] See Section.

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [No] The code and the data are proprietary.
Revised: [No] The code and the data are proprietary.

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] See Section(c)
Revised: [Yes] See Section(c)

Original: Did you discuss any potential negative societal impacts of your work?
Revised: Did you discuss any potential negative societal impacts of your work?

Original: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 3.
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] See Supplementary A (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: [Yes] See Supplementary A (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See Section 4.
Revised: [Yes] See Section 4.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [No] Repeated evaluations not applicable due to the limitation in amount of computational resources required for training (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [No] Repeated evaluations not applicable due to the limitation in amount of computational resources required for training (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Section 4.
Revised: [Yes] See Section 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See Section 4.
Revised: [Yes] See Section 4.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] See Section 4.
Revised: [Yes] See Section 4.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] See Supplementary A (d) Did you discuss whether
Revised: [Yes] See Supplementary A (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [Yes] Consent obtained from license (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [Yes] Consent obtained from license (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] Only personally identifiable information would be the "curls" dataset, which is obtained from Nerfies [39] and used under their license.
Revised: [N/A] Only personally identifiable information would be the "curls" dataset, which is obtained from Nerfies [39] and used under their license.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?[N/A] D
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video

Original: Supplementary Material

Original: A Code and Data

Original: All coda and data, as well as additional video results can be found on our project page: d2nerf.github.io.

Original: We also include a static copy of the code and website as zip files in the supplementary.

Original: B Hyperparameters

Original: As we do not have large TPUs available for training, we incorporate a light-weighted HyperNeRF

Original: [40] as the dynamic component to reduce training time.

Original: Compared to the original hyperparameters described in the paper, we reduce the number of samples per ray (64 vs. 128 in [40]), batch size (1024 vs. 6144 in [40]), and number of iterations (100k vs. 250k in [40]).

Original: We also do not apply the background regularization, as it requires a set of known background 3D points, which rely on accurate dynamic masks during COLMAP registration.

Original: We experimentally established a set of hyperparameters applicable for a variety of scenes.

Original: In total, we used five sets of hyperparameters for the evaluation on the real-world dataset, and four on the synthetic dataset.

Original: To ensure the various scenes are fully separated into different components, we increase λ s during training, where → indicates the value is linearly increased and ⇒ indicates it is exponentially increased. - entry for λ ρ indicates that the shadow field is not applied.

Original: [Table]
Revised: [N/A]

Original: C Scene Decoupling – Figure 11, Figure 13, Figure

Original: We demonstrate additional qualitative results on scene decoupling task on both real-world and synthetic scenes; see Figure 11, 12, 13, and 14.

Original: D Video Segmentation – Table 4, Figure

Original: As our method learns a density distribution of the dynamic objects in the scene, we can further produce an alpha mask of the objects.

Original: We therefore also evaluate the correctness of object segmentation at the image level.

Original: Existing benchmarks on video segmentation [42, 56, 24] either contain too few video frames for reliable SfM reconstruction, or do not have correct ground truth masks for all of the dynamic objects and effects in the scene.

Original: Similarly, the dataset from NeuralDiff [54] focuses on egocentric videos and the over-exaggerated difference between frames in the videos is not suitable for HyperNeRF which we use as the dynamic component.

Original: Hence, to highlight the ability of our method to decouple dynamic objects and shadows from video sequences with large viewpoint shifts, we evaluate on our synthetic dataset.

Original: In addition to the NeRF-like baselines, we also compare with

Original: Motion Grouping (MG) [64], a motion-based 2D image segmentation method.

Original: We fine-tuned the

Original: [Table]

Original: pre-trained MG model on each scene for 5k iterations.

Original: For other NeRF-based methods, we used the same settings as in Section 4.4 and produced the alpha masks as the normalized radiance weights of the time-varying component, and then applied a threshold of 0.1 to obtain the binary masks.

Original: See

Original: Table 4, Figure 15 for the results.

Original: E Novel View Synthesis – Table 5, Figure

Original: Although the aim of our method is not to improve the quality of time-varying scene reconstruction, as a by-product, we find that by introducing a static component to fully utilize the network capacity, our method achieves more robust reconstruction for both the dynamic objects and background.

Original: We therefore also evaluate our method on the ability to synthesize the whole scene from novel views.

Original: We compare several approaches for dynamic scene reconstruction, including NeuralDiff [54] and a baseline version of HyperNeRF [40], which has the same architecture as our dynamic component.

Original: Since our method uses an additional static component and naturally has more network parameters and capacity, we also compare with a fair version of HyperNeRF with roughly the same number of total parameters by extending the NeRF MLP width to 375.

Original: Unlike novel view experiments in [40], we do not interleave between two cameras, but use only the right camera as training view and the left where the dynamic object moves rapidly or training/validation views differ largely, HyperNeRF

Original: [40] fails to reconstruct a reasonable shape for the dynamic object, while ours might potentially predict a shifted object pose, but can still render the view with high fidelity.

Original: We additionally show the decoupled dynamic object from our method.

Original: The quality is slightly degraded compared to the decoupling results in Figure 7 and 11 as we are rendering from the more challenging novel views.

Original: camera as validation view.

Original: This presents greater challenges to all the methods.

Original: See Table 5, Figurefor the results.

Original: [Table]

Original: F Ambiguity between Dynamic Component and Shadow

Original: The shadow field network represents the density-less shadows in a more physically realistic way, and resolves the ambiguity in their motion.

Original: However, the aim of our method is to achieve decoupling of dynamic occluders from the static environment, and we do not over-extend to consider further decoupling between objects and shadows, which would require more priors related to environmental lighting conditions and background texture.

Original: We empirically found that shadow field is not necessary for scenes with strong and fast-moving shadows, where they can be directly learned by the dynamic component as thin layers on top of the static geometry; see Figure 17.

Original: On the other hand, there exists unsolvable ambiguity between the shadow and dynamic object, especially for which with a similar or darker color to the background, and hence could be potentially explained as a moving shadow instead of an actual 3D shape due to our monocular camera setting; see Figure 18.

Original: As the later case causes failed dynamic geometry reconstruction, leading to a severe decrease in novel view synthesis performance, we deliberately choose a large value of λ ρ to suppress the shadow field for scenes containing a mixture of dynamic objects and shadows.

Original: Although this potentially favors the former case and causes more shadows to be interpreted as thin-layers, we found that such setting has minimal impact on performance of both scene decoupling and reconstruction, and is still sufficient to achieve correct shadow decoupling, as the shadow field resolves the ambiguity in shadow in very early stage of the training.

Original: though such representation could potentially represent more than just shadows, it still tends to exclude unnecessary texture from static background and learns only the darkening effects