Original: Streaming Radiance Fields for 3D Video Synthesis
Revised: Streaming Radiance Fields for 3D Video Synthesis

Original: Abstract
Revised: Abstract

Original: We present an explicit-grid based method for efficiently reconstructing streaming radiance fields for novel view synthesis of real world dynamic scenes.
Revised: We present an explicit-grid based method for efficiently reconstructing streaming radiance fields for novel view synthesis of real world dynamic scenes.

Original: Instead of training a single model that combines all the frames, we formulate the dynamic modeling problem with an incremental learning paradigm in which per-frame model difference is trained to complement the adaption of a base model on the current frame.
Revised: Instead of training a single model that combines all the frames, we formulate the dynamic modeling problem with an incremental learning paradigm in which per-frame model difference is trained to complement the adaption of a base model on the current frame.

Original: By exploiting the simple yet effectivestrategy of tuning  with narrow bands, the proposed method achieves a feasible framework for handlingon-thefly video sequences  with high training efficiency.
Revised: By exploiting the simple yet effective tuning strategy with narrow bands, the proposed method realizes a feasible framework for handling video sequences on-the-fly with high training efficiency.

Original: The storage overhead induced by using explicit grid  can be significantly reduced by using model difference based compression.
Revised: The storage overhead induced by using explicit grid representations can be significantly reduced by using model difference based compression.

Original: We also introduce a curriculum learning strategy to further accelerate model optimization for each frame.
Revised: We also introduce an efficient strategy to further accelerate model optimization for each frame.

Original: Experiments on challenging video sequences demonstrate that our approach is capable of achieving a training speed of 15 seconds per-frame with competitive rendering quality, which attains 1000 × speedup over the state-of-the-art implicit methods.
Revised: Experiments on challenging video sequences demonstrate that our approach is capable of achieving a training speed of 15 seconds per-frame with competitive rendering quality, which attains 1000 × speedup over the state-of-the-art implicit methods.

Original:  Introduction
Revised:  Introduction

Original: 3D video synthesis aims to realize free-viewpoint photo-realistic rendering for dynamic scenes, which are typically recorded via a set of cameras (with known poses) from multiple views.
Revised: 3D video synthesis aims to realize free-viewpoint photo-realistic rendering for dynamic scenes, which are typically recorded via a set of cameras (with known poses) from multiple views.

Original: The topic has attracted much research effort because of its potential value in a wide range of applications in VR/AR.
Revised: The topic has attracted much research effort because of its potential value in a wide range of applications in VR/AR.

Original: Traditional techniques by estimating surfaces [1], multi-sphere images [2] or depth [3, 4] via multi-view stereo [5, 6] are usually used for modeling and representing dynamic scenes.
Revised: Traditional techniques by estimating surfaces [1], multi-sphere images [2] or depth [3, 4] via multi-view stereo [5, 6] are usually used for modeling and representing dynamic scenes.

Original: However, arbitrary geometry and complex appearance exhibiting in real-world scenarios pose challenges for leveraging a general methodology to pursue high-performance modeling.
Revised: However, arbitrary geometry and complex appearance exhibiting in real-world scenarios pose challenges for leveraging a general methodology to pursue high-performance modeling.

Original: Neural radiance fields (NeRF) [7, 8] have recently emerged as a new methodology for effectively reconstructing and rendering static scenes via neural networks.
Revised: Neural radiance fields (NeRF) [7, 8] have recently emerged as a new methodology for effectively reconstructing and rendering static scenes via neural networks.

Original: These methods learn a continuous mapping between 3D points (given view directions) and their corresponding radiance colors and opacity, to realize high-fidelity rendering results through the use of volumetric rendering techniques [9].
Revised: These methods learn a continuous mapping between 3D points (given view directions) and their corresponding radiance colors and opacity, to realize high-fidelity rendering results through the use of volumetric rendering techniques [9].

Original: However, they often suffer from costly training and inference due to a tremendous amount of computations through neural networks.
Revised: However, they often suffer from costly training and inference due to a tremendous amount of computations through neural networks.

Original: When extending the implicit formulation to a dynamic scene [10], time steps are embedded as additional input for training models across frames and achieving time-dependent rendering.
Revised: When extending the implicit formulation to a dynamic scene [10], time steps are embedded as additional input for training models across frames and achieving time-dependent rendering.

Original: Compared to training on static scenes, training overhead increases significantly with respect to sequence length, e.g., costing about 56 GPU days for training 300 frames, that would be prohibitive when handling long sequences.
Revised: Compared to training on static scenes, training overhead increases significantly with respect to sequence length, e.g., costing about 56 GPU days for training 300 frames, that would be prohibitive when handling long sequences.

Original: More importantly, such learning paradigm is restricted to offline modeling, thus unable to tackle online scenarios.
Revised: More importantly, such learning paradigm is restricted to offline modeling, thus unable to tackle online scenarios.

Original: These novel view results are rendered at interactive speed ( ∼ 10 FPS) by StreamRF.
Revised: These novel view results are rendered at interactive speed ( ∼ 10 FPS) by StreamRF.

Original: In this work, we propose a novel method for reconstructing streaming radiance fields for novel view synthesis of 3D videos.
Revised: In this work, we propose a novel method for reconstructing streaming radiance fields for novel view synthesis of 3D videos.

Original: We formalize the modeling of a dynamic scene in an incremental learning framework, enabling it to tackle video sequences on-the-fly.
Revised: We formalize the modeling of a dynamic scene in an incremental learning framework, enabling it to tackle video sequences on-the-fly.

Original: Inspired by the recent success achieved by Plenoxel [11] on training and inference efficiency, our method is built on explicit grid representations.
Revised: Inspired by the recent success achieved by Plenoxel [11] on training and inference efficiency, our method is built on explicit grid representations.

Original: Specifically, the overall training process is decomposed to the complete training of a basic voxel grid model at the first frame, with an online tuning mechanism for learning and storing model difference for each subsequent frame.
Revised: Specifically, the overall training process is decomposed to the complete training of a basic voxel grid model at the first frame, with an online tuning mechanism for learning and storing model difference for each subsequent frame.

Original: A complete grid model given a time step i can be simply attained by adding the model difference to the previous model at i − 1 (or accumulating all the model differences before i with the basic grid model).
Revised: A complete grid model given a time step i can be simply attained by adding the model difference to the previous model at i − 1 (or accumulating all the model differences before i with the basic grid model).

Original: By exploiting the prior of exhibiting local continuity between adjacent frames, we present a narrow-band tuning strategy for capturing and training the model difference efficiently, associated with a compression strategy for significantly reducing memory cost that explicit grid methods struggle with.
Revised: By exploiting the prior of exhibiting local continuity between adjacent frames, we present a narrow-band tuning strategy for capturing and training the model difference efficiently, associated with a compression strategy for significantly reducing memory cost that explicit grid methods struggle with.

Original: To further improve the training efficiency, we propose to learning pilot models to guide the optimization of full-scale models.
Revised: To further improve the training efficiency, we propose to learning pilot models to guide the optimization of full-scale models.

Original: The overall framework is simple yet efficient.
Revised: The overall framework is simple yet efficient.

Original: Quantitative and qualitative experimental results demonstrate the effectiveness of our approach.
Revised: Quantitative and qualitative experimental results demonstrate the effectiveness of our approach.

Original: The approach can achieve the training speed of 15 seconds and inference time of 120 ms per frame for rendering a 1k resolution image with competitive quality, achieving three orders of magnitude faster than the state-of-the-art dynamic neural radiance field methods (costing  GPU-days for training a sequence with 300 frames).
Revised: The approach can achieve the training speed of 15 seconds and inference time of 120 ms per frame for rendering a 1k resolution image with competitive quality, achieving three orders of magnitude faster than the state-of-the-art dynamic neural radiance field methods (costing 56 GPU-days for training a sequence with 300 frames).

Original: Compared to the efficient explicit-grid baseline by per-frame training, it still can obtain two orders of magnitude acceleration with a much lower storage cost.
Revised: Compared to the efficient explicit-grid baseline by per-frame training, it still can obtain ~ 100 × acceleration with a much lower storage cost.

Original: The empirical analysis further show that using a proper incremental training mechanism would not degrade performance when dealing with relatively long sequences.
Revised: The empirical analysis further demonstrates that using a proper incremental training mechanism would not degrade performance when dealing with relatively long sequences.

Original: code and dataset will be released at https://github.com/AlgoHunt/StreamRF
Revised: Code is available at https://github.com/AlgoHunt/StreamRF.

Original:  Problem Formulation
Revised:  Problem Formulation

Original: Neural Radiance fields (NeRF) [7] model a static scene implicitly by learning a continuous mapping from a 3D spatial position x ∈ R 3 specified in a viewing direction d ∈ R 3 (represented as Cartesian unit vectors) to RGB radiance color c ∈ R 3 and opacity σ ∈ R .
Revised: Neural Radiance fields (NeRF) [7] model a static scene implicitly by learning a continuous mapping from a 3D spatial position x ∈ R 3 specified in a viewing direction d ∈ R 3 (represented as Cartesian unit vectors) to RGB radiance color c ∈ R 3 and opacity σ ∈ R .

Original: The mapping function is typically implemented by a multilayer perceptron (MLP)
Revised: The mapping function is typically implemented by a multilayer perceptron (MLP)

Original: f m : ( x , d ) (cid:55)→ ( c , σ ) , trained on a set of images with known camera pose.
Revised: f m : ( x , d ) (cid:55)→ ( c , σ ) , trained on a set of images with known camera pose.

Original: Given a ray r casting on the image plane through a pixel, by using a traditional volume rendering technique [12], the expected color ˆ C ( r ) of the pixel is rendered by integrating the opacity and the color of sampling points along the ray:
Revised: Given a ray r casting on the image plane through a pixel, by using a traditional volume rendering technique [12], the expected color ˆ C ( r ) of the pixel is rendered by integrating the opacity and the color of sampling points along the ray:

Original: [Equation]
Revised: [Equation]

Original: where δ i denotes the distance to adjacent points.
Revised: where δ i denotes the distance to adjacent points.

Original: These methods are subjected to slow training and rendering speed, since the ray casting involved in both training and rendering stage requires hundreds of forward propagation (and backward propagation in training) through MLPs for the sampling points on each ray.
Revised: These methods are subjected to slow training and rendering speed, since the ray casting involved in both training and rendering stage requires hundreds of forward propagation (and backward propagation in training) through MLPs for the sampling points on each ray.

Original: For example, JaxNeRF [13], though optimized through parallelization, needs overseconds to render an image with 800 × 800 resolution with over 10 hours of training.
Revised: For example, JaxNeRF [13], though optimized through parallelization, needs overseconds to render an image with 800 × 800 resolution with over 10 hours of training.

Original: When extending the mechanism to model a dynamic scene, one straightforward solution is to embed the time variable t into the representation, i.e., f t : ( x , d , t ) (cid:55)→ ( c , σ ) , while training time would dramatically increase when dealing with a large number of frames.
Revised: When extending the mechanism to model a dynamic scene, one straightforward solution is to embed the time variable t into the representation, i.e., f t : ( x , d , t ) (cid:55)→ ( c , σ ) , while training time would dramatically increase when dealing with a large number of frames.

Original: Training a video with original
Revised: Training a video with original

Original: NeRF frameworks may cost over 600 GPU-days.
Revised: NeRF frameworks may cost over 600 GPU-days.

Original: N3DV [10] improves training efficiency by virtue of sophistical designs, however it still needs about 56 GPU-days for pursuing a model with reliable rendering quality.
Revised: N3DV [10] improves training efficiency by virtue of sophistical designs, however it still needs about 56 GPU-days for pursuing a model with reliable rendering quality.

Original: More importantly, these implicit formulations are naturally suitable for offline processing which feeds all the (key)frames in advance for training, and may not be easily generalized to online training scenarios.
Revised: More importantly, these implicit formulations are naturally suitable for offline processing which feeds all the (key)frames in advance for training, and may not be easily generalized to online training scenarios.

Original: Recent progress on explicit representations has demonstrated compelling improvement on training and render efficiency for modelling a static scene.
Revised: Recent progress on explicit representations has demonstrated compelling improvement on training and render efficiency for modelling a static scene.

Original: The scene space is discretized and represented explicitly with a sparse voxel grid instead of learning representations implicitly with large MLPs.
Revised: The scene space is discretized and represented explicitly with a sparse voxel grid instead of learning representations implicitly with large MLPs.

Original: As a representative work, Plenoxel [11] optimizes voxel grids without neural networks.
Revised: As a representative work, Plenoxel [11] optimizes voxel grids without neural networks.

Original: Each voxel in the grid consists of the scalar opacity σ and the coefficient vector k of spherical harmonics (SH) for respectively describing geometric and color information.
Revised: Each voxel in the grid consists of the scalar opacity σ and the coefficient vector k of spherical harmonics (SH) for respectively describing geometric and color information.

Original: The color at the voxel v is obtained by,
Revised: The color at the voxel v is obtained by,

Original: [Equation]
Revised: [Equation]

Original: where ξ is the sigmoid function and { Y ml } denotes the set of SH basis.
Revised: where ξ is the sigmoid function and { Y ml } denotes the set of SH basis.

Original: To render the expected color of the pixel through a ray, the opacity and the color of a sampling point x are calculated by performing trilinear interpolation among the voxels within its neighborhood N ( x ) ,
Revised: To render the expected color of the pixel through a ray, the opacity and the color of a sampling point x are calculated by performing trilinear interpolation among the voxels within its neighborhood N ( x ) ,

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Then the pixel color is produced by integrating the sampling points along the ray based on Eq.
Revised: Then the pixel color is produced by integrating the sampling points along the ray based on Eq.

Original: The voxels with zero opacity (or truncated by a minimal threshold) are pruned after sparsification, and an occupancy mask M with binary values is used for indicating if the space is occupied.
Revised: The voxels with zero opacity (or truncated by a minimal threshold) are pruned after sparsification, and an occupancy mask M with binary values is used for indicating if the space is occupied.

Original: More details can refer to the paper of Plenoxels [11].
Revised: More details can refer to the paper of Plenoxels [11].

Original: In theory, the mechanism could significantly reduce the overall computational complexity as it omits the requirement of numerous MLP-based costly computation during training and inference.
Revised: In theory, the mechanism could significantly reduce the overall computational complexity as it omits the requirement of numerous MLP-based costly computation during training and inference.

Original: It is able to train a static scene with good rendering quality within 15 minutes on single GPU and render a 1 k resolution image in 100 ms.
Revised: It is able to train a static scene with good rendering quality within 15 minutes on single GPU and render a 1 k resolution image in 100 ms.

Original: These efficiency gains would benefit developing a highly efficient framework for modeling and rendering a dynamic scene.
Revised: These efficiency gains would benefit developing a highly efficient framework for modeling and rendering a dynamic scene.

Original:  Streaming Radiance Fields for Dynamic Scenes
Revised:  Streaming Radiance Fields for Dynamic Scenes

Original: To realize sparse grids for the modeling of a dynamic scene, one intuitive way is to directly extend 3D spatial dimensions (denoted by H × W × D ) with the 4-th dimension indicating the time steps t for frames.
Revised: To realize sparse grids for the modeling of a dynamic scene, one intuitive way is to directly extend 3D spatial dimensions (denoted by H × W × D ) with the 4-th dimension indicating the time steps t for frames.

Original: However, using explicit grids for a static scene have faced challenges on storage although progressive training and sparsifying (i.e, pruning off empty voxels) [11] are used to alleviate the issue of costly overhead.
Revised: However, using explicit grids for a static scene have faced challenges on storage although progressive training and sparsifying (i.e, pruning off empty voxels) [11] are used to alleviate the issue of costly overhead.

Original: Simply expanding the grid into a space-time manner would make storage overhead increase linearly with t , which is infeasible for handling videos with long sequences.
Revised: Simply expanding the grid into a space-time manner would make storage overhead increase linearly with t , which is infeasible for handling videos with long sequences.

Original: On the other hand, per-frame training individually without considering temporal correlation may be feasible for achieving a tradeoff between storage and efficiency, while it is still far from the objective of pursing a high-performing framework for 3D video synthesis.
Revised: On the other hand, per-frame training individually without considering temporal correlation may be feasible for achieving a tradeoff between storage and efficiency, while it is still far from the objective of pursing a high-performing framework for 3D video synthesis.

Original: In this regard, we propose a novel method, named StreamRF, to realize a streaming radiance field for effectively rendering dynamic scenes.
Revised: In this regard, we propose a novel method, named StreamRF, to realize a streaming radiance field for effectively rendering dynamic scenes.

Original: Our approach is built on the benefits of utilizing an explicit representation via sparse voxel grids.
Revised: Our approach is built on the benefits of utilizing an explicit representation via sparse voxel grids.

Original: Particularly, by leveraging some intrinsic priors exhibiting in videos, we develop an incremental training framework which enables a faster training convergence and lower storage over per-frame grid modelling.
Revised: Particularly, by leveraging some intrinsic priors exhibiting in videos, we develop an incremental training framework which enables a faster training convergence and lower storage over per-frame grid modelling.

Original: Videos in forward-facing scenarios are typically composed of time-invariant components (e.g., static background) and time-variant components (e.g., moving objects) with small variation that exhibits local continuity between adjacent frames.
Revised: Videos in forward-facing scenarios are typically composed of time-invariant components (e.g., static background) and time-variant components (e.g., moving objects) with small variation that exhibits local continuity between adjacent frames.

Original: By exploiting the fact, we decompose the problem of sequential modeling into two sub-problems.
Revised: By exploiting the fact, we decompose the problem of sequential modeling into two sub-problems.

Original: We first fully train a base grid model given the first frame, denoted as V 

Revised: We first fully train a base grid model given the first frame, denoted as V

Original: = { V 0 c , V 0 σ } which stores the corresponding color features (the coefficients of spherical harmonics) and opacity values.
Revised: = { V 0 c , V 0 σ } which stores the corresponding color features (the coefficients of spherical harmonics) and opacity values.

Original: Then the model is adapted to succeeding frames through the use of an effective tuning strategy.
Revised: Then the model is adapted to succeeding frames through the use of an effective tuning strategy.

Original: Per-frame tuning is expected to perform only on the changes between adjacent frames.
Revised: Per-frame tuning is expected to perform only on the changes between adjacent frames.

Original: The grid model in the current frame is produced by adding the model difference, which is optimized and stored during training phase, to the grid model in the previous frame.
Revised: The grid model in the current frame is produced by adding the model difference, which is optimized and stored during training phase, to the grid model in the previous frame.

Original: Formally, assume the model difference between the i − 1 -th and i -th frames is denoted as δ i =
Revised: Formally, assume the model difference between the i − 1 -th and i -th frames is denoted as δ i =

Original: ∆ ( V i , V i − 1 ) .
Revised: ∆ ( V i , V i − 1 ) .

Original: In the inference phase, the grid model at the time step i can be obtained by,
Revised: In the inference phase, the grid model at the time step i can be obtained by,

Original: [Equation]
Revised: [Equation]

Original: Such an incremental formulation indeed presents an online training paradigm, enabling the proposed framework to handle video sequences on-the-fly.
Revised: Such an incremental formulation indeed presents an online training paradigm, enabling the proposed framework to handle video sequences on-the-fly.

Original: 3.1 Tuning with Narrow Bands
Revised: 3.1 Tuning with Narrow Bands

Original: A problem arises: how to effectively capture the model difference between adjacent frames i − 1 and i .
Revised: A problem arises: how to effectively capture the model difference between adjacent frames i − 1 and i .

Original: Given the sparse grid model V i − 1 , any object motion at the time step i may result in changes on the opacity and colors of a part of voxels.
Revised: Given the sparse grid model V i − 1 , any object motion at the time step i may result in changes on the opacity and colors of a part of voxels.

Original: It is hence likely to happen that informative points occur in the originally empty space as well as some voxels originally occupied becomes empty.
Revised: It is hence likely to happen that informative points occur in the originally empty space as well as some voxels originally occupied becomes empty.

Original: If we directly tune the sparse grid, only the non-empty voxels could be changed or pruned in a one-way manner while unable to add informative voxels.
Revised: If we directly tune the sparse grid, only the non-empty voxels could be changed or pruned in a one-way manner while unable to add informative voxels.

Original: It is problematic when handling long sequences (shown by the ablation study on the sparse grid in Fig. 2).
Revised: It is problematic when handling long sequences (shown by the ablation study on the sparse grid in Fig. 2).

Original: To address the issue, we propose a tuning strategy based on narrow bands.
Revised: To address the issue, we propose a tuning strategy based on narrow bands.

Original: We first retrieve and activate the narrow band for the frame i , i.e., the region of interest in the grid
Revised: We first retrieve and activate the narrow band for the frame i , i.e., the region of interest in the grid

Original: R i , based on the occupancy mask M i − 1 (with binary values) which indicates non-empty voxels at the frame i − 1 ,
Revised: R i , based on the occupancy mask M i − 1 (with binary values) which indicates non-empty voxels at the frame i − 1 ,

Original: [Equation]
Revised: [Equation]

Original: where F di and F er correspond to dilation and erosion operations, and ρ d and ρ e denote the corresponding radius parameters.
Revised: where F di and F er correspond to dilation and erosion operations, and ρ d and ρ e denote the corresponding radius parameters.

Original: ⊕ denotes the element-wise xor operation on the masks to extract the additional region of the dilated area compared to the eroded one.
Revised: ⊕ denotes the element-wise xor operation on the masks to extract the additional region of the dilated area compared to the eroded one.

Original: The narrow band actually represents a thin area around the surface of scenes (or objects in a composite scene).
Revised: The narrow band actually represents a thin area around the surface of scenes (or objects in a composite scene).

Original: We use it for two purposes:
Revised: We use it for two purposes:

Original: • For every empty voxel that falls into the band, we restore them to the grid and initialize the opacity and color features with zero values.
Revised: • For every empty voxel that falls into the band, we restore them to the grid and initialize the opacity and color features with zero values.

Original: • Only the voxels within the band are tuned and the rest ones are not involved during the training stage for the frame i .
Revised: • Only the voxels within the band are tuned and the rest ones are not involved during the training stage for the frame i .

Original: In other words, the tuning always performs on a relative small set of voxels in the frame i .
Revised: In other words, the tuning always performs on a relative small set of voxels in the frame i .

Original: We also prune empty voxels off to get a compact grid
Revised: We also prune empty voxels off to get a compact grid

Original: V i with an occupancy mask M i after tuning.
Revised: V i with an occupancy mask M i after tuning.

Original: The key idea of narrow band-based tuning is built upon leveraging the intrinsic priors: motions induced by moving objects typically show small changes and local continuity within neighboring between adjacent frames.
Revised: The key idea of narrow band-based tuning is built upon leveraging the intrinsic priors: motions induced by moving objects typically show small changes and local continuity within neighboring between adjacent frames.

Original: These assumptions have been successfully exploited a wide of video processing [14] and video understanding tasks [15, 16], e.g, traditional mean-shift method for tracking [17].
Revised: These assumptions have been successfully exploited a wide of video processing [14] and video understanding tasks [15, 16], e.g, traditional mean-shift method for tracking [17].

Original: We demonstrate that the strategy can effectively benefit to training convergence and be capable of achieving the speed of ∼ 15 s per-frame training for achieving a competitive quality (shown in section 5.4).
Revised: We demonstrate that the strategy can effectively benefit to training convergence and be capable of achieving the speed of ∼ 15 s per-frame training for achieving a competitive quality (shown in section 5.4).

Original: Compared to the state-of-the-art implicit method [10] with the training speed of 4.5 GPU-hours for each frame in average, our method is about three orders of magnitude faster on training and can achieve over 500 × speedup on rendering efficiency (over 60 s s VS 121 ms).
Revised: Compared to the state-of-the-art implicit method [10] with the training speed of 4.5 GPU-hours for each frame in average, our method is about three orders of magnitude faster on training and can achieve over 500 × speedup on rendering efficiency (over 60 s s VS 121 ms).

Original: 3.2 Difference-based Compression
Revised: 3.2 Difference-based Compression

Original: Using explicit grid modelling may pose the issue of large storage cost.
Revised: Using explicit grid modelling may pose the issue of large storage cost.

Original: Taking a grid size ( 1408 × 1156 × 128 in the experimental setting) for example, the overall storage for a conventional sparse grid costs about 1015 MB after sparsity for one frame.
Revised: Taking a grid size ( 1408 × 1156 × 128 in the experimental setting) for example, the overall storage for a conventional sparse grid costs about 1015 MB after sparsity for one frame.

Original: By virtue of narrow band tuning, we only need to store the model difference δ i with a significantly smaller size compared to the complete grid model.
Revised: By virtue of narrow band tuning, we only need to store the model difference δ i with a significantly smaller size compared to the complete grid model.

Original: The change of storage cost only happens in the following two scenarios, i.e., when adding voxels into the grid (i.e., appearing in the empty space) and pruning voxels off from the grid (i.e., the previously occupied space becomes empty).
Revised: The change of storage cost only happens in the following two scenarios, i.e., when adding voxels into the grid (i.e., appearing in the empty space) and pruning voxels off from the grid (i.e., the previously occupied space becomes empty).

Original: In this regard, we introduce a difference-based compression strategy to further reduce the storage overhead.
Revised: In this regard, we introduce a difference-based compression strategy to further reduce the storage overhead.

Original: We can efficiently find out the change areas by using the occupancy masks M i − 1 and M i between the adjacent frames i − 1 and i ,
Revised: We can efficiently find out the change areas by using the occupancy masks M i − 1 and M i between the adjacent frames i − 1 and i ,

Original: [Equation]
Revised: [Equation]

Original: where M ia , M ie and M ir respectively denote the masks of adding voxels, erasing voxels and remaining ones while the features at the voxels may change.
Revised: where M ia , M ie and M ir respectively denote the masks of adding voxels, erasing voxels and remaining ones while the features at the voxels may change.

Original: γ ( · ) defines the function of getting the L1 norm along the axis of color features (SH coefficients) on the model difference, and ϵ denotes a threshold.
Revised: γ ( · ) defines the function of getting the L1 norm along the axis of color features (SH coefficients) on the model difference, and ϵ denotes a threshold.

Original: M ir actually involves the voxels with relatively large modification.
Revised: M ir actually involves the voxels with relatively large modification.

Original: Instead of storing model difference δ i directly, we store the three masks, the voxel representations (SH coefficients and opacity) belonging to the adding area M ia and the representation differences for the voxels within the remaining area
Revised: Instead of storing model difference δ i directly, we store the three masks, the voxel representations (SH coefficients and opacity) belonging to the adding area M ia and the representation differences for the voxels within the remaining area

Original: M ir .
Revised: M ir .

Original: It can achieve a remarkable reduction rate (178 times) on storage that reduces 1015 MB to 5.
Revised: It can achieve a remarkable reduction rate (178 times) on storage that reduces 1015 MB to 5.

Original: MB for each frame in average.
Revised: MB for each frame in average.

Original:  Efficient Training with Pilot Model Guidance
Revised:  Efficient Training with Pilot Model Guidance

Original: To further improve the training efficiency of StreamRF, we exploit an optimization strategy inspired from curriculum learning to train from an easier problem.
Revised: To further improve the training efficiency of StreamRF, we exploit an optimization strategy inspired from curriculum learning to train from an easier problem.

Original: Formally, given the grid model V i − 1 , we adopt down-sampling operations to get a pilot model of a smaller size,
Revised: Formally, given the grid model V i − 1 , we adopt down-sampling operations to get a pilot model of a smaller size,

Original: [Equation]
Revised: [Equation]

Original: When setting the S to 1 / 2 , the capacity of pilot model is roughly 1 / 8 of the full grid.
Revised: When setting the S to 1 / 2 , the capacity of pilot model is roughly 1 / 8 of the full grid.

Original: Training on a smaller model would reduce the computational complexity at every iteration (with the same batch size) and also ease optimization.
Revised: Training on a smaller model would reduce the computational complexity at every iteration (with the same batch size) and also ease optimization.

Original: The model difference on the pilot models (cid:98) δ i =
Revised: The model difference on the pilot models (cid:98) δ i =

Original: ∆ ( (cid:98) V i , (cid:98) V i − 1 ) are implemented by using { (cid:99) M ia , (cid:99) M ie , (cid:99) M ir } as following Eq. 6.
Revised: ∆ ( (cid:98) V i , (cid:98) V i − 1 ) are implemented by using { (cid:99) M ia , (cid:99) M ie , (cid:99) M ir } as following Eq. 6.

Original: Then we can get a guidance mask (cid:98) G i by merging the three mask via elementwise "or" operator and upsampling to the full scale,
Revised: Then we can get a guidance mask (cid:98) G i by merging the three mask via elementwise "or" operator and upsampling to the full scale,

Original: [Equation]
Revised: [Equation]

Original: The guidance mask is used for regularizing the gradient update when tuning the full-scale model.
Revised: The guidance mask is used for regularizing the gradient update when tuning the full-scale model.

Original: The values of added and erased voxels are filled back to the full-scale grid.
Revised: The values of added and erased voxels are filled back to the full-scale grid.

Original: All the voxels outside the mask are frozen without optimization when training at the frame i .
Revised: All the voxels outside the mask are frozen without optimization when training at the frame i .

Original: The design of the strategy comes from the insight that training at a small size converges much faster and the induced difference of pilot model indicates the voxels that need to be modified.
Revised: The design of the strategy comes from the insight that training at a small size converges much faster and the induced difference of pilot model indicates the voxels that need to be modified.

Original: Using this strategy can prevent unnecessary modification in the full-scale model and avoid the loss of high-frequency details during downsample-upsample process.
Revised: Using this strategy can prevent unnecessary modification in the full-scale model and avoid the loss of high-frequency details during downsample-upsample process.

Original: We observe that using this strategy facilitates pursuing a higher fidelity model compared to directly training on the full-scale grid with the same training time.
Revised: We observe that using this strategy facilitates pursuing a higher fidelity model compared to directly training on the full-scale grid with the same training time.

Original: The qualitative results (in Fig. 4) reveals that the model via directly training on the full-scale grid may encounter flicker issues when rendering along time steps, i.e., sparkling noise may occur in the stable background area when training period is short.
Revised: The qualitative results (in Fig. 4) reveals that the model via directly training on the full-scale grid may encounter flicker issues when rendering along time steps, i.e., sparkling noise may occur in the stable background area when training period is short.

Original: The issue could be addressed by lengthening the training schedule with a small learning rate.
Revised: The issue could be addressed by lengthening the training schedule with a small learning rate.

Original: While training with the pilot model guidance can effectively prohibit flicker artifacts from happening, enabling a more efficient and stable optimization for the streaming paradigm.
Revised: While training with the pilot model guidance can effectively prohibit flicker artifacts from happening, enabling a more efficient and stable optimization for the streaming paradigm.

Original:  Experiments 5.1 Datasets
Revised:  Experiments 5.1 Datasets

Original: Meet Room Dataset We build a multi-view capture system using 13 Azure Kinect cameras.
Revised: Meet Room Dataset. We build a multi-view capture system using 13 Azure Kinect cameras.

Original: We use twelve cameras for training and one for testing, with an external pulse signal to synchronize all camera shutter.
Revised: We use twelve cameras for training and one for testing, with an external pulse signal to synchronize all camera shutter.

Original: Though this system is capable of depth capture, we do not use depth images in this research.
Revised: Though this system is capable of depth capture, we do not use depth images in this research.

Original: The videos are recorded at a resolution of 1280 × 720 and a frame rate of 30 FPS.
Revised: The videos are recorded at a resolution of 1280 × 720 and a frame rate of 30 FPS.

Original: The total size of the capture system is 100 cm × 75 cm and the captured views are aligned much sparser than some common datasets such as LLFF  and N3DV.
Revised: The total size of the capture system is 100 cm × 75 cm and the captured views are aligned much sparser than some common datasets such as LLFF [18] and N3DV [10].

Original: We will release this dataset for research purposes.
Revised: We will release this dataset for research purposes.

Original: Neural 3D Video (N3DV) Dataset [10] It contains 6 dynamic indoor scenes with varying illuminations, view-dependent and highly volumetric effects.
Revised: Neural 3D Video (N3DV) Dataset [10]. It contains 6 dynamic indoor scenes with varying illuminations, view-dependent and highly volumetric effects.

Original: Videos are captured at a resolution of 2704 × 2078 and a frame rate of 30 FPS.
Revised: Videos are captured at a resolution of 2704 × 2078 and a frame rate of 30 FPS.

Original: Following the setting in the original paper [10], the quantitative score is produced on the designated sequence (“flame_salmon") by downsampling it to 1352 × 1039 for a fair comparison, where the frames captured by 18 views are used for training and the rest one for testing.
Revised: Following the setting in the original paper [10], the quantitative score is produced on the designated sequence (“flame_salmon") by downsampling it to 1352 × 1039 for a fair comparison, where the frames captured by 18 views are used for training and the rest one for testing.

Original: 5.2 Implementation Details
Revised: 5.2 Implementation Details

Original: Initialization.
Revised: Initialization.

Original: We follow the standard training protocol of a forward-facing scene in [11] to obtain the base model at the first frame.
Revised: We follow the standard training protocol of a forward-facing scene in [11] to obtain the base model at the first frame.

Original: We start by a 256 × 256 × 128 grid which is upsampled and sparsified for every 38.4K iterations until the model reaches the final resolution of 1408 × 1156 × 128 .
Revised: We start by a 256 × 256 × 128 grid which is upsampled and sparsified for every 38.4K iterations until the model reaches the final resolution of 1408 × 1156 × 128 .

Original: The total training takes 128K iterations with a batch size of 5K rays.
Revised: The total training takes 128K iterations with a batch size of 5K rays.

Original: We adopt the RMSProp [18] optimizer with a decay parameter of 0 .
Revised: We adopt the RMSProp [19] optimizer with a decay parameter of 0 .

Original: 95 for optimization.
Revised: 95 for optimization.

Original: We adopt a slighter larger TV penalty to mitigate foggy issues, where λ TV is set to 5 × 10 − 4 for opacity and 5 × 10 − 3 for color features.
Revised: We adopt a slighter larger TV penalty to mitigate foggy issues, where λ TV is set to 5 × 10 − 4 for opacity and 5 × 10 − 3 for color features.

Original: Per Frame Tuning.
Revised: Per Frame Tuning.

Original: The full-scale model and the pilot model are trained in parallel.
Revised: The full-scale model and the pilot model are trained in parallel.

Original: For the experiments on the Meet Room dataset, we train the pilot model with 1000 iterations and then tune the full-scale model with 500 iterations.
Revised: For the experiments on the Meet Room dataset, we train the pilot model with 1000 iterations and then tune the full-scale model with 500 iterations.

Original: For the experiments on the N3DV dataset, we train the pilot model with 750 iterations and then tune the full model with 500 iterations.
Revised: For the experiments on the N3DV dataset, we train the pilot model with 750 iterations and then tune the full model with 500 iterations.

Original: We use RMSprop to optimize both models with a batch size of 20k rays.
Revised: We use RMSprop to optimize both models with a batch size of 20k rays.

Original: To promote the pilot model to have a better convergence, we enlarge λ TV by 10 times when training the pilot model. 
Revised: To promote the pilot model to have a better convergence, we enlarge λ TV by 10 times when training the pilot model. [Table]

Original: Diff Based Compression.
Revised: Diff Based Compression.

Original: We set the SH coefficient threshold ϵ to 1.5/27 and 1/27 for Meet Room
Revised: We set the SH coefficient threshold ϵ to 1.5/27 and 1/27 for Meet Room

Original: Dataset and N3DV dataset respectively by the consideration of coefficient dimension (i.e., 27).
Revised: Dataset and N3DV dataset respectively by the consideration of coefficient dimension (i.e., 27).

Original: We[Table] found using too small value would drastically increase saving size while the gain of rendering quality is marginal.
Revised: We found using too small value would drastically increase saving size while the gain of rendering quality is marginal.

Original: In the experiments we use them by default to achieve a proper trade-off between storage cost and rendering quality.
Revised: In the experiments we use the about setting by default to achieve a proper trade-off between storage cost and rendering quality.

Original: In order to further reduce storage cost, we convert all saving SH coefficients and opacity to float16 and merge M ia and M ie to one single mask to save more space in the implementation.
Revised: In order to further reduce storage cost, we convert all saving SH coefficients and opacity to float16 and merge M ia and M ie to one single mask to save more space in the implementation.

Original: Moreover, as the mask is mostly comprised of consecutive binary values, it can be compressed with a very high compression ratio (over 99%) simply with a standard compression library ( e.g. ZLIB[19]).
Revised: Moreover, as the mask is mostly comprised of consecutive binary values, it can be compressed with a very high compression ratio (over 99%) simply with a standard compression library ( e.g. ZLIB[20]).

Original: We pack all tensors together and compress them with ZLIB to get the final on-disk storage cost for each frame.
Revised: We pack all tensors together and compress them with ZLIB to get the final on-disk storage cost for each frame.

Original: Pilot Model Guidance.
Revised: Pilot Model Guidance.

Original: We first downsample the grid model and apply the standard training pipeline (narrow band and difference-based compression) for pilot model training.
Revised: We first downsample the grid model and apply the standard training pipeline (narrow band and difference-based compression) for pilot model training.

Original: The induced model difference is used in two purposes, including filling the values of added and erased voxels back to the full-scale model and tuning the full-scale model with the guidance mask.
Revised: The induced model difference is used in two purposes, including filling the values of added and erased voxels back to the full-scale model and tuning the full-scale model with the guidance mask.

Original: 5.3 Experimental Comparison
Revised: 5.3 Experimental Comparison

Original: We first quantitatively compare the method with some representative works including N3DV which achieves the state-of-the-art rendering quality.
Revised: We first quantitatively compare the method with some representative works including N3DV which achieves the state-of-the-art rendering quality.

Original: For the rest baselines, i.e., JaxNeRF [13], LLFF [20] and Plenoxels [11] which are originally designed and trained on static scenes, we train them in a per-frame independent manner, following the default setting as in their original papers.
Revised: For the rest baselines, i.e., JaxNeRF [13], LLFF [18] and Plenoxels [11] which are originally designed and trained on static scenes, we train them in a per-frame independent manner, following the default setting as in their original papers.

Original: We use peak signal-to-noise ratio (PSNR), training and inference time and storage cost as metrics for a comprehensive evaluation, and report the mean scores per-frame.
Revised: We use peak signal-to-noise ratio (PSNR), training and inference time and storage cost as metrics for a comprehensive evaluation, and report the mean scores per-frame.

Original: We refer to the scores reported in N3DV [10] for a fair comparison and estimate the rendering time of the method according to its network design.
Revised: We refer to the scores reported in N3DV [10] for a fair comparison and estimate the rendering time of the method according to its network design.

Original: As it used a larger MLP than the one in NeRF [7], its inference time is supposed to be longer than NeRF.
Revised: As it uses a larger MLP than the one in NeRF [7], its inference time is supposed to be longer than NeRF.

Original: All the baselines are evaluated every ten frames as in [10].

Original: As shown in the Table 1 , the implicit methods JaxNeRF and N3DV present fairly high computation cost, requiring over 8 GPU hours and 4 GPU hours for per-frame training, respectively. As for testing speed, the LLFF achieve the fastest record. Because it only needs blending the multi-plane image to render a novel view.
Revised: As shown in the Table 1 , the implicit methods JaxNeRF and N3DV present fairly high computation cost, requiring over 8 GPU hours and 4 GPU hours for per-frame training, respectively. Our method can obtain remarkable advantage on training efficiency, which is three orders of magnitude faster than SOTA method N3DV.

Original: However, it fails to produce a satisfactory result.
Revised: The inference efficiency also surpass them significantly.

Original: We achieve the optimization speed of costing 15 seconds only, outperforming all the baseline methods in terms of training efficiency. Meanwhile our method can achieve a comparable PSNR to the previous SOTA.
Revised: Unlike explicit baselines, our method can effectively decrease the need of storage. Plenoxels and LLFF show the advantage on training and inference time benefiting from explicit representations, whereas they suffer from considerable storage cost (over several gigabytes perframe).

Original: For the meet room dataset, we use the same sequence as in Fig 2 for testing and following the same test protocol as above.
Revised: Our streaming modeling framework is capable of further enhancing training efficiency, i.e., obtaining ~ 100 × acceleration over Plenoxels with comparable rendering quality, and ~ 20 × acceleration over LLFF with obvious superiority in rendering quality.

Original: Because our dataset contains fast motion and camera spacing is larger than N3DV dataset, we could see a clear PSNR drop on all four methods. Nevertheless, our method still achieves a comparable performance with per frame training counterpart, around 0.4dB drop in 
Revised: Moreover, our method can also surpass them obviously in storage cost. In a word, our method can outperform all the baseline methods substantially in terms of training efficiency, reaching a super fast speed (about 15 seconds and 10 seconds respectively) for per-frame optimization on the datasets.

Original: PSNR compared to JaxNeRF and Plenoxels, while obtaining the acceleration over 2 , 700 × and 80 × , respectively. More experimental results  and analysis can be referred to supplementary materials.
Revised: It is meanwhile able to attain competitive results on the metrics of rendering quality, inference time and storage. [Figure]

Original: 5.4 Ablation Studies
Revised: 5.4 Ablation Studies

Revised: We conduct a series of ablation studies to further analyze and understand the method.

Revised: More detailed experimental results and analysis can be referred to supplementary material.

Original: Narrow Band Finetune.
Revised: Narrow Band Finetune.

Original: We compare the  narrow band  with the following two counterparts, both quantitative and visually: (1) directly adopt the original sparse voxels grid, (2) switch theoriginal sparsevoxels grid  to its dense counterpart.
Revised: We validate the effect of using narrow band in Meet Room dataset, with the following two experimental configurations: (1) directly adopt the original sparse voxel grid, (2) switch the sparse grid of base model to its dense counterpart and tune on the dense grid.

Original: It can be seen in  Fig.2.
Revised: The curve of PSNR across frames and the visualization examples are shown in the Fig.2.

Original: The sparse grid failed immediately 
Revised: We can observe that directly tuning a sparse grid is intractable to model difference caused by moving, resulting the rendering quality dropped immediately when motion happens out of active voxel grids, as shown in Fig.2 (a), though the right person only moves at a small distance.

Original: [Figure] (Fig.2 (a)), though the right person only move at a small distance. This artifact is because the subject has moved out of the trainable area.
Revised: The rendering quality may be damaged obviously when there is a large movement, as shown in the Fig. 2 (b).

Original: A dense voxel grid allows every voxelto be trainable.
Revised: As for training on a dense grid, it allows every voxel trainable.

Original: It still fails when the right person has a large movement(Fig.2 (b)).
Revised: However, it still fails for modeling large movements with limited optimization steps.

Original: This artifact could be caused by the rapidly increasing number of active voxels after switching from sparse to dense  (5% vs. 100% of all voxels).
Revised: Compared to the sparse grid, a much large set of active voxels are tuned in the dense grid (5% vs. 100% of all the voxels) which might incur difficulty on convergence when training iterations are restricted.

Original: Both PSNR curve and rendering result show our method achieves comparable PSNR with  a dense grid and can handle large movement in a dynamic scene.
Revised: In contrast, finetuning with the narrow band strategy can effectively reach a stable and relatively high rendering quality across frames, even handling large movements.

Original: Moreover,in this ablation study, our method trains around 40% faster and requires 63% less memory consumption than  dense finetune.
Revised: Moreover, our method can train around 40% faster and require 63% less memory consumption than tuning on the dense grid.

Original: Diff-based Compression.
Revised: Diff-based Compression.

Original: We validate the performanceand effectiveness of diff-based compression in Fig. In this ablation, we first train our StreamRF on an entire sequence. The trained StreamRF is then saved withdiff-based compression enabled and disabled.
Revised: We validate the performance of diff-based compression in Meet Room dataset, with the following configurations, i.e., training on the entire sequence, saving the models with and without diff-based compression.

Original: We then compare the PSNR and saving size between the two test view sequences.
Revised: We then compare their performance on rendering quality (PSNR) and saving size (shown in Fig.

Original: We present the per frame PSNR to show whether diff-based compression impairs the performance in the long run.
Revised: 3 to investigate whether diff-based compression impairs the performance along with time steps (frames).

Original: The quantitative result shows that our compression method has negligible impact after a very long propagation.
Revised: The quantitative result shows that our compression method is effective with negligible loss on rendering quality compared to un-compressed manner, and the performance can be highly preserved without degeneration to time.

Original: Besides deducting overall storage size from 1015 MB to an average of 5.7 MB for each frame,we can observe that performance of our method does not downgrade with time and the incremental storage for each frame is roughly stable along time steps.
Revised: Compared to un-compressed version, using diff-based compression can effectively decrease the overall storage size from 1015 MB to an average of 5.7 MB for each frame, and the incremental storage for each frame is roughly stable along time steps.

Original: More details of model size reduction can be referred to supplementary materials.
Revised: More details of model size reduction can be referred to supplementary material.

Original: Pilot Model Guidance.
Revised: Pilot Model Guidance.

Original: We validate our pilot model guidance visually in Fig. 
Revised: We validate the strategy of using pilot model guidance visually in Fig. 4.

Original: For  base setting (right), we train the pilot model with 1,000 iterations and the full model with 500 iterations.
Revised: For the base setting (right), we train the pilot model with 1,000 iterations and the full model with 500 iterations.

Original: For the ablation setting (left), we directly train the full model with 1,000 iterations to make per frame training time equivalent in both settings.
Revised: For the ablation setting (left), we directly train the full model with 1,000 iterations to make per frame training time equivalent in both settings.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 4, pilot model guidance help reduce various kinds of artifacts: (top) vanishing hands and blurry elbow. (bottom) sparkle noise in a white wall, the fuzzy area between two people.
Revised: 4, pilot model guidance help reduce various kinds of artifacts: (top) vanishing hands and blurry elbow; (bottom) sparkle noise in a white wall, the fuzzy area between two people, and achieve better visual quality than lengthening full-scale training with the same iterations (1500 iters with a ~30% increase on time cost).

Original: These results verified our intention that we could use an easy-to-optimized small model to provide a more optimal initialization to the full model.
Revised: These results verified our intention that we could use an easy-to-optimized small model for benefiting full-scale model training. Pilot models can effectively indicate the voxels which need to be modified.

Revised: Using this strategy can prevent unnecessary modification in the full-scale model and reduce the loss of details in traditional downsample-upsample process.

Original: Related Work
Revised: 5.5 Limitation and Discussion

Revised: As shown in the Fig. 5, we can still observe artifacts like the loss of reconstructing high-frequency details and transparent/translucent objects in the rendering images which may "inherit" from using an explicit representation. It could be improved by extending the framework with explicit-implicit hybrid representations while remaining the benefit on training efficiency.

Revised: Moreover, the rendering quality may also benefit from the use of some sophisticated designs, e.g., assigning adaptive bandwidth to different parts by employing high-level understanding on scenes via extra supervisions or leveraging multiple key-frames when applying it in real-word scenarios with extremely long sequences.

Original: Static Novel View Synthesis Novel view synthesis  from a set of input images with known camera poses has been studied intensively in the field of computer vision and computer graphics.
Revised: Static Novel View Synthesis. Novel view synthesis learned from a set of input images is an important research topic in the field of computer vision and computer graphics.

Original: Earlier works like Lumigraph [21, 22] and Light-Field [23, 24] focus on interpolating the sampled light rays for realistic rendering.
Revised: Earlier works like Lumigraph [21, 22] and Light-Field [23, 24] focus on interpolating the sampled light rays for realistic rendering.

Original: It has been proved [25, 26] thatthe sampling density heavily depends on the underlying geometry of the captured scene.
Revised: Some methods show that sampling density heavily depends on the underlying geometry of the captured scene [25, 26].

Original: Thus, many works strive torecover an approximate [27, 28] or precise geometry model [29] to facilitate novel view rendering.
Revised: In this regard, many approaches are proposed to approximate [27, 28] or precise geometry model [29] to facilitate novel view rendering.

Original: Among those works, multiplane image (MPI) [20, 30–32] is a promising approximate geometry model that enables realistic rendering of generic scenes.
Revised: Multi-plane image (MPI) [18, 30–32] as a representative direction is used to represent geometric information and enable realistic rendering of generic scenes.

Original: More recently, NeRF [7] represents the radiance field with a multi-layer perceptron (MLP) network and combines it with differentiable volume rendering for view interpolation.
Revised: More recently, NeRF [7] represents the radiance field with a multi-layer perceptron (MLP) network learned by differentiable volume rendering and realizes view interpolation with impressive quality.

Original: This approach achieved tremendous success and there are many variants to push its limits towards every aspect.
Revised: A series of methods are developed to improve it from different aspects.

Original: The works  [8, 33–35] focus on improving rendering quality, while the works [36–38] strive to accelerate rendering speed, and other works [11, 39–42] propose to replace implicit function by an explicit or hybird counterpart to  boost  training convergence.
Revised: The works in [8, 33–35] focus on improving rendering quality by employing some tailored designs. Some methods [36, 37] are proposed to accelerate rendering speed by converting the large neural network to efficient representations with octree-based structure [38] or radiance maps [36] or a set of tiny MLPs [37]. Some approaches realize the boost of training efficiency by replacing the implicit formulation with explicit or hybrid representations [11, 39–42].

Original: Dynamic Novel View Synthesis Novel view synthesis has also been heavily studied with dynamic scenes.
Revised: Dynamic Novel View Synthesis.

Original: Early effort of generating 3D video can trace back to the 90s [4, 43], which computes explicit depth maps to facilitate novel view interpolation.
Revised: Early efforts of generating 3D video can trace back to the 90s [4, 43] by computing explicit depth maps to realize novel view interpolation.

Original: This approach has been combined with more sophisticated capturing hardware to generate  streamable free-viewpoint video in [1].
Revised: By equipping with more sophisticated capturing hardware, the work in [1] can generate high-quality streamable free-viewpoint video.

Original: Matusik and Pfister [44] further integrate camera arrays and 3D displays to build an end-to-end 3D TV system.
Revised: Matusik and Pfister [44] further integrate camera arrays and 3D displays to build an end-to-end 3D TV system.

Original: More recent works learn a neural network to capture radiance fields of dynamic scenes.
Revised: More recent works learn a neural network to capture radiance fields of dynamic scenes.

Original: A family of methods [45–51] realize view synthesis by training on monocular videos captured by a moving camera, i.e., viewpoints change with respect to time steps.
Revised: A family of methods [45–51] realize view synthesis by training on monocular videos captured by a moving camera, i.e., viewpoints change with respect to time steps.

Original: The works in [48–50] decouple the overall radiance field reconstruction to the learning of a static canonical template and a deformation field warping to the template.
Revised: The works in [48–50] decouple the overall radiance field reconstruction to the learning of a static canonical template and a deformation field warping to the template.

Original: Park et al.
Revised: Park et al.

Original: [51] lifts canonical templates into hyper-spaces to induce flexibility on modelling variations in topology.
Revised: [51] lifts canonical templates into hyper-spaces to induce flexibility on modeling variations in topology.

Original: Several methods model dynamics through the use of scene flows [45, 47] or depth prior [46], therefore they highly reply on the regularization of extra supervision (e.g., optical flow, depth or foreground-background segmentation estimated by well-trained models).
Revised: Several methods model dynamics through the use of scene flows [45, 47] or depth prior [46], therefore they highly rely on the regularization of extra supervision (e.g., optical flow, depth, or foreground-background segmentation estimated by welltrained models).

Original: Broxton et al.
Revised: Broxton et al.

Original: [2] and Attal et al.
Revised: [2] and Attal et al.

Original: [52] use multi-camera stereo rig and extend
Revised: [52] use multi-cameras stereo rig and extend

Original: MPI to MSI to support view interpolation with a larger field of view.
Revised: MPI to MSI to support view interpolation with a larger field of view.

Original: Bansal et al.
Revised: Bansal et al.

Original: [53] use multiple mobile phones to capture a dynamic scene and compose the static and dynamic components with neural networks for 4D space-time visualization.
Revised: [53] use multiple mobile phones to capture a dynamic scene and compose the static and dynamic components with neural networks for 4D space-time visualization.

Original: The closest work to our setting is Neural 3D 
Revised: The closest work to our setting is Neural 3D Video

Original: Video (N3DV)[10], which captures dynamic scenes with synchronous video sequences from multiple cameras.
Revised: (N3DV) [10], which captures dynamic scenes with synchronous video sequences from multiple cameras.

Original: It directly extends NeRF with an additional timecode input, and accelerates computation with a hybrid importance sampling and hierarchical training.
Revised: It directly extends NeRF with an additional timecode input and accelerates computation with a hybrid importance sampling and hierarchical training.

Original: N3DV can produce strong visual results given enough optimization, while it is very time consuming, requiring over per frame 5 GPU hours in training and 10+ seconds in rendering, making it impractical for real-world applications.
Revised: N3DV can produce strong visual results given enough optimization, while it is very time-consuming, requiring over per frame 5 GPU hours in training, making it impractical for real-world applications.

Original: On the other hand, some approaches adapt the advances in neural rendering for the modeling of specific objects, e.g., human bodies [54, 55] by integrating with domain-specific priors, e.g., skeleton or SMPL mesh [56].
Revised: On the other hand, some approaches adopt the advances in neural rendering for the modeling of specific objects, e.g., human bodies [54, 55] by integrating with domain-specific priors, e.g., skeleton or SMPL mesh [56].

Original: In contrast, our method imposes no prior in objects and is capable of representing more general scenes with complex deformation.
Revised: In contrast, our method imposes no prior in objects and is capable of representing more general scenes with complex deformation.

Original: Limitation & Future work

Original: [Figure]
Revised: [Figure]

Original: In Fig.

Original: 5 we can still observe artifacts like translucent foreground and texture sticking in the rendering images which may "inherit" from using an explicit representation.

Original: We expect to improve the rendering quality by extending the framework with explicit-implicit hybrid representations while remaining the benefit on training efficiency.

Original: Moreover, assigning adaptive bandwidth to different parts by employing high-level understanding on scenes via extra supervisions, accelerating to support real-time training, or leveraging multiple key-frames would be valuable directions to improve the framework when applying it in real-word scenarios with extremely long sequences.

Original:  Conclusion
Revised:  Conclusion

Original: We propose a novel streaming radiance field method for effectively reconstructing and rendering dynamic scenes with explicit grid representations.
Revised: We propose a novel streaming radiance field method for effectively reconstructing and rendering dynamic scenes with explicit grid representations.

Original: The modelling of dynamic scenes is formalized as an incremental learning paradigm which allows the method for handling on-the-fly video sequences without need of recording scenes in advance.
Revised: The modelling of dynamic scenes is formalized as an incremental learning paradigm which allows the method for handling on-the-fly video sequences without need of recording scenes in advance.

Original: By virtue of narrow band tuning, our approach can achieve a  fast training convergence.
Revised: By virtue of narrow band tuning, our approach can achieve a super fast training convergence.

Original: The storage cost induced by using explicit grids can be obviously reduced by performing difference-base compression.
Revised: The storage cost induced by using explicit grids can be obviously reduced by performing difference-base compression.

Original: We also present an efficient training mechanism with pilot model guidance to further improve model optimization.
Revised: We also present an efficient training mechanism with pilot model guidance to further improve model optimization.

Original: Experiments demonstrate that our approach is capable of training a high-performing model for dynamic scenes with the speed of 15s for tuning every frame, achieving  1000 × speedup over the state-of-the-art implicit dynamic methods. 
Revised: Experiments demonstrate that our approach is capable of training a high-performing model for dynamic scenes with the speed of 15s for tuning every frame, achieving ~ 1000 × speedup over the state-of-the-art implicit dynamic methods. As a direction of future work, we expect to further accelerate the framework to support real-time training. Checklist

Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Revised: [Yes] (b) Did you describe the limitations of your work?

Revised: [Yes] We show and discuss the limitation on visual quality in the experimental section.

Revised: (c) Did you discuss any potential negative societal impacts of your work?

Revised: [No] There is no obvious potential negative societal impact for this work as the task 3D video synthesis is a general extension of traditional videos.

Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Revised: [Yes] 2.

Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Revised: [N/A]

Revised: There are no theoretical results in this paper.

Revised: (b) Did you include complete proofs of all theoretical results?

Revised: [N/A]

Revised: There are no theoretical proofs in this paper.

Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Revised: [Yes] We provide the code for reproducing.

Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Revised: [Yes]

Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Revised: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Revised: [Yes] 4.

Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Revised: [Yes] We use the public datasets presented in existing works.

Revised: (b) Did you mention the license of the assets?

Revised: [Yes] CC-BY-NC 4.0 for N3DV dataset (c)

Revised: Did you include any new assets either in the supplemental material or as a URL?

Revised: [Yes] Our research code and a video for better showing qualitative results.

Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Revised: [Yes] We use publicly datasets presented in existing works.

Revised: We also use the data captured by our own.

Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Revised: [N/A]

Revised: The data does not contain personally identifiable information or offensive content.

Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Revised: [N/A]