DeVRF: Fast Deformable Voxel Radiance Fields for Dynamic Scenes
Abstract
Modeling dynamic scenes is important for many applications such as virtual reality and telepresence.
Despite achieving unprecedented fidelity for novel view synthesis in dynamic scenes, existing methods based on Neural Radiance Fields (NeRF) suffer from slow convergence ( i.e. , model training time measured in days ).
In this paper, we present DeVRF, a novel representation to accelerate learning dynamic radiance fields.
The core of DeVRF is to model both the 3D canonical space and 4D deformation field of a dynamic, non-rigid scene with explicit and discrete voxelbased representations.
However, it is quite challenging to train such a representation which has a large number of model parameters, often resulting in overfitting issues.
To overcome this challenge, we devise a novel static → dynamic learning paradigm together with a new data capture setup that is convenient to deploy in practice.
This paradigm unlocks efficient learning of deformable radiance fields via utilizing the 3D volumetric canonical space learnt from multi-view static images to ease the learning of 4D voxel deformation field with only few-view dynamic sequences.
To further improve the efficiency of our DeVRF and its synthesized novel view’s quality, we conduct thorough explorations and identify a set of strategies.
We evaluate DeVRF on both synthetic and real-world dynamic scenes with different types of deformation.
Experiments demonstrate that DeVRF achieves two orders of magnitude speedup ( 100× faster ) with on-par high-fidelity results compared to the previous state-of-the-art approaches.
The code and dataset are released in https://github.com/showlab/DeVRF .
Introduction
Free-viewpoint photorealistic view synthesis techniques from a set of captured images unleash new opportunities for immersive applications such as virtual reality, telepresence, and 3D animation production.
Recent advances in this domain mainly focus on static scenes, e.g., Neural Radiance
Fields (NeRF) [20], which implicitly represent rigid static scenes using 5D (spatial locations ( x, y, z ) and view directions ( θ, φ ) ) neural radiance fields.
Although achieving unprecedented fidelity for novel view synthesis, NeRF were mainly exploited under static scenes.
To unlock dynamic view synthesis, existing NeRF-based approaches either learn an additional MLP-based deformation field that maps coordinates in dynamic fields to NeRF-based canonical spaces [27, 24, 25, 38] or model dynamic scenes as 4D spatio-temporal radiance fields with relatively large MLPs [16, 7].
[Figure]
Despite being promising, NeRF is notoriously known for suffering from lengthy optimization time.
This issue becomes particularly prominent for non-rigid, dynamic scenes because the aforementioned
NeRF-based approaches require extra computation for the deformation MLPs [27, 24, 25, 38] or time-varying texture/density querying [16, 7], resulting in quite long training time (in “days”).
This motivates us to improve the learning efficiency of dynamic radiance fields.
Recent advances in static NeRF [35, 45] show that employing voxel grids, such a volumetric representation, can achieve fast convergence.
To adapt for dynamic scenes, one straightforward approach is to incorporate such a volumetric representation into the dynamic radiance field for fast neural modeling.
In this paper, we present a novel deformable voxel radiance field (DeVRF) that models both the 3D canonical space and 4D deformation field of a non-rigid, dynamic scene with explicit and discrete voxel-based representations, as illustrated in Fig.
1 (a-c).
However, we empirically observe that recklessly learning such a representation in dynamic radiance fields tends to plunge into the local optimum, i.e. , the overfitting issue, due to the large number of parameters in DeVRF.
To overcome this overfitting issue, we power our DeVRF with two novel designs: (1) We devise an efficient and practical learning paradigm, i.e. , static → dynamic , for learning deformable radiance fields.
The key idea behind this is that the 3D volumetric canonical space learned from multi-view static images can introduce inductive bias [3] to unlock efficient learning of deformable radiance fields.
Further, with such 3D priors, a dynamic scene can be effectively modeled with only a few fixed cameras.
We argue that such a few-fixed-cameras setup for dynamic scene data capture is more convenient than the moving camera (such as the setup used in D-NeRF [27]) in practice.
(2) Based on the static → dynamic paradigm, we conduct extensive explorations and identify a set of strategies customised for DeVRF to improve its efficiency and effectiveness.
These include a coarse-to-fine training strategy for the 4D deformation field to further improve efficiency, and three objectives to encourage our DeVRF to reconstruct dynamic radiance fields with high fidelity: deformation cycle consistency, optical flow supervisions, and total variation regularization.
Fig.
1 (d) shows that on five inward-facing synthetic scenes, two forward-facing real-world scenes and one inward-facing real-world scene, our approach enables fast dynamic radiance field modeling in about 10 minutes on a single NVIDIA GeForce RTX3090 GPU.
This is 100 × faster than SOTA approaches with comparable novel view synthesis quality.
To summarize, the major contributions of our paper are:
• A novel perspective of DeVRF is presented that enables fast non-rigid neural scene reconstruction, which achieves an impressive 100 × speedup compared to SOTA approaches with on-par high-fidelity.
• To the best of our knowledge, we are the first to incorporate the 4D voxel deformation field into dynamic radiance fields.
• We devise a static → dynamic learning paradigm that can boost performances with a low-cost yet effective capture setup.
Related Work
Novel View Synthesis for Static Scenes.
Earlier approaches [14, 31, 4, 6, 9, 21] tackle novel view synthesis by first building an explicit 3D reconstruction of a static scene, such as voxels and meshes, and then rendering novel views based on the reconstructed model.
On the other hand, multi-plane images [48, 19] represent a scene with multiple images at different depths and can reconstruct scenes with complex structures.
Recently, NeRF [20] achieves unprecedented fidelity for novel view synthesis by modeling static scenes with neural radiance fields.
Subsequent works have extended
NeRF to different scenarios, such as few-view novel view synthesis [12], multi-scale representation [2], and larger scenes [36, 43, 29].
However, these methods mainly focus on static scenes while the dynamic radiance fields reconstruction is more practical.
Novel View Synthesis for Dynamic Scenes.
In order to capture dynamic scenes with non-rigidly deforming objects, traditional non-rigid reconstruction approaches require depth information as additional input or only reconstruct sparse geometry [23, 11, 44, 5, 34].
VolumeDeform [11] introduces a unified volumetric representation to encode both the scene’s geometry and its motion but requires RGB-D sequences as input.
Neural Volumes [18] represents dynamic objects with a 3D voxel grid plus an implicit warp field, but requires an expensive multi-view capture rig and days to train.
Recent studies have built upon NeRF [20] and extended it to dynamic neural radiance field reconstruction by learning a mapping from dynamic to canonical field [27, 24, 25, 38] or building a 4D spatio-temporal radiance field [42, 16, 7, 15].
D-NeRF [27] learns a deformation field that maps coordinates in a dynamic field to a NeRF-based canonical space.
Nerfies [24] further associates latent codes in the deformation MLP and the canonical NeRF to tackle more challenging scenes such as moving humans.
HyperNeRF
[25] proposes to model the motion in a higher dimension space, representing the time-dependent radiance field by slicing through the hyperspace.
In contrast,
Video-NeRF [42] models the dynamic scene as 4D spatio-temporal radiance fields and addresses motion ambiguity using scene depth.
Sharing a similar idea on the 4D spatio-temporal field, NSFF [16] represents a dynamic scene as a time-variant function of geometry and appearance, and warps dynamic scene with 3D scene motion.
Lastly, several NeRF-based approaches have been proposed for modeling dynamic humans [8, 41, 17, 26, 32] but can not directly generalize to other scenes.
Although achieving promising results, existing methods require days of GPU training time, which is undesirable in real-world applications.
NeRF Acceleration.
In the light of NeRF’s substantial computational requirements for training and rendering, recent papers have proposed methods to improve its efficiency.
A line of work [46, 28, 10] focuses on NeRF rendering acceleration and has achieved encouraging results.
As for training acceleration, DVGO [35] models the radiance field with explicit and discretized volume representations, reducing training time to minutes .
Plenoxels [45] employs sparse voxel grids as the scene representation and uses spherical harmonics to model view-dependent appearance, reaching a similar training speedup.
Finally, Instant-ngp [22] proposes multiresolution hash encoding; together with a highly optimized GPU implementation, it can produce competitive results after seconds of training.
However, existing acceleration methods only focus on static scenes, while hardly any research, to our best knowledge, has studied NeRF acceleration for dynamic scenes.
Very recently,
Fourier PlenOctrees [39] extends PlenOctrees [46] to dynamic scenes by processing time-varying density and color in the frequency domain; however, the data capturing setup is expensive, and it still requires hours of training.
Instead, our proposed algorithm, DeVRF, offers a superior training speed while only requires a few cameras for data capture.
Method 3.1 Capture Setup
Deformable scenes undergo various types of deformations and motions, which can result in different scene properties such as object poses, shapes, and occlusions.
Therefore, capturing and modeling deformable scenes is nontrivial even for professional photographic studios.
Existing approaches [39, 13, 49, 1] attempt to capture 360° inward-facing dynamic scenes with multi-view sequences and thus require dozens of high-quality cameras.
On the other hand, D-NeRF [27] reconstructs deformable radiance fields from a sparse set of synthetic images rendered from a moving monocular camera.
However, in practice, it is particularly challenging to capture real-world 360° inward-facing dynamic scenes with a single moving camera due to various types of deformations and resulting occlusions in dynamic scenes, especially for scenes undergoing fast deformations.
As a result, subsequent studies [24, 25, 16, 7] only capture forward-facing videos of real-world dynamic scenes with a monocular camera.
Approach
No. of cameras Cost
Supported real-world use cases
D-NeRF[27], Nerfies[24]
Monocular
Low
Forward
[Table]
facing scenes, slow reconstruction in days .
Neural Volumes[18]
Multiple (34) High 360° inward-facing scenes, slow reconstruction in days .
Fourier PlenOctrees[39] Multiple (60) High 360° inward-facing scenes, fast reconstruction in 2hrs .
Ours
Few (4)
Low
[Table]
inward-facing and forward-facing scenes, super-fast reconstruction in 10mins .
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
Compared to dynamic scenes, it is much easier in practice to do multi-view capture for real-world static scenes with a monocular moving camera.
Therefore, we propose to separate the capture process of a dynamic scene into two stages: the first stage captures a static state using a moving monocular camera, and the second stage captures the scene in motion using a few fixed cameras.
In this capture setup, the multi-view static images provide complete 3D geometry and appearance information of the scene, while few-view dynamic sequences show how the scene deforms in 3D space over time; the entire capture process only requires a few cameras.
Tab. 1 compares our capture process with existing approaches in terms of the number of required cameras, cost, and supported real-world use cases.
3.2 Deformable Voxel Radiance Fields
As illustrated in Fig.
2, we present DeVRF to model both the 3D canonical space and 4D deformation field of a non-rigid scene with explicit and discrete voxel-based representations.
This volumetric representation allows us to efficiently query the deformation, density, and color of any 3D point at any time step in a deformable scene, thus largely improving the training and rendering efficiency.
In addition, we devise a static → dynamic learning paradigm that first learns a 3D volumetric canonical prior from multi-view static images (Fig.
2(a-b))
and transfers such prior to dynamic radiance fields reconstruction (Fig. 2(c-f)).
3D Volumetric Canonical Space.
We take inspiration from the volumetric representation of DVGO [35] and model the scene properties such as density and color of our 3D canonical space into voxel grids.
Such representation enables us to efficiently query the scene property of any 3D point via trilinear interpolation of its neighboring voxels,
[Equation]
where C is the dimension of scene property V p .
N x , N y , and N z are the voxel resolutions of V p in x - , y - , z - dimension.
As shown in Fig.
2(a-b), we learn the 3D volumetric canonical prior, i.e. , density grid V density and color grid V color , with multi-view static images { I S } via volumetric rendering.
Following DVGO [35], we employ softplus and post-activation after the trilinear interpolation of a 3D point in V density , as they are critical for sharp boundary and high-frequency geometry reconstruction.
We also apply a shallow MLP after the trilinear interpolation of a 3D point in V color to enable view-dependent color effects [35].
In our static → dynamic learning paradigm, the learned 3D volumetric canonical prior provides critical knowledge of the 3D geometry and appearance of the target dynamic scene, as few-view dynamic sequences alone struggle to reconstruct a complete deformable radiance field with high fidelity (as shown in Section 4).
4D Voxel Deformation Field.
We employ a 4D voxel deformation field V motion to efficiently represent the motion of a deformable scene.
As shown in Fig. 2(d), the arrow directions represent the motions of voxels, the color denotes the motion direction, and the arrow magnitude denotes the motion scale.
To synthesize a novel view at time step t , we shoot rays through image pixels and sample ray points X t
= { X ti } in 3D space.
The 3D motion ∆ X t → 0 = { ∆ X t → 0 i } from X t to their corresponding 3D points in the canonical space X
= {
X
i | X 0 i
= X ti + ∆ X t
→ 0 i } can be efficiently queried through quadruple interpolation of their neighboring voxels at neighboring time steps in the 4D backward deformation field,
[Equation]
where C is the degrees of freedom (DoFs) of the sample point motion.
We use C = 3 in this paper, i.e. , assign a displacement vector to each sample point.
N t is the number of key time steps that can be user-defined based on the scene motion properties.
Therefore, scene properties of X t can then be obtained by querying the scene properties of their corresponding canonical points X 0 through trilinear interpolation in the volumetric canonical space.
Finally, pixel colors can be calculated through volume rendering with the sampled scene properties along each ray, as illustrated in Fig.
2(e-f).
3.3 Optimization
Training the DeVRF is quite challenging, mainly because a large number of model parameters may lead to overfitting or suboptimal solutions.
This section describes the training strategy and optimization losses that we design to facilitate fast optimization of the DeVRF.
Coarse-to-Fine Optimization.
For a dense 4D voxel deformation field with N t × C × N x × N y × N z resolution, there could be millions of free parameters, which are prone to overfitting and suboptimal solutions.
To solve this problem, we employ a coarse-to-fine training strategy.
Specifically, in our experiments, we progressively up-scale the x - y - z resolution of the 4D voxel deformation field from 10 × 10 × 10 to 160 × 160 × 160 .
With this strategy, the 4D voxel deformation field first learns a rough motion at the coarse stage, which is thereafter progressively refined in finer stages.
Our experiments demonstrate that the coarse-to-fine strategy can effectively smoothen the optimization landscape of the 4D voxel deformation field and remove most suboptimal solutions, thus largely improving the training efficiency and accuracy.
Re-rendering Loss.
With sampled properties at X t , the color of a pixel can be calculated through volume rendering, i.e. , by integrating the density and color of X t along a ray r [20]:
[Equation]
where N r is the number of sampled deformed points along the ray, T i represents the probability of light transmitting through ray r to the i - th sampled point, and 1 − exp ( − σ i δ i ) is the probability that light terminates at the i - th point.
δ i is the distance between adjacent sampled points, and σ i ,
c i denote the density and color of deformed point i , respectively.
c bg is the pre-defined background color.
Given the few-view training dynamic sequences with calibrated poses { I D } , DeVRF is optimized by minimizing the photometric MSE loss between the observed pixels color C ( r ) and the rendered pixels color ˆ C ( r ) :
[Equation]
where R is the set of rays in a mini-batch.
4D Deformation Cycle Consistency.
As illustrated in Fig. 2(d), the forward deformation is modeled as 4D forward deformation voxels with the same resolution as the 4D backward deformation voxels, and we enforce 4D deformation cycle consistency between backward and forward motion, which regularizes the learned deformation field.
In the 4D deformation cycle, backward motion vectors ∆ X t → 0 models the motion from X t to X 0 ; in contrast, forward motion vectors ∆ X
→ t models the motion from X 0 to their corresponding 3D points in the dynamic space ˜ X t
= { ˜ X ti | ˜
X ti =
X
i + ∆ X
→ t i } .
The 4D motion cycle consistency can now be realized by minimizing the following cycle consistency loss L Cycle ( t ) ,
[Equation]
where N s is the number of sampled 3D points in a mini-batch.
Optical Flow Supervision.
The DeVRF is indirectly supervised by 2D optical flows estimated from consecutive frames of each dynamic sequence using a pre-trained RAFT model [37].
For X t and their corresponding X 0 , we first compute the corresponding 3D points of X 0 at t − 1 time step via forward motion
˜ X t − 1 = { ˜ X t − 1 i | ˜
X t − 1 i =
[Equation]
After that, we project ˜ X t − 1 onto the reference camera and get their pixel locations ˜
P t − 1 = { ˜ P t − 1 i } , and compute the induced optical flow with respect to the pixel location P t = { P ti } from which the rays of X t are cast.
We enforce the induced flow to be the same as the estimated flow by minimizing L Flow ( t ) ,
[Equation]
where w r ,i =
T i (1 − exp ( − σ i δ i )) is the ray termination weights from Eq.
(3), and f P t r ,i is the estimated 2D backward optical flow at pixel P t r ,i .
Total Variation Regularization.
We additionally employ a total variation prior [30] when training the 4D voxel deformation field to enforce the motion smoothness between neighboring voxels.
At time step t ,
[Equation]
where ∆ 2 x,y,z is the squared difference of motion vectors between voxel v i and its neighbors along x, y, z axes.
¯ N = N x × N y × N z denotes the number of voxels.
Training Objective.
The overall training objective of DeVRF is the combination of per-pixel rerendering loss L Render , cycle consistency loss L Cycle , optical flow loss L Flow , and total variation regularization L TV :
[Equation]
where ω Render , ω Cycle , ω Flow , ω TV are weights for corresponding losses.
[Table]
Experiments
We extensively evaluate the DeVRF on various types of datasets, including five synthetic 3 360° inward-facing dynamic scenes, two real-world forward-facing dynamic scenes
[Table]
and one real-world 360° inward-facing dynamic scene.
We run all experiments on a single NVIDIA GeForce RTX
GPU.
During training, we set ω
Render
= 1 , ω
Cycle = 100 , ω Flow = 0 .
005 , and ω TV = 1 for all scenes.
4.1 Comparisons with SOTA Approaches
To demonstrate the performance of DeVRF, we compare DeVRF to various types of SOTA approaches, including a volumetric method Neural Volumes [18], NeRF-based methods D-NeRF [27], Nerfies [24], HyperNeRF
[25], and a time-modulated method NSFF [16].
For a fair comparison, since
DeVRF follows a static → dynamic learning paradigm, we additionally implement 2-stage versions of D-NeRF, Nerfies, and HyperNeRF to learn a canonical space prior in the first stage and then optimize a deformation network in the second stage.
To show the effectiveness of our low-cost capture strategy for dynamic scenes, we also train these baselines using only a few-view dynamic sequences and observe a significant performance drop compared to those trained with both static and dynamic data.
For quantitative comparison, peak signal-to-noise ratio (PSNR), structural similarity index (SSIM) [40], and Learned Perceptual Image Patch Similarity (LPIPS) [47] with VGG [33] are employed as evaluation metrics 4 .
Evaluation on inward-facing synthetic and real-world deformable scenes.
We selected five synthetic dynamic scenes with various types of deformations and motions, and rendered synthetic images in 400 × 400 pixels under the 360° inward-facing setup.
For each scene, we use 100-view static images and 4-view dynamic sequences with 50 frames ( i.e. , time steps) as training data for all approaches, and randomly select another 2 views at each time step for test.
In addition, we collected one 360° inward-facing real-world deformable scene in 540 × 960 pixels.
With our data capture setup, only 4 cameras are required to capture dynamic scenes, and we choose 3 views of them as training data and the other view as test data.
We report the metrics of the real-world scene as well as the average metrics of five synthetic scenes for all approaches in Tab. 2 and leave the per-scene metrics to supplementary material.
As shown in
Tab. 2, for synthetic and real-world scenes, DeVRF achieves the best performance in terms of PSNR and LPIPS, and the second- or third-best in terms of SSIM among all approaches.
Most importantly, our per-scene optimization only takes less than 10mins with 4 . 4GB to 6 .
6GB GPU memory on a
P l an t
R abb i t
F l o w e r360 °
K u k a
D-NeRF
Nerfies
HyperNeRF
NSFF
Ours Groundtruth
Lego
F l oa t
i ng r obo t
D a i sy
G l o v e single NVIDIA GeForce RTX3090 GPU, which is about two orders of magnitude faster than other approaches.
The above quantitative comparison demonstrates the efficiency and effectiveness of
DeVRF.
Besides, the qualitative results of DeVRF and baselines on synthetic and real-world scenes are illustrated in Fig.
3, where DeVRF achieves on-par high-fidelity in comparison to SOTA methods.
Please see the supplementary video for more results.
For a fair comparison, we additionally report the results of the 2-stage versions for D-NeRF [27],
Nerfies [24], and HyperNeRF [25] in Tab. 2.
Since these approaches are not designed to separately learn a canonical space and a deformation field, there is no significant difference in the results between their 2-stage versions and 1-stage versions using the training dataset.
Furthermore, we also report the results of these baselines trained with dynamic data only (baseline-dynamic) in Tab. 2.
Their performances drop significantly compared to the results trained with both static and dynamic data.
In addition, since Neural Volumes [18] requires dozens of dynamic sequences as input, its performance is poor with our few-view dynamic sequences.
The experimental results not only show the effectiveness of our low-cost data capture process and the proposed DeVRF model; but also validate our observation that few-view dynamic sequences alone fail to provide complete information about the dynamic scene, while static multi-view data can favorably serve as a supplement.
[Table]
Evaluation on forward-facing real-world deformable
scenes.
We collected two forward-facing real-world deformable scenes in 540 × 960 pixels using 4 cameras, and we chose 3 views of them as training data and the other view as test data.
To handle forward-facing scenes, we adapt DeVRF to use normalized device coordinates (NDC) and multi-plane images (MPI) as in DVGO [35].
As shown in Tab. 3, DeVRF achieves the best result in the plant scene in terms of LPIPS metric and the second-best result in the rabbit scene in terms of all metrics.
Fig.
3 also demonstrates qualitative comparisons on these two scenes.
4.2 Ablation Study
Ablation study of DeVRF components.
We carry out ablation studies on both synthetic and realworld scenes to evaluate the effectiveness of each proposed component in DeVRF.
We progressively ablate each component from optical flow, cycle consistency, total variation, to coarse-to-fine strategy.
As shown in Tab. 2 and 3, the performance of DeVRF progressively drops with the disabling of each component, where disabling the coarse-to-fine training strategy causes the most significant performance drop.
This is as expected since the coarse-to-fine training strategy is critical to reducing local minimums during optimization.
1 View 2 Views
Lego
F l oa t
i ng r obo t
D a i sy
K u k a
G l o v e
Groundtruth
Lego o t
F l oa t
i ng r obo
D a i sy
Ablation study of dynamic training views on synthetic dataset.
Our static → dynamic learning paradigm is based on a low-cost yet effective capture setup: the multi-view static images provide complete 3D geometry and appearance information of the scene, while few-view dynamic sequences show how the scene deforms in 3D space over time.
The entire capture process only requires a few cameras that are convenient to deploy in practice.
To evaluate the influence of the number of dynamic views, we conduct additional ablations for DeVRF on five inward-facing synthetic dynamic scenes and report the per-scene metrics as well as the average metrics with respect to the number of dynamic training views.
As shown in Fig.
5, given the same multi-view static images, the performance of
DeVRF largely improves with the increment of dynamic training views and almost saturates at six dynamic views, and the four dynamic training views used in our paper can yield comparable results compared to six dynamic views.
We additionally visualize the qualitative results of DeVRF with different numbers of dynamic training views in Fig.
Therefore, in our static → dynamic learning paradigm, with static multi-view data as a supplement, only a few (e.g., four) dynamic views are required to significantly boost the performance of dynamic neural radiance fields reconstruction.
This further demonstrates the effectiveness of our low-cost data capture process and the DeVRF model.
[Figure]
Conclusion
We introduced DeVRF, a novel approach to tackle the challenging task of fast non-rigid radiance field reconstruction by modeling both the 3D canonical space and 4D deformation field of a dynamic scene with voxel-based representations.
The DeVRF can be efficiently optimized in two major steps.
We first proposed a static → dynamic learning paradigm to pinpoint that the 3D volumetric canonical prior can be effectively transferred into the 4D voxel deformation field.
Second, based on this learning paradigm, we developed a series of optimization strategies, including coarse-to-fine learning, deformation cycle consistency, optical flow supervisions, and total variation priors.
Such DeVRF finally produced a 100 × faster training efficiency with on-par high-fidelity results in comparison to
SOTA approaches.
We believe our DeVRF can provide a complement to existing literature and new insights into the view synthesis community.
Limitations and Future Work.
Although DeVRF achieves fast deformable radiance field reconstruction, the model size is large due to its large number of parameters.
In addition, DeVRF currently does not synchronously optimize the 3D canonical space prior during the second stage, and thus may not be able to model drastic deformations.
We consider these limitations as faithful future directions.
Acknowledgements
This project is supported by the National Research Foundation, Singapore under its NRFF Award
NRF-NRFF13-2021-0008, Mike Zheng Shou’s Start-Up Grant from NUS, and National Research
Foundation, Singapore and A*STAR, under its RIE2020 Industry Alignment Fund – Industry Collaboration Projects (IAF-ICP) grant call (Grant No. I2001E0059).
The computational work for this article was partially performed on resources of the National Supercomputing Centre, Singapore.
Jia-Wei Liu is supported by NUS IDS-ISEP scholarship.
[-Most importantly, our per-scene optimization only takes less than 10mins with less than 5 . 
-]