SGAM: Building a Virtual 3D World through Simultaneous Generation and Mapping
Abstract
We present simultaneous generation and mapping (SGAM), a novel 3D scene generation algorithm.
Our goal is to produce a realistic, globally consistent 3D world on a large scale.
Achieving this goal is challenging and goes beyond the capacities of existing 3D generation methods, which fail to scale up to create large scenes.
Video generation and view synthesis methods cannot produce a globally consistent 3D scene structure.
Towards tackling the challenges, we take a hybrid approach that integrates generative sensor modeling with 3D reconstruction.
Our proposed approach is an autoregressive generative framework that simultaneously generates sensor data at novel viewpoints and builds a 3D map at each timestamp.
Given an arbitrary camera trajectory, our method repeatedly applies this generationand-mapping process for thousands of steps, allowing us to create a gigantic virtual world.
Our model can be trained from RGB-D sequences without having access to the complete 3D scene structure.
The generated scenes are readily compatible with various interactive environments and rendering engines.
Building upon the CLEVR dataset, we propose a large-scale 3D scene generation benchmark, CLEVR-Inﬁnite dataset, and demonstrate ours can generate consistent, realistic, and geometricallyplausible scenes that compare favorably to existing view synthesis methods. Our code is available at this link and supplementary material at this link.
Introduction
Human perception goes way beyond simple recognition and reconstruction.
Our extraordinary abilities not only allow us to make sense of what we see, but also enable us to extrapolate and imagine what we do not ( e.g. , hallucinate what the scene may look like outside the image).
With just a simple glance, we can effortlessly re-build a mental world, that may not be exactly like the original, but is perceived by our brain to be the same.
In fact, it is such a profound ability that draws us apart from existing AI systems.
The goal of this paper is to equip computational machines with similar capabilities.
We aim to develop a program that can create an arbitrarily large, realistic, and consistent 3D world from a single snapshot of the scene .
The task is of paramount interest to many applications in computer vision (60; 40; 74), computer graphics (65; 65), geography (5), and robotics (13; 44), since it unlocks numerous potentials.
For instance, it may allow us to build an interactive virtual environment without any costly and laborious 3D modeling pipeline (12).
Indeed, there has been a consistent pursuit within the computer vision and graphics community in the past few decades (17; 16; 85; 65; 9; 68; 72), where people attempt to design algorithms and models that are capable of extrapolating and hallucinating the world from a single input image.
Unfortunately, a large body of efforts have been focusing on modeling in the 2D image space (17; 16; 85; 18; 37; 9), or require external 3D assets and manually speciﬁed rules for restricted 3D generation (10).
Recently, with the advent of deep generative models (36; 76; 40; 61; 60), researchers have made great strides 36th Conference on Neural Information Processing Systems (NeurIPS 2022). in unconstrained 3D syntheses.
However, due to the highly complex structure of the task, these approaches mostly focus on object-centric scenarios, or generating small-scale environments.
With these problems in mind, we present Simultaneous Generation and Mapping (SGAM), a novel 3D scene generation algorithm.
SGAM builds upon insights from state-space representation in
SLAM (48; 23) as well as recent generative models (73; 18; 66).
At its core lies two key modules: (i) a generative sensor module that takes a scene representation and a novel query viewpoint as input and renders a photo-realistic RGB-D image of a 3D scene; and (ii) a mapping module that exploits the newly generated RGB-D observation at a given camera pose and updates the scene representation.
Given an arbitrary camera trajectory, our method iteratively applies this procedure to construct the 3D scene.
By grounding scene generation with “a map”, we can produce a realistic and spatially consistent 3D world at a signiﬁcantly larger scale than any previous 3D generation effort without drifting or collapsing.
Additionally, our generated scenes can be represented explicitly as textured 3D models, making them readily compatible with various interactive environments and rendering engines.
We note that we are not the ﬁrst to tackle the task of scene synthesis under large viewpoint differences (56; 60; 40).
Several recent efforts have demonstrated promising results in this direction, from which our method is inspired.
Nonetheless, these works do not produce a consistent 3D scene representation of the inﬁnite world and potentially suffer from mode collapse (60; 40).
We validate the efﬁcacy of our method on two novel large-scale 3D scene datasets, Clevr-inﬁnite and GoogleEarth.
Both benchmarks produce substantially larger 3D scenes than existing datasets.
This allows us to benchmark large-scale 3D scene generation.
We evaluate the standard metrics for reference-based and non-reference-based perceptual image quality, such as PSNR, SSIM (75),
LPIPS (82), IS (63), and FID (27).
We further benchmark the realism of the generated 3D scenes in terms of Jensen-Shannon divergence and maximum-mean discrepancy (22).
Our experimental results suggest that 1) our method could produce more realistic results than existing single-image novel view synthesis methods; 2) our approach can generate a more meaningful 3D world than the prior perpetual view generation algorithms.
Related Work
Image generation: How to synthesize an image realistically has been a long-standing problem in computer vision.
The task dates back to 60s (32; 4) where researchers attempted to generate textures by matching statistics (42; 26).
Through parametric sampling (85) or non-parametric matching (17; 16), they were able to synthesize an inﬁnite amount of high-ﬁdelity texture images.
Unfortunately, these approaches fall short when applied to natural images, since the images have much higher complexity.
Recently, with the help of deep generative models (21; 35; 55; 28; 66), researchers have demonstrated promising results on generating photo-realistic images (30; 33; 34).
With proper design and inductive biases (73; 18; 11), they are even able to scale the output to mega-pixel level (70; 9).
In this paper, we build our generation module on top of VQ-GAN (18).
However, instead of treating scene generation as a pure 2D task as in the past, we explicitly consider the relationship between appearance and geometry — both at the input level and the output level.
By encoding both information into the quantized codebook, we are able to “grow” the 3D scene from an initial seed image and generate a boundless world, which is not exactly like the original, but will be perceived by humans to be the same region. 3D Generation: 3D modeling and synthesis have been an active yet challenging research problem for decades (8; 6; 68).
Recently, with the development of image generation techniques (21; 35; 57), the ﬁeld has experienced a rapid growth (78; 49; 69).
Drawing inspiration from its corresponding
Generative Sensing …
Mapping
Input observation
Graphics compatible 3D scene …
SGAM:
Simultaneous Generation and Mapping the new to update initial a trajectory, we repeatedly run this generation- to create the virtual world. 2D analogue, researchers have been able to generate high quality point clouds (38; 79; 7), voxels (19; 77), meshes (24; 20), etc. Unfortunately, since 3D solution space is much more intricate than that of 2D, these approaches are typically object-centric; also, they mostly focus on generating common objects in our daily lives (25; 39; 14; 15).
To enable scene-level synthesis, researchers have sought to incorporate more structures into the generation pipeline (47; 54; 51; 53), such as reducing the output space from full 3D to predeﬁned, compact representation (52; 46; 67).
While these strategies greatly alleviate the issue, the generated scene scale is still rather limited ( e.g. , an indoor environment).
In this work, we push the boundary of 3D generation systems and present a model that is capable of generating coherent 3D scene structure over thousands of meters.
The produced high-quality 3D world not only allows us to build a large-scale map, but also provides geometric grounding for visual appearance synthesis.
Generative sensor models:
Generative sensor models serve critical roles in existing graphics/robotics simulation systems (44; 10).
Their goal is to faithfully simulate the sensor measurement in real world.
Depending on the sensor modalities ( e.g. , rgb/depth cameras, LiDAR scanner), the output can be either photometric or geometric quantities.
These methods usually assume an underlying (proxy) 3D geometry is given (80; 59) and then perform conditional generation (43).
The geometry can take various forms ( e.g. , point clouds (2), surfels (80; 44), meshes (58), etc), as long as it can effectively ground the system.
Our full model is a combination of a generative sensor module and a mapping module.
The former takes as input RGBD images and simulates both the appearances and the geometry of the scene.
Single image novel view synthesis (NVS): Our work is also related to single image NVS approaches (62; 71; 76; 29), whose goal is to simulate novel views of a scene using only visual cues from one input image.
These methods typically rely on neural networks to learn statistical priors from data.
Together with carefully designed scene representations ( e.g. , radiance ﬁelds (81; 74), multi-plane images (71), layered depth images (72; 64), etc), they are able to synthesize scene appearance with mild viewpoint change.
To push the frontier even further, more recently, researchers have proposed to go beyond local view synthesis, and instead simulate the scene under larger viewpoint change (60; 40; 36; 61; 56).
For instance, generate one room from the other.
The task is extremely challenging as it requires the model to learn not only the underlying scene representations, but also the long-range dependencies.
Without proper state/map representations, the generated output will easily drift away and the constructed world will no longer be coherent (40; 61).
To address this issue, in this work, we develop a model that simultaneously generates novel views and builds a map.
By integrating synthesized results into the map, we are able to query relevant information whenever we re-visit a spot and then decode, allowing us to produce realistic and consistent video footage. [Figure]
SGAM: Simultaneous Generation and Mapping
In this paper, we seek to devise a method that can generate an arbitrarily large, realistic, and consistent 3D world from a single snapshot of the scene .
Based on the observation that current 3D scene generation approaches are prone to drifting and often produce inconsistent 3D worlds, we present a novel framework, dubbed as SGAM, that simultaneously generates novel scenes and constructs world maps.
By grounding the generation and the mapping process in a tightly coupled fashion, we are able to produce a 3D world at a signiﬁcantly larger scale than any previous 3D generation effort.
We unfold this section by formulating our task in a probabilistic framework.
Then we describe how each module can be implemented with a neural network and how we conduct inference to construct the virtual 3D world.
Finally, we discuss the learning procedure, our design choices, and the relationships to existing methods.
3.1 Problem Formulation
Given a camera trajectory { ⇠ t 2 SE (3) } over discrete time steps { t } and an initial RGB-D sensor observation x 0 2 R W ⇥ H ⇥ 4 , SGAM aims to compute an estimate of the sensor observation x t R W ⇥ H ⇥ 4 and a perceived map of the environment M t as a set of 3D surface points and associated colors { p i 2 R 3 , c
i 2 R 3 } 1 ...N t .
Due to the inherent randomness, we consider all the quantities to be random variables, so the objective is to model:
[Equation]
Using Bayes rule and Markov property, SGAM factors the joint probability into a product of conditional probability over all discrete time steps and concerns the following auto-regressive model at each step t :
[Equation]
As shown from this equation, at each step SGAM needs to estimate two conditional probabilities: 1) a generative sensor module p ✓ ( x t |M t  1 , ⇠ t ) which renders a realistic and complete sensor observation x t given an (incomplete) scene representation M t and a query viewpoint ⇠ t ; 2) a mapping module p ✓ ( M t |M t  1 , x t , ⇠ t ) , which integrates the new sensory observation to update the scene.
Next we will describe each module in detail.
Frame
Frame
Frame 450 Frame Frame
Frame
Frame 450 Frame
G F V S
E xp li c it
G F V S
I m p li c
it
I n ﬁ n it e
N a t u r e
[Table]
u r s
[Table]
We ﬁrst describe our generative sensing module p ( x t |M t  1 , ⇠ t ) , which samples a new RGBD observation x t given a query viewpoint ⇠ t and the current scene representation M t  1 .
Three desired properties of sensor data generation are 1) consistency the generated sensory data is consistent with existing knowledge about the 3D scene; 2) realism : the resulting images look realistic, regardless of how many auto-regressive steps we have run; 3) completeness : the sensing module should produce complete observation over both explored and unexplored areas, allowing our scene representation to expand the virtual world as camera moves.
Towards these goals, we propose a two-step process including neural rendering and conditional generative reﬁnement as shown in Figure 2.
Scene rendering: We ﬁrstly create a guidance image at the target pose by rendering the current scene x 0 t = ⇡
( M t  1 , ⇠ t ) , where ⇡ is an image-based rendering procedure.
Given a triangular mesh extracted from the scene representation M t  1 , the image-based rendering pipeline ﬁrst produces a dense depth map at the target pose ⇠ t through rasterization.
The rendered depth map is then used for inverse warping, producing an RGB image by transferring color appearance from existing sensor observations to the target image.
A geometry consistency check will be conducted between each pixel at the rendered target depth and source depth, ensuring the color comes from the same point in 3D.
The image-based rendering step gives us an incomplete RGBD observation, where the missing pixels are due to the incompleteness of the current scene representation.
Nevertheless, this step promotes the consistency between the generated sensory data and our current knowledge about the scene, built upon the past generated sensory data.
Generative reﬁnement: Next, we generate the ﬁnal complete virtual sensor observation x t based on the incomplete rendering x 0 t .
Inspired by the success of the discrete latent variable models (73; 18; 9) in conditional image generation, we propose a lightweight latent VQGAN-based encoder and decoder module.
Prior work (61; 18), introduces extra structures like a latent transformer sampler, to out-paint the missing regions. To avoid heavy computation, we simply enforce our SGAM encoder to take dual responsibility of encoding input observation and extrapolation in empty regions. The detailed procedure is as follows.
The incomplete image x 0 t is fed into an encoder network during inference, which outputs a continuous-valued latent feature map.
For a latent feature on this map g enc , we simply select the quantized codeword embedding that is the closest to the latent continuous features in terms of Euclidean distance.
Finally, we generate the outpainted sensory output by sampling each latent code from the proposal distribution, create a quantized feature map based on the codebook, and feed the quantized feature into the decoder:
[Equation]
where q ( z = i | x 0 t ) is the probability of i -th latent code following a K-
[Table]
Compared to prevailing VQGAN-based work for conditional view synthesis, our method does not require comprehensive transformer-based latent code sampling, leading to a signiﬁcant boost in speed.
Furthermore, we ﬁnd our conditional generation produces comparable or better performance in perceptual quality.
3.3 Mapping
Representation Maintaining an efﬁcient and ﬂexible scene representation is crucial for high ﬁdelity 3D virtual world generation at a large scale.
Inspired by its success in depth-based SLAM (48) and 3D reconstruction (83), we leverage hashing based volumetric representation (50) for representing
M t .
Each voxel p in the space maintains a signed distance value M t ( p ) : R 3 ! R , where the magnitude encodes the distance to the surface boundary, and the sign represents whether it is interior or exterior.
Such representation is ﬂexible, allowing us to update online at each step when new sensory data is generated.
Additionally, the voxel-hashing structure enables sparse storage, allowing the map to scale to huge scenes.
Integration At each time t , given a new sensor observation x t , we update our map M t following
KinectFusion (48).
Speciﬁcally, for each value, a weighted sum update is conducted for points along the camera ray:
[Equation]
where O ( x t , p ) is the observed signed distance function at p , calculated by the new sensor observation x t ; w t is a counter that summarizes how many steps that p has been updated.
After each integration step, we will also conduct marching cube (41) to extract triangular mesh, allowing us to render images from a new viewpoint during the generative sensing process.
3.4 Learning
SGAM adopts a two-stage training pipeline.
In the ﬁrst stage, inspired by VQGAN (18), we ﬁrst pre-trained a discrete-latent encoder, decoder, and codebook on the clean RGB-D image dataset in an auto-encoding fashion.
During training, we perform online re-initialization of the codebook embedding by performing k-means clustering over a past feature cache on the unquantized feature embedding space, whenever the active codeword hit rate within a ﬁxed time interval drops below a pre-deﬁned threshold.
Following the practice in GFVS (61), we use L 1 reconstruction loss, perceptual loss (82), patch-gan-based discriminator loss (30) and commitment loss (18).
Our second stage aims to train the encoder so that it learns to complete the latent code space.
Speciﬁcally, we ﬁne-tune the encoder with incomplete/complete image pairs.
The incomplete RGB-D images are simulated through inverse warping from neighboring viewpoints.
We use perceptual loss (82) and patch-gan-based discriminator loss (30) at this stage.
In this stage, we freeze the codebook and decoder parameters.
We ﬁnd this mechanism helps maintain the output images realistic even after many unrolling steps during inference.
We use Adam to train our entire network.
Learning through a non-differentiable quantization step is hard.
To stabilize training, we ﬁrst train the encoder-decoder network without quantization for the ﬁrst 30k iterations and then add it back and train the entire network for another 60k iterations using the straight-through gradient estimator trick (3) to back-propagate gradient. [Figure]
Experiments
We ﬁrst evaluate the effectiveness of SGAM on a large-scale synthetic dataset.
Next, we comprehensively study the characteristics of our method.
Finally we showcase how SGAM can be applied in real-world scenarios to generate high-quality 3D worlds.
4.1 Setup
Dataset:
SGAM requires a myriad of RGB-D images to train.
Unfortunately, existing static, large-scale 3D datasets, such as ACID (40) and RealEstate10K (84), do not provide accurate depth.
We thus exploit Blender and the assets from CLEVR (31) to render an extremely large-scale synthetic benchmark, which we called CLEVR-Inﬁnite.
CLEVR-Inﬁnite contains 72 training scenes and 8 test scenes.
For each scene, we sampled a ﬁxed number of objects from the asset bank with replacement and randomly placed them on the ground plane.
The asset bank consists of four primitive shapes, each of which with three different scales.
We set all object material to diffuse, and determine their color randomly.
We use the same environmental light across all scenes.
The cameras are distributed uniformly among the 50 ⇥ 50 grid plane right above and parallel to the ground.
We set the pitch ( i.e. , the angle between the look-at vector and the plane) of the cameras to be 80  .
In total, the dataset contains 200k RGB-D images at the resolution of 256 ⇥ 256 .
Metrics:
We evaluate the generated 3D world both locally at the image level and globally at the scene level .
The former measures to what degree the synthetic scene reproduces the local details, while the latter quantiﬁes how well the virtual world matches the statistics of the original counterpart.
Following previous work (45; 81), we ﬁrst use Peak Signal-to-Noise Ratio (PSNR), Structural
Similarity (SSIM), and LPIPS (82) to measure the image level similarity with respect to the GT.
However, since our model is generative, we might synthesize scenes that are plausible yet different from the original one.
We thus further employ Fréchet inception distance (FID) (27) to measure the generative quality.
Speciﬁcally, two FID score are reported: (i) ref-FID, which we compare against the GT; and (ii) FID, which we compare against a separate set of images from test set.
As for scene-level evaluation, we follow prior art in 3D generative models (1; 79) to gauge the overall geometry quality with a diverse set of metrics: Jensen-Shannon divergence (JSD), maximum mean discrepancy (MMD), coverage (COV), minimum matching distance (MinDist), and 1-nearest neighbor accuracy (1-NNA).
Speciﬁcally, we ﬁrst divide the whole scene into non-overlapping regions.
Each region is then discretized into 15 3 voxels.
Finally, we compute the 3D histogram within each local grid and employ cosine distance to compute the metrics.
To ensure the generated scene as a whole looks realistic, we also compute the Inception Score (IS) and FID against random sampled scenes from the test set.
Lastly, to demonstrate the efﬁciency of our method, we report the runtime of a single prediction step.
The numbers are benchmarked on Nvidia Quadro 8000.
We sample the latent code that is closest to image features during one-step and scene-level prediction for our method.
[Figure]
Frame Frame
Frame Frame
[Figure] [Figure]
[Figure]
Frame
G F V S
E xp li c it
G F V S
I m p li c
it
I n ﬁ n it e
N a t u r e
O u r s
Baselines: We compare SGAM against state-of-the-art approaches in long range scene generation: GFVS (61) and InﬁniteNature (40). For GFVS, we consider both the explicit-image and implicitdepth variants with the following modiﬁcations: ﬁrst, we use a single codebook to encode RGB-D rather than two, which better combines texture and geometry during scene expansion; second, to support multi-image input, we conduct average pooling over the features of source images before passing it to an MLP to infer the target code.
For InﬁniteNature, since the authors did not release their training scripts, we re-implement their method in Pytorch.
We remove the geometric grounding step since we ﬁnd it harmful when we unroll the scene in CLEVR-Inﬁnite.
Implementation details:
During full scene evaluation, we generate the scene in a zig-zag fashion on a 30 ⇥ 30 grid.
We start from coordinate (0, 0) and unroll for 900 steps.
The above procedure is repeated for 10 times with different seed images.
For a fair comparison, GFVS-implicit, GFVSexplicit and SGAM share the same codebook embedding and the same weights for decoder.
We set the vocabulary size to 16384.
We implement SGAM in PyTorch, and train with a batch size ofon 2 Nvidia A40 GPU until convergence.
For other baselines, we maximize batch size to ﬁt GPU memory, and train until convergence.
During the ﬁrst-stage codebook training, we follow the exact training conﬁg as GFVS ﬁrst-stage training in the ofﬁcial codebase.
GFVS-implicit, GFVS-explicit and SGAM share the same learning rate at 0.0625 for the second-stage trainig.
4.2 Experimental results
As shown in Tab. 1, SGAM achieves superior or comparable performance across all image level metrics.
Comparing to the second best approach, GFVS-Implicit, we are much more efﬁcient.
We use less CPU and GPU memory and our runtime is 150 times faster.
Fig.
5 shows a few one step prediction results.
SGAM is able to faithfully reconstruct the color and the shape of the assets within the visible area, and generate the invisible region in a coherent, harmonic fashion.
When unrolling multiple steps (see.
Fig.
4 and Tab. 2), baselines such as InﬁniteNature and GFVS-explicit suffer from error accumulation, and collapse catastrophically.
GFVS-implicit generates objects that resemble
the
GT data, yet it sacriﬁces the speed and fails to enforce consistency across frames (see Fig. 5).
Our approach, in contrast, is able to produce consistent and realistic results in an efﬁcient manner.
[Figure]
[Figure]
4.3 Analysis
Importance of two-stage training:
Two-stage training is critical.
Removing either stage degrades our performance.
Without the ﬁrst-stage codebook training, the generative sensing module will have to learn the codebook from warped inputs, which could inevitably encode missing holes into the codebook.
On the other hand, by ﬁnetuning the encoder from a converged ﬁrst-stage checkpoint, we consistently improve the performance of our model across all image level metrics.
We refer the readers to the supp.
materials for detailed numbers.
Real-world scene:
We also apply SGAM and the baselines to real-world images that we scraped from Google Earth (see Fig. 7).
The global mapping visualization is presented in Fig.
Similar as before, both InﬁniteNature and GFVS-explicit suffer from severe domain shift.
Since the images contain more diverse and complicated scene structures, GFVS-implicit fails to capture the viewpoint shift precisely during unrolling.
In contrast, SGAM can produce consistent and realistic results.
Please refer to supp.
material for more details on the data.
Diversity: SGAM allows us to create diverse 3D world by varying camera trajectories and by varying the step size.
As shown in Fig.
6 and Fig.
8, both of which are very effective.
Greedy samples the next target pose based on the L1 distance of the visible region of the warped views.
Effectiveness of Global Mapping:
The global mapping aims to resolve the geometric inconsistency from multi-view RGB-D sensor observation, and provides a denoised warped sensor observation in the reﬁnement stage. Without global mapping, direct warping noisy multi-view sensor observations to target poses hurdles inference quality at the generative reﬁnement stage.
To denoise the mapping
M t  1 , we ﬁnd it effective to reﬁne and grow the global scene with RGBD-integrated 3D mesh using
KinectFusion (48).
We demonstrate in Fig.
9 that RGBD-integration denoised mapping can achieve much more diverse scenarios than the one without, which scene degrades to large empty airport ground around 200 steps of unrolling becomes it unable to ﬁt latent code to ﬁt noisy features.
Failure Cases: We identify several typical failure cases appear on the GoogleEarth dataset in Fig. Despite producing realistic footage and 3D scene in most cases, SGAM tends to expand the building footprints to an unrealistic scale during the recursive sampling (row 1 and row 2).
Additionally, SGAM outpaints better along the vertical direction than moving horizontally on the GoogleEarth dataset (row 3) since the same camera moving step results in a larger outpainting gap if moving horizontally than moving forward.
We anticipate this could be resolved through a minor relative motion step. Finally, compared with natural objects, e.g., trees, SGAM is likely to struggle to generate complex structures (row 4). Training a stronger-capacity codebook might alleviate.
Societal impact:
Realistic 3D generation technology has many applications, such as visual content creation and simulation.
Unfortunately, it may also be used to spread misinformation.
Conclusion
In this paper we present a novel 3D scene generation algorithm, SGAM, which create a consistent, realistic, and large-scale 3D virtual world through simultaneous generation and mapping.
Our result demonstrates we can achieve similar (in synthetic data) or even more realistic (in real-world data) sensor observation than methods with Transformer-based latent code sampler during unrolling
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[Yes] Please see supp.
material for the discussion.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] We plan to release the code after acceptance.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] Our SGAM is built on top of VQGAN codebase.
We re-use the hyperparameters among all baselines.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
We randomly generate multiple scene with different seed images, i.e., initiall sensor observation to capture overall model performance.
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] We plan to release our dataset upon acceptance.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]