Single-Shot Scene Reconstruction
Abstract: We introduce a novel scene reconstruction method to infer a fully editable and re-renderable model of a 3D road scene from a single image.
We represent movable objects separately from the immovable background, and recover a full 3D model of each distinct object as well as their spatial relations in the scene.
We leverage transformer-based detectors and neural implicit 3D representations and we build a Scene Decomposition Network (SDN) that reconstructs the scene in 3D. Furthermore, we show that this reconstruction can be used in an analysisby-synthesis setting via differentiable rendering.
Trained only on simulated road scenes, our method generalizes well to real data in the same class without any adaptation thanks to its strong inductive priors.
Experiments on two synthetic-real dataset pairs (PD-DDAD and VKITTI-KITTI) show that our method can robustly recover scene geometry and appearance, as well as reconstruct and re-render the scene from novel viewpoints.
Introduction
Decomposing images into disjoint symbolic representations is the ultimate goal of robotics and computer vision, as it allows semantic reasoning over all scene parts.
Such a decomposition beneﬁts different possible applications (e g., robotics, augmented reality, autonomous driving) where a decomposed scene can be reassembled in different ways, enabling interaction or reenactment.
Unfortunately, single image scene reconstruction is inherently ill-posed due to a variety of reasons.
Most prominently, ambiguity in projective geometry can only be resolved with prior knowledge about the observed scene.
Current state-of-the-art methods either do not provide a full decomposition of the scene and regress per-pixel depth [1, 2], or estimate the poses of already known objects and ignore the rest of the geometry [3, 4].
In this work we design a pipeline to obtain an interpretable, disentangled scene representation for 3D-aware manipulation of driving scenes.
Our system is capable of reconstructing full geometries and appearances of detected object instances from a single RGB image.
Such a decomposition is made possible by three components: (1) a scene decomposition network (SDN) that detects object instances, recovers their partial geometries and poses, and predicts the full geometry of the background and RGB color behind the occluders; (2) a differentiable database of colored object priors that encodes their full geometry and appearance in the form of signed distance ﬁelds (SDF) and luminance ﬁelds (LF); and (3) a 2D/3D optimization pipeline that uses surfel-based differentiable rendering to ﬁt initial observations to the object database, leveraging their shape and appearance.
Our contributions are summarized as follows:
• A holistic single-shot scene reconstruction system that splits the scene into background and object instances and recovers their full geometry and appearance; • An optimization pipeline based on NOCS-SDF and RGB-LF ﬁtting as well as surfel-based differentiable rendering to reﬁne geometries, appearances, and poses of detected instances; • A scene manipulation and generation pipeline allowing to place and render objects of various geometry and appearance, and change view points.
[Figure]
MLPs
ID
E
D
[Figure]
NN
ID ID
[Figure]
Lat
Lat Lat
[Figure]
(B)
Pose Estimation
[Figure]
2D/3D Optimization
DR
[Figure]
Recovered 3D
[Figure]
Related Work
Single-Shot Scene Reconstruction Holistic single image scene reconstruction is a highly illposed problem, tackled from different angles by a variety of works.
The most common approach is monocular depth estimation that predicts per-pixel depth values [5, 1, 2].
While achieving remarkable results, these methods reconstruct the scene as a whole and do not incorporate knowledge about the scene’s objects and layout.
Alternatively, 3D detection pipelines detect separate objects and recover their masks and 3D bounding boxes [3, 6, 7, 4], or incorporate relationships between objects by using a graph or physical simulation [8, 9, 10].
While having understanding about the object appearance in 2D and their 3D pose, these methods do not recover the shapes of the objects nor the rest of the scene.
There are also pipelines aiming at reconstructing objects from detections by aligning 3D CAD models to ﬁt the predicted masks and input images [11, 12, 13], albeit without generalization to unknown instances outside the model database.
Lastly, there are recent works that recover room layouts together with full object geometries [14, 15], but restrict themselves to the geometric components of the recovered instances.
In our work we recover the geometry and appearance for both background and detected objects by using a differentiable database of object priors allowing us to smoothly interpolate between plausible object shapes.
6DoF Object Detection via Correspondence Regression Object detection
and pose estimation from RGB images is a well-established but ongoing research problem due to, among others, the inherent ambiguity of perspective projection.
The myriad of existing solutions can loosely be grouped in three categories: direct pose regression (using a neural net to output the pose directly by learning from data) [16, 17, 18, 19], template-matching (comparing patches with a predeﬁned set of templates either in pixel space or feature space)
[20, 21, 22, 23, 24, 25], and correspondence-based methods (directly utilizing correspondences between image coordinates and 3D models of interest)
[26, 27, 3, 4] - each having their own beneﬁts and drawbacks.
The current state-of-the-art methods in object pose estimation almost exclusively belong to the latter group with such representatives as PVNet [7], CDPN [28], EPOS [27], Pix2Pose [4], GDR-Net [29] and DPOD [3, 6].
They all regress per-pixel correspondences and use a P n P solver to estimate a 6DoF pose given a set of 2D-3D correspondences and camera parameters, minimizing reprojection errors.
While some works regress manually deﬁned UV correspondences, others regress
Normalized Object Coordinate
Space (NOCS) maps [30] that represent visible xyz object coordinates in RGB space.
The beneﬁts of NOCS are their straightforward creation through rendering, and their simultaneous deﬁnition of both the object’s pose and shape.
In our work we show how to effectively leverage this representation for geometry retrieval as well.
Neural Implicit Representations Recently, a new prominent direction for 3D shape and appearance representation has emerged with DeepSDF [31], Occupancy networks [32], and IM-Net [33], using neural networks for scalar function approximation.
All three methods take 3D coordinates as input and output either a binary occupancy estimate or a continuous SDF value, encoding the object’s surface.
Subsequently, follow-up work such as Deep Local Shapes (DLS)
[34], Neural Geometric
Level of Detail (NGLOD) [35] or NASA [36] improved the direction further.
While DLS stores
SDFs as voxel grid cells to allow larger scenes, NGLOD uses an efﬁcient Octgrid representation and a feature composition scheme based on trilinear interpolation to further improve the reconstruction quality, whereas NASA extends the approach to articulated objects.
Similar representations have been extended to represent object appearance.
Scene Representation
Networks (SRN) [37] add a ray marching routine and also regress RGB colors at surface intersections to learn from multi-view images, and Differentiable Volumetric Rendering [38] which couples an implicit shape representation with differentiable rendering.
Alternatively, instead of regressing SDF or RGB values for predeﬁned 3D coordinates, NeRF [39] proposes to regress density and color values along rays (5D coordinates) and compute the true color via numerical integration.
This approach, drawing inspiration from Neural Volumes [40] and their volumetric representation, boosted interest in implicit volumetric rendering and resulted in a multitude of works tackling problems from training and rendering time performance [41, 42, 43, 44], to covering dynamic scenes [45, 46, 47, 48], scene relighting [49, 50, 51], and composition [52, 53, 54].
While providing realistic renderings, most NeRF-based methods overﬁt to a single scene (with the exception of PixelNeRF [55]) and require a set of very dense view point annotations.
Our pipeline, on the other hand, is capable of reconstructing scenes never seen during training, and is also not limited to viewpoints seen during training.
Methodology
Given a single RGB image of a typical driving scene, our pipeline holistically reconstructs the scene by not only explaining every visible pixel, but also predicting the full geometry and appearance of detected objects and the occluded areas that are not visible in the image.
This is made possible by a combination of three components: a scene decomposition network (SDN), a differentiable database of object priors, and a 2D/3D optimization pipeline.
3.1 Scene Decomposition Network
The ﬁrst component of our pipeline (Fig. 1) is the scene decomposition network, or SDN.
Its main role is to estimate the layout of the scene, detect and identify visible objects, and recover the geometry of both the background and the detected objects.
For clarity, it is divided into three separate blocks: a detection transformer-based decomposition block, an object reasoning block, and a 3D reasoning block.
Detection Transformer Block (Fig. 1a)
The core of our SDN is a transformer-based decomposition.
An encoder-decoder object detector based on transformers, a popular architecture for sequence prediction and has been ﬁrst applied to computer vision in DETR [56].
DETR ﬁrst generates distinctive image features from an input RGB image, and feeds them to a transformer module to build attention maps between features.
This self-attention mechanism explicitly models all pairwise interactions between elements in a sequence, making this architecture particularly suitable for speciﬁc constraints of set prediction (e.g. duplicate suppression).
Using the notion of object queries, we are able to retrieve output features for each of the detected objects, which are then used as input to our object reasoning block and 3D reasoning block.
Object Reasoning Block (Fig. 1b)
Our object reasoning block takes output features of the detection transformer and uses a collection of MLPs to predict important object properties.
Following [56] we use a 3-layer perceptron with ReLU activation and a linear projection layer to regress object class
IDs and 2D bounding boxes.
Additionally, we also regress SDF and LF feature vectors, denoted by z sdf and z lf respectively.
These feature vectors provide an initial state for object reconstruction and are essential to effectively reason about and reﬁne the 3D component of these objects, as described in detail in Section 3.2.
DR Z SDF
Z LF
SDF MLP
RGB MLP +
P
Recovered Surface
Recovered Surface Recovered Surface
NOCS + RGB
Rendered Predicted (A) (B)
Octgrid
Z SDF +
[Figure]
2D 3D
3D Reasoning Block (Fig. 1c)
Our 3D reasoning block aims to recover 3D scene information by splitting it into two parts: a background containing road surfaces, buildings, and other objects not detected by the detection transformer, and a foreground entirely consisting of detected objects.
We use the output of the transformer decoder for each object to compute multi-head attention scores of this embedding over the output of the encoder, generating M attention heatmaps per object.
These masks are then used to regress the geometry for both foreground and background.
The foreground is predicted as a set of normalized shape maps.
Since they encode 3D coordinates via RGB, visualizing each 3D color component in 3D space allows us to recover a partial 3D shape of the object in its normalized coordinate space (Fig. 3).
This representation is crucial for our pipeline to retrieve the object’s full geometry and pose.
The background, on the other hand, is represented as a depth map since we solely care about its geometry.
For view synthesis applications, we also predict the depth and appearance behind the detected objects by utilizing a GAN-like encoder-decoder generator architecture.
This architecture takes masked RGB and depth images and inpaints RGB and depth values for occluded regions, employing a fully-convolutional PatchGAN discriminator to judge the genuineness of the inpainted maps.
Given estimated object masks and NOCS maps, we recover three things enabling pose estimation: (1) 2D coordinates of the object in the image, (2) 3D points of the partial object shape in the canonical frame of reference, (3) and their 2D-3D correspondences.
This unique representation frees us from the need of storing a collection of 3D primitives and identifying them to ﬁnd a detected model, because both 3D and 2D information is contained in the form of a 3-channel map.
We multiply the recovered normalized shape by the per class scale factor to recover absolute scale.
The 6DoF pose is then estimated using P n P [57] that predicts the object pose from given correspondences and camera intrinsics.
Since we get a large set of correspondences for each model, we also combine P n P with
RANSAC to robustify against outliers.
For the results presented in the evaluation section, we run 1000 RANSAC iterations per pose with a 1 px reprojection error threshold.
3.2 Shape, Pose, and Appearance Optimization
The second crucial component of our pipeline is a 2D/3D optimization that is enabled by the differentiable database of object priors (Fig. 2a) as well as our surfel-based differentiable rendering pipeline (Fig 2b).
Given output features corresponding to detected objects regressed by the object reasoning block (Fig. 1b), our optimization procedure generates full shapes, appearances and poses of all the objects in the scene.
We leverage a differentiable database of object priors - PriorDB - which encodes the shape and luminance of the input models as Signed Distance Fields (SDFs) and Luminance Fields (LFs) with associated latent feature vectors.
Given a partial shape observation, we differentiate against PriorDB and ﬁnd the maximum likelihood latent vector that best explains the observed shape.
The RGB component is optimized similarly, with the complete procedure described in detail in Sec 3.2.1.
While NOCS-based SDF and LF optimization allows us to reconstruct plausible objects from images, it does not account for the pose component.
It might very well be the case that estimated poses are incorrect and are harming the full scene reconstruction.
This can be alleviated via a 2D alignment step, which can further be used as a complimentary source for shape optimization when predicted
NOCS maps are noisy, resulting in a lower reprojection error and better reconstructions.
Thus, to further constrain and improve the optimization, we deﬁne a differentiable rendering pipeline that allows us to impose 2D image losses based on renderings of the recovered SDF shape and appearance, as described in detail in Sec. 3.2.2.
3.2.1 Differentiable Database of Object Priors (Fig.
1d
Our differential database of object priors or PriorDB represents objects as signed distance ﬁelds (with positive and negative values pointing to exterior and interior areas respectively), in which each value corresponds to the distance to the closest surface.
A single MLP can be used to represent multiple geometries using a single feature vector z sdf and query 3D locations x
= { x 1 , ..., x N } as f sdf ( x ; z sdf )
=
s .
Object appearances, on the other hand, are represented as luminance ﬁelds (LF), deﬁning the prior perceived luminance of the object as seen in the training set.
Similarly to SDN, the LF module is implemented as an MLP, but it takes a feature vector by concatenating z sdf and z lf as well as query locations x as input, and outputs resulting luminance as f lf ( x ; z sdf ; z lf )
= l .
Both SDF and LF modules are trained using synthetic data provided by Parallel Domain [58], containing high-quality realistic road scene renderings together with ground truth annotations.
To train the SDF module, we use a collection of canonical ground truth point clouds and associated surface normals representing rendered objects.
In total, we encode 27 cars in our PriorDB .
We ﬁrst deﬁne a multi-level octree spanning the volume [ − 1 , 1] 3 , with points located at the corners of the imaginary voxels of various sizes depending on the level.
For each of the point clouds we deﬁne unsigned distances by computing nearest neighbors to the octree points.
Next, we deﬁne the loss L sdf as the distance to the surface at grid points starting from the coarsest level, gradually increasing the resolution up to the highest level (we use 3 levels of granularity as shown in Fig. 4), while only considering points located at pre-deﬁned per-level distances to the surface.
To ensure that we learn a correct signed ﬁeld, we additionally minimize the distance between the ground truth surface normals and analytically-derived surface normals from the SDF module.
Moreover, to ensure that we always regress plausible shapes regardless the input feature vector, we do an additional reﬁnement round by sampling random feature vectors and enforcing the output to be equal to the shape of the nearest plausible neighbor.
Once the SDF module is trained, we store SDF features z sdf associated with objects and use them to train the LF module.
Differently, it is trained on partial canonical shapes recovered from provided
RGB renderings (see Fig.
2a), associated depth maps and poses.
We compute the luminance component from RGB renderings and enforce the LF of each feature to be the same.
The network weights are then optimized to ﬁt the average luminance per object using an L1 loss.
Additionally, we deﬁne a symmetry loss L symm enforcing colors on the left side of the car to be similar to those on the right one.
For that we compute nearest neighbors between the left side of the car and an inverted right side and then minimize the L1 distance between these correspondences.
Optimization 3D Our optimizer takes partial canonical object shape and appearance observations (recovered from predicted NOCS maps as shown in Fig.
1b), as well as initial PriorDB features.
Due to the robustness of NOCS regression, and despite the occasional noisy outliers, we set it as our main optimization prior.
The initially predicted SDF is used to recover the full surface of the object in the form of a point cloud with surface normals using a 0-isosurface projection.
Then we estimate nearest neighbors between the two point clouds and minimize the distance between the points by optimizing the feature vector z sdf .
The RGB component is optimized similarly by minimizing the color difference between the correspondences, but we also allow the LF module weights to vary for a ﬁner level of detail.
In the case of RGB optimization we also use the L symm loss between the left and right sides of the predicted shape.
3.2.2 Surfel-based Differentiable Renderer
Differentiable rendering allows to optimize objects with respect to pose and shape, and is an essential component of our optimization scheme.
Since we use an SDF representation, we cannot directly use differentiable renderers for triangulated meshes.
Instead, we implemented a renderer that uses surfels as the primary representation (inspired by [59]), estimating surface points and normals using an 0-isosurface projection, followed by a surfel-based rasterizer.
Surface Tangent Discs Surface elements or surfels are a common concept in computer graphics [60] as an alternative to connected triangular primitives.
In order to render watertight surfaces, the individual surface normals must sufﬁciently approximate the local region of the object’s geometry.
[Equation]
To construct surface discs we ﬁrst estimate the 3D coordinates of the resulting tangent plane given the normal n i =
∂f ( p i ; z ) ∂p i of a projected point p i .
The distance d of the plane to each 2D pixel ( u, v ) is computed by solving a system of linear equations for the plane and camera projection as deﬁned in Eq.
1a, where K − 1 is the inverse camera matrix.
Then, we get a get a 3D plane coordinate by backprojection (Eq. 1b).
Finally, we estimate the distance between the plane vertex and surface point and clamp if it is larger than a disc diameter to get ﬁnal discs M = max( diam − || p
i − P
|| 2 , 0) .
Rendering Function Similarly to [59, 61], we combine colors from different surfel primitives based on their depth values to compose a ﬁnal rendering.
The distance of the primitive to the camera deﬁnes its ﬁnal intensity contribution to the image.
To ensure that all primitive contributions sum up to 1 at each pixel we use a modiﬁed softmax function.
The ﬁnal rendering function is given in
Eq.
2a, where I is the output image, S the estimated NOCS map, and w i the weighting masks,
[Equation]
Eq.
2b deﬁnes weighting masks w i , where ˜ D is the normalized depth and σ is a transparency weight with σ −→ ∞ deﬁning a completely opaque rendering as only the closest primitive is rendered.
Optimization 2D Formally, we deﬁne three losses on the rendering output L nocs 2 D , L lf 2 D , and L mask 2 D .
L nocs 2 D compares the output of the renderer with predicted NOCS, L lf 2 D compares the output of the renderer with input RGB images, and L mask 2 D compares the rendered and the predicted masks.
The exact deﬁnitions of these losses are provided in the appendix.
The ﬁnal optimization loss is a weighted sum of both 2D and 3D components (see Sec. 3.2.1).
While 3D components ensure a robust shape optimization, 2D losses help to better align the object in 3D space and allow a better luminance matching.
Experiments
In this section, we ﬁrst evaluate the individual components of the SDN on common benchmarks for stereo view synthesis, depth estimation and 3D object detection.
Then we demonstrate the capabilities of our full pipeline for the tasks of scene reconstruction, object recovery and manipulation.
The evaluation is performed on four datasets: PD, VKITTI2, DDAD, and KITTI.
VKITTI2 [62] and KITTI [63] Virtual KITTI [64] is a synthetic proxy to the KITTI dataset – the standard benchmark for autonomous computer vision.
It contains ﬁve virtual worlds rendered under ten different conditions of weather and camera conﬁguration, in total comprising 21,260 images.
VKITTI2 extends the original dataset with more photo-realistic renderings.
In the experiments we use the
80% consecutive frames of each conﬁguration for training and the last 20% of the clone we
3D [65] split consisting of 3769 images.
Parallel Domain (PD) [58] and DDAD [2] The Parallel Domain dataset is a procedurallygenerated collection of fully annotated photo-realistic renderings of urban driving scenes.
It contains 200 180 -frame sequences resulting in a total of 36000 images.
Each sequence also contains an additional frame with extra ground truth modalities which we use to form the PD test split with a total of 200 images.
The Dense Depth for Automated Driving (DDAD) dataset is a real-world counterpart of the PD dataset mainly used as a depth evaluation benchmark.
We consider only the front camera, resulting in 50 validation sequences with 3950 images.
4.1 Scene Decomposition and 3D Reasoning
Stereo View Synthesis
In this experiment we quantitatively evaluate our SDN for the task of view synthesis by recovering a stereo-pair of the input image.
We compare our results with state-of-theart generalizable NeRF-based view synthesis method PixelNeRF [55] and depth estimation method
PackNet-SfM [2].
All pipelines were trained on the same VKITTI2 training split and evaluated on the VKITTI2 test split.
From the results in Table 2, we observe that SDN can reliably re-generate the scene from a novel view point thanks to its integrated prior and geometric nature, and outperforms both baselines.
Please refer to the appendix for qualitative results.
Depth Evaluation Next we evaluate the quality of our geometric output using common metrics for depth evaluation.
We argue that our NOCS maps, given estimated poses, can be transformed
[Table]
[Figure]
[Table]
We evaluate the performance of our SDN for the task of stereo-pair recovery on the
VKITTI2 dataset.
[Table] [Table]
50% IoU threshold.
Both methods are trained on VKITTI2 train set
[Table]
to depth maps and signiﬁcantly outperform standard depth estimation pipelines due to the prior geometry knowledge.
To conduct this experiment, we choose current state-of-the-art depth estimation pipelines, Monodepth
[1] and PackNet-SfM [2], and train them on 2 synthetic datasets - VKITTI and PD - similarly to our method.
Then we evaluate the depth quality for foreground objects on real datasets - KITTI and DDAD - using a number of standard depth estimation metrics.
This experiment evaluates three aspects of our model: 1) the reconstructed object geometry, 2) the accuracy of the estimated poses, and 3) transferrability across domains.
Table 1 shows that our model consistently outperforms the baselines for monocular depth estimation, demonstrating the superior geometric accuracy and generalization qualities achieved thanks to the encoded object prior.
3D Detection
In this experiment we aim to demonstrate how well our method generalizes to other datasets in terms of 3D detection quality.
We compare against a state-of-the-art method for KITTI pose estimation, SMOKE [66], which is trained on the same VKITTI2 train set as our method.
To compare with our algorithm that has access to ground-truth size of 3D boxes, we substitute the size of predicted boxes with the ones of nearest-neighbor ground-truth, when evaluating SMOKE.
Table 3 shows that we perform competitively with SMOKE, and demonstrates that our underlying
NOCS representation generalizes to diverse shapes and poses of objects in the real-world.
4.2 Scene Manipulation and Object Recovery
The section aims to demonstrate the capability of our full pipeline to reconstruct and manipulate the scene.
In Fig.
6, we show the scene decomposition results on the KITTI dataset.
Our SDN takes as input a single reference image (Fig.
6a) and recovers full 3D objects (Fig.
6c) as well as the scene background (Fig.
6d).
Recovered objects can be freely manipulated as shown in Fig.
6b.
The full 3D scene reconstruction is shown in Fig.
6e.
Note that our pipeline has been trained entirely on synthetic data, and generalizes to real images without any adaptation.
We refer the readers to our supplementary videos for the visualization of the full capability of scene manipulation.
Optimization ablation Lastly, we investigate how different components of our optimization pipeline affect the end result in terms of full shape and luminance estimation, as well as 2D amodal mask alignment.
All evaluations are performed on the PD test split.
We observe a signiﬁcant performance increase for all considered metrics when compared to an unoptimized result as shown in Table 4.
While 3D optimization enables robust shape estimation, it does not align luminance in the projective space and also ignores object’s pose since it is performed in the object’s canonical frame.
The 2D optimization step on the other hand is performed in the projective space and allows for exact luminance matching and good mask alignment.
However, it performs worse with respect to shape optimization due to an additional ambiguity introduced by the pose component, which also affects the amodal mask metric.
Finally, using both 2D and 3D optimization combines their strengths and yields the best overall performance.
Conclusion
We present a novel holistic scene reconstruction pipeline that disentangles the scene by representing objects separately from the scene.
Our Scene Decomposition Network detects and recovers partial geometries and poses of objects, and estimates depth of the background.
The partial geometries are then completed using our differentiable database of object priors and a 2D-3D optimization procedure.
As a result, our pipeline explains every pixel on the image and more - the full object geometries and appearances.
Despite being trained only on synthetic data our experiments show good generalization to real data on KITTI and DDAD datasets.