D IFFERENTIABLE G RADIENT S AMPLING FOR L EARNING I MPLICIT 3D S CENE R ECONSTRUCTIONS FROM A S INGLE I MAGE
A BSTRACT
Implicit neural shape functions, e.g. occupancy fields or signed distance functions, are promising 3D representations for modeling arbitrary 3D surfaces.
However, existing approaches that use these representations for single-view 3D reconstruction require 3D supervision signals at every location in the scene, posing difficulties when extending to real-world scenes where ideal watertight geometry necessary to compute dense supervision is difficult to obtain.
In such cases, constraints on the spatial gradient of the implicit field, rather than the value itself, can provide a training signal, but this has not been employed as a source of supervision for single-view reconstruction in part due to the difficulties of differentiably sampling a spatial gradient from a feature map.
In this paper, we derive a novel closed-form Differentiable Gradient Sampling (DGS) solution that enables backpropagation of the loss on spatial gradients to the feature maps, thus allowing training on large-scale scenes without dense 3D supervision.
As a result, we demonstrate single view implicit 3D surface reconstructions on real-world scenes via learning directly from a scanned dataset.
Our model performs well when generalizing to unseen images from Pix3D or downloaded directly from the Internet (Fig. 1).
Extensive quantitative analysis confirms that our proposed DGS module plays an essential role in our learning framework.
Video and code are available at https://github.com/zhusz/ICLR22-DGS.
[Figure]
I NTRODUCTION
Recent studies have shown promising learning capabilities of implicit models for 3D geometry representations (Park et al., 2019;
Mescheder et al., 2019; Mildenhall et al., 2020; Sitzmann et al., 2019;
Gropp et al., 2020;
Saito et al., 2019; Yu et al., 2020).
Rather than explicitly representing 3D geometry with a textured mesh (Kato et al., 2018) or point cloud (Fan et al., 2017), implicit representations express a function of points in the space.
Among many recent promising implicit models, the occupancy field (Mescheder et al., 2019; Popov et al., 2020; Chen & Zhang, 2019b) and the Signed
Distance Field (SDF) (Park et al., 2019; Xu et al., 2019; Li & Zhang, 2021) are particularly suitable for high resolution topology free surface reconstruction.
Given a query point ( x, y, z ) ∈ R 3 in the space, the occupancy function returns the occupancy status of the point o ( x, y, z ) ∈ { + , −} , while the SDF value returns the closest distance from a point to the surface of the given 3D geometry, as well as its sign for its occupancy.
The surface is defined by the classification boundary of the occupancy field or the zero level-set of the SDF (Lorensen & Cline, 1987).
Despite the strong representational power of these models and their success on predicting 3D objects from single image, learning to predict complex 3D geometry, such as scenes, from a real world image remains a challenge for this class of methods.
Obtaining the occupancy label, or the sign, from noisy non-watertight meshes is non-trivial (Atzmon & Lipman, 2020)
(Fig. 2).
Furthermore, producing the label on the fly from a large number of triangle meshes, as encountered in typical real-world scenes with complex geometry, can be computationally prohibitive (Straub et al., 2019;
Chang et al., 2017; Dai et al., 2017; Song et al., 2017).
Storing dense query points sufficiently with pre-computed occupancies or SDF for complex geometries, or meta data of meshes like octrees or hashing, is a possibility, but incurs severe engineering and computational burdens on storage and runtime loading (Yu et al., 2021).
The challenge is exacerbated for predicting scenes in particular, since unlike objects scenes in the view frustrum typically cannot be enclosed by a pre-defined range of space and do not have a categorical canonical coordinate system (Popov et al., 2020).
Recent studies on fitting a single scene geometry to partial 3D observations (Gropp et al., 2020; Sitzmann et al., 2020) have pointed out the value of the 3D spatial gradient of the implicit field.
For instance, the gradients of the SDF on surface points are actually the surface normals and the 2-norm of the spatial gradients for any point in the space is 1, known as the Eikonal equation (Crandall & Lions, 1983).
Similarly, in this work, we propose to regularize the gradient of the occupancy field to be zero away from the surface (Fig. 3(c)).
Thus, one can train the full occupancy field or SDF with only the surface point clouds, thanks to these constraints on the gradients of the implicit shape models, even when the value of the occupancy or the SDF itself is not available everywhere.
However, existing approaches (Gropp et al., 2020; Sitzmann et al., 2020) have only demonstrated this benefit in the cases of geometry inputs only, where a network is fit to a particular scene per model without any conditioning on input images.
When locally sampled image features are used in the feed-forward prediction, it is necessary to derive a differentiable gradient sampling solution over the feature pixel map.
In this paper, we tackle this challenge by deriving a closed-form differentiable gradient sampling (DGS) solution for learning a single-view 3D implicit reconstruction model (Fig. 4).
Our novel propagation of the loss gradient of the spatial gradient back to the feature maps (Eq. 6) serves in addition to the existing spatial gradient sampling (Eq. 5) and loss gradient back-propagation (Eq. 4) used in existing deep learning frameworks (Paszke et al., 2019; Abadi et al., 2016).
The resulting end-to-end learning scheme with supervision over the spatial gradients opens the potential to train a model generalizable to unseen test cases of complex scenes with only surface point cloud supervision — a typical setting for real-world data — without requiring ground truth per-query-point labels.
Our contributions include a framework to propagate spatial gradients through a spatial feature sampling procedure, and a novel closed-form DGS solution.
Experiments on real-scanned data (ScannetV2 (Dai et al., 2017))
shows that DGS enables training a single-image 3D implicit reconstruction model that can generalize to unseen scenes, with only imperfect surface annotations as the supervision (e.g. non-watertight meshes).
Experiments on synthetic data (ShapeNet Chang et al.
(2015)) indicate that our learning framework without using dense per-query-point training labels demonstrates competitive performance compared to the oracle scenario where dense occupancy labels are available.
To the best of our knowledge, DGS-enabled shape inference provides the first singleview implicit shape reconstruction on real scene datasets which can generalize accurately to unseen scenes from different datasets or domains (see Figure 1).
[Figure]
R ELATED W ORKS
+ - + + + + + + + + + + - - - - - - - + + + + - - - - - - - - - - - - - - - + + + + + + + - + + - + + + + - ∇ Occupied
[Figure]
∇ !
,#,$ o % =
[Figure]
Illustration: ∇ [Equation] ∇ ∇
3D Implicit Representations Among all the 3D representations, implicit models are advantageous for arbitrarily high resolution modeling (unlike voxels with fixed resolution and no detailed surface modeling) and easy learning (unlike meshes which assume a fixed topology).
Implicit models typically learn to establish a mapping between a query point and the prediction of the point.
Groueix et al. proposed to learn a mapping from the uv texture map to the 3D surface point.
More recent works (Park et al., 2019; Mescheder et al., 2019; Xu et al., 2019; Popov et al., 2020; Chen & Zhang, 2019a) focus on mapping the query point coordinates to the signed distance field or the occupancy field.
In addition to these pure geometry modeling, recent works like (Saito et al., 2019) also model the surface texture via learning a mapping from the surface point to the RGB value, or use the RGB loss as the supervision signal (Niemeyer et al., 2020; Yariv et al., 2020).
In contrast to defining textures explicitly on the surface, Mildenhall et al.
(2020) proposed the volumetric rendering representation which maps the point coordinate and the viewing angle to volumetric textures.
Oechsle et al.
; Wang et al.
; Yariv et al.
(2021) use volume renderings for fitting high quality geometry of single scenes, but these are not generalizable to unseen scenes.
In multi-view stereo setting, Murez et al.
Sun et al.
(2021); Boˇziˇc et al.
(2021) proposed to accumulate the multi-view predictions in a TSDF voxel volume.
We believe our proposed DGS module is necessary when extending most of the existing implicit surface representations into feed-forward models that are conditioned on input image(s), and training these models with a loss on the gradient of the predicted field.
In this work, we focus on 3D geometry reconstruction reconstructions from single view, as an example of the application of the proposed DGS module.
Differentiable Operations End-to-end deep learning requires all modules in a computation path to be differentiable.
Many differentiable modules are proposed to serve a particular functionality.
The spatial transformer layer (Jaderberg et al., 2015) is one of the early approaches for differentiable sampling of the feature map.
Kato et al.
(2018) proposed to render the mesh into image in a differentiable manner to optimize the mesh or the camera parameters to fit to a particular image.
A similar idea has also been applied for rendering point clouds (Wiles et al., 2020; Yifan et al., 2019).
Tulsiani et al.
(2017) and Mildenhall et al.
(2020) proposed to compute the ray rendering in an accumulating manner.
One common feature of these differentiable operations is a strategy to soften the existing explicit modeling (Liu et al., 2019), and replace the gradient of the loss with approximate signals, making sure those signals provide guidance in the right direction.
Our differentiable gradient sampler also seeks to soften the spatial gradient between the adjacent pixels.
L EARNING F RAMEWORK
Problem definition and notations.
We aim to train a feed-forward deep model for predicting the 3D implicit surface reconstruction conditioned on a single image.
We denote the input RGB image as
I ∈ R m × n × 3 .
Like most feed-forward models, our model employs an image encoder.
We denote the extracted 2D features as ϕ .
Our model ˆ f Θ ( · ) , parameterized by Θ , takes in the image feature ϕ and predicts for each 3D location ( x, y, z ) the implicit value ˆ f Θ ( ϕ ; x, y, z ) .
We denote the ground truth value as f ( ϕ ; x, y, z ) .
For the occupancy field, the implicit field value f represents the occupancy probability.
We denote the predicted occupancy probability as ˆ o Θ ( ϕ ; x, y, z ) ∈ [0 , 1] and the ground truth binary occupancy label as o ( ϕ ; x, y, z ) ∈ { “ − ” , “ + ” } , where “ − ” represents unoccupied space and “ + ” vice versa.
For SDF, the implicit field value f represents the signed distance between the query point and its projection on the surface.
We denote the predicted and the ground truth signed distance as ˆ s Θ ( ϕ ; x, y, z ) ∈ R and s ( ϕ ; x, y, z ) ∈ R respectively.
Background.
Typical fully supervised training (Xu et al., 2019; Popov et al., 2020) imposes the loss to each individual sampled query points by comparing the predicted field value ˆ f Θ ( ϕ ; x, y, z ) with the ground truth value f ( ϕ ; x, y, z ) .
For occupancy predictions, we compute the loss as (cid:80) { I, P I }∈D (cid:80) ( x,y,z ) ∈P
I BCE (ˆ o Θ ( ϕ ; x, y, z ) , o Θ ( ϕ ; x, y, z ))
(Fig. 3(a)), where “BCE” represents the binary cross entropy loss, D represents the whole single-view image training set, and P I represents the set of all the possible query point sampling locations ( x, y, z ) within the view frustum.
For
SDF, we compute the loss as (cid:80) { I, P I }∈D (cid:80) ( x,y,z ) ∈P I | ˆ s Θ ( ϕ ; x, y, z ) − s ( ϕ ; x, y, z ) | .
3. P ROPOSED F RAMEWORK
As obtaining the full labels f ( ϕ ; x, y, z ) for query points ( x, y, z ) from every location in P I is nontrivial for complex geometry from real-world (Fig. 2) and due to additional engineering constraints on runtime loading and memory bottleneck for the sufficiently dense pre-computed query point occupancy labels, we propose to incorporate the loss w.r.t.
the spatial gradients.
For occupancy of the implicit field,
[Equation]
[Equation]
where ∇ x,y,z ˆ o Θ ( ϕ ; x, y, z ) =
[ ∇ x ˆ o Θ ( ϕ ; x, y, z ) , ∇ y ˆ o Θ ( ϕ ; x, y, z ) , ∇ z ˆ o Θ ( ϕ ; x, y, z )]
⊤ denotes the spatial gradient of the occupancy prediction, P
+ I and P 0 − I denote inward and outward near-surface query points, and λ or represents the loss weight of the occupancy geometric regularization.
The facing direction of the mesh surface (determining inward or outward) can be determined by normals (if available), or via rendering the surface in all views as in the RGBD captures (Dai et al., 2017) (with the surface facing toward camera as the “outward” side).
The three terms in Eq.
1 correspond to the three types of losses in Fig. 3(c) (“ ∇ ”, “ + ”, “ − ”) respectively.
For SDF, we set the loss similar to Gropp et al.
[Equation]
[-where P 
-]
[Figure]
𝑥𝑦𝑧 𝑖𝑗 ∇ !
,#,$ 𝜙 %,
[Figure]
𝜙 %,
[Figure]
QueryLocation ∇ !
,#,$ 𝑓 % (𝜙;𝑥,𝑦,𝑧) 𝑓 % (𝜙;𝑥,𝑦,𝑧) …
Loss 𝜙 ! 𝜙 ! 𝜙 " 𝜙 # 𝜙 $ 𝜙 !,# 𝜙 " 𝜙 # 𝜙 $ 𝛽𝑤 (1−𝛽)𝑤 𝛼ℎ (1−𝛼)ℎ 𝜙 ! 𝜙 " 𝜙 # 𝜙 $ 𝜙 %,’ 𝜙 ! 𝜙 " 𝜙 # 𝜙 $ 𝜙 !
𝜙 " 𝜙 # 𝜙 $ ∇ % 𝜙 %,’ ∇ ’ 𝜙 %,’ 𝜙 ! −𝜙 " 𝜙
[Figure]
−𝜙 $ 𝜙 $ −𝜙
[Figure]
{+where P 
 I denotes the query points on the ground truth surface only, and λ sr and λ sn represents the loss weight for the signed distance geometric regularization term and normal term respectively. The three terms are the Eikonal regularization (Crandall & Lions, 1983), surface zero-SDF loss, and the surface normal loss respectively, where the last term is optional (Gropp et al., 2020).+}
Note now the loss functions in Eq. 1, 2 no longer require the implicit ground truth label for query points far away from the ground truth surface, enabling training with the real-world imperfect scanned data.
In both cases, the availability of surface normals is optional.
Practically, we found only the proposed loss for the occupancy model (Fig. 3(c), Eq. 1) to be effective in the real-world single-view feed-forward scenario, and use Eq. 1 rather than Eq. 2 in all of our experiments.
There are a few reasons.
First, learning with the Eikonal term (Eq. 2) suffers severely from its sensitivity of model initialization.
We found either the SDF or the Truncated SDF (TSDF) representation model cannot easily converge during training.
Specifically, we found it poses numerical difficulties when the model is learned to predict a large distance value or a constant truncation value for the majority of the query points in the air that is far from any surfaces.
This becomes a major problem when we depart from the single objects scenario (Mescheder et al., 2019; Popov et al., 2020) or single scene fitting (Gropp et al., 2020) to large schale scenes, which is our focus.
Second, our loss function (Eq. 1) can significantly save memory footprint during training and enable large batch size training, which is crucial in our learning framework.
Unlike in Eq.
2 where all the query points require spatial gradient computation, which leads to 3 × higher of memory footprint, in our loss function (Eq. 1), only the non-surface query points do.
We set the query point batch size of the non-surface points (512 in practice) to be much smaller than the critical near-surface points (4096 in practice), enabling learning with the image batch size of 32 in our real-scene training.
3. S AMPLING WITH D IFFERENTIABLE G RADIENTS
A distinct difference compared to existing works is that our spatial gradient ∇ x,y,z
ˆ f Θ ( ϕ ; x, y, z ) is also conditioned on the pixels in ϕ .
Gropp et al.
(2020) devised the model f Θ to fit to a single scene, and the spatial gradient ∇ x,y,z ˆ f Θ ( x, y, z ) can be conveniently computed because it does not involve the image sampling procedure.
Jiang et al.
Lin et al.
(2020) used a non-spatial global feature for inference and hence bypassed 2D sampling.
In our learning framework, the spatial gradient computation must undergo the sampling procedure.
We name our gradient computation involving the sampling operation as the Pixel Conditioned Gradients and derive a closed-form solution, Differentiable Gradient Sampling (DGS), for handling forward and backward propagation.
Figure 4(a) provides an illustration of our training pipeline.
Each layer in our network tracks both the response of the layer as well as its spatial gradient w.r.t.
( x, y, z ) .
While it is well established to track the layer-wise spatial gradient for fully-connected (FC) layers or convolutions in existing works (Gropp et al., 2020), tracking the spatial gradients and back-propagating the loss to the feature map pixels ϕ through the sampling module has not been studied.
To this end, we derive the closed-form sampling scheme for tracking and propagating the spatial gradients ∇ x,y,z ϕ = [ ∇ x ϕ, ∇ y ϕ, ∇ z ϕ ] ⊤ through the sampling layer.
Background - 2D Differentiable Sampling.
Differentiably sampling pixel values from a grid of 2D feature map with the given pixel locations ( i, j ) is a common operation.
Throughout our paper, we define the pixel coordinates in the normalized coordinate system that ranges from -1 to 1.
As illustrated in Fig. 4(d), given the feature map ϕ and the sampling locations ( i, j ) , the resulting sampled value ϕ ( i, j ) is
[Equation]
Please refer to Fig. 4(c) for the definitions of α, β and ϕ A , ϕ B , ϕ C , ϕ D .
Without loss of generality we use bilinear interpolation.
During training, the gradient from the loss can be back-propagated via
[Equation]
Equation 4 is for Pixel A and similarly to the other 3 pixels (please refer Eq. 9).
2D Differentiable Gradient Sampling.
Our learning framework (Sec. 3.1, Fig. 4(a)) requires the extension of the sampling capability from just the feature value response ϕ
i,j to its spatial gradient ∇ i,j ϕ ( i, j ) = [ ∇ i ϕ ( i, j ) , ∇ j ϕ ( i, j )]
⊤ .
During the forward and the backward propagation, both the sampled feature response ϕ
i,j and its spatial gradient ∇ i,j ϕ ( i, j ) are recorded for further propagation (Fig. 4(e)):
[Equation]
Please refer to Eq.
10 for ∇ j ϕ
( i, j ) .
Hence, during the forward pass, we compute the spatial gradient via Eq.
5 in addition to the existing value sampling (Eq. 3).
During the backward pass, we compute the loss gradient over the spatial gradient via
[Equation]
where w and h are the width and height of a pixel in normalized coordinate system, s.t. w = 2 /W and h = 2 /H for the feature map with the size W × H .
Please refer to Eq.
11 for ∂L∂ϕ B , ∂L∂ϕ C and ∂L∂ϕ D .
3D Differentiable Gradient Sampling.
We now extend the sampling to 3D. We model the camera as the pin-hole camera.
For any point ( x, y, z ) in the camera space, we seek for its projected 2D locations ( i, j ) based on the focal length f 0 via i = yf {+
 z ,+} j = xf 0 z .
The feed forward pass is
[Equation]
[Equation]
Note that here, our notation of the sampled 2D image feature ϕ ( x, y, z ) refers to the same as ϕ ( i, j ) , as ( i, j ) is the projected pixel coordinates of ( x, y, z ) that represent the exact projected 2D locations when extracting the 2D features.
During the backward propagation procedure, the DGS accumulates the gradient via ∂L
[Equation]
Our approach (the last 4 rows) demonstrates competitive performance compared to state-of-the-art approaches (the top 4 rows) even trained without the dense occupancy labels as used in the oracle settings of these existing works.
Our comparisons with the state-of-the-art approaches are direct ablations as we maintain exactly the same experimental setups except for the loss function.
In addition, our approach also comfortably outperforms the ablation baselines (the middle 5 approaches)
[Table]
E XPERIMENTS 4. L EARNING FROM S YNTHETIC D ATA (S HAPE N ET )
Dataset.
Following Popov et al.
, we conduct experiments on ShapeNet with the low-realism and the high-realsim renderings.
For low-realism, we use the renderings of various models from
Choy et al.
Following Mescheder et al.
(2019), we split the dataset into the training set of 30661 models, the validation set of 4371 models, and the test set of 8751 models.
During testing, we follow Mescheder et al.
(2019) to only test the first rendering of each model.
Similar to all the prior works, we use the same 13 classes to report any performance via grouping all the test cases into the same class.
For high-realism, we follow the split and evaluation protocol of the single-object scenes
Popov et al.
(2020) - we only use the 13 categories used by Choy et al.
(2016), filter out repeated samples, and finally construct the data with 666,565 models from the training set, 96,084 from the validation set, and 189,748 from the test set.
Following Popov et al.
(2020), we only evaluate the first 1% test cases of the test set (1898 samples).
Metrics. We use the Intersection-of-Union (IoU) for evaluating the model performances. For lowrealism renderings, we follow the protocol of IoU benchmarking from Mescheder et al. that evaluating occupancy of the specified 100K query points in the space.
For high-realism renderings, we follow the protocol from Popov et al. (2020) to evaluate occupancy prediction of the 128 × 128 × 128 grid. Following Popov et al.
(2020), the grid cube is a unit cube that spans from − 0 . 5 to 0 .
5 for x and y , and from f/ 2 to 1 + f/ 2 for z in the camera coordinate system. Comparison with state-of-the-art approaches. On high-realism ShapeNet, we build our model on top of the representative state-of-the-art models - OccNet (Mescheder et al., 2019), DISN (Xu et al., 2019), CoReNet (Popov et al., 2020) for their 3D volumetric implicit modeling capability.
We also incorporate a strong baseline named “DISN (ResNet50) + DVR (Niemeyer et al., 2020)”, where we replace the originally used VGG (Simonyan & Zisserman, 2014) with ResNet-50 as the image encoder as used in CoReNet (Popov et al., 2020), and devised the 5-layer fully connected ResNet as the decoder as used in DVR (Niemeyer et al., 2020). In each DGS experiment, we maintain exactly the same encoder and decoder architecture, optimization parameters (e.g. the learning rate and the epsilon of Adam Kingma & Ba (2014)) and the prediction format (per-query occupancy probability). We set λ or to be 0.01 in our loss function (Eq. 1). Note that the DGS experiments can only access nearsurface training signals, in contrast to the oracle learning setting as in the stateof-the-art approaches where models were trained with dense occupancy labels.
We name the DGS version of the CoReNet model as the DGS model, and the DISN+DVR counterpart as the DGSBest model.
Quantitative results are reported in Tab. 4 for the low-realism evaluation setting, and in
Tab. 1 for the high-realism setting 1 . Our competitive experimental results indicate that our learning
[Figure]
[Table]
framework along with the differentiable gradient sampling layer implementation plays the critical role when learning with only the near-surface training labels
[Table]
and can achieve similar performance without the dense occupancy training labels.
Please refer to Sec.
A for ablation baselines details and low-realism ShapeNet experimental settings.
4. L EARNING FROM R EAL S CANNED D ATASETS (S CANNET V2)
Dataset.
We use ScannetV2 (Dai et al., 2017) for training and evaluating the performance of the models on the real images.
We follow the standard training / testing split as used in Sun et al.
(2021) and Murez et al.
(2020), where 1513 scenes are for trainval (with 1201 for training and 312 for validation), and 100 for testing.
Each scene is provided with multiple image capture as well as the associated camera pose.
We train the models with all the views given in the training / validation set (2423872 frames in total, after filtering out frames with invalid extrinsic poses), while for testing, we select 10 frames with different extrinsic poses for each test scene.
Practically, since all the frames of the scenes are in the form of video clips, with adjacent frames associated with similar extrinsic cameras poses, we select the 10 frames for each frame via extracting every 100 frames from each scene video (e.g. Frame 1, 101, 201, ..., 901), resulting in 1000 frames in total in our test set ( frames per scene, with 100 test scenes).
Metrics.
We use the same evaluation metrics following Sun et al.
(2021) and Murez et al.
Since we are the first, to our knowledge, to evaluate single view 3D implicit reconstruction on the
ScannetV2 benchmark, and here we only evaluate the geometries within the camera view frustum rather than the whole scene geometries as in Sun et al.
; Murez et al.
In addition, we also only evaluate the geometries in front of the amodal depth for each pixel ray, where only the space in front of the wall, bounded within the ceiling and the floor, is evaluated.
We define the amodal depth for a pixel ray to be delineated by the minimum between the closest structure-category surface (e.g. walls and doors, etc) and the farthest surface.
In practice, in order to accommodate the evaluation of the surfaces right on the amodal depth, we slack the evaluation scope with a factor λ (1.05 in our case) multiplied with the amodal depth.
This evaluation protocol would be equivalent to the “single-layered” protocol used in Sun et al.
(2021), within our single-view scenario.
[Figure] [Figure]
Due to the inherent ambiguity of the scaling and shifting of the predicted 3D single view geometry (Ranftl et al., 2019; Yin et al., 2021), we follow them by computing the best scale and shift comparing the predicted depth with the ground truth depth.
For approaches that predict 3D surfaces, the rendered depth map from the predicted mesh would be used for calculating the scale and shift.
Please refer to Sec. B for details on baselines.
Results.
We provide quantitative comparison in Tab. 2 and qualitative comparison in Fig.
5 respectively.
Our model outperforms all state-of-the-art approaches as well as the ablation models.
Compared to the synthetic data scenario in Tab. 1 and 4, our model demonstrates large advantages when compared to existing state-of-the-art approaches, as our motivation stems from addressing learning from imperfect 3D labels directly from the real scan data.
Compared to single-view depth prediction approaches trained on massive data (Lasinger et al., 2019; Yin et al., 2021), our approach does not prevail on “Acc” and “Prec”.
This is due to the fact that these two metrics only project the predicted surface to the ground truth surface, giving advantages to approaches that only predict the visible surface.
Our approach still prevails for other metrics (Chamfer Distance and F1), which are considered as the most important metrics (Boˇziˇc et al., 2021;
Sun et al., 2021; Murez et al., 2020).
Generalization to Unseen Scenes (Pix3d and Open-Domain Images).
We further test our model without further finetuning directly on unseen scenes for evaluating the generalizability of the learned model.
We run our model on Pix3d Sun et al.
(2018) as well as test images downloaded directly from the internet.
We provide a detailed qualitative comparison in Fig.
6 and more results in Fig.
The results further indicate our learning framework exhibits promise for unseen scene generalization.
Failure Cases.
We provide two representative failure cases of our approach in Fig.
The first case (the first row) demonstrates difficulties in predicting the floor occupancy as a result of the noisy and non-watertight mesh during training.
The second case (the second row) shows that our model cannot identify small objects (e.g. chairs) and not predict the invisible partition of these objects.
C ONCLUSIONS
We have presented our learning framework for real-world 3D implicit surface reconstruction from a single view image.
Owing to our unique learning framework that directly trains the model from the raw scan data and our novel occupancy loss function over the gradients, we are able to go beyond the existing works for single-objects reconstruction or single view fitting.
Thanks to our differentiable gradient sampling module we enable efficient and memory-efficient end-to-end training from images and demonstrated single view 3D surface reconstruction results on scenes for the first time, to our knowledge.
A PPENDIX
In this document, we provide additional experimental details and results on ShapeNet in Sec. A, additional qualitative results on ScannetV2 in Sec.
B, analysis of numerical gradient approximation in Sec. C, additional results generalizing to external unseen scenes in Sec. D and E, as well as the details of the formulation and derivations of DGS in Sec. F.
Additional Quantitative Results. We provide our original results on high-realism ShapeNet in
Tab. 3. These results are evaluated with the 64 × 64 × 64 volume grids due to an error in our prior code base.
Based on the results in both Tab. 1 and Tab. 3, we found our proposed learning framework performs competitively with the oracle training setting, even though our approach does not utilize dense occupancy labels as used in the oracle state-of-the-art approaches (OccNet, DISN, CoReNet and DISN (ResNet50) + DVR).
On the other hand, we observe that our model demonstrates slight improvements when evaluated with the resolution of 64 × 64 × 64 , while evaluating the same models with the resolution of 128 × 128 × 128 would reverse the course. In particular, we found categories like “car” exhibited major performance decrease on all the approaches with the higher resolution evaluating setting. This might be due to the fact that these categories demonstrate empty ground truth inside the objects, and and at a higher resolution, the metrics are biased to penalize models that predict “occupie” inside the object.
However, the competitive performance in both resolution settings indicates that our learning with only the surface labels does not deteriorate the performance, and can be faithfully applied toward the real-scene settings where the dense occupancy labels are truly unavailable. [Table] Ablation Study Details. We conduct ablation studies to further validate the importance of derived
DGS module, via attempting work-around training methods without DGS. i)
NoGrad - To test the performance of the baseline when only the surface data are provided, we train this ablation model in exactly the same way compared to its original model, with the only exceptions that only the near-surface points are equipped with training labels, and we do not use any training labels from the non-surface query points (where it is not necessary to backpropagate the loss gradient of the spatial gradient using DGS). To further evaluate how the rate of known voxels affects the learning performance, we enlarge the near-surface region and evaluate when the rate is 10%, 30% and 50%. An increase in the performances among these baselines would indicate the importance of knowing more voxel labels if our proposed gradient loss (Eq. 1) is not imposed.
ii) FixedE - We train with both the near-surface as well as the spatial gradient supervision, without DGS - meaning the loss gradient of the spatial gradient would not back-propagate to any module before the sampling module - in our case, the feature encoder network. Note all the other losses without gradient sampling can still back-propagated to the feature maps. We report the ablation results in Tab. 1. Both experiments are conducted with the high-realism
ShapeNet data. We comfortably outperform all the in-house ablation baselines, validating the essential roles of DGS in our learning framework. Experiments on low-realism ShapeNet. For our low-realism evaluation setting, all the baseline approaches reported their results in the papers.
For OccNet and DISN, the reported results are based on knowing the category canonical view prior, which demonstrates considerable privilege with respect to accuracy Popov et al.
Hence, we mark the results from the literature as OccNet-Privilege and DISN-Privilege (OccNet-Priv. and DISN-Priv. for short). Compared to their privileged setting, our results demonstrate superior results even without the category canonical view prior privilege.
To further provide the the baseline results where these two approaches are without the category canonical view prior, we retrain their models with the released codes, and report the results in the “OccNet” and “DISN” row respectively. The results further provide evidence that the category canonical view prior demonstrates privilege on accuracy, as observed in Popov et al.
Qualitative Results.
We provide qualitative results in Fig.
9-13 as the additional illustration of the performance of all the approaches on ShapeNet.
[Table]
B A DDITIONAL R ESULTS ON S CANNET V
Baselines.
Since we demonstrate the first attempt to predict 3D implicit surface of scenes from a single image, very few exiting works provide a direct baseline performance to our task. For OccNet (Mescheder et al., 2019), since its image feature extraction is not local, and its gradient propagation does not require sampling, we use its direct application with Gropp et al. (2020) “OccNet + GeoReg” as one baseline.
We train DISN (Xu et al., 2019) with the TSDF voxelization labels (Murez et al., 2020; Sun et al., 2021; Bo ˇ zi ˇ c et al., 2021).
For CoReNet (Popov et al., 2020), we stick to its own voxelization and internal filling toolbox for obtaining the training labels.
Since CoReNet can only predict geometries within the fixed range of space, we tried our best to pick the best cube location based on the dataset statistics.
We also incorporate depth approaches (Ranftl et al., 2019;
Yin et al., 2021) for comparison by finetuning their weights on ScannetV2.
Lastly, we compare with the two ablation models NoGrad and FixedE as introduced in Sec. 4.1.
Implementation details for DGS.
Since the DISN variant architectures (Xu et al., 2019; Niemeyer et al., 2020; Yu et al., 2020) demonstrate higher degress of flexibility of representing any point in the space, in contrast to 3D convolutions that are fixed for a particular range of space (Popov et al., 2020), we build our model based on the former architectures.
Similar to Niemeyer et al.
Yu et al.
(2020), we use 5 residual blocks in the fully connected part with the first 3 blocks receiving the 2D features and its spatial gradients.
Our image encoder uses the same RexNext architecture (Xie et al., 2017) as in Yin et al.
and starts the training with the pre-trained weights.
We use a batch size of 32 images with 2048 near-surface points and 512 non-surface points per image.
Similar to the ShapeNet experiments, we set λ or to be 0.01.
We found in the real-world setting, this parameter setting is crucial for the convergence of the learning procedure.
Quantitative Results for 2.5D evaluation metrics.
We provide the quantitative 2.5D evaluation results in Tab. 5.
Our approach constantly outperforms all other implicit based baselines.
While our approach falls short slightly when compared to the existing depth-based approaches, we claim that our approach is not directly trained with the massive depth training data.
Our results indicate that our depth performance is still on par with the state-of-the-art depth prediciton approaches.
Additional qualitative results.
We provide additional qualitative results on ScannetV2 Fig.
[Table]
C A NALYSIS WITH N UMERICAL G RADIENT A PPROXIMATION
As part of our proposed learning framework with the loss imposed on the spatial gradient, the differentiable gradient sampling module (Sec. 3.2) has a numerical alternative where we can perturb the query points to get its simulated numerical gradients.
We further compare the closed-form performance with the numerical counterpart in Tab. 6.
Please find qualitative comparison in Fig.
We provide a closer look at our comparison to the simulated gradient baseline.
We found in our experiments that the simulated gradient baseline demonstrates relatively severe convergence difficulties.
As shown in Fig.
8, the simulated gradient model (red) does not observe loss drop in the first 10k training iterations.
Despite its subsequent loss drop, which indicates the simulated gradient model can still learn to predict the geometry in moderate accuracy, its converged training loss is still higher than our DGS (blue).
This aligns with our benchmarking evaluations in Tab. 6 that the closed-form variant (DGS) achieve better performance.
[Table]
T h e T o t a l
[Figure]
D A DDITIONAL G ENERALIZABILITY Q UALITATIVE R ESULTS
We provide more qualitative results on our model generalizing to unseen images downloaded from the Internet in Fig.
Each result is visualized in 6 views.
Once again, these additional visual results further indicate that our model demonstrates good generalizability capability for handling unseen indoor scene images. [Table]
E E VALUATION OF THE G ENERALIZABILITY TO M ATTERPORT 3D DATASET
To quantatively evaluate our generaizability to new datasets
[Table]
we provide our results on the Matterport3D dataset (Chang et al., 2017) using our model trained from ScannetV2 as used in Sec. 4.2.
The Matterport3d dataset (Chang et al., 2017) collects indoor scenes of 90 full houses.
Textured meshes are provided along with the scanned real images.
Since the Matterport3D dataset does not provide an official train/test split, and our model is not trained on Matterport3d, we evaluate our model over all the 90 scenes in the dataset.
In particular, we use the 129600 images (without the elevated pitch views where the camera is looking up at the ceiling) from the dataset, use thefirst images from each scene (resulting in 900 test images in total), and test their reconstruction performance in the same way as we tested on ScannetV2.
We provide the quantative evaluation result in Tab. 7.
We can see from the table that the results align with our evaluation in Tab. 2 that our model demonstrates clear advantages.
We also noticed that the AdelaiDepth Yin et al.
(2021) baseline (finetuned on ScannetV2) performance is closer to our results (F1) score compared to our
ScannetV2 evaluation.
This is probably due to the fact that AdelaiDepth was directly pre-trained from a massive number of training images and potentially gives it some privilege when generalize to a new dataset.
F D ETAILS OF THE FORMULATION AND DERIVATION OF DGS
In this section, we provide the full formulation as well as the derivation procedure of our DGS in this section as the supplementary to Sec. 3.2.
In Eq.
4, we provided the backward gradient for differentiable sampling for Pixel A.
The backward gradient for Pixel B, C and D are ∂L
[Equation]
In Eq.
5, we provided the forward gradient in the vertical pixel direction.
The forward gradient of the horizontal pixel direction is
[Equation]
In Eq.
6, we provided the backward gradient for only Pixel A.
The backward gradient for Pixel B, C and D are
[Equation]
In Eq.
[Equation] we provided the backward gradient for only Pixel A in the 3D setting.
The backward gradient for Pixel B, C and D are ∂L
[Equation]
∂L
[Equation]
As an illustration of the derivation for Eq.
8, we can obtain the backward gradient via
[Equation]
The four partial derivatives with respect to ϕ A in Eq.
13 can be determined based on the sampling rules.
For example, based on Eq.
3, we can quickly obtain the first derivative with respect to ϕ A via
[Equation]
For the other three derivatives with respect to ϕ A in Eq. 13, we can substitute Eq.
5, 10 into Eq.
7, and obtain them via
[Equation]
We can then obtain Eq. 8 via substituting Eq.
14, 15 into Eq.
[Figure]
[Figure]
[Figure]
[Figure] [Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure]