TANDEM: Tracking and Dense Mapping in Real-time using Deep Multi-view Stereo
Abstract:
In this paper, we present TANDEM, a real-time monocular tracking and dense mapping framework.
For pose estimation, TANDEM performs photometric bundle adjustment based on a sliding window of keyframes.
To increase the robustness, we propose a novel tracking front-end that performs dense direct image alignment using depth maps rendered from a global model that is built incrementally from dense depth predictions.
To predict the dense depth maps, we propose Cascade View-Aggregation MVSNet (CVA-MVSNet) that utilizes the entire active keyframe window by hierarchically constructing 3D cost volumes with adaptive view aggregation to balance the different stereo baselines between the keyframes.
Finally, the predicted depth maps are fused into a consistent global map represented as a truncated signed distance function (TSDF) voxel grid.
Our experimental results show that TANDEM outperforms other state-of-the-art traditional and learning-based monocular visual odometry (VO) methods in terms of camera tracking.
Moreover, TANDEM shows state-of-the-art real-time 3D reconstruction performance.
Webpage: https://go.vision.in.tum.de/tandem
Introduction
Real-time dense 3D mapping is one of the major challenges in computer vision and robotics.
This problem, known as dense SLAM, includes both estimating the 6DoF pose of a sensor and a dense reconstruction of the surroundings.
While there exist numerous well-working and robust RGB-D mapping solutions [1, 2, 3], real-time dense reconstruction from monocular cameras is a signiﬁcantly more difﬁcult challenge as depth values cannot be simply read out from the sensor and fused.
Nevertheless, it is a very important problem, as monocular approaches offer signiﬁcant advantages over RGB-D-based methods [1] which are usually limited to indoor environment due to the nearrange sensing or LiDAR-based [4] solutions which are expensive and heavyweight.
Several deep neural network (DNN) based approaches have been proposed to tackle the problem of monocular tracking and dense mapping by utilizing monocular depth estimation [5], variational auto-encoders [6, 7, 8], or end-to-end neural networks [9, 10].
Unlike the aforementioned works, in this paper, we propose a novel monocular dense SLAM method, TANDEM, which, for the ﬁrst time, integrates learning-based multi-view stereo (MVS) into a traditional optimization-based VO.
This novel design of dense SLAM shows state-of-the-art tracking and dense reconstruction accuracy as well as strong generalization capability on challenging real-world datasets with the model trained only on synthetic data.
Figure 1 shows the 3D reconstructions delivered by TANDEM on unseen sequences.
Our contributions.
(1) a novel real-time monocular dense SLAM framework that seamlessly couples classical direct VO and learning-based MVS reconstruction; (2) to our knowledge, the ﬁrst monocular dense tracking front-end that utilizes depth rendered from a global TSDF model; (3) a novel MVS network, CVA-MVSNet, which is able to leverage the entire keyframe window by utilizing view aggregation and multi-stage depth prediction; (4) state-of-the-art tracking and reconstruction results on both synthetic and real-world data.
Related Work
There are two different work streams related to the proposed method.
On one side, there is pure 3D reconstruction based on posed images and, on the other side, there are full SLAM or VO frameworks that simultaneously estimate camera poses and a 3D reconstruction of the environment. 3D Reconstruction.
Most dense 3D reconstruction approaches consider images and corresponding reference poses as inputs and estimate a dense or partially dense reconstruction of the environment.
Over the last decade, several classical methods have been proposed [14, 15, 16, 17].
Recently, deep-learning-based methods have shown superior performance over classical methods.
These methods regress a 3D model of the environment utilizing DNNs.
This 3D model can be either in the form of a volumetric representation [9, 18, 19, 20], a 3D point cloud [21] or a set of depth maps [22, 23, 24].
Nowadays, most popular are methods which predict the ﬁnal model from 3D cost volumes.
Huang et al.
[22] proposed one of the ﬁrst cost-volume-based approaches.
Zhou et al.
[24] aggregate multiple image-pair-wise volumes to a single cost volume and use a 3D CNN for depth prediction.
Yao et al.
[23] propose to directly calculate a single volume based on 2D deep feature maps predicted from the input images.
In a follow-up work, Yao et al.
[25] replace the depth prediction CNN by a recurrent network.
To improve run-time and memory consumption, Gu et al.
[26] propose a cascade cost volume.
Yi et al.
[27] introduce a self-adaptive view aggregation to weigh the voxel-wise contribution for each input image.
The proposed CVA-MVSNet is built upon the two aforementioned works [26, 27] and largely inspired by them.
However, only by their combination and adaption to the SLAM setting we achieve better performance and real-time capability.
While all previous methods are based on a set of selected frames, Murez et al.
[9] instead directly predict a TSDF model from a single global 3D cost volume.
Weder et al.
[18] propose a learningbased alternative to classical TSDF fusion of depth maps.
While these volumetric representations, in general, are rather memory intense, Niessner et al.
[28] propose voxel-hashing to overcome this limitation and Steinbr¨ucker et al.
[29] perform depth map fusion on a CPU using an octree.
RGB-D SLAM.
In the area of visual SLAM, RGB-D approaches by nature provide dense depth maps along with the camera trajectory and therefore target to solve a similar problem as our approach.
Bylow et al.
[30] and Kerl et al.
[2, 31] mainly focus on accurate trajectory estimation from
RGB-D images.
In addition to camera tracking, Newcombe et al.
[1] integrate the depth maps into a global TSDF representation.
Whelan et al.
[3] perform surfel-based fusion and non-rigid surface deformation for globally consistent reconstruction.
K¨ahler
et al.
[32] use a TSDF map representation which is split into sub maps to model loop closures.
While most previous methods optimize only for the frame pose, Sch¨ops et al.
[33] propose a full bundle adjustment based direct RGB-D SLAM
[Figure]
Warp
VA
Warp
VA 3D U-Net [Figure] U [Figure]
Net 3D U-Net [Figure]
DifferentiableWarping
VA ViewAggregation
[Figure]
Monocular SLAM.
Compared to RGB-D methods, for monocular approaches both tracking and mapping become much more challenging.
Using a single monocular camera, Newcombe et al.
[35] perform optimization based on a photometric cost volume to jointly estimate the camera pose and dense depth maps in real-time on a GPU.
Engel et al.
[36] propose the ﬁrst large-scale photometric SLAM formulation including loop closure detection and pose graph optimization.
By using a sparse representation, Engel et al.
[37] were able to formulate the ﬁrst fully photometric VO framework which jointly estimates pose and depth in real-time.
To obtain denser reconstructions, Mur-Artal et al.
[38] perform semi-dense probabilistic depth estimation on top of feature-based SLAM [39].
Sch ¨ ops et al.
[40] perform temporal, plane-sweep-based depth estimation using the poses and images obtained from a mobile tracking device.
Tateno et al.
[5] and Yang et al.
[41, 42] leverage DNNs in a traditional direct SLAM framework to improve tracking performance and overcome the problem of scale ambiguity.
While traditional geometric and hybrid approaches still achieve superior tracking performance, there are several fully learned SLAM frameworks [43, 44, 45, 24], which are superior in terms of reconstruction completeness.
Jatavallabhula et al.
[46] propose a differential optimizer which has the potential to bridge the gap between traditional and learning-based SLAM.
A novel idea for deep-learning-based SLAM is proposed by Bloesch et al.
The authors propose to learn a frame-wise code representation for the scene depth, which is the jointly optimized together with the camera pose.
The work of Czarnowski et al.
[7] is an extension of [6], where the code representation is integrated into a full SLAM system.
Zuo et al.
[8] make use of a similar code representation in a visual-inertial setup and furthermore feed sparse, tracked features into a DNN.
TANDEM
The proposed TANDEM is comprised of three components: monocular visual odometry (section 3.1), dense depth estimation with CVA-MVSNet (section 3.2), and volumetric mapping (suppl.).
Figure 2a shows an overview of the system.
The visual odometry utilizes the monocular video stream and the dense depth rendered from the 3D TSDF model to estimate camera poses in a sliding-window manner.
Given the keyframes and their estimated poses, the proposed CVA-MVSNet predicts a dense depth map for the reference keyframe.
To reconstruct a complete and globally consistent 3D model of the environment, the depth maps are then fused into the TSDF voxel grid [47] with voxelhashing [28].
By seamlessly integrating these components, the resultant system TANDEM enables real-time tracking and high-quality dense mapping from a monocular camera.
Further details, including on the TSDF volume initialization, are given in the supplementary material.
3.1 Visual Odometry
Estimating camera poses by tracking a sparse set of points across multiple frames has shown great performance in recent VO systems [37, 39].
Using more points for the joint optimization, however, does not necessarily further improve the accuracy of the estimated poses while signiﬁcantly increases the runtime [37].
Therefore, in the proposed VO system we make use of a direct sparse windowed optimization back-end as described in Direct Sparse Odometry (DSO) [37].
However, we utilize dense depth maps rendered from the global TSDF model, which we build up incrementally, in the direct image alignment front-end.
In numerous experiments, we conﬁrm that this combination of dense tracking front-end and sparse back-end optimization signiﬁcantly improves the tracking performance (cf.
Table 1) while maintaining a fast runtime.
Dense Front-end Tracking.
The front-end tracking provides camera-rate pose estimations and serves as initialization for the windowed optimization back-end.
In the original DSO, a new frame is tracked against the last keyframe n by direct image alignment using a sparse depth map D DSO n [ p ] generated from all points in the optimization window.
This approach, however, lacks robustness (cf.
Table 1) due to the sparsity of the depth map.
We alleviate the issue by incorporating a dense depth map D TSDF n which is rendered from the constructed TSDF model.
For each pixel p in the current keyframe n , we assign a depth value either based on the sparse VO points D DSO n [ p ] , if available, or based on the rendered dense depth D TSDF n [ p ] , otherwise.
Due to the incrementally-built TSDF model, the combined depth buffer might not contain valid depth values for all pixels but it is much denser in comparison to using the sparse depth values only.
The nearly-dense combined depth map is used for two-frame direct image alignment.
3.2 CVA-MVSNet
Let { ( I i , T i ) } ni =1 be the set of active keyframes where I i is the image of size ( H, W ) and T i is the corresponding global pose estimated by the VO.
CVA-MVSNet is based on the principles of multi-view stereo [48] and further leverages deep neural networks [23] to estimate a depth map for the reference frame I n − 1 .
CVA-MVSNet overcomes the prohibitive memory requirement of deep
MVS networks by hierarchically estimating the depth using cascaded cost volumes and aggregates the deep features of all the keyframes effectively with a self-adaptive view aggregation module.
As shown in Figure 2b, the multi-scale deep features F si of the keyframes are ﬁrstly extracted by 2D U-Nets with shared weights, where i ∈ [1 , n ] is the frame index and s ∈ [1 , 3] is the scale index.
As a result, F si is of the shape ( F s , H s , W s ) where F s is the feature dimension of the scale s , H s = H/ 2 3 − s , and W s = W/
2 3 − s .
The depth map of the reference frame is estimated hierarchically with 3 stages each of which takes the set of features { F si } ni =1 as the inputs and predicts the reference depth map D s of shape ( H s , W s ) .
For clarity, we ﬁrst explain how a single stage estimates the depth and then describe how multiple stages are assembled hierarchically.
Single Stage Depth Estimation.
For each stage, a cost volume C s needs to be constructed using the deep features { F si } ni =1 .
For each pixel of the reference frame, we deﬁne D s depth hypotheses, which results in a tensor D shyp of shape ( D s , H s , W s ) .
The deep features F si of each frame are geometrically transformed with differentiable warping [49] using the depth hypotheses, the relative pose T ij =
T − 1 i T j and the camera intrinsics.
As a result, a feature volume V si of shape ( F s , D s , H s , W s ) is constructed for every frame.
In order to aggregate the information from multi-view feature volumes into one cost volume C s , most prior deep MVS methods treat different views equally and use a variance-based cost metric: [Equation] n
[Equation]
where
[Equation]
However, in the sliding-window SLAM setting, the keyframes are not evenly distributed within the optimization window – typically the distance between newer keyframes is much smaller than between older keyframes.
This causes considerable occlusion and non-overlapping images.
The variance-based cost volume, which weighs different views equally, is thus inappropriate.
To alleviate this issue, we employ self-adaptive view aggregation [27] to construct the cost volume:
[Equation]
where the view aggregation weights W si have the shape (1 , D s , H s , W s ) and (cid:12) is element-wise multiplication with broadcasting.
The view aggregation weights W si are estimated by a shallow 3D convolutional network for each V si separately by taking ( V si − V sj ) 2 as the input.
This aggregation module allows the network to adaptively downweight erroneous information.
The cost volume C s is then regularized using a 3D U-Net and ﬁnally passed through a softmax non-linearity to obtain a probability volume P s of shape ( D s , H s , W s ) .
Given the per-pixel depth hypotheses D shyp of shape ( D s , H s , W s ) the estimated depth is given as the expected value D s [
[Equation]
Hierarchical Depth Estimation.
The network leverages the depth estimated from the previous stage D s − 1 ( s > 1) to deﬁne a ﬁne-grained depth hypothesis tensor D shyp with a small D s .
Since no prior stage exists for the ﬁrst stage, each pixel of D 1 hyp has the same depth range deﬁned by [ d min , d max ] with D 1 = 48 depth values.
For the later stages ( s > 1 ), the depth D s − 1 is upsampled and then used as a prior to deﬁne D shyp .
Speciﬁcally, for the pixel location ( h, w ) , D shyp ( · , h, w ) is deﬁned using the upsampled D s − 1 ( h, w ) as the center and then sampled D s values around it using a pre-deﬁned offset [26].
In this way, fewer depth planes are needed for the stage with a higher resolution, i.e., D 1 ≥ D 2 ≥ D 3 .
We train the network using the L 1 loss applied on all three stages with respect to the ground-truth depth and use the sum as the ﬁnal loss function.
3.3 Implementation Details
To guarantee real-time execution, TANDEM leverages parallelism on multiple levels.
Dense tracking and bundle adjustment are executed in parallel threads on the CPU while, asynchronously and in parallel, TSDF fusion and DNN inference are run on the GPU.
We train our CVA-MVSNet in
PyTorch [50] and perform inference in C++ using TorchScript.
TANDEM can processes images at ca.
20 FPS while running on a desktop with an Nvidia RTX 2080 super with 8 GB VRAM, and an Intel i7-9700K CPU.
This includes tracking and dense TSDF mapping but no visualization or mesh generation.
We refer to the supplementary for further details, including how to potentially scale the network for deployment on embedded platforms.
Experimental Results
As TANDEM is a complete dense SLAM system, we evaluate it for both monocular camera tracking and dense 3D reconstruction.
Speciﬁcally, for the camera tracking, we compare with state-of-the-art traditional sparse monocular odometry, DSO [37] and ORB-SLAM [39], as well as learning-based dense SLAM methods, DeepFactors [7] and CodeVIO [8].
For the 3D reconstruction, we compare with a state-of-the-art deep multi-view stereo method, Cas-MVSNet [26], end-to-end reconstruction method, Atlas [9], learning-based dense SLAM methods, CNN-SLAM [5], DeepFactors [7], and CodeVIO [8], as well as iMAP [34], a recently proposed RGB-D dense SLAM method using a deep implicit map representation [51, 52].
In the following, we will ﬁrst introduce the datasets we use for training and evaluation.
Note that only CVA-MVSNet needs to be trained while the dense tracking part of TANDEM is purely optimization-based and does not require training on speciﬁc datasets.
Then, the ablation study for
TANDEM demonstrates different design choices.
In the end, we show quantitative and qualitative comparisons with other state-of-the-art methods.
Due to the limited space, we demonstrate only part of the conducted evaluation and refer to the supplementary material for additional experiments.
4.1 Datasets
Training sets .
We train two models for CVA-MVSNet: One on the real-world ScanNet [53] dataset and the other one on the synthetic Replica dataset [11].
The ScanNet dataset consists ofindoor scenes and we use the ofﬁcial train and test split for training CVA-MVSNet to give a fair comparisons with DeepFactors and Atlas.
However, the geometry and texture in ScanNet show noticeable artifacts and incompleteness [11], which limits its potential to training high-quality dense reconstruction methods.
To complement the ScanNet dataset, we build upon the recently proposed
Replica dataset [11], which consists of 18 photorealistic scenes.
These scenes were captured from real-world rooms using a state-of-the-art 3D reconstruction pipeline and show very high-quality texture and geometry.
Because the authors did not release any sequences, we extend the dataset by manually creating realistic camera trajectories that yield 55 thousand poses and images. i c l / o ff i c e Image
DeepFactors
Atlas TANDEM (Replica) GT Depth i c l / li v i n ge u r o c / v D e p t h [ m ]
D e p t h [ m ] D e p t h [ m ] Evaluation sets . We use the ICL-NUIM [13] dataset and the Vicon Room sequences of the EuRoC dataset [12] for evaluating the tracking and the dense 3D reconstruction.
Note that TANDEM is not trained on either of the datasets.
ICL-NUIM is a synthetic indoor dataset with pose and dense depth ground-truth.
It contains low-texture scenes which are challenging for monocular visual odometry and dense depth estimation.
EuRoC is a real-world dataset recorded by a micro aerial vehicle (MAV).
4.2 Camera Pose Estimation
We evaluate TANDEM for pose estimation against other state-of-the-art monocular SLAM methods on EuRoC and ICL-NUIM.
On EuRoC, we evaluate against DSO [37], ORB-SLAM2 [39], DeepFactors [7], and CodeVIO [8] uses a camera and an IMU sensor. Note that we turn off the global optimization and relocalization of ORB-SLAM2 for a fair comparison.
We also implement a variant of DSO (DSO + Dense Depth) which uses all the pixels of the dense depth maps estimated by CVA-MVSNet for the front-end direct image alignment.
Note that the difference between TANDEM and DSO + Dense Depth is that TANDEM tracks against the global 3D model by rendering the depth maps from the TSDF grid.
TANDEM achieves overall better tracking accuracy and robustness than the other monocular methods on both ICL-NUIM and EurRoC. Due to the limited space, we show the results on EuRoC in Table 1 of the main paper and kindly refer to the supplementary material for the results on ICL-NUIM where we also show the comparision with DeepTAM [24].
All the methods except for CodeVIO [8] are run ﬁve times for each sequence and reported with the mean RMSE error and standard deviation in terms of absolute pose estimations after Sim(3) alignment with ground-truth.
For CodeVIO, we directly take the numbers reported in their paper.
The comparison with DSO and DSO + Dense Depth indicates that the proposed dense tracking against the global 3D model improves the camera pose estimations, especially on the more challenging sequences (V102 and V202).
However, we should admit that TANDEM still cannot compete with
CodeVIO which uses an IMU sensor for pose estimations.
4.3 Ablation Study
We conduct the ablation study of CVA-MVSNet on the test split of Replica and show the results in Table 2.
Speciﬁcally, we evaluate the effectiveness of using the full VO window with [Table] [Table] keyframes (Win), the view aggregation module (VA), and fewer depth planes (S) with (48 , 4 , 4) for ( D 1 , D 2 , D 3 ) .
The baseline method is the original Cas-MVSNet using 3 multi-view images as the inputs, no view aggregation module, and more depth planes (48 , 32 , 8) .
We use the absolute difference (Abs), and the percentage of inliers with different thresholds ( a 1 , a 2 , a 3 ) as the metrics for the depth map evaluation.
Please refer to the supplementary material for the formulas of the metrics.
In addition, we also show the inference time and the memory usage of different models.
From the table, we can see that using the entire keyframe window with more frames does not improve the accuracy over the baseline model while increasing the runtime and memory usage.
With the view aggregation module, the accuracy is signiﬁcantly improved, but the runtime further increases.
Using fewer depth planes does not show a signiﬁcant drop in accuracy but improves the runtime and memory usage a lot.
Therefore, to guarantee the real-time performance of TANDEM, we use the fewer-plane model as the ﬁnal CVA-MVSNet and all other experiments in the paper are conducted with this model.
4.4 3D Reconstruction
We evaluate the reconstruction accuracy on both ICL-NUIM and EuRoC.
On ICL-NUIM we compare with DeepFactors [7], CNN-SLAM [5], Atlas [9], and Cas-MVSNet [26].
Table 3 shows the evaluation results.
Since Atlas, a pure 3D reconstruction method, does not estimate poses, we provide ground-truth poses as the input.
Note that Atlas estimates a TSDF volume directly from a 3D CNN, so we render the depth maps for evaluation against other methods.
CNN-SLAM, DeepFactors, and Atlas are trained on ScanNet.
For Cas-MVSNet, we re-train it on the Replica dataset and use the same poses as for our CVA-MVSNet. We show the results of the evaluations for our
ScanNet-trained model and our Replica-trained model to facilitate a fair comparison.
The depth maps of monocular methods are aligned in scale based on the trajectory and further details are given in the supplementary.
We use the a 1 metric as the major measurement for accuracy.
From Table 3, we can see that our method shows a notable improvement in comparison to the other methods and delivers the best result on average.
Note that CVA-MVSNet achieves better results with the model trained on the synthetic Replica than the model trained on ScanNet.
On EuRoC, we cannot compare with CNN-SLAM since the numbers on EuRoC are not provided on the paper and code is not publicly available.
We further add CodeVIO [8] into the evaluation on
EuRoC as it is a recent dense SLAM system and it was also evaluated on EuRoC.
Please note that
CodeVIO uses a monocular camera and an inertial sensor for tracking, while TANDEM and other
SLAM methods rely only on monocular cameras.
Table 4 shows the evaluation results.
We further evaluate TANDEM against iMAP [34], an RGB-D dense SLAM system that leverages deep implicit map representation.
Please note two major differences between iMAP and TANDEM: on one hand, iMAP uses an RGB-D sensor while TANDEM needs only a monocular camera; on
[Table]
[Table] [Table]
the other hand, the DNN of iMAP does not require any pre-training and it is trained purely online with the RGB-D inputs, while the depth estimation network of TANDEM requires ofﬂine training.
Since iMAP was also evaluated on Replica, we, therefore, compare TANDEM with iMAP on the two sequences of their dataset which are not included in the training set of our Replica split.
We use the evaluation metrics from iMAP and show the results in Table 5.
In general, TANDEM achieves similar results to iMAP while using a monocular camera.
In Figure 3 we show qualitative depth maps estimated by DeepFactors, Atlas, and TANDEM.
Both
DeepFactors and Atlas can recover the geometry of the underlying scene well, but our method generally manages to capture more ﬁne-scale details.
We further show the complete scene reconstruction as meshes in Figure 4.
As DeepFactors does not generate a complete 3D model by itself, we only compare TANDEM with Atlas for this experiment.
From the ﬁgure we can see that, similarly to the depth maps, TANDEM is able to reconstruct more ﬁne-scale details than Atlas.
Conclusion
We presented TANDEM, a real-time dense monocular SLAM system with a novel design that couples direct photometric visual odometry and deep multi-view stereo.
In particular, we propose
CVA-MVSNet which leverages the whole keyframe window effectively and predicts high-quality depth maps.
Further, the proposed dense tracking scheme bridges camera pose estimation and dense 3D reconstruction by tracking against the global 3D model created with TSDF fusion.
The quantitative and qualitative experiments show that TANDEM achieves better results than other state-of-theart methods for both 3D reconstruction and visual odometry on synthetic and real-world data.
We believe that TANDEM further bridges the gap between RGB-D mapping and monocular mapping. Acknowledgments
We thank the anonymous reviewers for providing helpful comments that improved the paper.
We express our appreciation to our colleagues, who have supported us, speciﬁcally we thank Mariia Gladkova (Technical University of Munich) and Simon Klenk (Technical University of Munich) for proof reading, as well as, Stefan Leutenegger (Technical University of Munich & Imperial College London) and Sotiris Papatheodorou (Imperial College London) for their help with setting up a demo for the conference. We thank the authors of iMAP, speciﬁcally Edgar Sucar (Imperial College London), for providing us with evaluation data and scripts from their paper.