Advancing Self-supervised Monocular Depth Learning with Sparse LiDAR
Anonymous CoRL Submission
Paper ID
Abstract: Self-supervised monocular depth prediction provides a cost-effectivesolution to obtain the 3D location of each pixel.
However, the existing approachesusually lead to unsatisfactory accuracy, which is critical for autonomous robots.
Inthis paper, we propose a novel two-stage network to advance the self-supervisedmonocular dense depth learning by leveraging low-cost sparse (e.g. 4-beam) LiDAR.
Unlike the existing methods that use sparse LiDAR mainly in a mannerof time-consuming
iterative post-processing, our model fuses monocular imagefeatures and sparse LiDAR features to predict initial depth maps.
Then, an efﬁcient feed-forward reﬁne network is further designed to correct the errors in theseinitial depth maps in pseudo-3D space with real-time performance.
Extensiveexperiments show that our proposed model signiﬁcantly outperforms all the stateof-the-art self-supervised methods, as well as the sparse-LiDAR-based methodson both self-supervised monocular depth prediction and completion tasks.
Withthe accurate dense depth prediction, our model outperforms the state-of-the-artsparse-LiDAR-based method (Pseudo-LiDAR++ [1]) by more than 68 % for thedownstream task monocular 3D object detection on the KITTI Leaderboard.
Introduction
Obtaining the 3D location of objects is an essential task for autonomous robots.
However, accurate dense depth perception with LiDAR is normally expensive, thus limit it for mass production.
The depth prediction from monocular images [2, 3] is cost-effective and attracting more and moreattention from both research and industry communities.
Although depth prediction is an ill-posed problem since one image can be mapped to multiple plausible 3D scenes, many methods have been proposed and remarkable progress has been achievedin recent years [4, 5, 6, 7].
Unlike other computer vision tasks in which the annotations are relatively easy to obtain [8, 9], it is impractical to obtain large-scale dense depth labels.
Therefore,self-supervised monocular depth prediction has been a promising solution.
Typically networks aretrained to predict both the depth and ego-motion of the camera, while the re-projection photo-metricloss is calculated as an intermediary constraint to optimize the networks.
However, these methodsusually suffer from multiple challenges due to the loss function design.
The most signiﬁcant oneis that the re-projection constraint assumes the scene is static and without occlusions between theneighboring frames.
In fact, most of the vital objects (e.g., vehicles, pedestrians, and cyclists in thedriving scenario) are dynamic, and the occlusions are almost inevitable.
To handle these challenges, a potential direction is to perform the monocular depth prediction withother low-cost sensors like sparse (e.g. 4-beam) LiDAR.
Compared to the 64-beam LiDAR, thecost of 4-beam LiDAR is at least two orders lower while providing very sparse yet accurate depth.
These 4-beam sensors alone are too sparse to be directly used for high-level perception tasks likedetection and tracking, but they potentially can be used with images to guide the network for betterdense depth prediction Pseudo-LiDAR++ [1] employed the sparse LiDAR in the post-processingusing a graph-based depth correction (GDC) module to improve the performance of stereo 3Ddetection.
It achieved excellent performance for the detection task.
However, this approach hastwo signiﬁcant limitations: 1) the quality of its predicted dense depth is non-optimal since thesparse LiDAR data is not utilized in the depth prediction network; 2) the GDC post-processing
[Figure]
directly used such as detection due to the sparsity monocular depth prediction can be signiﬁcantly improved by effectively fusing the features the sparse Li
DAR self-supervised our high- remarkably improved did not utilize the visual context information
and is too slow ( 1 to 2 FPS) for real-time applications like autonomous robots.
Pursing anaccurate and real-time self-supervised monocular depth prediction, we propose a two-stagenetwork for depth prediction by fully
in the the
Our framework can learn the complementaryinformation from the two distinct types of features for the dense depth prediction tasks.
Toovercome the sparsity issue of the sparse LiDAR, we transform the sparse LiDAR pointsinto pseudo dense representations, which aremore suitable for networks to extract features,and then fuse the features with the image features to predict the initial depth.
To further improve the quality of initial depth, we train a ReﬁneNet to efﬁciently correct the high-order errors in the 3D space to obtain high-quality dense depth maps.
The ultimate goal of depth prediction is to provide 3D information for downstream tasks such asmonocular 3D detection and re-construction.
However, the relation between the performance ofdepth prediction and the high-level downstream tasks has not yet been explored.
To thoroughlyevaluate our proposed method, we report the performance for both low-level tasks, including selfsupervised dense depth prediction and completion, and a downstream high-level perception task:monocular 3D object detection.
On all these tasks, our proposed model signiﬁcantly outperformsthe state-of-the-art methods that rely on or do not rely on sparse LiDAR.
To summarize, our keycontributions are as follows:
• We propose a novel two-stage self-supervised network to predict and reﬁne dense depthmaps by fusing the features of 2D monocular images and 3D sparse
LiDAR points.
Ourexperiments demonstrates that our model achieves state-of-the-art performance in the depthprediction and completion tasks on the KITTI dataset.
• To overcome the sparsity issue of the sparse LiDAR, we propose to transform the sparsepoints into a novel pseudo dense representation (PDR) which can be more effectively fusedwith monocular image
features.•
With the improved predicted depth maps, the performance of the downstream task monocular 3D object detection is signiﬁcantly improved.
Our model outperforms the state-of-theart sparse-LiDAR-based 3D detection model (Pseudo-LiDAR++ [1]) by more than 68 % onthe KITTI dataset.
Related Work
Self-supervised Monocular Depth Prediction : Early work for depth prediction is usually supervised methods [4, 3, 10, 11, 12, 13, 5, 6, 7, 14].
Pixel-level labeled dense depth is rarely available,in recent years, self-supervised methods [15, 16, 17, 18] became more and more popular.
They haveachieved great success but still suffer from dynamic objects and scale ambiguity.
In contrast, wefuse the feature from sparse LiDAR points to help our method predict a more accurate depth foreach pixel.
Depth Prediction with Sparse LiDAR : Recently, many researchers proposed to use few-beam
LiDAR for better depth prediction [19, 20, 21, 22, 1].
The Pseudo LiDAR++ [1] achieved excellentperformance by a GDC post-processing module to optimize the predicted depth with 4-beam LiDARdata.
However, the potential of the sparse LiDAR is not fully discovered since the sparse LiDARpoints are unused in the initial depth prediction stage.
To effectively utilize the sparse LiDARand monocular images, we fuse them in both feature and prediction level for more accurate depthpredictions.
[Figure]
Depth Completion : The depth completion is a task to generate per-pixel dense depth maps from therelative sparse depths.
Most depth completion models [23, 24, 25, 26, 27, 28] are trained with labeleddata which require intensive human labors.
To utilize the massive unlabeled data, self-superviseddepth completion methods were developed in recent years [29, 30, 31, 32, 33] to generate depth mapsfrom 64-beams dense
LiDAR points.
Our proposed method is designed to predict depth maps from4-beams sparse LiDAR points, however, with the generalizability, our method is also applicable forthe depth completion task.
Monocular 3D Object Detection: Monocular 3D object detection is to directly predict the 3Dcoordinates of objects from monocular images.
There are mainly two types of methods: RGB imagebased and pseudo-LiDAR based.
Former employ detection networks like CenterNet [34] to predictthe bounding boxes [35, 36, 37, 38] directly from images.
The latter perform the detection overthe pseudo-LiDAR representation, which is lifted from the dense depth prediction [39, 40, 41, 42].
Beneﬁted from the 2D-to-3D mapping, the pseudo-LiDAR-based methods achieved much betterperformance [41].
Our experiments demonstrate that the performance of the monocular 3D objectdetection [1, 41] can be signiﬁcantly improved using our depth prediction.
The Proposed Method
The overview of our proposed self-supervised framework is shown in Fig.
Our framework predictsa dense depth map for each monocular image by taking two types of data as input: monocular imageand its corresponding sparse LiDAR points.
Our framework consists of two steps: initial depthprediction based on the fused multi-scale features from both the monocular image and sparse LiDARpoints, and depth reﬁnement to correct the high-order errors in the initial depth maps.
The details ofeach component are described in the following sections.
3.1 Initial Depth Generation
For each monocular image I H
× W × 3 , the corresponding sparse
LiDAR points are P N × 3 capturedby few-beam LiDAR (e.g. 4-beam) where N is the number of the points.
Each point p i consistsof three values X, Y, Z , representing the location in the 3D space.
The network predicts an initialdepth map for each image based on the fusion of features from image
I and the corresponding sparse
LiDAR points P .
The main challenge here is to effectively fuse the features from a 2D image and the features from aset of unordered LiDAR points.
When projecting all the sparse LiDAR points into the image plane,only 1 . 4 % of the pixels have corresponding depth values.
We observe that simply concatenating theimage data and the projected sparse depth map can only negligibly improve the performance dueto the sparsity of the representation.
To resolve the sparsity issue, we transform the sparse LiDAR points into pseudo dense representations (PDRs), which can be effectively encoded and fused withthe monocular image features by our model.
Pseudo Dense Representations Generation:
For each image I H
× W × 3 , all its corresponding sparse
LiDAR points P N × 3 are transformed into two-channel PDRs with a size of H × W × 2 , including adepth channel to present the absolute depth values of each pixel, and a conﬁdence channel presentsthe reliability of the corresponding depth.
Based on the assumption that the depth values should besimilar for most neighboring pixels, we generate the depth channel by ﬁrst projecting each sparse
LiDAR point p
i into the image plane at the position of ( u i , v i ) and then dilate it into a circular areawith a radius of R .
The depth channel ( D ) is generated by setting the depth value of pixel ( x, y )within this circular area as Z i :
[Equation]
Although the depth channel transfers the sparse LiDAR points into dense representation, it inevitablyintroduces noises.
To mitigate the impact of these noises, we further generate the conﬁdence channel to indicate the reliability of depth for each pixel.
The conﬁdence of each pixel is inverselyproportional to the distance to its circular center:
[Equation]
If more than one sparse data point is close to an PDR pixel, the conﬁdence and depth scores generated from these multiple data points will be averaged.
These two channels jointly provide alternativedense representations that can be encoded by the convolutional neural networks more effectively.
DepthNet for Initial Depth Map Prediction Based on Fused Features.
After transforming thesparse LiDAR points into the two-channel PDRs, the features of PDRs and monocular images canbe fused together for the initial depth prediction.
To enable our network to thoroughly learn thecomplementary information from two distinct features, we adopt the intermediate fusion to combinethe multi-scale deep features layer by layer.
As shown in Fig.
2, there are two feature encoders: PDRfeature encoder for extracting features that explicitly encodes depth information from the sparse
LiDAR pseudo dense representation, and monocular image feature encoder to extract feature whichimplicitly encodes semantic information from images.
The decoder network takes the two typesof multi-scale features and concatenates them together to predict the initial depth map at multiplescales.
With the effective feature fusion, our DepthNet can predict more accurate dense depths.
PoseNet for Ego-Motion Prediction.
The PoseNet is essential for self-supervised monocular depthprediction since its predicted ego-motion makes the cross-frame projection possible, which is usedas geometry constraints to train the network.
It is directly related to the quality of the pixel correspondence across frames.
To predict accurate ego-motions, as shown in Fig.
2, the PoseNet is alsodesigned to take the complementary monocular images and the PDRs as the input.
The ego-motionis formulated as 6 degrees of freedom, consisting the camera rotation and translation.
3.2 Depth Reﬁnement
Our network produces relatively accurate initial depth predictions by fusing the features of monocular images and sparse LiDAR points.
However, the network still makes various errors, such asinconsistent depth prediction for different parts of the same object and the systematic depth bias.
To further resolve the inconsistent depth prediction and improve the correction computing efﬁciency,we propose a ReﬁneNet to correct the initial depth errors in the pseudo 3D space [41].
Our ReﬁneNetis a multi-scale fully convolutional network, takes the PDR feature, image feature, and initial depthas input, outputs the reﬁned ﬁnal depth prediction.
To leverage the 3D information predicted from
DepthNet, each initial depth map is ﬁrst transformed into a 3-channel x-y-z map representation.
Fora pixel at location ( u, v ) with depth d at the initial depth map, the transformation is based on thecamera intrinsics f x , f y , C x , C y :
[Equation]
Our ReﬁneNet is trained by distilling knowledge from a ofﬂine depth correction module in theway of self-training.
As shown in Fig.
2, for each initial depth map, a depth correction moduleis applied to produce a more accurate depth named ‘Enhanced Depth’ with the guidance of sparse
LiDAR points.
By training with pairs of the pseudo 3D initial depth, PDR and image features, andthe enhanced depth, the ReﬁneNet can improve the initial depth quality by correcting its errors.
Compared to the depth correction module GDC [1] we used as teacher module, our network isvery computationally efﬁcient for real-time systems.
Moreover, the conventional depth correctionmodule can still be further applied as post-processing after our ReﬁneNet to improve the accuracyby sacriﬁcing the real-time performance.
3.3 Loss Functions
Our model is jointly trained with two loss functions: the re-projection loss L re to utilize the interframe geometric constraints and the scale-invariant loss L si to distill the knowledge from the enhanced depth maps generated by the ofﬂine depth correction model.
The re-projection loss L re is a linear combination of two parts: the photo-metric loss L p with aﬁltering mask µ , and the smoothness loss L smooth .
The photo-metric loss L p is to evaluate thepixel-level similarity between the re-projected fake image I t
→ t (cid:48) with the real image
I (cid:48) t at adjacenttime frames, based on the photo-metric reconstruction error pe which consists of the SSIM and L 1distance to penalize the errors of the re-projected image.
We choose to use the frames I t +
and I t − 1as
I t (cid:48) .
The proj () projects current frame dense depth D t to frame t (cid:48) with camera intrinsic matrix Kand the camera ego-motion estimated by the PoseNet, (cid:10)(cid:11) is the sampling operator.
[Equation]
To eliminate the shrinking of the depth map, we further adopt the edge-aware metric from [43] intoour smoothness loss function L smooth , which is formulated as:
[Equation]
while d ∗ t = d t /d t is the mean-normalized inverse depth.
This normalization makes the smoothnessloss invariant to output scale.
For the photo-metric loss L p , we follow [17] to apply a ﬁltering mask µ to ﬁlter out the occlusionand stationary pixels, and then interpolate the depth predictions at each scales to the input resolutionbefore computing our re-projection loss to eliminate the ’holes’ at the low-texture area.
The ﬁlteringmask µ is formulated as:
[Equation]
To distill the knowledge from the ofﬂine correction model, the scale-invariant loss is employed foroptimization, which is formulated as:
[Equation]
(12) where y and y ∗ indicate the predicted depth and enhanced depth respectively, n is the number ofpixels.
The Si loss penalizes the relative depth-differences between each pixel pairs.
Our framework is trained with a linear combination of the re-projection loss L re and the scaleinvariant loss L si :
[Equation]
while α and β are the weights for re-projection loss L re and scale-invariant loss L si respectively.
The implementation details can be found in the supplementary materials.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
with methods that also rely on sparse LiDAR points.
The input point clouds are randomly sampled from 64-beam LiDAR points.
Our model outperforms all other methods with a large gap even without reﬁnement.
[Table]
for the ing set
*The depth module of Pseudo LiDAR++ [1] is stereo, which can greatly improve the detection result.
For fair comparison, we replace the it with Monodepth2 [17], and refer as
MonoPL++ [1]
Experiments
4.1 Depth Prediction
Dataset.
Following the state-of-the-art methods [17, 18, 15], we evaluate the performance of densedepth prediction on the Eigen split [57] of the KITTI original dataset.
We did not evaluate on the
KITTI testing benchmark which is for vision-only methods.
The 4-beams data are sampled fromoriginal 64-beams LiDAR data same as Pseudo LiDAR++ [1].
Results.
To extensively evaluate the performance, we compare with four types of methods including: (1) self-supervised monocular-based methods ( M ) [58, 44, 43, 59, 60, 17, 18, 15], (2) selfsupervised stereo-based methods ( S ) [61, 62, 16, 63, 64], (3) supervised methods ( Sup ) [45, 46],and (4) methods that use LiDAR signal as guidance ( L ) [1, 47].
The performance comparison withthe state-of-the-art methods for these groups is shown in Table 1.
Note that the initial depth module in Pseudo LiDAR++ [1] is supervised and stereo.
For fair comparison, we replace it with thestate-of-the-art unsupervised monocular depth module Monodepth
We will refer this modelas MonoPL++.
Due to the limitations of the re-project photo-metric loss, the self-supervised monocular (M) [58,44, 43, 59, 60, 17, 18, 15] and stereo-based methods (S) [61, 62, 16, 63, 64] usually have Abs Relover 0 .
With the advantage of using the sparse LiDAR, the performance of our initial depth maps
[Table]
is already better than all these methods and even outperforms the supervised methods (Sup) [45, 46].
With our ReﬁneNet, our performance is further improved and outperforms all the sparse-LiDARbased methods [1, 47].
Using the GDC for post-processing, our ﬁnal results signiﬁcantly outperformall other methods, including the above mentioned most recent work MonoPL++ [1] which has accessto the same amount of the sparse LiDAR points and same post-processing.
The results indicate aneffective fusion of sparse LiDAR points and monocular images achieves more accurate dense depthpredictions.
More quantitative and qualitative comparison and error analysis can be found in thesupplementary materials.
To extensively compare with the state-of-the-art methods under the same settings, as shown in Table 2, we compare with methods that were originally designed to use sparse LiDAR for depth prediction, including full-MAE [19], Parse-a-Line [20], Sparse-to-Dense [21], and MonoPL++ [1].
Foreach group of experiments, the same amount of sparse LiDAR points is used for a fair comparison.
Under the same settings, our proposed model consistently signiﬁcantly outperforms all thestate-of-the-art methods.
These results further conﬁrm the effectiveness of our proposed method.
4.2 Monocular 3D Object Detection
The ultimate goal of the monocular depth prediction is to provide 3D representation for downstreamtasks.
To demonstrate the impact of our improvement on the depth metrics to downstream tasks, weevaluate the performance of the monocular 3D object detection task with our predicted depth mapsas input on the KITTI detection dataset.
Following the state-of-the-art methods [41], we report 3D
Average Precision (AP) as the evaluation metrics.
Table 4 shows the performance comparison for the monocular 3D object detection task.
The samedetection model is employed for all the experiments while the only difference is the input depth.
The monodepth2 [17] is used as baseline model.
Our model use the same CNN backbone (ResNet18 [9]) as Monodepth2 [17].
By effectively using the sparse LiDAR, the detection performancecan be improved by more than 40 % compared to the baseline which only uses monocular images.
Compared to the recently proposed MonoPL++ [1], our model signiﬁcantly outperforms it by morethan 98 .
2 % in terms of the mAP over all the three categories.
Without the time-consuming iterativereﬁnement module, our model is 50 times faster than the MonoPL++ [1].
Furthermore, our ReﬁneNet obtains a signiﬁcantly increased performance on the detection metricsthan the depth metrics.
The improvement of applying ReﬁneNet on the depth score is only around3 .
4 % in terms of the relative error, while the improvement is more than 23 % on the detection score.
We observe that our improvement in the dense depth prediction can yield a even more signiﬁcantimprovement in the downstream task.
This indicates the importance of using the downstream tasksto evaluate the quality of the learned dense depth.
Table 3 further shows the comparison of our method with other state-of-the-art for monocular 3Ddetection tasks on the car category.
As an unsupervised learning method, our method signiﬁcantlyoutperforms all the state-of-the-art methods by a large margin, including the sparse-LiDAR-based
MonoPL++ [1].
Note that the performance of our model can be further improved by extending it tothe supervised setting.
More qualitative analysis can be found in the supplementary materials.
[Table]
4.3 Depth Completion
We evaluate our model on the KITTI depth completion task.
Most (over 95 %) of the methods inthe KITTI depth completion benchmark is under supervised training. Following other state-of-theart self-supervised methods [29, 30, 31, 32, 33], we test our model on the KITTI validation set,as shown in Table 6.
By effectively utilizing the LiDAR features, our model outperforms all otherself-supervised methods by a large gap with all the metrics demonstrating the generalizability of ourproposed method.
4.4 Ablation Study
[Table]
We evaluate our method on the KITTI depth completion task validation set, comparing it to the state-of-the-art self-supervised methods.
All metrics are the lower, the better.
The best results are in bold
To evaluate the effectiveness of each componentof our framework, we perform three groups of experiments to evaluate the impact of pseudo denserepresentation, camera-LiDAR fusion, and the reﬁnement on the dense depth prediction task.
Theresults of the ablation studies are shown in Table 5.
The impact of the LiDAR sparsity andthe error analysis by semantic categories can befound in the supplementary materials.
Pseudo Dense Representation.
When the sparse
LiDAR are directly used as input, the performance for depth prediction is only slightly improved by around 6 %.
However, by directly usingour proposed pseudo dense presentation as input,the improvement is doubled ( 12 %), conﬁrming itsimportance.
PDR Feature and Image Feature Fusion.
We conduct three types of feature fusions, includinginput level, feature level, and output level fusion.
The best performance is achieved by performingthe feature level fusion of the PDR features and image features from our proposed encoders.
Reﬁnement.
The third group shows that the depth prediction performance can be improved by our
ReﬁneNet evenly either with or without GDC.
Although the ReﬁneNet can only slightly improve theperformance of dense depth prediction, it can signiﬁcantly improve the monocular 3D object detection task (+ 23 %), demonstrating the effectiveness of the ReﬁneNet.
More qualitative and efﬁciencyanalysis of the ReﬁneNet can be found in the supplementary materials.
Conclusion
We proposed a two-stage self-supervised framework that effectively fuses features from monocularcamera images and sparse LiDAR data.
Our method outperforms all the state-of-the-art methods onboth depth prediction and completion tasks.
Also shows a remarkable advantage for downstreamtasks like monocular 3D object detection.