P SEUDO -L I DAR++: A CCURATE D EPTH FOR 3D O BJECT D ETECTION IN A UTONOMOUS D RIVING
A BSTRACT
Detecting objects such as cars and pedestrians in 3D plays an indispensable role in autonomous driving.
Existing approaches largely rely on expensive LiDAR sensors for accurate depth information.
While recently pseudo-LiDAR has been introduced as a promising alternative, at a much lower cost based solely on stereo images, there is still a notable performance gap.
In this paper we provide substantial advances to the pseudo-LiDAR framework through improvements in stereo depth estimation.
Concretely, we adapt the stereo network architecture and loss function to be more aligned with accurate depth estimation of faraway objects — currently the primary weakness of pseudo-LiDAR.
Further, we explore the idea to leverage cheaper but extremely sparse LiDAR sensors, which alone provide insufﬁcient information for 3D detection, to de-bias our depth estimation.
We propose a depthpropagation algorithm, guided by the initial depth estimates, to diffuse these few exact measurements across the entire depth map.
We show on the KITTI object detection benchmark that our combined approach yields substantial improvements in depth estimation and stereo-based 3D object detection — outperforming the previous state-of-the-art detection accuracy for faraway objects by 40% . Our code is available at https://github.com/mileyan/Pseudo_Lidar_V2 .
I NTRODUCTION
Safe driving in autonomous cars requires accurate 3D detection and localization of cars, pedestrians and other objects.
This in turn requires accurate depth information, which can be obtained from LiDAR (Light Detection
And
Ranging) sensors.
Although highly precise and reliable, LiDAR sensors are notoriously expensive: a 64-beam model can cost around $75, (USD) 1 .
The alternative is to measure depth through inexpensive commodity cameras.
However, in spite of recent dramatic progress in stereo-based 3D object detection brought by pseudo-LiDAR (Wang et al., 2019), a significant performance gap remains especially for faraway objects (which we want to detect early to allow time for reaction).
The trade-off between affordability and safety creates an ethical dilemma.
In this paper we propose a possible solution to this remaining challenge that combines insights from both perspectives.
We observe that the higher 3D object localization error of stereo-based systems, compared to LiDAR-based ones, stems entirely from the higher error in depth estimation (after the 3D point cloud is obtained the two approaches are identical (Wang et al., 2019)).
Importantly, this error is not random but systematic : we observe that stereo methods do indeed detect objects with high reliability, yet they estimate the depth of the entire object as either too far or too close.
See Figurefor an illustration: the red stereo points capture the car but are shifted by about 2m completely outside the ground-truth location (green box).
If we can de-bias these depth estimates it should be possible to obtain accurate 3D localization even for distant objects without exorbitant costs.
We start by revisiting the depth estimation routine embedded at the heart of state-of-the-art stereobased 3D detection approach (Wang et al., 2019).
A major contributor to the systematic depth bias comes from the fact that depth is typically not computed directly.
Instead, one ﬁrst estimates the disparity — the horizontal shift of a pixel between the left and right images — and then inverts it to obtain pixel-wise depth.
While the use of deep neural networks has largely improved disparity estimation (Chang & Chen, 2018; Cheng et al., 2018; Mayer et al., 2016; Wang et al., 2018b), designing and learning the networks to optimize the accuracy of disparity estimation simply overemphasizes nearby objects due to the reciprocal transformation.
For instance, a unit disparity error (in pixels) for a 5-meter-away object means a 10cm error in depth: the length of a side mirror.
The same disparity error for a 50-meter-away object, however, becomes a 5.8m error in depth: the length of an entire car.
Penalizing both errors equally means that the network spends more time correcting subtle errors on nearby objects than gross errors on faraway objects, resulting in degraded depth estimates and ultimately poor detection and localization for faraway objects.
We thus propose to adapt the stereo network architecture and loss function for direct depth estimation.
Concretely, the cost volume that fuses the left-right images and the subsequent 3D convolutions are the key components in stereo networks.
Taking the central assumption of convolutions — all neighborhoods can be operated in an identical manner — we propose to construct the cost volume on the grid of depth rather than disparity, enabling 3D convolutions and the loss function to perform exactly on the right scale for depth estimation.
We refer to our network as stereo depth network (SDN).
See Figure 1 for a comparison of 3D points obtained with SDN (purple) and disparity estimation (red).
Although our SDN improves the depth estimates signiﬁcantly, stereo images are still inherently 2D and it is unclear if they can ever match the accuracy and reliability of a true 3D LiDAR sensor.
Although LiDAR sensors with 32 or 64 beams are expensive, LiDAR sensors with only 4 beams are two orders of magnitude cheaper 2 and thus easily affordable.
The 4 laser beams are very sparse and ill-suited to capture 3D object shapes by themselves, but if paired with stereo images they become the ideal tool to de-bias our dense stereo depth estimates: a single high-precision laser beam may inform us how to correct the depth of an entire car or pedestrian in its path.
To this end, we present a novel depth-propagation algorithm, inspired by graph-based manifold learning (Weinberger et al., 2005;
Roweis & Saul, 2000; Xiaojin & Zoubin, 2002).
In a nutshell, we connect our estimated 3D stereo point cloud locally by a nearest neighbor graph, such that points corresponding to the same object will share many local paths with each other.
We match the few but exact LiDAR measurements ﬁrst with pixels (irrespective of depth) and then with their corresponding 3D points to obtain accurate depth estimates for several nodes in the graph.
Finally, we propagate this exact depth information along the graph using a label diffusion mechanism — resulting in a dense and accurate depth map at negligible cost .
In Figure 1 we see that the few (yellow)
LiDAR measurements are sufﬁcient to position almost all ﬁnal (blue) points of the entire car within the green ground truth box.
We conduct extensive empirical studies of our approaches on the KITTI object detection benchmark (Geiger et al., 2012; 2013) and achieve remarkable results.
With solely stereo images, we outperform the previous state of the art (Wang et al., 2019) by 10% .
Further adding a cheap 4-beam
LiDAR brings another 27% relative improvement — on some metrics, our approach is nearly on par with those based on a 64-beam LiDAR but can potentially save 95% in cost.
B ACKGROUND
3D object detection.
Most work on 3D object detection operates on 3D point clouds from LiDAR as input (Li, 2017;
Li et al., 2016; Meyer et al., 2019b;
Yang et al., 2018a; Du et al., 2018; Shi et al., 2019; Engelcke et al., 2017; Yan et al., 2018; Lang et al., 2019).
Frustum PointNet (Qi et al., 2018) applies PointNet (Qi et al., 2017a;b) to the points directly, while Voxelnet (Zhou & Tuzel, 2018) quantizes them into 3D grids.
For street scenes, several work ﬁnds that processing points from the bird’s-eye view can already capture object contours and locations (Chen et al., 2017; Yang et al., 2018b; Ku et al., 2018).
Images have also been used, but mainly to supplement LiDAR (Meyer et al., 2019a;
Xu et al., 2018; Liang et al., 2018; Chen et al., 2017; Ku et al., 2018).
Early work based solely on images — mostly built on the 2D frontal-view detection pipeline (Ren et al., 2015; He et al., 2017;
Lin et al., 2017) — fell far behind in localizing objects in 3D (Li et al., 2019a;
Xiang et al., 2015; 2017; Chabot et al., 2017; Mousavian et al., 2017; Chen et al., 2015; Xu & Chen, 2018; Chen et al., 2016; Pham & Jeon, 2017; Chen et al., 2018).
Pseudo-LiDAR.
This gap has been reduced signiﬁcantly recently with the introduction of the pseudoLiDAR framework proposed in (Wang et al., 2019).
This framework applies a drastically different approach from previous image-based 3D object detectors.
Instead of directly detecting the 3D bounding boxes from the frontal view of a scene, pseudo-LiDAR begins with image-based depth estimation, predicting the depth Z ( u, v ) of each image pixel ( u, v ) .
The resulting depth map Z is then back-projected into a 3D point cloud: a pixel ( u, v ) will be transformed to ( x, y, z ) in 3D by
[Equation]
where ( c U , c V ) is the camera center and f U and f
V are the horizontal and vertical focal length.
The 3D point cloud is then treated exactly as LiDAR signal — any LiDAR-based 3D detector can be applied seamlessly.
By taking the state-of-the-art algorithms from both ends (Chang & Chen, 2018;
Ku et al., 2018; Qi et al., 2018), pseudo-LiDAR obtains the highest image-based performance on the
KITTI object detection benchmark (Geiger et al., 2012; 2013).
Our work builds upon this framework.
Stereo disparity estimation.
Pseudo-LiDAR relies heavily on the quality of depth estimation.
Essentially, if the estimated pixel depths match those provided by LiDAR, pseudo-LiDAR with any
LiDAR-based detector should be able to achieve the same performance as that obtained by applying the same detector to the LiDAR signal.
According to (Wang et al., 2019), depth estimation from stereo pairs of images (Mayer et al., 2016; Yamaguchi et al., 2014; Chang & Chen, 2018) are more accurate than that from monocular (i.e., single) images (Fu et al., 2018; Godard et al., 2017) for 3D object detection.
We therefore focus on stereo depth estimation, which is routinely obtained from estimating disparity between images.
A disparity estimation algorithm takes a pair of left-right images I l
and I r as input, captured from a pair of cameras with a horizontal offset (i.e., baseline) b .
Without loss of generality, we assume that the algorithm treats the left image, I l , as reference and outputs a disparity map D recording the horizontal disparity to I r for each pixel ( u, v ) .
Ideally, I l ( u, v ) and I r ( u, v + D ( u, v )) will picture the same 3D location.
We can therefore derive the depth map Z via the following transform,
[Equation]
A common pipeline of disparity estimation is to ﬁrst construct a 4D disparity cost volume C disp , in which C disp ( u, v, d, :) is a feature vector that captures the pixel difference between I l ( u, v ) and
I r ( u, v + d ) .
It then estimates the disparity D ( u, v ) for each pixel ( u, v ) according to the cost volume
C disp .
One basic algorithm is to build a 3D cost volume with C disp ( u, v, d )
= (cid:107) I
l ( u, v ) − I r ( u, v + d ) (cid:107)and determine D ( u, v ) as arg min d C disp ( u, v, d ) .
Advanced algorithms exploit more robust features in constructing C disp and perform structured prediction for D .
In what follows, we give an introduction of PSMNet (Chang & Chen, 2018), a state-of-the-art algorithm used in (Wang et al., 2019).
PSMNet begins with extracting deep feature maps h l and h r from I l
and I r , respectively.
It then constructs C disp ( u, v, d, :) by concatenating features of h l ( u, v ) and h r ( u, v + d ) , followed by layers of 3D convolutions.
The resulting 3D tensor S disp , with the feature channel size ending up being one, is then used to derive the pixel disparity via the following weighted combination,
[Equation]
where softmax is performed along the 3 rd dimension of S disp .
PSMNet can be learned end-to-end, including the image feature extractor and 3D convolution kernels, to minimize the disparity error (cid:88)
[Equation]
where (cid:96) is the smooth L1 loss, D (cid:63) is the ground truth map, and A contains pixels with ground truths.  S TEREO D EPTH N ETWORK (SDN)
[Figure]
A stereo network designed and learned to minimize the disparity error (cf.
Equation 4) may over-emphasize nearby objects with smaller depths and therefore perform poorly in estimating depths for faraway objects.
To see this, note that Equation 2 implies that for a given error in disparity δD , the error in depth δZ increases quadratically with depth:
[Equation]
The middle term is obtained by differentiating Z ( D ) w.r.t.
D .
In particular, using the settings on the KITTI dataset (Geiger et al., 2012; 2013), a single pixel error in disparity implies only a 0.1m error in depth at a depth of 5 meters, but a 5.8m error at a depth of 50 meters.
See Figure 2 for a mapping from disparity to depth.
Depth Loss.
We propose two changes to adapt stereo networks for direct depth estimation.
First, we learn stereo networks to directly optimize the depth loss
[Equation]
Z and Z (cid:63) can be obtained from D and D (cid:63) using Equation 2.
The change from the disparity loss to the depth loss corrects the disproportionally strong emphasis on tiny depth errors of nearby objects — a necessary but still insufﬁcient change to overcome the problems of disparity estimation.
Depth Cost Volume.
To facilitate accurate depth learning (rather than disparity)
we need to further address the internals of the depth estimation pipeline.
A crucial source of error is the 3D convolutions within the 4D disparity cost volume, where the same kernels are applied for the entire cost volume.
This is highly problematic as it implicitly assumes that the effect of a convolution is homogeneous throughout — which is clearly violated by the reciprocal depth to disparity relation (Figure 2).
For example, it may be completely appropriate to locally smooth two neighboring pixels with disparity 85 and 86 (changing the depth by a few cm to smooth out a surface), whereas applying the same kernel for two pixels with disparity 5 and 6 could easily move the 3D points by 10m or more.
Taking this insight and the central assumption of convolutions — all neighborhoods can be operated upon in an identical manner — into account, we propose to instead construct the depth cost volume [Figure]
[Table] [Figure] C depth , in which C depth ( u, v, z, :) will encode features describing how likely the depth Z ( u, v ) of pixel ( u, v ) is z .
The subsequent 3D convolutions will then operate on the grid of depth, rather than disparity, affecting neighboring depths identically, independent of their location.
The resulting 3D tensor S depth is then used to predict the pixel depth similar to Equation
[Equation]
We construct the new depth volume, C depth , based on the intuition that C depth ( u, v, z, :) and
C disp
[Equation]
should lead to equivalent “cost”.
To this end, we apply a bilinear interpolation to construct C depth from C disp using the depth-to-disparity transform in Equation 2.
Speciﬁcally, we consider disparity in the range of [0 , 191] following PSMNet (Chang & Chen, 2018), and consider depth in the range of [1 m , 80 m ] and set the grid of depth in C depth to be 1m. Figure 5 (top) depicts our stereo depth network (SDN) pipeline.
Crucially, all convolution operations are operated on
C depth exclusively.
Figure 4 compares the median values of absolute depth estimation errors using the disparity cost volume (i.e., PSMNet) and the depth cost volume (SDN) (see subsection D.5 for detailed numbers).
As expected, for faraway depth, SDN leads to drastically smaller errors with only marginal increases in the very near range (which disparity based methods over-optimize).
See the appendix for the detailed setup and more discussions.
D EPTH C ORRECTION
Our SDN signiﬁcantly improves depth estimation and more precisely renders the object contours (see Figure 3).
However, there is a fundamental limitation in stereo because of the discrete nature of pixels: the disparity, being the difference in the horizontal coordinate between corresponding pixels, has to be quantized at the level of individual pixels while the depth is continuous .
Although the quantization error can be alleviated with higher resolution images, the computational depth prediction cost scales cubically with resolution— pushing the limits of GPUs in autonomous vehicles.
We therefore explore a hybrid approach by leveraging a cheap LiDAR with extremely sparse (e.g., 4 beams) but accurate depth measurements to correct this bias.
We note that such sensors are too sparse to capture object shapes and cannot be used alone for detection.
However, by projecting the
LiDAR points into the image plane we obtain exact depths on a small portion of “landmark” pixels.
We present a graph-based depth correction (GDC) algorithm that effectively combines the dense stereo depth that has rendered object shapes and the sparse accurate LiDAR measurements.
Conceptually, we expect the corrected depth map to have the following properties: globally, landmark pixels associated with LiDAR points should possess the exact depths; locally, object shapes captured by neighboring 3D points, back-projected from the input depth map (cf.
Equation 1), should be preserved.
Figure 5 (bottom) illustrates the algorithm.
Input Matching.
We take as input the two point clouds from LiDAR (L) and Pseudo-LiDAR (PL) by stereo depth estimation.
The latter is obtained by converting pixels ( u, v ) with depth z to 3D points ( x u , y v , z ) .
First, we characterize the local shapes by the directed K-nearest-neighbor (KNN) graph in the PL point cloud (using accelerated KD-Trees (Shevtsov et al., 2007)) that connects each 3D point to its KNNs with appropriate weights.
Similarly, we can project the 3D LiDAR points onto pixel locations ( u, v ) and match them to corresponding 3D stereo points.
Without loss of generality, we assume that we are given “ground truth” LiDAR depth for the ﬁrst n points and no ground truth for the remaining m points.
We refer to the 3D stereo depth estimates as Z ∈ R n + m and the LiDAR depth ground-truth as G ∈ R n .
Edge weights.
To construct the KNN graph in 3D we ignore the LiDAR information on the ﬁrst n points and only use their predicted stereo depth in Z .
Let N i denote the set of k neighbors of the i th point.
Further, let W ∈ R ( n + m ) × ( n + m ) denote the weight matrix, where W ij denotes the edge-weight between points i and j .
Inspired by prior work in manifold learning (Roweis & Saul, 2000; Weinberger et al., 2005)
we choose the weights to be the coefﬁcients that reconstruct the depth of any point from the depths of its neighbors in N i .
We can solve for these weights with the following constrained quadratic optimization problem:
[Equation]
Here 1 ∈ R n + m denotes the all-ones vector.
As long as we pick k > 3 and the points are in general position there are inﬁnitely many solutions that satisfy Z = WZ , and we pick the solution with the minimum L 2 norm (obtained with slight L 2 regularization).
Depth Correction.
Let us denote the corrected depth values as Z (cid:48)
∈ R n + m , with Z (cid:48) =
[ Z (cid:48)
L ; Z (cid:48) PL ] and Z (cid:48)
L ∈ R n and Z (cid:48) PL ∈ R m , where Z (cid:48)
L are the depth values of points with LiDAR ground-truth and Z (cid:48) PL otherwise.
For the n points with LiDAR measurements we update the depth to the (ground truth) values Z (cid:48)
L = G .
We then solve for Z (cid:48)
PL given G and the weighted KNN graph encoded in W .
Concretely, we update the remaining depths Z (cid:48) PL such that the depth of any point i can still be be reconstructed with high ﬁdelity as a weighted sum of its KNNs’ depths using the learned weights
W ; i.e. if point i : 1 ≤ i ≤ n is moved to its new depth G i , then its neighbors in N i must also be corrected such that G i ≈ (cid:80) j ∈N i W ij Z (cid:48) j .
Further, the neighbors’ neighbors must be corrected and the depth of the few n points propagates across the entire graph.
We can solve for the ﬁnal Z (cid:48) directly with another quadratic optimization:
[Equation]
To illustrate the correction process, imagine the simplest case where the depth of only a single point ( n = 1 ) is updated to G
=
Z 1 + δ .
A new optimal depth for Equation 8 is to move all the remaining points similarly, i.e. Z (cid:48)
=
Z + 1 δ : as Z = WZ and W 1 = 1 we must have W ( Z + 1 δ ) =
Z + 1 δ .
In the setting with n > 1 , the least-squares loss ensures a soft diffusion between the different LiDAR depth estimates.
Both optimization problems in Equation 7 and Equation 8 can be solved exactly and efﬁciently with sparse matrix solvers.
We summarize the procedure as an algorithm in the appendix.
From the view of graph-based manifold learning, our GDC algorithm is reminiscent of locally linear embeddings (Roweis & Saul, 2000) with landmarks to guide the ﬁnal solution (Weinberger et al., 2005).
Figure 1 illustrates vividly how the initial 3D point cloud from SDN (purple) of a car in the KITTI dataset is corrected with a few sparse LiDAR measurements (yellow).
The resulting points (blue) are right inside the ground-truth box and clearly show the contour of the car.
Figure 4 shows the additional improvement from the GDC (blue) over the pure SDN depth estimates (see subsection D. for detailed numbers).
The error (calculated only on non-landmark pixels) is corrected over the entire image where many regions have no LiDAR measurements.
This is because that the pseudo-LiDAR point cloud is sufﬁciently dense and we choose k to be large enough (in practice, we use k = 10 ) such that the KNN graph is typically connected (or consists of few large connected components).
See subsection D.6 for more analysis.
For objects such as cars the improvements through GDC are far more pronounced, as these typically are touched by the four LiDAR beams and can be corrected effectively. [Table]
E XPERIMENTS 5. S ETUP
We refer to our combined method (SDN and GDC) for 3D object detection as PSEUDO -L
I DAR++ (PL++ in short).
To analyze the contribution of each component [Table] we evaluate SDN and GDC independently and jointly across several settings.
For GDC we set k = 10 and consider adding signal from a (simulated) 4-beam LiDAR, unless stated otherwise.
, Metrics, and Baselines.
We evaluate on the KITTI dataset (Geiger et al., 2013; 2012), which contains 7,481 and 7,518 images for training and testing.
We follow (Chen et al., 2015) to separate the 7,481 images into 3,712 for training and 3,769 validation.
For each (left) image, KITTI provides the corresponding right image, the 64-beam Velodyne LiDAR point cloud, the camera calibration matrices, and the bounding boxes.
We focus on 3D object detection and bird’s-eye-view (BEV) localization and report results on the validation set .
Speciﬁcally, we focus on the “car” category, following Chen et al.
(2017) and Xu et al.
We report average precision (AP) with
IoU (Intersection over Union) thresholds at 0.5 and 0.7.
We denote AP for the 3D and BEV tasks by AP 3D and AP BEV .
KITTI deﬁnes the easy, moderate, and hard settings, in which objects with 2D box heights smaller than or occlusion/truncation levels larger than certain thresholds are disregarded.
We compare to four stereo-based detectors: PSEUDO -L I DAR (PL in short) (Wang et al., 2019), 3DOP (Chen et al., 2015), S-RCNN (Li et al., 2019b), and MLF- STEREO (Xu & Chen, 2018).
Stereo depth network (SDN).
We use PSMN ET (Chang & Chen, 2018) as the backbone for our stereo depth estimation network (SDN).
We follow Wang et al.
(2019) to pre-train SDN on the synthetic Scene Flow dataset (Mayer et al., 2016) and ﬁne-tune it on the 3,712 training images of
KITTI.
We obtain the depth ground truth by projecting the corresponding LiDAR points onto images.
We also train a PSMN ET in the same way for comparison, which minimizes disparity error.
3D object detection.
We apply three algorithms: AVOD (Ku et al., 2018), PIXOR (Yang et al., 2018b), and P-RCNN (Shi et al., 2019).
All utilize information from LiDAR and/or monocular images.
We use the released implementations of AVOD ( speciﬁcally, AVOD-FPN) and P-RCNN.
We implement PIXOR ourselves with a slight modiﬁcation to include visual information (denoted as PIXOR (cid:63) ).
We train all models on the 3,712 training data from scratch by replacing the LiDAR points with pseudo-LiDAR data generated from stereo depth estimation.
See the appendix for details.
Sparser LiDAR.
We simulate sparser LiDAR signal with fewer beams by ﬁrst projecting the 64-beam
LiDAR points onto a 2D plane of horizontal and vertical angles.
We quantize the vertical angles into 64 levels with an interval of 0 .
4 ◦ , which is close to the SPEC of the 64-beam LiDAR.
We keep points fallen into a subset of beams to mimic the sparser signal.
See the appendix for details.
[Table]
[Table]
[Table]
[Table] [Table] [Table] 5. E XPERIMENTAL RESULTS
Results on the KITTI val set.
We summarize the main results on KITTI object detection in Table 1.
Several important trends can be observed: 1) Our PL++ with enhanced depth estimations by SDN and GDC yields consistent improvement over PL across all settings; 2) PL++ with GDC reﬁnement of 4-beam LiDAR (Input: L# + S) performs signiﬁcantly better than PL++ with only stereo inputs (Input: S); 3) PL experiences a substantial drop in accuracy from IoU at 0.5 to 0.7 for the hard setting.
This suggests that while PL detects faraway objects, it mislocalizes them, likely placing them at the wrong depth.
This causes the object to be considered a missed detection at higher overlap thresholds.
Interestingly, here is where we experience the largest gain — from PL: P-RCNN (AP BEV = 52 . 7 ) to PL++: P-RCNN (AP BEV = 73 . 4 ) with input as L# + S. Note that the majority of the gain comes from GDC, as PL++ with the stereo-only version only improving the score to 57 .
3 AP BEV .
The gap between PL++ and LiDAR is at most 13% AP BEV , even at the hard setting under IoU at 0.7. 5)
For
IoU at 0.5, with the aid of only 4 LiDAR beams, PL++ (SDN + GDC) achieves results comparable to models with 64-beam LiDAR signals.
Results on the KITTI test set.
Table 2 summarizes results on the car category on the KITTI test set.
We see a similar gap between our methods and LiDAR as on the validation set, suggesting that our improvement is not particular to the validation data.
Our approach without LiDAR reﬁnement (pure
SDN) is placed at the top position among all the image-based algorithms on the KITTI leaderboard.
In the following, we conduct a series of experiments to analyze the performance gain by our approaches and discuss several key observations.
We mainly experiment with P-RCNN :
we ﬁnd that the results with AVOD and PIXOR (cid:63) follow similar trends and thus include them in the appendix.
Depth loss and depth cost volume.
To turn a disparity network (e.g., PSMN ET ) into SDN, there are two changes: 1) change the disparity loss into the depth loss; 2) change the disparity cost volume into the depth cost volume.
In Table 3, we uncover the effect of these two changes separately.
On the
AP BEV /AP
3D (moderate) metric, the depth loss gives us a 6% / 2% improvement and the depth cost volume brings another 2 ∼ 3% gain 3 .
Impact of sparse LiDAR beams.
We leverage 4-beam LiDAR to correct stereo depth using GDC.
However, it is possible that gains in 3D object detection come entirely from the new LiDAR sensor and that the stereo estimates are immaterial.
In Table 4, we study this question by comparing the detection results against those of models using 1) sole 4-beam LiDAR point clouds and 2) pseudo-LiDAR point clouds with depths of landmark pixels replaced by 4-beam LiDAR: i.e., in depth correction, we only correct depths of the landmark pixels without propagation. It can be seen that 4-beam LiDAR that the depth loss provides more gain than the depth cost volume (for P-RCNN).
In Table 6, we, however, see that the depth cost volume provides comparable or even bigger gain than the depth loss (for PIXOR (cid:63) and AVOD).
Nevertheless, Table 3 and Table 6 both suggest the compatibility of the two approaches: combining them leads to the best performance.
itself performs fairly well on locating faraway objects but cannot capture nearby objects precisely, while simply replacing pseudo-LiDAR with LiDAR at the landmark pixels prevents the model from detecting faraway object accurately.
In contrast, our proposed GDC method effectively combines the merits of the two signals, achieving superior performance than using them alone.
Pedestrian and cyclist detection.
For a fair comparison to (Wang et al., 2019), we apply FP OINT N ET (Qi et al., 2018) for detecting pedestrians and cyclists.
Table 5 shows the results: our methods signiﬁcantly boosts the performance. [Equation]
Qualitative visualization.
In Figure 6, we show an qualitative comparison of detection results on a randomly chosen scene in the KITTI object validation set, using P-RCNN (with conﬁdence > 0 . 95 ) with different input signals.
Speciﬁcally, we show the results from the frontal-view images and the bird’s-eye view (BEV) point clouds.
In the BEV map, the observer is on the left-hand side looking to the right.
It can be seen that the point clouds generated by PSEUDO -L I DAR ++ (SDN alone or SDN +GDC) align better with LiDAR than that generated by PSEUDO -L I DAR (PSMN ET ). For nearby objects (i.e., bounding boxes close to the left in the BEV map), we see that P-RCNN with any point cloud performs fairly well in localization. However, for Additional results, analyses, qualitative visualization and discussions.
We provide results of PSEUDO -L I DAR ++ with fewer LiDAR beams, comparisons to depth completion methods, analysis on depth quality and detection accuracy, run time, failure cases, and more qualitative results in the appendix.
With simple optimizations, GDC runs in 90 ms/frame using a single GPU ( 7 . 7 ms for KD-tree construction and search).
C ONCLUSION
In this paper we made two contributions to improve the 3D object detection in autonomous vehicles without expensive LiDAR.
First, we identify the disparity estimation as a main source of error for stereo-based systems and propose a novel approach to learn depth directly end-to-end instead of through disparity estimates.
Second, we advocate that one should not use expensive LiDAR sensors to learn the local structure and depth of objects.
Instead one can use commodity stereo cameras for the former and a cheap sparse LiDAR to correct the systematic bias in the resulting depth estimates.
We provide a novel graph propagation algorithm that integrates the two data modalities and propagates the sparse yet accurate depth estimates using two sparse matrix solvers.
The resulting system, PSEUDO -L I DAR ++ (SDN + GDC), performs almost on par with 64-beam LiDAR systems for $75,000 but only requires 4 beams and two commodity cameras, which could be obtained with a total cost of less than $1,000. A CKNOWLEDGMENTS
This research is supported by grants from the National Science Foundation NSF (III-1618134, III1526012, IIS-1149882, IIS-1724282, and TRIPODS-1740822), the Ofﬁce of Naval Research DOD (N00014-17-1-2175), the Bill and Melinda Gates Foundation, and the Cornell Center for Materials
Research with funding from the NSF MRSEC program (DMR-1719875).
We are thankful for generous support by Zillow and SAP America Inc. We thank Gao Huang for helpful discussion. Appendix
We provide details omitted in the main text.
• Appendix A: details on constructing the depth cost volume (section 3 of the main paper).
• Appendix B: detailed implementation of the GDC algorithm (section 4 of the main paper).
• Appendix C: additional details of experimental setups (subsection 5.1 of the main paper).
• Appendix D: additional results, analyses, and discussions (subsection 5.2 of the main paper).
A D EPTH C OST V OLUME
With Equation 2, we know where each grid ( u, v, z ) in C depth corresponds to in C disp (may not be on a grid).
We can then obtain features for each grid in C depth (i.e., C depth ( u, v, z, :) ) by bilinear interpolation over features on grids of C disp around the non-grid location (i.e., (cid:18) u, v, f U × b z (cid:19) ).
We applied the “grid_sample” function in PyTorch for bilinear interpolation.
We use PSMN ET (Chang & Chen, 2018) as the backbone for our stereo depth estimation network (SDN).
The only change is to construct the depth cost volume before performing 3D convolutions.
B G RAPH - BASED D EPTH C ORRECTION (GDC) A LGORITHM
Here we present the GDC algorithm in detail (see algorithm 1).
The two steps described in the main paper can be easily turned into two (sparse) linear systems and then solved by using Lagrange multipliers.
For the ﬁrst step (i.e., Equation 7), we solve the same problem as in the main text but we switch the objective to minimizing the L 2 -norm of W and set Z − WZ = 0 as a constraint 4 .
For the second step (i.e., Equation 8), we use the Conjugate Gradient (CG) to iteratively solve the sparse linear system.
Input
R ( n + m ) × 1 , the corresponding pseudo-LiDAR (PL) point cloud
P ∈ R ( n + m ) × 3 , and LiDAR depths G ∈ R n × 1 on the ﬁrst the n pixels.
Output: Corrected depth map Z (cid:48) ∈ R ( n + m ) ×function GDC ( Z, P, G, K )
Solve: W = arg min W ∈ R ( n + m ) × ( n + m ) (cid:107) W (cid:107)s.t.
Z − W · Z = 0 ,
W ij =
if j / ∈ N i (i.e., the set of neighbors of the i th point) according to P , (cid:80) j W ij = 1 for ∀ i = 1 , . .
. , n + m .
Solve: Z (cid:48)
PL = arg min
Z (cid:48) PL ∈ R m × 1 (cid:107) [ G ; Z (cid:48) PL ] − W [ G ; Z (cid:48) PL ] (cid:107)return [ G ; Z (cid:48)
PL ] end
C E XPERIMENTAL S ETUP C. S PARSE L I DAR GENERATION
In this section, we explain how we generate sparser LiDAR with fewer beams from a 64-beam LiDAR point cloud from KITTI dataset in detail.
For every point ( x i , y i , z i ) ∈ R 3 of the point cloud in one scene is the location )), we compute the elevation angle θ i to the LiDAR sensor as
[Equation]
We order the points by their elevation angles and slice them into separate lines by step 0 .
4 ◦ , starting from − 23 .
6 ◦ (close to the Velodyne 64-beam LiDAR SPEC).
We select LiDAR points whose elevation angles fall within [ − 2 .
4 ◦ , − 2 . 0 ◦ ) ∪ [ − 0 .
8 ◦ , − 0 .
4 ◦ ) to be the 2-beam LiDAR signal, and similarly [ − 2 . 4 ◦ , − 2 . 0 ◦ ) ∪ [ − 1 .
6 ◦ , − 1 . 2 ◦ ) ∪ [ − 0 .
8 ◦ , − 0 .
4 ◦ )
∪ [0 . 0 ◦ , 0 . 4 ◦ ) to be the 4-beam LiDAR signal.
We choose them in such a way that consecutive lines has a 0 .
8 ◦ interval, following the SPEC of the “cheap” 4-beam LiDAR ScaLa.
We visualize these sparsiﬁed LiDAR point clouds from the bird’s-eye view on one example scene in Figure 7.
[Figure]
C. 3D OBJECT DETECTION ALGORITHMS
In this section, we provide more details about the way we train 3D object detection models on pseudo-LiDAR point clouds.
For AVOD, we use the same model as in (Wang et al., 2019).
For
P-RCNN, we use the implementation provided by the authors.
Since the P-RCNN model exploits the sparse nature of LiDAR point clouds, when training it with pseudo-LiDAR input, we will ﬁrst sparsify the point clouds into 64 beams using the method described in subsection C.1.
For PIXOR (cid:63) , we implement the same base model structure and data augmentation speciﬁed by Yang et al.
(2018b), but without the “decode ﬁne-tune” step and focal loss.
Inspired by the trick in (Liang et al., 2018), we add another image feature (ResNet-18 by He et al.
(2016)) branch along the LiDAR branch, and concatenate the corresponding image features onto the LiDAR branch at each stage.
We train
PIXOR (cid:63) using RMSProp with momentum 0 . 9 , learning rate 10 − 5 (decay by 10 after 50 andepochs) for 90 epochs.
The BEV evaluation results are similar to the reported results (see Table 1).
D A DDITIONAL R ESULTS , A NALYSES , AND D ISCUSSIONS D. A BLATION STUDY
In Table 6 and Table 7 we provide more experimental results aligned with experiments in subsection 5.2 of the main paper.
We conduct the same experiments on two other models, AVOD and
PIXOR (cid:63) , and observe similar trends of improvements brought by learning with the depth loss (from
PSMN ET to PSMN ET +DL), constructing the depth cost volume (from PSMN ET +DL to SDN), and applying GDC to correct the bias in stereo depth estimation (comparing SDN +GDC with SDN).
We note that, in Table 7, results of AVOD (or PIXOR (cid:63) ) with SDN + L# are worse than those with L# at the moderate and hard settings.
This observation is different from that in Table 4, where P-RCNN with SDN + L# outperforms P-RCNN with L# in 5 out of 6 comparisons.
We hypothesize that this is because P-RCNN takes sparsiﬁed inputs (see subsection C.2) while AVOD and PIXOR (cid:63) take dense inputs.
In the later case, the four replaced LiDAR beams in SDN + L# will be dominated by the dense stereo depths so that SDN + L# is worse than L#.
D. U SING FEWER L I DAR BEAMS
In PL++ (i.e., SDN + GDC), we use 4-beam LiDAR to correct the predicted point cloud.
In Table 8, we investigate using fewer (and also potentially cheaper) LiDAR beams for depth correction.
We observe that even with 2 beams, GDC can already manage to combine the two signals and yield a better performance than using 2-beam LiDAR or pseudo-LiDAR alone
[Table]
[Table] [Table] [Table]
[Table]
D. D EPTH CORRECTION VS . DEPTH COMPLETION
We compare our GDC algorithm for depth correction to depth completion algorithms, which aim to “densify” LiDAR data beyond the beam lines (Wang et al., 2018a;
Tomasello et al., 2018; Ma et al., 2019; Yang et al., 2019; Cheng et al., 2018; Torres-Mendez & Dudek, 2004) 5 .
We note that most depth completion approaches take as input a 64-beam LiDAR and a single image, while our focus is on fusing a much sparser 4-beam LiDAR and stereo depths.
As such, the two problems are not [Figure]
m e d i a n e rr o r i n a s c e n e [Figure] [Table] commensurate.
Also, our GDC algorithm is a general, simple, inference-time approach that requires no training , unlike prior learning-based approaches to depth completion.
Here we empirically compare to P N P (Wang et al., 2018a), a recently proposed depth completion algorithm compatible with any (even stereo) depth estimation network, similar to GDC.
We use
SDN for initial depth estimation, and evaluate GDC and P N P by randomly selecting a fraction of
LiDAR points as provided ground truths and calculating the median absolute depth errors on the remaining LiDAR points.
As shown in Figure 8, GDC outperforms P N P by a large margin.
Tableshows a further comparison to P N P on 3D object detection.
We apply P N P and GDC respectively to correct the depth estimates obtained from SDN, train a P-RCNN or PIXOR (cid:63) using the resulting pseudo-LiDAR points on the KITTI training set, and compare the detection results on the KITTI validation set.
In either case, SDN + GDC outperforms SDN + P N P by a notable margin.
D. R UN TIME
With the following optimizations for implementation,
Sub-sampling pseudo-LiDAR points:
keeping at most one point within a cubic of size 0 . 1 m2.
Limiting the pseudo-LiDAR points for depth correction: keeping only those whose elevation angles are within [ − 3 . 0 ◦ , 0 . 4 ◦ ) (the range of 4-beam LiDAR plus 0 .
6 ◦ ; see subsection C. for details) 3.
After performing GDC for depth correction, combining the corrected pseudo-LiDAR points with those outsides the elevation angles of [ − 3 . 0 ◦ , 0 . 4 ◦ )
GDC runs in 90 ms/frame using a single GPU ( 7 . 7 ms for KD-tree construction and search, 46 .
5 ms for solving W , and 26 .
9 ms for solving Z (cid:48) PL ) with negligible performance difference (see Table 10).
For consistency, all results reported in the main paper are based on the naive implementation.
Further speedups can be achieved by CUDA programming for GPUs.
D. S TEREO DEPTH VS . DETECTION
We quantitatively evaluate the stereo depths by median errors in Figure 4 of the main text (numerical values are listed in Table 11).
In Table 12 we further show mean errors with standard deviation (the large standard deviation likely results from outliers such as occluded pixels around object boundaries).
[Table]
[Table]
For both tables
[Table] we divide pixels into beams according to their truth depths [Table] and evaluate on pixels not on the 4-beam LiDAR.
The improvement of SDN (+ GDC) over PSMN ET becomes larger as we consider pixels farther away.
Table 13 further demonstrates the relationship between depth quality and detection accuracy: SDN (+ GDC) signiﬁcantly outperforms PSMN ET for detecting faraway cars.
We note that, for very faraway cars (i.e., 50-70 m), the number of training object instances are extremely small, which suggests that the very poor performance might partially cause by over-ﬁtting.
Further, we apply the same evaluation procedure but group the errors by the shortest distance between each PSEUDO -L I DAR point and the 4-beam LiDAR points in Figure 9. We can see that the closer the PSEUDO -L I DAR points are to the 4-beam LiDAR points, the bigger improvement GDC can bring. D. C ONNECTED C OMPONENTS IN KNN GRAPHS OF PSEUDO -L I DAR POINTS BY SDN
Here, we provide empirical analysis on the relationship between the k we choose in building the K-nearest-neighbor graph of PSEUDO -L I DAR points and the number of connected components of that graph.
We show the results on KITTI validation set in Figure 11. It can be seen that with k ≥ 9 , the average number of connected components in the graph is smaller than 2 .
D. F AILURE CASES AND WEAKNESS
There is still a gap between our approach and LiDAR for faraway objects (see Table 13).
We further analyze AP BEV at different IoU in Figure 10.
For low IoU (0.2-0.5), SDN (+GDC) is on par with LiDAR, but the gap increases signiﬁcantly at high IoU thresholds.
This suggests that the predominant gap between our approach and LiDAR is because of mislocalization, perhaps due to residual inaccuracies in depth. [Table]
D. Q UALITATIVE RESULTS
In Figure 6,12,13 and Figure 14, we show detection results using P-RCNN (with conﬁdence > 0 . 95 ) with different input signals on four randomly chosen scenes in the KITTI object validation set.
Speciﬁcally, we show the results from the frontal-view images and the bird’s-eye view (BEV) point clouds.
In the BEV map, the observer is on the left-hand side looking to the right.
It can be seen that the point clouds generated by PSEUDO -L I DAR ++ (SDN alone or SDN +GDC) align better with
LiDAR than those generated by PSEUDO -L I DAR (PSMN ET ).
For nearby objects (i.e., bounding boxes close to the left in the BEV map), we see that P-RCNN with any point cloud performs fairly well in localization.
However, for faraway objects (i.e., bounding boxes close to the right), PSEUDO -L I DAR with depth estimated from PSMN ET predicts objects (red boxes) deviated from the ground truths (green boxes).
Moreover, the noisy PSMN ET points also leads to several false positives or negatives.
In contrast, the detected boxes by our PSEUDO -L I DAR ++, either with SDN alone or with SDN +GDC, align pretty well with the ground truth boxes, justifying our targeted improvement in estimating faraway depths.
In Figure 12, we see one failure case for both PSEUDO -L I DAR and PSEUDO -L I DAR ++: the most faraway car is missed, while LiDAR signal can still detect it, suggesting that for very faraway objects stereo-based methods may still have limitation.
[Figure]
[Equation] [Figure] [Equation]
[Figure] [Figure]
[Equation] [Figure] [Equation]