Probabilistic and Geometric Depth: Detecting Objects in Perspective
Abstract: 3D object detection is an important capability needed in various prac- tical applications such as driver assistance systems.
Monocular 3D detection, as a representative general setting among image-based approaches, provides a more economical solution than conventional settings relying on LiDARs but still yields unsatisfactory results.
This paper ﬁrst presents a systematic study on this prob- lem.
We observe that the current monocular 3D detection can be simpliﬁed as an instance depth estimation problem: The inaccurate instance depth blocks all the other 3D attribute predictions from improving the overall detection performance.
Moreover, recent methods directly estimate the depth based on isolated instances or pixels while ignoring the geometric relations across different objects.
To this end, we construct geometric relation graphs across predicted objects and use the graph to facilitate depth estimation.
As the preliminary depth estimation of each instance is usually inaccurate in this ill-posed setting, we incorporate a probabilis- tic representation to capture the uncertainty.
It provides an important indicator to identify conﬁdent predictions and further guide the depth propagation.
Despite the simplicity of the basic idea, our method, PGD, obtains signiﬁcant improvements on KITTI and nuScenes benchmarks, achieving 1st place out of all monocular vision-only methods while still maintaining real-time efﬁciency.
Code and models will be released at https://github.com/open-mmlab/mmdetection3d .
Introduction
3D object detection is an essential task for many robotic systems such as autonomous vehicles.
Recent advanced methods in this ﬁeld typically resort to various sensors, such as LiDAR [1, 2, 3, 4, 5, 6], Radar [7], binocular vision [8, 9], or their combinations for accurate depth information.
Nevertheless, these perceptual systems are complicated, expensive, and difﬁcult to maintain in complex environments.
In contrast, monocular 3D detection, a setting that aims at perceiveing 3D objects from 2D monocular images, has drawn increasing attention due to its low costs.
However, as the depth information is not directly manifest in the input, this task is inherently ill-posed, making the problem particularly challenging.
This paper starts from a systematic study about this problem on two authoritative benchmarks in a quantitative way.
Although we already knew the depth information is critical to this task, the study surprisingly shows that inaccurate depth estimation blocks all the other localization predictions from improving the ﬁnal results.
As instance depth has shown to be the bottleneck, we can simplify monocular 3D detection as an instance depth estimation problem to tackle it essentially.
Previous methods [10, 11] ﬁrst use an extra cumbersome depth estimation model to complement 2D detectors on depth information.
The following methods [12, 13, 14] simplify the frameworks by directly regarding depth as one dimension of the 3D localization task.
However, they still use simple methods that estimate depth from isolated instances or pixels in a regression manner.
We observe that aside from each object itself, other objects are co-existing in an image and the geometric relations across them can be valuable constraints to guarantee accurate estimation.
Motivated by these observations, we propose Probabilistic and Geometric Depth (PGD) that jointly leverages probabilistic depth uncertainty and geometric relationships across co-existed objects for accurate depth estimation.
Speciﬁcally, as the preliminary depth estimation of each instance is usually inaccurate in this ill-posed setting, we incorporate a probabilistic representation to capture the uncertainty of the estimated depth.
We ﬁrst bucket the depth values into a set of intervals and calculate the depth by the expectation of the distribution (Fig. 1(a)).
The average of top-k conﬁdence scores from the distribution is taken as the uncertainty of the depth.
To model the geometric relations, we further construct a depth propagation graph to enhance the estimations with their contextual relationship.
The uncertainty of each instance depth provides useful guidance for the propagation therein.
Beneﬁting from this overall scheme, we can easily identify the predictions with higher conﬁdence, and more importantly, estimate their depths more accurately with the graph-based synergistic mechanism.
We implement the methods on a simple monocular 3D object detector FCOS3D
Despite the simplicity of the basic idea, our PGD results in signiﬁcant improvements on KITTI [16] and nuScenes [17] with different benchmark settings and evaluation metrics.
It achieves 1st place out of all monocular vision-only methods while still maintaining real-time efﬁciency.
The simple yet effective method proves that with only designs tailored to depth, a 2D detector can be capable of detecting objects in perspective.
Related Work
2D Object Detection
According to the base of initial guesses, modern 2D detection methods can be divided into two branches, anchor-based and anchor-free.
Anchor-based methods [18, 19, 20, 21] beneﬁt from the predeﬁned anchors in terms of much easier regression, while anchor-free methods [22, 23, 24, 25, 26] do not need complicated prior settings and thus have better universality.
For simplicity, we take FCOS3D [15], the 3D adapted version of FCOS [24], as the baseline considering its capability of handling overlapped ground truths and scale variance problem.
Monocular Depth Estimation Monocular depth estimation is also a challenging ill-posed problem like monocular 3D detection.
It aims at predicting dense and global depth ﬁeld at pixel level given an RGB image.
Early works [27, 28] predict depth from hand-crafted features with non-parametric optimization methods.
With the rapid progress of CNNs, fully supervised methods [29, 30, 31], selfsupervised methods based on stereo pairs [32, 33] and monocular videos [34, 35] gradually emerged.
Although this problem has been explored for a long time, there are very few works [36] studying it in a speciﬁc task, like detection, where the dense depth supervision is always not guaranteed and we only care about the accuracy of instance depth instead of the global depth ﬁeld.
As for the reformulation of depth learning problems, there are a few attempts in this ﬁeld.
For example, DORN [37] recasts the depth learning problem as ordinal regression and proposes a spacingincreasing discretization (SID) strategy to improve network training and reduce computations.
It is similar to the underlying idea of our probabilistic representation for uncertainty modeling while different in terms of motivation and design details.
Monocular 3D Object Detection Monocular 3D detection is more complicated than the 2D case.
The underlying problem is the inconsistency of input 2D data modal and the output 3D predictions.
Methods involving sub-networks Earlier work uses sub-networks to assist 3D detection.
3DOP [38] and MLFusion [10] use a depth estimation network while Deep3DBox [39] uses a 2D object detector.
They rely on the design and performance of these sub-networks, even external data and pre-trained models, which makes the training inconvenient and introduces additional system complexity.
Transform to 3D representation Another category is to convert the RGB input to 3D representations like OFTNet [40] and Pseudo-Lidar [11].
Although these methods have shown promising performance, they actually rely on dense depth labels and hence are not regarded as pure monocular approaches.
There are also domain gaps between different depth sensors, making them hard to generalize smoothly to a new practical setting.
Furthermore, the efﬁciency of processing a large number of point clouds is also a signiﬁcant issue to deal with in practical applications.
End-to-end designs like 2D detection Recent work notices these drawbacks, and end-to-end frameworks are thus proposed.
M3D-RPN [13] implements a single-stage multi-class detector with an end-to-end region proposal network and depth-aware convolution.
SS3D [41] proposes to detect 2D key points and further predicts object characteristics with uncertainties.
MonoDIS [12] introduces a disentangling loss to reduce the instability of the training procedure.
Some of them still have multiple training stages or post-optimization phases.
In addition, they all follow anchor-based manners, and thus the consistency of 2D and 3D anchors is needed to be determined.
In contrast, anchor-free methods [26, 14, 42, 15] do not need to make statistics on the given data and have better generalized ability to more various classes or different intrinsic settings, so we choose to follow this paradigm.
Nevertheless, all of these works rarely have customized designs for instance depth estimation in particular, and only take it as one common regression target for isolated points or instances.
It actually hinders the breakthrough of this problem, which will be discussed in our quantitative study and speciﬁcally addressed in our approach.
Preliminary and Motivating Study
In this section, we aim at making an in-depth quantitative error analysis on top of a basic adapted monocular 3D detector to investigate the key challenge in the speciﬁc 3D detection setting.
Typically, conventional 2D detection expects the model to predict 2D bounding boxes and category labels for each object of interest, while a monocular 3D detector needs to predict 7-DoF 3D boxes given the same input.
From this perspective of problem formulation, the main difference lies on the regression targets.
An intuitive reason for the much worse performance of monocular 3D detection compared to 2D is that there exist much more difﬁcult targets to regress in the localization .
Hence, we choose a simple detector FCOS3D
[15] to study the speciﬁc problem, which keeps the welldeveloped designs for 2D feature extraction and is adapted for this 3D task with only basic designs for speciﬁc 3D detection targets.
As shown in the left part of Fig.
3, there are overall two branches for classiﬁcation and localization respectively.
Formally, for the regression branch, the detector predicts 3D attributes, including offsets ∆ x, ∆ y to the projected 3D center, depths d , 3D size w 3 D , l 3 D , h 3 D , sin value of rotation θ , direction class C θ , center-ness c , and distances to four sides of 2D boxes l , r , t , b , for each location on the output dense map.
We further equip it with a basic consistency loss between 3D and 2D localization, which will be detailed in supplemental materials.
On this basis, we apply this baseline on two representative benchmarks, KITTI and nuScenes, and replace the predictions with ground truths step by step to identify the performance bottleneck (Fig. 2).
Unexpectedly, the inaccurate depth blocks all the other sub-task predictions from improving the overall detection performance, on both datasets under different metrics.
Hence, current monocular 3D detection, especially 3D localization, can be reduced to the dominating instance depth estimation problem to a great extent, which will be the focus of our method to be presented next.
See more details about the explanation of oracle analyses in supplemental materials.
Our Approach
Given images collected from similar cameras, previous work typically resorts to direct regression for instance depth estimation and expects the model to directly learn that objects with certain appearances and sizes always exist at locations with certain depths.
Our baseline also follows this way.
However, it is hard to learn due to the large variance and also obviously not enough for the accuracy needed in 3D detection.
Given the inherent downside of hard regression for isolated points, in our approach, we aim at constructing an uncertainty-aware depth propagation graph to enhance the estimation from contextual connections among instances.
Next, we will ﬁrst elaborate on the adopted probabilistic representation and technical details of the constructed geometric graph, and ﬁnally present how we integrate these obtained depth estimations.
4.1 Uncertainty Modeling with Probabilistic Representation
For a one-stage detector, a general design for direct depth estimation is a small head along the regression branch expected to output a dense depth map.
Formally, as shown in Fig. 3, suppose the input feature map has shape H × W , then the direct depth regression output 2 can be denoted as
D R ∈ R H × W .
On this basis, to establish an effective depth propagation mechanism, modeling the uncertainty of depth estimation for each instance is an important preliminary, which can provide useful guidance for weighing the propagation among instances.
We adopt a simple yet effective probabilistic representation to achieve this: Considering the depth value is continuous in a certain range, we uniformly quantize the depth interval into a set of discrete values and represent the prediction with the expectation of the distribution.
Suppose the detection range is 0 ∼ D max , the discretized unit is U , then we have C = (cid:98) D max /U (cid:99) + 1 split points.
Denote the set of points as a weight vector ω ∈ R C , and then we introduce a new head parallel with direct regression to produce a probabilistic output map D PM , which will be decoded with:
[Equation]
where D P is the so-called probabilistic depth.
It is equivalent to compute the expectation of the probabilistic distribution formed by softmax ( D PM ) .
Apart from the D P , we can further obtain the depth conﬁdence score, denoted as s d ∈ S D , from the depth distribution of each instance.
In practice, we take the average of top-2 conﬁdence as the depth score for U = 10 m .
It will be multiplied by the center-ness and classiﬁcation score as the ﬁnal ranking criterion for predictions during inference.
Subsequently, we fuse D R and D P with the sigmoid response of a data-agnostic single parameter λ :
[Equation]
Here D L is regarded as a local depth estimation for each isolated instance, which together with the depth score derived from D PM serve as the foundation of constructing the depth propagation graph.
It is worth noting that our implementation is different from the typical way used in monocular depth estimation [37], which usually adopts a ﬁne-grained quantization for the depth interval and further estimates the value with classiﬁcation and residual regression.
In comparison, our method is more memory-efﬁcient, more straightforward for regressing continuous value, and provides a natural indicator for uncertainty estimation.
Please refer to the supplemental for empirical results about comparison with other depth interval division methods.
reliable depth predictions from other objects (car 1, 3, 4) with perspective geometry can enhance the reasoning from the global context.
4.2 Depth Propagation from Perspective Geometry
With the depth prediction D L for isolated instances and their depth conﬁdence scores S D for uncertainty estimation, we can further construct the propagation graph based on the contextual geometric relationship.
Consider the typical driving scenarios: a general constraint can be leveraged, i.e. , almost all the objects are on the ground.
Early in [43], Hoiem et al.
utilized the scene projection with this constraint to put objects in the context of the overall 3D scene by modeling the relationship of different elements.
Here, targeting the depth estimation problem, we instead propose a geometric depth propagation mechanism with consideration of interdependence between instances.
Next, we will ﬁrst derive the perspective relationship between two instances, and then present the details of the graph-based depth propagation scheme with edge pruning and gating.
Perspective Relationship Consider in the general perspective projection, suppose the camera projection matrix P is:
[Equation]
where f is the focal length, c u and c v are the vertical and horizon position of camera in the image, b x , b y and b z denote the baseline with respect to the reference camera (non-zero in KITTI while zero in nuScenes).
Note that we represent the focal length with a single f considering most cameras share the same one for the u and v axis.
Then a 3D point x 3D =
( x, y, z, 1) T in the camera coordinates can be projected to a point x 2D = ( u (cid:48) , v (cid:48) , 1) T in the image with:
[Equation]
To simplify the result, we replace v (cid:48) with v + c v , then v represents the distance to the horizon line (Down is the positive direction in the Fig. 4).
Then we get:
[Equation]
The relation for u is similar.
Considering the constraint that all the objects are on the ground, the bottom centers of objects always share the same y (height in the camera coordinates), so we mainly consider this relation for v next.
Given two objects 1 and 2, the relationship between their center depths can be derived from Eqn. 5:
[Equation]
with which we can predict d 2 given d 1 precisely with the height difference between 3D centers.
Besides, we can also leverage an approximation of this relationship given the assumption that objects share the same bottom height, then y 2 − y 1 can be substituted by the difference of half heights of 3D boxes 12 ( h 3 D 1 − h 3 D 2 ) , deﬁned as d P 1 → 2 .
In this relation, when h 3 D= h 3 D 2 , v 1 d 1 = v 2 d 2 , which is easy to understand, i.e. , an object closer to vanishing line is farther away.
It is a clear relationship connecting different instances but also yields errors.
Suppose | ( y 2 − y 1 ) − 12 ( h 1 − h 2 ) | =
δ , the error of depth will be ∆ d = fv 2 δ .
When δ = 0 .
1 m , v 2 = 50 (pixels), ∆ d can be about 1 . 5 m .
Although it is acceptable for objects 30 m away (corresponding with v 2 = 50 ), we also need a mechanism to avoid possible large errors.
It consists of the edge pruning and gating scheme to be described next and the location-aware weight map to be mentioned in the Sec. 4.3.
Graph-Based Depth Propagation With the pairwise perspective relationship, we can estimate the depth of any object from the cues of other objects.
Then we can construct a dense directed graph with two bidirectional edges between any two objects representing the depth propagation
(Fig. 4).
Formally, suppose we have N predicted objects with indices from P = { 1 , 2 , ..., n } , we can estimate the depth of object i given d Pj → i for all the j ∈ P , deﬁned as the geometric depth d Gi ∈ D G .
Considering the computational efﬁciency and possible large errors mentioned previously, we propose an edge pruning and gating scheme to improve the propagation graph.
From our observation, the same category of nearby objects can well satisfy the ”same ground” condition, so we select the following 3 most important factors to decide which edges are inﬂuential and reliable, including the depth conﬁdence s dj , 2D distance score s 2 Dij , and classiﬁcation similarity s clsij .
The latter two and the overall edge score s ej → i are computed as follows:
[Equation]
where t 2 Dij is the 2D distance between projected centers of object i and j , t 2 Dmax is set to the length of image diagonal, f i and f j are the output conﬁdence vectors of two objects from classiﬁcation branch and k is the maximum number of edges to be kept after pruning (edges with top-k scores are kept).
The edge score is then used for gating so that each node attends its edges with their importance:
[Equation]
Note that obtaining the geometric depth map D G from this graph is free of learnable parameters.
To avoid inﬂuencing the learning of other components, we cut off the gradients backpropagated from this computation and only focus on how to integrate D L and D G , which will be discussed next.
4.3 Probabilistic and Geometric Depth Estimation
So far, we have obtained two depth predictions D L and D G from isolated and graph-based contextual estimations, respectively.
Then we integrate these two complementary components in a learning manner.
Unlike the data-agnostic single parameter used in the local estimation, integrating these two results should be more complex considering their ﬂexible roles in various complicated cases.
So we further introduce a branch to produce a location-aware weight map α ∈ R H × W to fuse them (Fig. 3):
[Equation]
The fused depth D will replace the direct regressed D R in the baseline and trained with the common smooth L1 loss in the same end-to-end way.
Note that adding intermediate supervisions empirically makes the training more stable but does not bring any performance gains.
Experiments
In this section, we present our experimental setup and implementation details, and then make the quantitative analysis on the KITTI and nuScenes dataset with details of both performance and efﬁciency.
Finally, detailed ablation studies are conducted to show the efﬁcacy of each component in our method.
Refer to supplemental materials for more qualitative analysis.
5.1 Datasets & Evaluation Metrics
We evaluate our method on two datasets, KITTI [16] and nuScenes [17].
There are 7481/ samples for training/testing respectively on KITTI, and the training samples are generally divided into 3712/3769 samples as training/validation splits.
We ﬁrst validate our method on this popular benchmark.
Nevertheless, the variety of scenes and categories is limited on KITTI, so we further test our approach on the large-scale nuScenes dataset.
NuScenes consists of multi-modal data collected from 1000 scenes, including RGB images from 6 cameras, points from 5 Radars, and 1 LiDAR.
It is split into 700/150/150 scenes for training/validation/testing.
There are overall 1.4M annotated 3D bounding boxes from 10 categories.
In addition, nuScenes uses different metrics, distance-based mAP and NDS, which can help evaluate our method from another perspective.
See more explanations about metrics in supplemental materials.
5.2 Implementation Details
Network Architectures As shown in the Fig.
3, our baseline framework basically follows the design of FCOS3D [15].
Given the input image, we utilize ResNet101 [44] as the feature extraction backbone followed by FPN [45] for generating multi-level predictions.
Detection heads are shared among multi-level feature maps except that three scale factors are used to differentiate some of their
[Table] [Table]
ﬁnal regressed results, including offsets, depths, and sizes, respectively.
For the hyperparameters in the depth estimation module, U is set to 10 m and k is set to 5.
The overall framework is built on top of MMDetection3D [46].
Please refer to FCOS3D [15] and supplemental materials for the design of loss and other implementation details.
Training Parameters For all the experiments, we trained randomly initialized networks from scratch following end-to-end manners.
Models are trained with SGD optimizer, in which gradient clip and warm-up policy are exploited with learning rate 0.001, number of warm-up iterations 500, warm-up ratio 0.33 and batch size 32/12 on 16/4 GTX 1080Ti GPUs for nuScenes/KITTI.
Data Augmentation We only implement image ﬂip for augmentation, where offset and 2D targets are ﬂipped for the 2D image while 3D boxes are transformed correspondingly in 3D space.
No other augmentation (right image augmentation, cropping, resizing, etc. )
methods are utilized.
5.3 Quantitative Analysis
We make quantitative analyses both on KITTI (Tab. 1 and 4, Fig. 1(c)) and much harder, less commonly validated nuScenes dataset (Tab. 2).
It can be seen that our method achieves the state-ofthe-art on both benchmarks with different settings and metrics while maintains outstanding speed.
We list part of early monocular methods with extra data or pre-trained models and recent image-only methods that have related results for comparison on the KITTI dataset.
Only the results for car detection are compared here because the performance of small objects is always unstable due to their limited samples.
Our framework based on the simple adapted FCOS3D achieves much better performance than others, especially considering M3D-RPN [13] and RTM3D [14] adopt stronger backbone and data augmentation.
Furthermore, our method can run at the speed of 36Hz to achieve this, thanks to most of our modules not introducing extra computational costs to inference.
It is an excellent trade-off between performance and efﬁciency.
Then for the nuScenes dataset, we also compare the results on the test set and validation set, respectively.
On the test set, we ﬁrst compared all the methods using RGB images as the input data.
Our single model achieved the best performance among them with mAP 37.0% and NDS 43.2%, in which we particularly exceeded the previous best method more than 3% in terms of mAP.
We also list benchmarks based on other data modality, including lightweight, real-time PointPillars [2] with
LiDAR, CenterFusion [7] with RGB image and Radar, and CenterPoint [52] ensemble results with all the sensors.
It can be seen that although our method has a certain gap with the high-performance
CenterPoint, it even surpasses PointPillars and CenterFusion on mAP, which shows that this ill-posed
[Table] [Table]
[Table] [Table]
problem can be solved decently with enough data. At the same time, the methods using other modal data usually yield better NDS, mainly because the mAVE is smaller.
The reason is that they can predict the speed of objects from continuous multi-frame point clouds or velocity measurement of
Radar.
In contrast, we only use the single-frame image in our experiments.
So how to mine the speed information from consecutive frame images will be a direction worthy of exploring in the future.
On the validation set, we compare our method with the best open-source center-based detector,
CenterNet.
Our method is not only much more efﬁcient to train and inference (3 days to train the
CenterNet vs. only one day to train our model with comparable performance), but also achieves better performance, especially in terms of the mAP and mAOE.
On this basis, we ﬁnally achieved an improvement of about 9% on NDS.
See more detailed results about depth estimation accuracy and per-class detection performance in supplementary materials.
[Table] [Table]
5.4 Ablation Studies
Finally, we conduct ablation studies to validate the efﬁcacy of our proposed key components on
KITTI (Tab. 3) and nuScenes (Tab. 5).
We can observe that local constraints can basically enhance the baseline, and our probabilistic and geometric depth further boost the performance signiﬁcantly, especially in terms of mAP and translation error (mATE).
Tab. 7 and Tab. 8 show more details of two core components for improving depth estimation with the metrics average precision under IOU ≥ 0.7.
It can be seen that combining the probabilistic representation (prop. branch in the Tab. 7) with direct regression (w/ direct) and leveraging the depth score in the inference (depth score) can ﬁnally make the most of this design.
For geometric depth, the basic fusion with local estimation can not bring the desirable gain.
Improving the propagation graph via edge pruning and gating (edge gating) and cutting off the unexpected gradients propagation (cut off grad.) can help remove possible noises and prompt the learning more focused on the ﬁnal integration, thus making the overall scheme much more effective.
As for alternative implementations, we compare feasible methods of computing the depth score from the probabilistic distribution (Tab. 6).
Compared to other more complicated ways, normalized entropy and standard deviation, our exploited top-2 score can achieve decent results with better efﬁciency.
See more results about different depth division methods and detailed analyses for these two datasets from other perspectives like the Precision-Recall curve in the supplemental.
Conclusion
This paper targets the key challenge lying in monocular 3D object detection, instance depth estimation.
Started from a basic adapted 3D detector, we ﬁrstly make in-depth oracle analyses.
We surprisingly ﬁnd that depth estimation is the dominating bottleneck for current 3D detection, especially in terms of localization.
To tackle the discovered challenge, we propose a novel approach, Probabilistic and
Geometric Depth (PGD), which leverages the geometric relationship in perspective to construct a graph connecting instance estimations with uncertainty and thus predicts depths more accurately.
The efﬁcacy of this solution is demonstrated on both KITTI and large-scale nuScenes datasets.
In the future, we will further extend the geometric depth scheme to more general cases by relaxing the ”ground” assumption via 2D height regression or ground normal estimation, and validate this pipeline on other 2D detectors.
How to better leverage temporal geometry information to address the difﬁculty of instance depth estimation is also a promising direction worthy of further exploration. Acknowledgments
This work is supported in part by Centre for Perceptual and Interactive Intelligence Limited, in part by the GRF through the Research Grants Council of Hong Kong under Grants (Nos. 14208417, 14207319 and 14203518) and ITS/431/18FX, in part by CUHK Strategic Fund and CUHK Agreement
TS1712093, in part by the Shanghai Committee of Science and Technology, China (Grant No.
20DZ1100800).