W EAK M3D: T OWARDS W EAKLY S UPERVISED M ONOCULAR 3D O BJECT D ETECTION
A BSTRACT
Monocular 3D object detection is one of the most challenging tasks in 3D scene understanding.
Due to the ill-posed nature of monocular imagery, existing monocular 3D detection methods highly rely on training with the manually annotated 3D box labels on the LiDAR point clouds.
This annotation process is very laborious and expensive.
To dispense with the reliance on 3D box labels, in this paper we explore the weakly supervised monocular 3D detection.
Speciﬁcally, we ﬁrst detect 2D boxes on the image.
Then, we adopt the generated 2D boxes to select corresponding RoI LiDAR points as the weak supervision.
Eventually, we adopt a network to predict 3D boxes which can tightly align with associated RoI LiDAR points.
This network is learned by minimizing our newly-proposed 3D alignment loss between the 3D box estimates and the corresponding RoI LiDAR points.
We will illustrate the potential challenges of the above learning problem and resolve these challenges by introducing several effective designs into our method.
Codes will be available at https://github.com/SPengLiang/WeakM3D.
I NTRODUCTION
3D object detection is essential for many applications in the real world, such as robot navigation and autonomous driving.
This task aims to detect objects in 3D space, bounding them with oriented 3D boxes.
Thanks to a low deployment cost, monocular-based methods Chen et al.
; Mousavian et al.
; Roddick et al.
Liu et al.
; Manhardt et al.
; Ma et al.
(2020; 2021) are drawing increasing attention in both academia and industry.
In recent years, monocular 3D detection has achieved remarkable progress.
However, it also requires numerous 3D box labels (locations, dimensions, and orientations of objects) for training.
These labels are labeled on LiDAR point clouds, where the manually annotating process is quite time-consuming and expensive.
The high annotation cost encourages us to dispense with the reliance on the 3D box annotations.
To this end, in this paper we propose WeakM3D, a novel method towards weakly supervised monocular 3D object detection.
Considering the well-developed 2D detection technology Redmon &
Farhadi (2018); Ren et al. (2015);
He et al.
; Qi et al.
; Zhou et al.
(2019), we use an off-the-shelf 2D detector Qi et al.
(2018) to obtain 2D boxes, which are then lifted to 3D boxes by predicting required 3D box
parameters
To learn the 3D information desired in the lifting process, we
employ the LiDAR point cloud as the weak supervision
since
it provides rich and accurate 3D points within the scene
Speciﬁcally, given a raw LiDAR point cloud
, we
select LiDAR points if their projections are inside the 2D object on the image plane.
We term these points object-LiDARpoints
which describe a
part
outline of the object in
3D
space
Therefore
, as shown in Figure
(a
object-LiDAR-points can be used for aligning with 3D box predictions in loss functions, consequently endowing the network with the ability of being trained without any 3D box label.
However, how to formulate the loss between 3D box predictions and object-LiDAR-points is considerably challenging.
We pose and summarize the main challenges and our solutions.
• Challenge 1 (Section 3.2): as shown in Figure 1 (a), the alignment gap between 3D box estimates and object-LiDAR-points should be measured appropriately.
We address this challenge by introduc-
ing a geometric point-to-box alignment loss, to minimize spatial geometric distances from 3D box predictions to object-LiDAR-points.
This loss allows the network to learn the object’s 3D location.
• Challenge 2 (Section 3.3): as shown in Figure 1 (b), an alignment ambiguity problem is caused if only geometrically aligning 3D box predictions with object-LiDAR-points.
Regarding objectLiDAR-points that are captured from only one surface of an object, it is not clear which surface of the 3D box prediction should be used for aligning, thus causing an alignment ambiguity issue.
Inspired by the constraints of camera imaging and LiDAR scanning, we eliminate this ambiguity by proposing a ray tracing loss.
In particular, we track each object-LiDAR-point from the camera optical center and use the resulting ray to make collision detection with predicted 3D boxes to ﬁnd the surface correspondence.
In this way, the alignment ambiguity issue can be alleviated greatly.
• Challenge 3 (Section 3.4): as shown in Figure 1 (c), LiDAR point clouds distribute in 3D space unevenly.
Considering that both geometric alignment and ray tracing losses are calculated pointwisely, unevenly distributed points can cause unbalanced point-wise losses in training, which is harmful since losses produced by sparse yet important points are overwhelmed by losses of dense points.
We use the point density to balance point-wise losses to resolve this problem.
• Challenge 4 (Section 3.5): a 3D box is parameterized by many estimates (locations, dimensions, and orientations).
Such entangled 3D box estimates result in a heavy learning burden during training.
To resolve this issue in training, we disentangle the learning of each group of estimates by freezing the object dimension and heuristically obtaining the orientation from object-LiDAR-points.
Such challenges and our method are detailed in the following sections.
Extensive experiments validate the effectiveness of our method.
In summary, our contributions can be listed as follows:
Firstly , we explore a novel method (WeakM3D) towards weakly supervised monocular 3D detection, removing the reliance on 3D box labels.
Secondly , we pose the main challenges in WeakM3D and correspondingly introduce four effective strategies to resolve them, including geometric alignment loss, ray tracing loss, loss balancing, and learning disentanglement.
Thirdly , evaluated on the KITTI benchmark, our method builds a strong baseline for weakly supervised monocular 3D detection, which even outperforms some existing fully supervised methods which use massive 3D box labels.
R ELATED W ORK 2. L I DAR- BASED 3D O BJECT D ETECTION
The LiDAR device is able to provide point clouds with precise depth measurements for the scene.
Thus, LiDAR-based methods Shi et al.
(2019b)
; Lang et al.
; He et al.
; Shi et al.
(2020); Shi & Rajkumar (2020); Shi et al.
(2019a)
; Zheng et al.
(2021) attain high accuracy and can be employed in autonomous driving.
Early methods project point clouds into the bird’s-eyeview Chen et al.
(2017b)
or front-view Li et al.
(2016), ignoring the nature of point clouds, thus resulting in sub-optimal performances.
LiDAR-based 3D detectors can be roughly divided into two categories: voxel-based methods Zhou & Tuzel (2018); Yan et al.
; Kuang et al.
(2020) and point-based methods Shi et al.
(2019a)
; Qi et al.
; Yang et al.
The former partition the 3D space into voxel grids, transforming the irregular raw point cloud to regular voxels so that 3D convolutions can be employed to extract more discriminative features.
Point-based methods directly design the network tailored to the raw point cloud representation.
Both two types of methods have achieved great success, but are inherently limited by the main shortcomings of the LiDAR device, i.e., the high price and limited working ranges.
2. M ONOCULAR - BASED 3D O BJECT D ETECTION
In recent years, monocular 3D object detection has achieved signiﬁcant improvements
Qin et al.
(2019); Brazil & Liu (2019)
; Ma et al.
Prior works such as Mono3D Chen et al.
and Deep3DBox Mousavian et al.
(2017) mainly take advantage of geometry constraints and auxiliary information.
More recently, Monodle Ma et al.
(2021) resorts to reducing the localization error in monocular 3D detection by three tailored strategies.
Furthermore, with the development of depth estimation, some other monocular methods attempt to use the explicit depth information generated from an off-the-shelf depth estimator.
Pseudo-LiDAR Wang et al.
; Weng & Kitani (2019) converts the image-only representation, to mimic the real LiDAR signal to utilize the existing
LiDAR-based 3D detector.
PatchNet Ma et al.
(2020) rethinks the underlying mechanism of pseudo
LiDAR, pointing out the effectiveness comes from the 3D coordinate transform.
Although recent monocular 3D object detection methods obtain exciting results, they heavily rely on a large number of manually labeled 3D boxes.
2. W EAKLY S UPERVISED 3D O BJECT D ETECTION
To alleviate heavy annotation costs, some weakly supervised methods are proposed.
WS3D Meng et al.
(2020) introduces a weakly supervised approach for LiDAR-based 3D object detection, which still requires a small set of weakly annotated scenes and a few precisely labeled object instances.
They use a two-stage architecture, where stage one learns to generate cylindrical object proposals under horizontal centers click-annotated in bird’s-eye-view, and stage two learns to reﬁne the cylindrical proposals to get 3D boxes and conﬁdence scores.
This weakly supervised design does not fully get rid of the dependence on 3D box labels and only works well for LiDAR point clouds input.
Another weakly supervised 3D detection method Qin et al.
(2020) also takes point clouds as input.
It proposes a 3D proposal module and utilizes an off-the-shelf 2D classiﬁed network to identify 3D box proposals generated from point clouds.
Both two methods cannot be directly applied to ﬁt the single RGB image input.
Also, Zakharov et al.
Zakharov et al.
(2020) propose an autolabeling pipeline.
Speciﬁcally, they apply a novel differentiable shape renderer to signed distance ﬁelds (SDF), leveraged together with normalized object coordinate spaces (NOCS).
Their autolabeling pipeline consists of six steps, with a curriculum learning strategy.
In contrast to WeakM3D, their method is not end-to-end and rather complicated.
M ETHODS 3. P ROBLEM D EFINITION
Given an RGB image captured from a single calibrated camera and the corresponding projection matrix P , the task, i.e. , monocular 3D object detection, aims to classify and localize objects within the scene in 3D space, where each object is represented by its category cls , 2D bounding box b 2 d , and 3D bounding box b 3 d .
In particular, we utilize a separate 2D detector to achieve 2D bounding boxes of objects, eliminating the extra attention on this well-developed area.
This simple design enables us to focus on solving the 3D bounding box, which is the most important and challenging in monocular 3D detection.
Note that we do not use any 3D box label during training.
Speciﬁcally, the 3D bounding box b 3 d is parameterized by the location ( x 3 d , y 3 d , z 3 d ), the dimension ( h 3 d , w 3 d , l 3 d ), and the orientation ( θ y ), which are all described at the camera coordinate system in 3D space.
We show our network architecture in Figure 2, and the forward pass is as follows: deep features are ﬁrstly extracted from a single RGB image via an encoder.
At the same time, the image is fed into an off-the-shelf 2D detector Qi et al.
(2018) to generate 2D bounding boxes.
The 2D boxes are then used to obtain speciﬁed object features for every object from deep features.
Finally, we achieve the 3D box b 3 d by regressing the required 3D box parameters from object features.
Object-LiDAR-points.
Object-LiDAR-points contain only LiDAR points located on the object surface, indicating much 3D information.
Thus they are ideal weakly supervised signals for monocular 3D detection.
We achieve object-LiDAR-points from raw LiDAR point clouds and an off-the-shelf
2D detector.
First, the ground plane is estimated from the raw LiDAR point cloud via RANSAC Fischler & Bolles (1981), which is used to remove ground points in the raw point cloud.
The remaining
LiDAR points whose projections on the image within the 2D box are selected as the initial object point cloud, which contains some points of no interest such as background points.
Alternatively, we can also use 2D instance masks from a pre-trained network to select the initial object point cloud.
Then, we obtain object-LiDAR-points by ﬁltering the initial point cloud via an unsupervised density clustering algorithm Ester et al.
(1996), which is detailed in the appendix.
3. G EOMETRICALLY A LIGNED 3D B OX P REDICTIONS
Without loss of generality, 3D object boxes should contain objectLiDAR-points and align with them along box edges.
To facilitate the learning of the object’s 3D location, we impose the location constraint between 3D box predictions from the network and objectLiDAR-points.
A naive solution refers to minimizing the euclidean distance from the object center to each LiDAR point.
Nevertheless, this trivial loss misleads the network due to its inaccurate nature for localization.
As shown in Figure 3, this center distance loss L center would push the predicted object center to the point cloud as close as possible.
Unfortunately, object-LiDAR-points are typically captured from visible surfaces of objects, meaning that the predicted center using only L center tends to be close to the groundtruth box edge, but not the groundtruth box center.
Based on the above analysis, we propose a geometric alignment loss between the predicted 3D box and associated object-LiDAR-points, in which the major obstacle lies in appropriately measuring the distance from points to the 3D box.
Aiming this, we create a ray from the 3D box center P 3 d to each object-LiDAR-point P , where the ray intersects with the edge of the box prediction at P I .
Therefore, the geometric alignment loss at each point among object-LiDAR-points is as follows:
[Equation]
where Intersect ( Ray P 3 d → P , b 3 d ) refers to the intersection point P I .
In particular, we do not directly predict the absolute object’s 3D center P 3 d , but its projection [ t x , t y ] on the image plane and the object instance depth z .
Thus, P 3 d = [ ( t x − c x ) f x z, ( t y − c y ) f
y z, z ] T , where f x , f y , c x , c y are the parameters from the camera projection matrix P .
We illustrate the geometry alignment loss in Figure 4.
This loss forces 3D box predictions to align with object-LiDAR-points in terms of geometry.
3. R AY T RACING FOR A LIGNMENT A MBIGUITY E LIMINATING
Although the geometric alignment bounds 3D box predictions and object-LiDAR-points, The ambiguity happens when object-LiDAR-points fail to represent an adequate 3D outline of the object, e.g. , object-LiDAR-points are captured from only one surface of an object.
Speciﬁcally, the ambiguity refers to how to semantically decide the correspondence between each object-LiDAR-point and edges of the box prediction in the aligning process.
We call this problem the alignment ambiguity.
Figure 5 shows an example.
Both the BEV box 0 and 1 commendably align with object-LiDAR-points by the edge 4 and 2, respectively.
They conduct the same geometric alignment losses, while their 3D locations are quite different.
However, we cannot decide which box is a better prediction if only concerning the geometric alignment because existing geometric clues cannot indicate the semantic correspondence of objectLiDAR-points.
In other words, there is a dilemma in choosing the correspondence between box edges and object-LiDARpoints.
This ambiguity brings adverse impacts in training, especially for hard samples (their associated object-LiDARpoints usually carry poor information).
Generally, we resolve the alignment ambiguity by considering occlusion constraints.
Similar to the process of camera imaging, when scanning a scene, the signal of a LiDAR device will be reﬂected if meeting obstacles.
Considering the reﬂected LiDAR signal inside the camera FOV (ﬁeld of view), we propose to implement ray tracing from the camera optical center P cam to each object-LiDARpoint, minimizing the distance from each object-LiDAR-point to the intersection point on the object box.
As shown in Figure 4, the LiDAR point is P and the intersection point is P R (with Z buffering, namely, the closer intersection point is chosen), the ray tracing loss is as follows:
[Equation]
if Ray P cam
[Equation]
P intersects with b
[Equation] [Equation]
otherwise
[Equation]
) { P 1 , P 2 } =
Intersect ( Ray P cam → P , b 3 d ) , and P R = P 1 if P 1 is closer to the camera, or P 2 otherwise.
Note that the ray tracing loss is zero if the ray does not intersect with the predicted 3D box, meaning that this loss here does not contribute to the gradient descent in back-propagation.
In this way, we eliminate the alignment ambiguity, encouraging 3D box predictions to follow occlusion constraints in parallel to geometrically aligning with object-LiDAR-points.
Let us back to the example shown in Figure 5.
The ray tracing losses produced by box 1 are much larger than losses of box 0, consequently leading the network to conduct a reasonable and precise result.
3. P OINT - WISE L OSS B ALANCING
The varied spatial distribution of object-LiDAR-points is also an obstacle, i.e. , the point density is somewhere high while somewhere low.
Point-wise losses such as the geometric alignment loss and ray tracing loss all suffer from the unevenly distributed nature.
Speciﬁcally, the loss produced by the dense region can dominate the total loss, ignoring the loss conducted by other relatively sparse yet essential points.
To balance the inﬂuence, we normalize the density of object-LiDAR-points when calculating loss.
Let us calculate the number of LiDAR points
E i in the neighborhood at point P i as follows:
[Equation]
where M is the number of object-LiDAR-points.
From Equation 3, we can know that points within the spatial range R towards the point P i are counted for the density.
Thus we balance point-wise losses by weighting each point with the density, as follows:
[Equation]
We illustrate an example in Figure 4.
The balanced loss alleviates the adverse impact brought by the unevenly distributed LiDAR point clouds.
Here we employ the center loss for regularization and empirically set λ to 0.1, and set the coefﬁcients of geometric alignment and ray tracing loss to 1 by default.
3. L EARNING D ISENTANGLEMENT FOR 3D B OX E STIMATES
As mentioned in Simonelli et al.
(2019), complicated interactions of 3D box estimates take a heavy learning burden during training.
Prior fully supervised methods can easily deal with this issue by disentangling the learning of each group of 3D box estimates, i.e. , imposing losses between each group of predictions and corresponding labels, respectively.
However, it is not trivial for the weakly supervised manner due to the absence of 3D box labels.
To disentangle the learning, we extract object orientations from LiDAR point clouds.
In addition, we use a-priori knowledge about dimensions on the speciﬁed category and freeze them because objects that belong to the same class have close sizes, e.g. , cars.
Such information allows us to solve each group of estimates independently, largely reducing the learning burden.
Orientation Estimated from Object-LiDAR-points.
We propose a simple yet effective method to obtain the global object orientation from object-LiDAR-points.
Intuitively, object-LiDAR-points describe a part 3D outline of the object, implicitly indicating the object orientation θ y .
We obtain the object orientation from the directions of paired points.
Speciﬁcally, as shown in Figure 6, we calculate the direction of each pair of object-LiDAR-points, drawing a histogram with respect to the direction range from 0 to π .
The direction of paired points varies near the object orientation, and the direction α that is highest in histogram refers to θ y or θ y ± π/ 2 .
The object orientation is further decided by the spatial offset d x of object-LiDAR-points along x axis in 3D space.
A low offset indicates the orientation close to π/ 2 while a high offset denotes the orientation close to 0 .
We set the offset threshold to 3.0 meters by default.
Therefore, we heuristically obtain the global object orientation, which can be used as the supervised signal for orientation estimates and employed in the process of 3D box alignment.
More details can be found in the appendix.
3. N ETWORK T RAINING
We conduct point-wise losses on the bird’s-eye-view (BEV) to simplify the task.
We can train the network to learn the object’s location ( x 3 d , z 3 d ) by L balancing , which is the most challenging and crucial.
To further obtain the 3D location, we averaging coordinates among object-LiDAR-points along y axis, denoted by y L , building the loss: L loc y =
SmoothL 1 (ˆ y, y L ) , where ˆ y is the network prediction.
Thus, we obtain the overall loss formulation as:
[Equation]
We empirically set coefﬁcients for each loss item to 1 by default.
For the orientation, only the local orientation, i.e. , the observation angle, can be directly estimated from the RGB image.
Thus the network conducts local orientation estimates and then converts to global orientations using the predicted 3D center ( x 3 d , z 3 d ).
The orientation loss L orient follows Deep3DBox Mousavian et al.
We provide more details in the appendix.
E XPERIMENTS 4. I MPLEMENTATION D ETAILS
Our method is implemented by PyTorch Paszke et al.
and trained on a Titan V GPU.
We use the Adam optimizer Kingma & Ba (2014) with an initial learning rate of 10 − 4 .
We train our network for 50 epochs.
To obtain an initial object point cloud, we adopt Mask-RCNN He et al.
(2017) pretrained on COCO Lin et al.
Alternatively, we can use an off-the-shelf 2D detector Qi et al.
Both manners lead comparable and satisfactory performance.
For the frozen dimensions for cars, we empirically adopt 1 . 6 , 1 . 8 , 4 . 0 meters as the height, width, and length, respectively.
R for the point density in Equation 3 is set to 0.4-meter.
We adjust y coordinate in the location according to 2D-3D consistency Brazil & Liu (2019).
We provide more experimental results and discussion in the appendix due to the space limitation.
4. D ATASET AND M ETRICS
Like most prior fully supervised works do, we conduct experiments on KITTI Geiger et al.
(2012) dataset.
KITTI object dataset provides 7,481 images for training and 7,518 images for testing, where groundtruths for testing are kept secret and inaccessible.
Following the common practice Chen et al.
(2017a)
, the 7,481 samples are further divided into training and validation splits, containing 3, and 3,769 images, respectively.
To utilize raw LiDAR point clouds since our method does not require 3D box labels, we use raw sequences for training that does not overlap with the validation and test
[Table] [Table]
set.
We do not perform data augmentation during the training.
Also, prior fully supervised mainly evaluate their methods on the category of car
[Table]
and we follow this line.
For evaluation metrics, APis commonly applied in many prior works
[Table]
while AP 40 Simonelli et al.
(2019) is suggested to use recently.
Most fully supervised methods use IOU 0.7 and weakly supervised methods use IOU 0.
criterion
[Table]
thus we follow them for comparisons
[Table]
respectively.
4. C OMPARING WITH O THER A PPROACHES
Comparing with monocular-based fully supervised methods.
As shown in Table
[Table]
although our method does not use any 3D box annotation
[Table]
it outperforms some prior fully supervised methods.
We apply our method to CenterNet for comparisons.
[Table]
For instance,
WeakM3D outperforms
FQNet Liu et al.
(2019) by 8.45 AP BEV under the moderate setting.
Table 2 also shows that our method outperforms some prior SOTA fully supervised methods.
We can also observe that there still remains a considerable gap between WeakM3D and current SOTA fully supervised methods
Ma et al.
(2021), we have faith in greatly pushing the performance of weakly supervised methods in the future due to the rapid development in the community.
Furthermore, we apply our method to PatchNet Ma et al.
(2020) in Tableand CenterNet Zhou et al.
in Table 3 while keeping other settings the same.
Interestingly, our weakly supervised method achieves comparable performance compared to the fully supervised manner, demonstrating its effectiveness.
Comparing with other weakly supervised methods.
We compare our method with other weakly supervised methods.
We report the results in Table 4.
To make fair comparisons, we employ our baseline network for Autolabels Zakharov et al.
Note that the original VS3D Qin et al.
[Table] [Table]
(2020) takes advantage of the per-trained network Dorn Fu et al.
(2018) to estimate image-based depth maps.
As mentioned in Simonelli et al.
; Peng et al.
(2021), Dorn utilizes the leaked data on the KITTI 3D object dataset.
Therefore we use the ﬁxed split Peng et al.
and keep the number of LiDAR point clouds the same as our method for fairness.
We can observe that our method performs the best, validating its effectiveness
[Table]
4. A BLATION S TUDY
[Table]
To investigate the impact of each component in our method, we conduct extensive ablation studies.
As shown in Table 5, the accuracy of the baseline is very low, indicating that the weakly supervised manner is challenging.
By adding the learning disentanglement and geometry alignment between 3D box predictions and object-LiDAR-points, we obtain signiﬁcant progress.
When employing ray tracing, which considerably eliminates the alignment ambiguity, the performance is further improved.
Finally, the balancing at point-wise losses in favor of accuracy and stability makes the network perform the best.
These results validate the power of our method.
We also compare the weakly supervised manner using different requirements in Table 6.
We can see that both requirements lead to good performance.
L IMITATIONS AND F UTURE W ORK
There are also some limitations in our work.
The loss for 3D localization is a little complicated, and the accuracy can be further improved by a more elegant design.
As an emerging area, there is ample room for improving the accuracy of weakly supervised monocular 3D object detection.
Therefore, we resort to alleviating the label reliance further and boosting the performance in future work.
C ONCLUSIONS
In this paper, we explore a novel weakly supervised monocular 3D object detection method.
Specifically, we take advantage of 2D boxes/masks to segment LiDAR point clouds to obtain objectLiDAR-points, which are employed as the weakly supervised signal for monocular 3D object detection.
To learn the object’s 3D location, we impose alignment constraints between such points and 3D box predictions.
Aiming this, we point out the main challenges of the weakly supervised manner, introducing several novel strategies to deal with them.
Extensive experiments show that our method builds a strong baseline for weakly supervised monocular 3D detection, which even outperforms some prior fully supervised
methods
A CKNOWLEDGMENTS
This work was supported in part by The National Key Research and Development Program of China (Grant Nos: 2018AAA0101400), in part by The National Nature Science Foundation of China (Grant Nos: 62036009, U1909203, 61936006, 62133013), in part by Innovation Capability Support Program of Shaanxi (Program No. 2021TD-05)
Appendix
Due to the space limitation, we provide details omitted in the main text in this appendix, which is organized as follows:
• Section A : Network and the running time.
• Section B : Detailed ablation studies.
• Section C : Performance on Waymo.
• Section D : Generalization ability.
• Section E : Details of object-LiDAR-points.
• Section F : Qualitative results.
• Section G : Orientation analysis.
• Section H : Statistics on 3D object labels.
• Section
I : Discussion on viewpoint differences.
• Section J : Discussion on difﬁcult cases.
• Section K : Discussion on different baselines.
• Section L : Results on NuScenes.
A N ETWORK AND R UNNING T IME
For the feature encoder, we adopt different backbones including ResNet18, ResNet34, and ResNet
He et al.
We train our network with a batch size of 8 by default.
Specifically, we remove the output layer and the last convolution stage of ResNet, and obtain 7 × 7 RoI
Aligned
He et al.
(2017) object features from extracted deep features.
As shown in
Figure 7, we reshape each output tensor to a 1-dim tensor, passing fully connected layers to regress corresponding outputs.
We show the quantitative results on different backbones in Table 7.
Under the AP BEV /AP
3 D (IoU=0.3) metric, different backbones all show satisfactory and comparable accuracy, demonstrating the robustness of our method.
If using the stricter AP BEV /AP
3 D (IoU=0.5) metric, a better backbone is more beneﬁcial for the overall performance as it encodes richer features for object 3D localization.
Table 7 shows the running time for the forward pass at different backbones.
Thanks to our lightweight design, the time-cost is little.
The adopted 2D detector takes extra 60ms in inference, meaning that the overall running time of our method is less than 80ms, meaning that our method is efﬁcient.
[Table]
B D ETAILED A BLATION S TUDIES
We conduct extensive experiments to validate the effectiveness of each component in WeakM3D
[Table]
As shown in Table
[Table]
the baseline (Experiment a) using only the center loss shows an extremely low
[Table]
detection accuracy, indicating the difﬁculty for the weakly supervised manner.
When only employing one of the components, we can observe that the geometric alignment (Experiment c) brings the most improvements, as it enables the network to learn the most 3D information.
When employing two of the components, we can know that the combination of geometric alignment and ray tracing (Experiment i) performs the best.
Such two strategies mutually reinforce each other, allowing the network to effectively learn the object’s 3D location.
Also, after applying loss balancing (Experiment o), unevenly distributed point-wise losses can be balanced to facilitate the learning process.
Finally, if we use all proposed strategies (Experiment p), we can achieve the best performance.
Furthermore, instead of freezing the object dimension, we let the network learn the dimension and show the results in Table 9.
We can see that adding more entangled learning objectives brings a heavier learning burden, thus making the network perform slightly worse.
Our detailed ablation studies comprehensively show the power of our method and how such strategies interact.
[Table]
C P ERFORMANCE ON W AYMO
We also evaluate our method on Waymo Sun et al.
(2020) dataset, which is a considerably large dataset, and provide the results in Table 10.
Waymo dataset provides many annotations, thus the fully supervised manner method M3D-RPN Brazil & Liu (2019) is supposed to perform much better than our method.
However, we can observe that our method obtains comparable performance compared to M3D-RPN.
Especially for medium and far objects, our method outperforms it with a signiﬁcant margin.
It indicates that our weakly supervised manner is more suitable on massive data.
Also,
[Table]
prior point-cloud-based weakly supervised methods and many fully supervised monocular methods do not evaluate their results on Waymo, thus our method sets a good baseline for future works.
Notably, in this comparison
[Table]
the performance of WeakM3D is still limited by the number of training samples.
Such samples can be easily collected in real applications, as we do not require the extra manual annotating process.
In other words
[Table]
our method is inherently suitable for the autonomous driving scenario, which automatically produces more and more unlabeled data, therefore it has much potential to be explored.
D G ENERALIZATION A BILITY
Our method is robust and works well for most object categories.
We follow a basic assumption adopted by almost all 3D detection works, i.e. , all objects in 3d detection are represented as cuboids.
[Table]
Please consider the underlying mechanism of human annotating.
People annotate each object by ﬁnding a reasonable minimum 3d bounding box.
Correspondingly, our loss functions exactly aim to learn this reasonable minimum 3D bounding box for the object.
Our method therefore works well for most object types.
To demonstrate this, we conduct experiments on pedestrian and cyclist categories in Table 11, and we obtain promising results.
To the best of our knowledge, we are the ﬁrst weakly supervised method that works for the two classes.
E D ETAILS OF O BJECT -L I DAR- POINTS
As mentioned in the main text, we can obtain an initial object point cloud by using the raw LiDAR point cloud and 2D boxes/masks.
This initial point cloud contains some noisy or background points, which should be removed.
As shown in Figure 8, we cluster the initial object point cloud using an unsupervised clustering algorithm Ester et al.
(1996), in which points are divided into different clusters in terms of the density.
Therefore, we select the cluster with most points as object-LiDARpoints and remove other points.
In particular, we further remove the points if their coordinates y 3 d are lower than the median of object-LiDAR-points.
These inner points from the bird’s-eye-view perspective do not contribute to the object’s 3D location.
Then, we randomly sample 100 object-
LiDAR-points.
Such points precisely describe a part outline of an object in 3D space, and they are ideal weak supervisions for corresponding 3D box prediction.
F Q UALITATIVE R ESULTS
We provide some qualitative results in KITTI validation set in Figure 9.
We can see that our method works well for most cases, while it can fail for the hard cases, e.g , truncated, heavily occluded, and far samples.
This inspires us to further improve the performance in future work.
G O RIENTATION A NALYSIS
appearances
In monocular 3D detection, the orientation consists of two types of orientations, i.e. , the global orientation θ y and the local orientation δ y , where the local orientation is also named the observation angle.
We show the relationship between θ y and δ y in Figure 10.
For monocular 3D detection, the global orientation θ y is hard to reason directly.
Even holding the same global orientation θ y , the appearance of an object shown on the image can change as the viewpoint varies.
We give an example in Figure 10.
For the location 1, 2, and 3, the mainly captured visible surfaces by the camera are different, consequently varied appearances on light of instead of directly regressing
, we estimate the local orientation δ y and then convert it to θ y by taking the camera viewpoint into account.
More details can be found in Mousavian et al.
[Figure]
We connect each pair of points and then calculate the direction of the resulting line.
Also, we provide details for obtaining the global orientation from
Object-LiDAR-points.
Object-LiDAR-points denote a part outline of the object, which is highly related to the global orientation θ y .
Speciﬁcally, we calculate the direction of each pair of object-LiDAR-points and then draw the corresponding histogram.
We show the paired points and the histogram in Figure 11 and Figure 12, respectively.
The direction α y that is highest in histogram refers to θ
y or the perpendicular direction of θ y .
To further determine the orientation, we use the spatial offset d x of object-LiDAR-points along x axis in 3D space.
Since a car in motion usually faces forward, we ﬁrst convert α y to the range ( π 4 , 3 π 4 ) by adding or subtracting π 2 , thus θ y is derived as follows:
[Equation]
where C is the offset threshold.
C is set to 3.0 meters by default.
From Figure 12 and Equation 6, we can know that a low offset indicates the orientation close to π/ 2 while a high offset denotes the orientation close to 0 or π .
H S TATISTICS ON 3D O BJECT L ABELS
We conduct statistics on 3D object labels for the car in KITTI Geiger et al.
and show the results in Figure 13.
Please note, we do not use the statistics on 3D box labels as the dimension priors.
Speciﬁcally, we draw the histogram for the object dimension and 3D location.
As mentioned in the main text, objects that belong to the same class usually have close sizes.
We can observe that the height of cars mainly varies from 1.4 to 1.6 meters.
Similarly, the width of cars mainly varies from 1.5 to 1.7 meters, and the width of cars mainly varies from 3.3 to 4.5 meters.
In contrast to the low-range varied dimensions, the object’s 3D locations vary a lot, especially for { x 3 d , z 3 d } , e.g. , the object depth ( z 3 d ) varies from 0 to 80 meters.
Considering the IoU (Intersection over Union) criterion and the ill-posed nature of monocular imagery, improving the localization accuracy is much more important than obtaining more precise object dimensions.
The satisfactory performance of our method also indicates that the manner of taking a ﬁxed object dimension is acceptable for weakly supervised monocular 3D detection.
Intuitively, in future work we can attempt to obtain more accurate object dimensions to further improve the accuracy.
[Figure]
Theoretical failure cases caused by viewpoint differences between the LiDAR and camera.
A well known yet important fact is that the LiDAR and camera are installed in the same car/robot, which indicates that their viewpoint differences are marginal compared to the extensive 3D space.
We summarize the theoretical failure cases caused by the viewpoint differences.
For case 1, it cannot happen since the camera and LiDAR always face towards objects.
The occlusion constraints for surface 1 and 3 are the same for the camera and LiDAR.
For case 2, the object width W object is lower than W cam − >lidar (the offset between camera and LiDAR along X axis)
meanwhile the object center is very close to the ego-car/robot center along X axis.
Such objects are few.
Also, the adverse impact brought by this type of mismatching can be ignored since W object is very small.
For case 3, the conditions are similar to case 2, but the object can have a bigger W object .
In this case, the
LiDAR mainly captures points from surface 3, such points dominate the loss since points captured from surface 2 are very few.
Thus mismatches on surface 2 can also be ignored in the overall loss.
Best viewed in color with zoom in.
I D ISCUSSION ON V IEWPOINT D IFFERENCES BETWEEN L I DAR AND C AMERA
In the real world, the LiDAR and camera have different viewpoints, which may have some impacts on the proposed losses.
Therefore, in this section we discuss the impact brought by viewpoint differences.
First, if an object exists in one sensor while invisible in the other sensor, we cannot obtain the associated object-LiDAR-points.
In this case, we directly ignore this object.
Second, if an object is visible in both camera and LiDAR, the viewpoint differences may bring some impacts in some conditions.
For this case, we provide a detailed discussion Figure 14.
In sum, we can ignore the impact brought by viewpoint differences.
J D ISCUSSION ON D IFFICULT C ASES
Potential noises and sparsity can exist in clustered points from faraway and small objects.
Such faraway and small objects are typical dilemmas for most 3D detection methods due to the inadequate visual clues on the image or few LiDAR points.
Unfortunately, our method is a general design and does not introduce special components for such objects.
We may use depth completion to enhance the raw LiDAR point cloud, to alleviate this issue.
This solution can be explored in future works.
Also, when only one object surface ( e.g. , only the back of a car) is observed, the network cannot learn the complete object dimensions, and the learning of object locations is also affected by object dimensions.
In other words, such LiDAR point clouds only provide part of weak supervision, thus the network trained with them performs suboptimally compared to using more complete LiDAR point clouds.
Fortunately, not all LiDAR point clouds are incomplete, and other good LiDAR points provide accurate and complete object 3D information, including object 3D locations and dimensions.
Therefore, the network still works well after being trained with different types of LiDAR point clouds.
[Table]
[Table]
K D ISCUSSION ON D IFFERENT B ASELINES
It is interesting to see the performance gap between the fully supervised and our weakly supervised manner on different baselines.
For the proposed baseline network in the paper
[Table]
we report the results in Table 12.
Interestingly, on this baseline network, we can see that our method performs better than using the full supervision.
This tendency is the same as Table
[Table]
since our method takes more geometry and scene constraints into consideration, beneﬁting simple baselines.
On the other hand, when using more advanced baselines
[Table]
our method performs worse than using the full supervision ( e.g. , WeakM3D-PatchNet vs. PatchNet in Table 1).
It is because advanced baselines weaken the usefulness of geometry and scene constraints of our method.
To comprehensively investigate the performance on advanced baselines, we report the results on PatchNet Ma et al.
(2020) under the
[Table]
IoU 0.5 criterion in Table 13.
We can observe that the advanced baseline can adequately use the full supervision, thus showing better results than our method.
L R ESULTS ON N U S CENES
To encourage future works on weakly supervised monocular 3D detection, we further provide results on the NuScenes dataset, which can serve as a baseline for future works.
We report the results in
Table 14.
We use the pre-trained Mask-RCNN, to obtain RoI LiDAR points in training, and to obtain 2D detections in inference.
Please note that the pre-trained Mask-RCNN cannot output all classes annotated in NuScenes, we only conduct experiments on the car category.
Additionally, we do not report the AVE (Average Velocity Error) and AOE (Average Orientation Error) because we can not obtain supervision of the velocity and the moving direction in the weakly supervised method.