4D Unsupervised Object Discovery
Anonymous Author(s)
AffiliationAddress email
Abstract
Object discovery is a core task in computer vision.
While tremendous success hasprogressed in supervised object detection with vast annotated data, its unsupervised counterpart remains largely unexplored.
With the growth of data volume,the expensive cost of annotations is the major limitation hindering further study.
Therefore, discovering the objects without annotations has great significance.
However, the task seems impractical on still-image or point cloud alone due to thelack of discriminative information.
Previous studies overlook the crucial temporalinformation and constraints naturally behind multi-modality.
In this paper, wepropose 4D unsupervised object discovery , jointly discovering objects from 4Ddata——3D point clouds and 2D RGB images with temporal information.
Wepresent the first practical approach for this task by proposing a ClusterNet on3D point clouds, which is joint iterative optimizing with a 2D localization network.
Extensive experiments on the large-scale Waymo Open Dataset suggestthat the localization network and ClusterNet achieve competitive performance onclass-agnostic 2D object detection and 3D instance segmentation, bridging the gapbetween unsupervised methods and full supervision.
Code will be released.
Introduction
Computer vision researchers have been trying to locate objects in complex scenes without humanannotations for a long time.
Current supervised methods achieve remarkable performance on 2Ddetection [25, 14, 24, 32, 5] and 3D detection [41, 22, 28, 27, 39], benefiting from high-capacitymodels and massive annotated data but tend to fail for scenarios that lack training data.
Therefore,unsupervised discovery is critical for relieving the demand for training labels in deep networks, whereraw data are infinite and cheap but annotations are limited and expensive.
However, unsupervised object discovery in complex scenes used to believe impractical.
Only a fewstudies pay attention to this field and achieve limited performance in simple scenarios, far inferiorto the supervised model.
Recent methods [29, 34] discover objects on 2D still-image utilizing theself-supervised learning [6, 35] to distinguish primary objects from the background, then fine-tune alocalization network using the pseudo label.
Although these methods have achieved superior results toprevious methods [33, 2, 42], the performance still lags a huge gap to supervised models.
Furthermore,such methods guided by contrastive learning have the problem of distinguishing different instanceswithin the same category.
Alternatively, the 3D point cloud can be decomposed into different classagnostic instances based on proximity cues [4, 3], but it is hard to identify the foreground.
Theseproblems can be mitigated by the complementary characteristics of 2D RGB images and 3D pointclouds.
The point cloud data provides accurate location information, while the RGB data contains richtexture and color information.
Therefore, [31] proposed to aid unsupervised object detection with
LiDAR clues, but it also depends on the performance of the self-supervised model [13] to identifyforeground objects.
In summary, all previous methods rely heavily on the self-supervised learningmodels and overlook the important information from time dimension .
To explore this field further, we propose a new task named 4D unsupervised object discovery , discovering objects utilizing 4D data——3D point clouds and 2D RGB images with temporal information [21].
The task needs to joint discover objects on RGB images as 2D Object Detection and 3D point cloudsas 3D Instance Segmentation .
Thanks to the popularization of LiDAR sensors in autonomous drivingand consumer electronics (e.g., iPad Pro), such 4D data has become much more readily available,indicating the great potential of this task for general application.
In this paper, we present the first practical solution for this task.
We proposed a joint iterativeoptimization for ClusterNet on 3D point cloud and localization network in RGB images, utilizingthe spatio-temporal consistency from multi-modality.
Specifically, the ClusterNet was trained bymotion cues initially, which can obtain from the consecutive point clouds using temporal information.
The 3D instance segmentation output by ClusterNet can be further projected to the 2D image assupervision to train the localization network.
Conversely, 2D detection can also help to refine the 3Dinstance segmentation.
In this way, the 2D localization network and 3D ClusterNet can benefit fromeach other.
Temporal information could serve as a constraint in the optimization.
Our main contributions are as follows: (1) we proposed a new task termed 4D Unsupervised Object
Discovery , aiming at jointly discovering the objects in the 2D image and 3D Point Cloud withoutmanual annotations; (2) we proposed a ClusterNet on 3D point clouds, which is jointly iterativeoptimized with a 2D localization network.
(3) Experiments on the Waymo Open Dataset [30] suggestthe feasibility of the task and the effectiveness of our approach.
We outperform the state-of-the-artunsupervised object discovery by a significant margin, superior to supervised methods with limitedannotations, and even comparable to supervised methods with full annotations.
Related work
2.1 Supervised object detection
Object detection from Image.
2D detection has made great progress in recent years.
Two-stagemethods represented by the RCNN family [12, 25, 14] extract region proposals first and refine themwith deep neural networks.
One-stage methods like YOLO [24], SSD [20] and RetinaNet [19] predictthe class-wise bounding box in one-shot based on the anchors.
FCOS [32] and CenterNet [40] furtherdetect objects without predefined anchors.
Object detection from Point Cloud.
LiDAR-based 3D object detection develops rapidly along withautonomous driving.
Point-based methods [23, 22, 38] directly estimated 3D bounding boxes frompoint clouds.
The computing efficiency is affected by the number of points, so these methods areusually suitable for indoor scenes.
Voxel-based methods [41, 37, 27] operate on the 3D voxelizedpoint cloud are capable for large outdoor scenes.
However, voxel resolution can greatly affectperformance but is limited by computational constraints.
Second [37] and PVRCNN [27] furtherapply sparse 3D convolutions to reduce compute.
CenterPoint [39] extends the idea of anchor-freefrom 2D detection and proposes a center-based representation of bird-eye view (BEV).
2.2 Unsupervised object discovery
Bottom-up clustering .
Clustering methods combine similar elements based on proximity cues, applicable to point clouds and image data.
Selective Search [33], MCG [2] and Edge Box [42] can proposea large number of candidate objects with the help of appearance cues, but it is difficult to identifyobjects from the background.
Similarly, point cloud data can decompose into distinct segmentsaccording to density-based methods [9, 4, 26] but is unable to determine which is foreground.
Top-down learning .
Recently, self-supervised learning [7, 13, 6, 35] are capable to learn discriminatefeatures without labels.
Therefore, many methods attempt to introduce such properties to discriminateforeground objects without manual annotations.
LOST [29] utilized a pre-trained DINO [6] to extractprimary object attention from the background as the pseudo label and then finetuned an objectdetector.
FreeSOLO [34] further proposed a unified framework for generating pseudo masks anditerative training.
However, the performance relies heavily on the pre-trained self-supervised model,which determines the upper limits of such methods.
Furthermore, such attention-based methodslearned by contrastive learning also have the problem of distinguishing different instances withinthe same category.
Our approach adopts top-down learning as well.
Instead of aiding by an externalself-supervised model, we look for motion cues to discover potential objects naturally.
𝐼 " 𝑃
[Figure]
𝑳
[Figure]
𝑵 𝜽
Movable
Static 2D View 3D View 2D View 3D View
T
T
Algorithm
3.1 Task definition and algorithm overview
The task of 4D Unsupervised Object Discovery is defined as follows.
As shown in Figure 1, the inputis a set of video clips recorded in both 2D video frames 𝐼 𝑡 and 3D point clouds 𝑃 𝑡 at frame 𝑡 duringtraining.
Since the point cloud and image data provide complementary information about locationand appearance, they can serve as the natural cues guiding the training process mutually.
Duringinference, the trained localization network 𝐿 𝜃 1 is applied to still-image for 2D object detection, andthe trained ClusterNet 𝑁 𝜃 2 is applied to point cloud for 3D instance segmentation.
[Equation]
{ 𝑏 𝑡
[Equation]
..., 𝑏 𝑡𝑘 } are the 2D bounding box predictions by localization network
𝐿 𝜃 1 at frame 𝑡 .
{ 𝜉 𝑡 1 , ..., 𝜉 𝑡 𝑛 }are the 3D instance segments output by ClusterNet 𝑁 𝜃 2 .
𝑘 and 𝑛 denotes the instance index.𝜃
[Equation]
𝜃
[Equation]
[Equation]
Our solution exploits the spatio-temporal consistency on 2D video frames and 3D point clouds.
Thealgorithm can be formulated into a joint optimization function 𝑓 in Eq.2.
𝜃 1 and 𝜃 2 are the parametersof the network need to optimize.
Temporal information 𝑡 serve as the natural constraint in function 𝑓 .
The localization network 𝐿 𝜃 1 utilized Faster R-CNN as default.
We propose a ClusterNet 𝑁 𝜃 2 for 3Dinstance segmentation.
Detail implementation will discuss in section 3.2.
The major challenge isthe optimization for function 𝑓 without annotations.
To overcome the challenge, we seek for motioncues , 2D-3D cues and temporal cues to serve as the supervision.
All these cues are extracted naturallyin the informative 4D data.
(1) motion cues , represented as 3D scene flow, can distinguish movablesegments from the background.
It uses to train the ClusterNet 𝑁 𝜃 2 initially.
(2) 2D-3D cues , reflectingthe mapping between LiDAR points and RGB pixels, can be used as a bridge to optimize the 𝐿 𝜃 1and 𝑁 𝜃 2 iteratively.
It indicates the output of either network can be further used to optimize anothernetwork.
(3) temporal cues , encouraging the temporal-consistent discovery in 2D and 3D view, canserve as the constraint to optimize the function together.
More details will introduce in section 3.3.
[Figure]
Vote Vote
Class
Center
C l
[Figure]
s
[Figure]
e r i n g
3.2 ClusterNet
ClusterNet generates 3D instance segmentation from raw point clouds.
As shown in Figure 2, given apoint cloud 𝑃 ∈ {( 𝑥, 𝑦, 𝑧 ) 𝑖 , 𝑖 = 1 , ..., 𝑁 } , the network is able to give each point a class type 𝑦 𝑖 ∈ { 1 , 0 }(foreground or background) and instance ID 𝑑 𝑖 ∈ { 1 , ..., 𝑛 } .
Thus we can obtain 𝑛 candidate segments𝜉 𝑖
= {( 𝑥, 𝑦, 𝑧 ) 𝑗 | 𝑦 𝑗 = 1 , 𝑑 𝑗 = 𝑖 } on the point
Network design.
The model first voxelized 3D points ( 𝑥, 𝑦, 𝑧 ) 𝑖 and extract voxelized features by atransformer-based feature extractor [10].
We further project these voxelized features back to eachpoint.
The feature dimensions of points become 3 + 𝐶 ( 3 means 𝑋𝑌𝑍 and 𝐶 denotes the embeddingdim).
Inspired by the VoteNet [22], we leverage a voting module to predict the class type and centeroffset for each point.
Specifically, the voting module is realized with a multi-layer perception (MLP)network.
The voting module takes point feature 𝒇 𝒊 ∈ R 3 + 𝐶 and outputs the Euclidean space offsetΔ 𝒙 𝒊 ∈ R 3 and class type prediction 𝑦 𝑖 .
The final loss is the weighted sum of the class prediction andcenter regression:
[Equation]
[Equation]
The class prediction loss L 𝑐𝑙𝑠 choose the focal loss [19] to balance the points of foreground andbackground.
The predicted 3D offset Δ 𝒙 𝒊 is supervised by a regression loss:
[Equation]
(4) where 1 [ 𝑦 ∗ 𝑖
= 1 ] indicates whether a point belongs to the foreground according to the ground truth𝑦 ∗ 𝑖 .
𝑀 is the total number of foreground points.
Δ 𝒙 ∗ 𝒊 is the ground truth offset from the point position𝒙 𝒊 to the instance center it belongs to.
According to spatial proximity, we could further group thepoints into candidate instance segments with the predicted class type and center offset.
Training by motion cues.
It is more challenging to obtain the supervision signal without manualannotation than the network design.
Motion provides strong cues for identifying foreground pointsand grouping parts into objects since moving points generally belong to objects and have the samemotion pattern if they belong to the same instance.
We could estimate the 3D scene flow 𝑆 𝑡 from thesequence of point clouds 𝑃 𝑡 using the unsupervised method [17] at frame 𝑡 .
3D scene flow describesthe motion of all the 3D points in the scene, represented as 𝑆 𝑡 = {( 𝑣 𝑥 , 𝑣 𝑦 , 𝑣 𝑧 ) 𝑡 𝑖 , 𝑖 = 1 , ..., 𝑁 } .
Combining the scene flow ( 𝑣 𝑥 , 𝑣 𝑦 , 𝑣 𝑧 ) 𝑖 and point location in 2D ( 𝑢, 𝑣 ) 𝑖 and 3D ( 𝑥, 𝑦, 𝑧 ) 𝑖 , we can obtain( 𝑢, 𝑣, 𝑥, 𝑦, 𝑧, 𝑣 𝑥 , 𝑣 𝑦 , 𝑣 𝑧 ) 𝑖 for each point 𝑝 𝑖 in 𝑃 𝑡 .
Then, we cluster the points with HDBSCAN [4] todivide the scan into 𝑚 segments which will be the instance candidates 𝜉 1 , ..., 𝜉 𝑚 .
However, thesesegments contain both foreground and background.
We further assign each point 𝑝 𝑖 of segment 𝜉 𝑗 abinary label 𝑦 ∗ 𝑖 to distinguish foreground points with the motion cues (scene flow), as shown in Eq. 5.
[Equation]
(5) in which 1 [] is the indicator function.
| 𝜉 𝑗 | represents the total number of points belong to segment𝜉 𝑗 .
𝑝 𝑖 ∈ 𝜉 𝑗 is a point in segment 𝜉 𝑗 .
∥ 𝑆 𝑡 ( 𝑝 𝑖 )∥ 2 represents the velocity of the point 𝑝 𝑖 , and 𝜎 denotesthe threshold for velocity.
𝜂 determine the ratio of being an foreground object.
𝜎 = 0 .
05 and 𝜂 = 0 .
8by default.
𝑦 ∗ 𝑖
= 1 means the point belongs to foreground segments.
When the proportion of movingpoints in the segment is greater than the threshold 𝜂 , we regard it as a foreground object, and all thepoints it contains are labeled as foreground.
These foreground segments serve as the pseudo groundtruth to train the ClusterNet initially.
3.3 Joint iterative optimization
ClusterNet trained with the motion cues serves as the initial weights for 𝜃 2 .
Although movable objectscan separate from the background, there are many static objects (e.g., parked cars or pedestrianswaiting for traffic lights) in the scenes.
Discovering both movable and static objects relies on furtherjoint optimization by 2D-3D cues and temporal cues .
Specifically, The 3D segments output by 𝑁 𝜃 2can project to the 2D image to train the 𝐿 𝜃 1 , and the 2D proposals output by 𝐿 𝜃 1 can transform backto 3D view to train the 𝑁 𝜃 2 .
Temporal consistency ensures the objects appear continuously in 2D and3D views, which is a critical constraint in the optimization.
The joint optimization can be iteratedmany times since the 2D localization network and 3D ClusterNet can benefit from each other.
[Equation] [Equation]
In Eq.
6, our goal is to optimize the 𝜃 1 and 𝜃 2 without annotations.
𝐼 𝑡 and 𝑃 𝑡 denote the RGB imageand point cloud at frame 𝑡 .
It is challenging to optimize both parameters simultaneously, so we dividethe optimization process into two iterative steps: 2D step and 3D step.2D step.
In this step, the 𝜃 2 is fixed and optimized 𝜃 1 .
Since the ClusterNet 𝑁 𝜃 2 are able to generate3D instance segments
𝜉 1 , ..., 𝜉 𝑛 in 3D space, we can further project the 3D instance segments to 2Dimage plane by the transformation 𝑇 𝑐𝑙 (from the LiDAR sensor to the camera) and projection matrix𝑃 𝑝𝑐 (from camera to pixels) defined by the camera intrinsic.
[Equation]
in which 𝒖 denotes the pixel location in the 2D image plane, and 𝒙 represents the 3D positionof LiDAR points.
Hence we can obtain the object point sets 𝜔 1 , .., 𝜔 𝑛 in the 2D image plane byprojecting the 3D instance segments 𝜉 1 , ..., 𝜉 𝑛 .
The 2D bounding boexes 𝑏 ∗ 1 , ..., 𝑏 ∗ 𝑛 derived fromprojected object point sets 𝜔 1 , .., 𝜔 𝑛 can use to optimize the weights of localization network
𝐿 𝜃 1 .3D step.
In this step, the 𝜃 1 is fixed and optimized 𝜃 2 .
The localization network 𝐿 𝜃 1 can generatecandidate 2D proposals 𝑏 𝑎 1 , ..., 𝑏 𝑎𝑘 based on the image appearance information.
It enables us todiscover more objects in the scene (e.g., parked cars regarded as background by motion cues).
Wecan get the new 2D object set 𝑏 ∗ by Eq.
8 (box IoU set to 0.3 in Non-Maximum Suppression).
[Equation]
Since many 3D instance segments may have been labeled as background before, we later refinedthe label with the help of the 2D object set.
Although the projection from LiDAR to the imageis non-invertible without dense depth maps, we could utilize the mapping between LiDAR pointand image pixels.
It suggests using the LiDAR points within the 2D bounding box to relabel the3D instance segments.
However, the bounding box contains many LiDAR points corresponding todifferent 3D instance segments.
Practically, we only consider the primary segment 𝜉 𝑗 (with mostpoints) inside the bounding box and relabel the primary segment as the foreground object.
With therefined label, we obtain the new set 𝜉 ∗ of 3D instance segments 𝜉 ∗ 1 , ..., 𝜉 ∗ 𝑛 , which utilize to optimizethe weights for ClusterNet 𝑁 𝜃 2 .
[Equation]
[Table]
(cid:125) 3D step (9)
Temporal cues.
Temporal information can be integrated into the 2D step and 3D step as constraints.
As shown in Eq.
9, 𝑏 𝑡 , 𝜉 𝑡 represent the predicted 2D bounding box set and predicted 3D segments setfor frame 𝑡 by 𝐿 𝜃 1 and
𝑁 𝜃 2 .
𝑏 𝑡 ∗ , 𝜉 𝑡 ∗ denote the pseudo annotation for 2D bounding boxes ( 𝑏 ∗ 1 , ..., 𝑏 ∗ 𝑛 )and 3D segments ( 𝜉 ∗ 1 , ..., 𝜉 ∗ 𝑛 ) from previous 2D step and 3D step.
L 2 𝐷 is the loss for localizationnetwork, and L 3 𝐷 is the loss for ClusterNet, which is the same as Eq. 3.
L 𝑠𝑚𝑜𝑜𝑡ℎ encourages thatthe same object have consistent object labels across frames.
Therefore, it can help find new objects orfilter out wrong annotations across time.
More details are illustrated in Appendix.
Experiments
4.1 Dataset and implementation details
We evaluate our approach on the challenging Waymo Open Dataset (WOD) [30], which provides 3Dpoint clouds and 2D RGB image data that is suitable for our task setting.
Dataset.
Waymo [30] is a recently released large-scale dataset for autonomous driving.
We utilizepoint clouds from the ‘top’ LiDAR (64 channels, a maximum distance of 75 meters), and videoframes (at a resolution of 1280 × 1920 pixels) from the ‘front’ camera.
The training and validationsets contain around 158k and 40k frames, respectively.
All training frames and validation frames aremanually annotated with 2D bounding boxes and 3D bounding boxes, which are capable of evaluatingthe performance of 2D object detection and 3D instance segmentation.
Furthermore, Waymo alsoprovides the scene flow annotation in the latest version [16], which can illustrate the upper potentialof our method.
Evaluation protocol.
Evaluation is conducted on the annotated validation set of WOD.
We evaluatethe performance of 2D object detection and 3D instance segmentation.
The dataset contains fourannotated object categories (‘vehicles’, ‘pedestrians’, ‘cyclists’, and ‘sign’).
We test the class-agnosticaverage precision (AP) score for vehicles, pedestrians, and cyclists.
For 2D object detection, the
AP score is reported at the box intersection-over-union (IoU) threshold of 0.5.
For better analysis,results are also evaluated on small (area < 32 2 pixels), medium ( 32 2 pixels < area < 96 2 pixels) andlarge objects (area > 96 2 pixels).
We also calculated the average recall (AR) to measure the ability ofobject discovery.
For 3D instance segmentation, no previous metrics have been proposed on Waymo.
Referring to the 2D AP metrics, we propose to compute 3D AP score based on the IoU betweenpredicted instance point sets and the ground truth.
The ground truth for the instance segmentationcan be obtained by labeling the point within 3D bounding boxes.
The 3D AP score is reported at thepoint sets IoU threshold of 0.7 and 0.9, denoted as 𝐴𝑃 70 and 𝐴𝑃 90 , respectively.
We also calculatedthe recall and foreground IoU for better analysis, which can measure the ability of object discoveryfrom more perspectives
Implementation details.
Our implementation is based on the open-sourced code of mmdetection3d [8] for 3D detection and detectron2 [36] for 2D detection.
For 2D localization network, weutilize Faster R-CNN [25] with FPN [18] by default, where ResNet-50 [15] is used as the backbone.
The network is trained on 8 GPUs (A100) with 2 images per GPU for 12k iterations.
The learningrate is initialized to 0.02 and is divided by 10 at the 6k and the 9k iterations.
The weight decay and themomentum parameters are set as 10 − 4 and 0 . 9 , respectively.
For 3D ClusterNet, the input raw pointclouds removed ground points first by [3] and remained the points that can only be seen on the frontcamera.
The cluster range is [ 0 𝑚, 74 . 88 𝑚 ] for the X-axis, [− 37 .
44 𝑚, 37 .
44 𝑚 ] for the Y-axis and[− 2 𝑚, 4 𝑚 ] for the Z-axis.
The voxel size is ( 0 .
32 𝑚, 0 .
32 𝑚, 6 𝑚 ) .
The feature extractor for voxelizedpoints is [10], and the embedding dim for 𝐶 is 128.
In the focal loss for class prediction, we set𝛾 = 2 . 0 , 𝛼 = 0 .
The balance weight 𝜆 for Eq.
3 is set to 5.
During inference, we set the minimumnumber of points to 5 for clustering.
The ClusterNet is trained on 8 GPUs (A100) with 2 point cloudsper GPU for 12 epochs.
The learning rate is initialized to 10 − 5 and adopts the cyclic cosine strategy(target learning rate is 10 − 3 ).
For hyper-parameters in HDBSCAN [4], we set the min cluster size to15, and the others follow the default.
For more implementation details, please refer to Appendix.
4.2 Main results
2D object detection.
Table 1 compares the results of manual annotation and ClusterNet for classagnostic 2D object detection.
All the experiments in Table 1 utilized Faster R-CNN [25].
For the 2Dbounding box annotations, the distant boxes are LiDAR invisible.
The manual-annotated supervisedbaseline is trained with LiDAR visible 2D box annotations for a fair comparison.
Even though ourmethod still has the gap 43.2 vs. 54.4 to supervised baseline using fully manual annotation (1137kbounding boxes), we can outperform the case when the annotation is limited.
Limited annotation isfrequent in real-world applications.
Compared with 10% manual annotation (127k bounding boxes),our method could achieve 43.2 AP without any manual annotations and beat the 33.8 AP by a largemargin.
Since our method relies on the motion cues estimated unsupervised, we proved that theperformance could increase to an incredible 51.8 AP with ground truth scene flow, which is veryclose to the performance of fully manual annotation but without bounding box annotation .
[Table]
Since the previous unsupervised methods only focused on still 2D images and could not extract theobjects from the background accurately, it is no surprise they could only achieve poor results in suchchallenging scenes.
LOST [29] can only extract one primary object from the background, whichdoes not apply to our driving scenes.
Felzenszwalb Segmentation [11] generate potential proposalsby graph-based segmentation, but lacks the ability to identify foreground.
Our method has greatperformance advantages over these still-image methods.3D instance segmentation.
Table 2 illustrates the effectiveness of our ClusterNet on 3D instancesegmentation.
Our ClusterNet achieved 26.2 AP 70 and 19.2 AP 90 without any annotation, superior to10% supervised baseline 23.6 AP 70 and 15.5 AP 90 with 397k 3D bounding boxes annotation.
Weproved that our method with accurate motion cues (ground truth scene flow) could achieve 42.0 AP 70and 33.2 AP 90 , even comparable to that supervised baseline with fully manual annotation (4268k 3Dbounding boxes).
No previous method can achieve such high performance under an unsupervisedsetting.
Figure 3 illustrates the object prediction of our approach on the Waymo validation set.
[Table]
2D instance segmentation.
We can also conduct instance segmentation by projecting the LiDARpoints of 3D instance segments to the 2D image plane.
The key difference is that instance segmentationmasks other than object bounding boxes deriving from 3D instance segments as pseudo annotations.
We utilized alpha shape [1] to generate the mask of object points (projected to the 2D image).
Thelocalization network can change conveniently to Mask R-CNN [14] for the instance segmentation without manual annotations.
Some predictions on validation set are illustrated in Figure 4 and
Appendix.
Because the Waymo Open Dataset did not provide the annotation for instance mask, theperformance of instance segmentation cannot be evaluated quantitatively.
4.3 Ablation studies
Analysis of multi cues for training.
Table 3 analyze the contributions of multi cues in our approachto the Waymo validation set.
Three iterations of joint optimizing train the final results.
AP 2 𝐷denotes the AP 50 score for 2D object detection, and AP 3 𝐷 denotes the AP 70 score for 3D instancesegmentation.
The ClusterNet was trained with the pseudo-annotations obtained by HDBSCAN
Clustering [4] for the first time.
So a simple baseline is directly using the HDBSCAN for 3D instancesegmentation and project on 2D for localization network training.
In comparison, we demonstrate theeffectiveness of using ClusterNet; the performance increased by 4.9 AP 2 𝐷 (from 25.1 to 30.0) and2.2 AP 3 𝐷 (from 4.6 to 6.8).
Furthermore, the performance improved significantly by joint optimizingfor the ClusterNet and localization network, with 2D-3D cues and temporal cues.
[Table]
Training strategy for localization network.
Training the localization network with the pseudoannotations generated by ClusterNet is quite different from training with manual annotations.
Specifically, the pseudo annotations will be noisy and incomplete before joint iterative training.
Therefore,two points we found crucial for the initial training: (1) sampling strategy in RPN (Region Proposal
Network) and (2) early stopping for training.
The following ablation experiments are conducted forthe first-time training the localization network.
Table 4 compares three different strategies: (a) sampleanchors with box IoU > 0 .
7 as positive example and box IoU < 0 .
3 as negative example (denoted as𝐼𝑜𝑈 + > 0 .
7 , 𝐼𝑜𝑈 − < 0 .
, as in the standard Faster R-CNN; (b) sample anchors with box IoU > 0 .
6as positive example and box IoU < 0 .
4 as negative example (denoted as 𝐼𝑜𝑈 + > 0 .
6 , 𝐼𝑜𝑈 − < 0 . 4 );(c) sample anchors with box IoU > 0 .
6 as positive example, and box 0 .
1 < IoU < 0 .
4 as negativeexample (denoted as 𝐼𝑜𝑈 + > 0 .
1 < 𝐼𝑜𝑈 − < 0 . 4 ).
In this way, the strategy considerably reducesthe chance of sampling missed objects as negative examples.
Table 5 compares different trainingiterations and shows that early stopping improves the generalization performance.
Since the pseudoannotations have noise, training for a long time may overfit the noise in the training set, leading tothe degradation of generalization performance.
[Table]
[Table]
Joint iterative optimization.
Table 6 presents the effectiveness of our joint iterative optimizationfor the 2D localization network and 3D ClusterNet.
Iteration 0 represents the initial performanceof ClusterNet trained by motion cues (estimated scene flow).
Next, each iteration means a 2D stepand a 3D step.
Even though the model did not perform well at the beginning, with joint iterativeoptimization, both AP 2 𝐷 and AP 3 𝐷 improved rapidly.
Applying more than one iteration improves theresults, indicating that the 2D localization network and 3D ClusterNet can benefit from each other.
We set the iteration number as 3 by default.
Minimum points for ClusterNet.
Minimum points determine the minimum number of LiDARpoints for a 3D instance.
Table 7 analyze the model performance under different parameters duringthe inference.
We set the min points to 5 by default.
[Table] [Table]
Discussion and conclusions
Discussion.
Unsupervised object discovery used to believe infeasible due to the ambiguity ofobjects and complexity of scenarios.
However, 4D data with the sequence of image frames andpoint clouds provide enough cues to discover the movable objects, even without manual annotation.
The complementary information behind the 3D LiDAR points and 2D image and constraints fromtemporal are the critical factors for the success of unsupervised object discovery.
With 4D sensordata readily available onboard, our approach shows extraordinary potential for scenarios with limitedor no annotation.
The only limitation is that our method is suitable for movable objects; static things(never move) like beds or chairs can not be discovered.
After all, the manual annotation is limited,and the costs are expensive.
We seek the real intelligence for the machine to naturally understand thescene by mutual supervision of different modalities rather than human annotations.
Conclusions.
In this work, we propose a new task named 4D Unsupervised Object Discovery .
The task needs to discover objects both on the image and point clouds without manual annotations.
We present the first practical approach for this task by proposing a ClusterNet for 3D instancesegmentation and joint iterative optimization.
Extensive experiments on the large-scale Waymo Open
Dataset demonstrate the effectiveness of our approach.
So far as we know, we are the first work toachieve such high performance for unsupervised 2D object detection and 3D instance segmentation,bridging the gap between unsupervised methods and supervised methods.
Our work sheds light on anew perspective on the future study of unsupervised object discovery.
Societal Impacts.
The development of unsupervised object discovery requires large datasets, introducing privacy issues.
The technology of unsupervised detection dramatically reduces the labelingcost; it may affect the people engaged in the labeling industry in the future.
The elimination of humanintervention may also cause some data annotators to lose their current jobs.
Our approach only testsin driving scenes for effectiveness, which may lead to some wrong detection in other scenes.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section.•
Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] See Section 5(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] SeeSection 5(d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] Code will bereleased upon acceptance(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Section 4.1 and Appendix
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]