4D Unsupervised Object Discovery
Abstract
Object discovery is a core task in computer vision.
While fast progresses have been made in supervised object detection, its unsupervised counterpart remains largely unexplored.
With the growth of data volume, the expensive cost of annotations is the major limitation hindering further study.
Therefore, discovering objects without annotations has great signiﬁcance.
However, this task seems impractical on still-image or point cloud alone due to the lack of discriminative information.
Previous studies underlook the crucial temporal information and constraints naturally behind multi-modal inputs.
In this paper, we propose 4D unsupervised object discovery , jointly discovering objects from 4D data – 3D point clouds and 2D
RGB images with temporal information.
We present the ﬁrst practical approach for this task by proposing a ClusterNet on 3D point clouds, which is jointly iteratively optimized with a 2D localization network.
Extensive experiments on the large-scale Waymo Open Dataset suggest that the localization network and
ClusterNet achieve competitive performance on both class-agnostic 2D object detection and 3D instance segmentation, bridging the gap between unsupervised methods and full supervised ones.
Codes and models will be made available at https://github.com/Robertwyq/LSMOL .
Introduction
Computer vision researchers have been trying to locate objects in complex scenes without human annotations for a long time.
Current supervised methods achieve remarkable performance on 2D detection [31, 15, 30, 38, 6] and 3D detection [49, 27, 34, 33, 47], beneﬁting from high-capacity models and massive annotated data, but tend to fail for scenarios that lack training data.
Therefore, unsupervised object discovery is critical for relieving the demand for training labels in deep networks, where raw data are inﬁnite and cheap, but annotations are limited and expensive.
However, unsupervised object discovery in complex scenes used to believe impractical.
Only a few studies pay attention to this ﬁeld and achieve limited performance in simple scenarios, far inferior to the supervised model.
Recent methods [35, 42] discover objects on 2D still-image utilizing the self-supervised learning [7, 43] to distinguish primary objects from the background, then ﬁne-tune a localization network using the pseudo label.
Although these methods outperform the previous generation of object proposal methods [39, 2, 50], their detection results are still far behind supervised models.
Furthermore, contrastive learning-guided methods have difﬁculty in distinguishing different instances within the same category.
Alternatively, the 3D point cloud can be 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
decomposed into different class-agnostic instances based on proximity cues [5, 4], but due to lack of semantic information, it is difﬁcult to identify the foreground instances.
These problems can be mitigated by the complementary characteristics of 2D RGB images and 3D point clouds.
The point cloud data provides accurate location information, while the RGB data contains rich texture and color information.
Therefore, [37] proposed to aid unsupervised object detection with LiDAR clues, but it still depends on self-supervised models [14] to identify foreground objects.
In summary, all previous methods rely heavily on the self-supervised learning models and overlook the important information from the time dimension .
To these ends, we propose a new task named 4D unsupervised object discovery , discovering objects utilizing 4D data – 3D point clouds and 2D RGB images with temporal information [25].
The task needs to joint discover objects on RGB images as in 2D Object Detection and objects on 3D point clouds as in 3D Instance Segmentation .
Thanks to the popularization of LiDAR sensors in autonomous driving and consumer electronics (e.g., iPad Pro), such 4D data has become much more readily available, indicating the great potential of this task for general application.
In this paper, we present the ﬁrst practical solution for 4D unsupervised object discovery.
We proposed a joint iterative optimization for ClusterNet on 3D point cloud and localization network in RGB images, utilizing the spatio-temporal consistency from multi-modality.
Speciﬁcally, the ClusterNet was trained with supervision from motion cues initially, which can be obtained from temporally consecutive point clouds.
The 3D instance segmentation output by ClusterNet can be further projected to the 2D image as supervision for the localization network.
Conversely, 2D detection can also help to reﬁne the 3D instance segmentation by utilizing appearance information.
In this way, the 2D localization network and 3D ClusterNet can beneﬁt from each other through joint optimization.
Temporal information could serve as a constraint in the optimization.
Our main contributions are as follows: (1) we proposed a new task termed 4D Unsupervised Object
Discovery , aiming at jointly discovering the objects in the 2D image and 3D Point Cloud without manual annotations.
(2) we proposed a ClusterNet on 3D point clouds for 3D instance segmentation, which is jointly iterative optimized with a 2D localization network.
(3) Experiments on the Waymo
Open Dataset [36] suggest the feasibility of the task and the effectiveness of our approach.
We outperform the state-of-the-art unsupervised object discovery by a signiﬁcant margin, superior to supervised methods with limited annotations, and even comparable to supervised methods with full annotations.
Related work 2.1 Supervised object detection
Object detection from Image.
2D object detection has made great progress in recent years.
Twostage methods represented by the RCNN family [13, 31, 15] extract region proposals ﬁrst and reﬁne them with deep neural networks.
One-stage methods like YOLO [30], SSD [22] and RetinaNet [21] predict the class-wise bounding box in one-shot based on the anchors.
FCOS [38] and CenterNet [48] further detect objects without predeﬁned anchors.
Object detection from Point Cloud.
LiDAR-based 3D object detection develops rapidly along with autonomous driving.
Point-based methods [28, 27, 46] directly estimated 3D bounding boxes from point clouds.
The computing efﬁciency is affected by the number of points, so these methods are usually suitable for indoor scenes.
Voxel-based methods [49, 45, 33] operate on the 3D voxelized point cloud are capable for large outdoor scenes.
However, voxel resolution can greatly affect performance but is limited by computational constraints.
Second [45] and PVRCNN [33] further apply sparse 3D convolutions to reduce compute.
CenterPoint [47] extends the idea of anchor-free from 2D detection and proposes a center-based representation of bird-eye view (BEV).
2.2 Unsupervised object discovery
Bottom-up clustering .
Clustering methods combine similar elements based on proximity cues, applicable to point clouds and image data.
Selective Search [39], MCG [2] and Edge Box [50] can propose a large number of candidate objects with the help of appearance cues, but it is difﬁcult to identify objects from the background.
Similarly, point cloud data can decompose into distinct segments according to density-based methods [10, 5, 32] but is unable to determine which is foreground.
[Figure]
Top-down learning .
Recently, self-supervised learning [8, 14, 7, 43] are capable to learn discriminate features without labels.
Therefore, many methods attempt to introduce such properties to discriminate foreground objects without manual annotations.
LOST [35] utilized a pre-trained DINO [7] to extract primary object attention from the background as the pseudo label and then ﬁnetuned an object detector.
FreeSOLO [42] further proposed a uniﬁed framework for generating pseudo masks and iterative training.
However, the performance relies heavily on the pre-trained self-supervised model, which determines the upper limits of such methods.
Furthermore, such attention-based methods learned by contrastive learning also have the problem of distinguishing different instances within the same category.
Our approach adopts top-down learning as well.
Instead of aiding by an external self-supervised model, we look for geometric information to discover objects in the scene naturally.
Algorithm 3.1 Task deﬁnition and algorithm overview
The task of 4D Unsupervised Object Discovery is deﬁned as follows.
As shown in Figure 1, the input is a set of video clips recorded in both 2D video frames  C and 3D point clouds % C at frame C during training.
Since the point cloud and image data provide complementary information about location and appearance, they can serve as the natural cues guiding the training process mutually.
During inference, the trained localization network !
\ 1 is applied to still-image for 2D object detection, and the trained ClusterNet # \ 2 is applied to the point cloud for 3D instance segmentation.
[Equation]
{ 1 C 1 , ..., 1 C: } are the 2D bounding box predictions by localization network !
\ 1 at frame C .
{ b C 1 , ..., b C= } are the 3D instance segments output by ClusterNet
and = denotes the instance index.
[Equation]
Our solution exploits the spatio-temporal consistency on 2D video frames and 3D point clouds.
The algorithm can be formulated into a joint optimization function 5 in Eq.2.
\ 1 and \ 2 are the parameters of the network need to optimize.
Temporal information C serve as the natural constraint in function 5 .
The localization network ! \ 1 utilized Faster R-CNN as default.
We propose a ClusterNet # \ 2 for 3D instance segmentation.
Detail implementation will discuss in section 3.2.
The major challenge is the optimization for function 5 without annotations.
To overcome the challenge, we seek for motion cues , 2D-3D cues and temporal cues to serve as the supervision.
All these cues are extracted naturally in the informative 4D data.
(1) motion cues , represented as 3D scene ﬂow, can distinguish movable segments from the background.
It uses to train the ClusterNet # \ 2 initially.
(2) 2D-3D cues , reﬂecting the mapping between LiDAR points and RGB pixels, can be used as a bridge to optimize the ! \and # \ 2 iteratively.
It indicates the output of either network can be further used to optimize another network.
(3) temporal cues , encouraging the temporal-consistent discovery in 2D and 3D view, can serve as the constraint to optimize the function together.
More details will introduce in section 3.3.
3.2 ClusterNet
ClusterNet generates 3D instance segmentation from raw point clouds.
As shown in Figure 2, given a point cloud % 2 {( G , H , I ) 8 , 8 = 1 , ..., # } , the network is able to give each point a class type H 8 2 { 1 , 0 } (indicating foreground or background) and instance ID 3 8 2 { 1 , ..., = } .
Thus we can obtain = candidate segments b 8 = {( G , H , I ) 9 | H 9 = 1 , 3 9 = 8 } on the point cloud.
= represents the number of instance segments in one frame of point cloud, and it is different in each frame.
Network design.
The model ﬁrst voxelized 3D points ( G , H , I ) 8 and extract voxelized features by a transformer-based feature extractor [11].
We further project these voxelized features back to each point.
The feature dimensions of points become 3 + ⇠ ( 3 means -./ and ⇠ denotes the embedding dim).
Inspired by the VoteNet [27], we leverage a voting module to predict the class type and center offset for each point.
Speciﬁcally, the voting module is realized with a multi-layer perception (MLP) network.
The voting module takes point feature f i 2 R 3 + ⇠ and outputs the Euclidean space offset  x i 2 R 3 and class type prediction H 8 .
The ﬁnal loss is the weighted sum of the class prediction and center regression:
[Equation]
The class prediction loss L 2;B choose the focal loss [21] to balance the points of foreground and background.
The predicted 3D offset  x i is supervised by a regression loss:
[Equation]
where [ H ⇤ 8 = 1 ] indicates whether a point belongs to the foreground according to the ground truth H ⇤ 8 .
" is the total number of foreground points.
x ⇤ i is the ground truth offset from the point position x i to the instance center it belongs to.
According to spatial proximity, we could further group the points into candidate instance segments with the predicted class type and center offset.
3D instance initialization.
It is more challenging to obtain the supervision signal without manual annotation than the network design.
The model was trained initially by motion cues.
Speciﬁcally, motion provides strong cues for identifying foreground points and grouping parts into objects since moving points generally belong to objects and have the same motion pattern if they belong to the same instance.
We could estimate the 3D scene ﬂow ( C from the sequence of point clouds % C using the unsupervised method [19] at frame C .
3D scene ﬂow describes the motion of all the 3D points in the scene, represented as ( C = {( E G , E H , E I ) C8 , 8 = 1 , ..., # } .
Combining the scene ﬂow ( E G , E H , E I ) 8 and point location in 2D ( D , E ) 8 and 3D ( G , H , I ) 8 , we can obtain ( D , E , G , H , I , E G , E H , E I ) 8 for each point ? 8 in % C .
Then, we cluster the points with HDBSCAN [5] to divide the scan into < segments, which will be the instance candidates b 1 , ..., b < .
However, these instance candidates contain both foreground and background segments.
We further assign each point ? 8 of segment b 9 a binary label H ⇤ 8 to distinguish foreground points using the motion cues (3D scene ﬂow), as shown in Eq.
[Equation]
B a c k b o n e
Vote Vote Class Center
C l u s t e r i n g in which [] is the indicator function.
| b 9 | represents the total to segment b 9 . ?
8 2 b 9 is a point in segment b 9 .
k ( C ( ? 8 ) k 2 represents the velocity of the point ? 8 , and f denotes the threshold for velocity.
[ determines the ratio of being a foreground object.
f = 0 .
05 and [ = 0 .by default.
H ⇤ 8 = 1 means the point belongs to foreground segments.
When the proportion of moving points in the segment is greater than the threshold [ , we regard it as a foreground object, and all the points it contains are labelled as foreground.
These foreground segments selected by motion serve as the pseudo ground truth to train the ClusterNet initially.
3.3 Joint iterative optimization
ClusterNet trained by the motion cues serves as the initial weights for \ 2 , which is the initialization (iter 0 ) of joint iterative optimization.
Although movable objects can separate from the background with the motion cues , there are many static objects (e.g., parked cars or pedestrians waiting for trafﬁc lights) in the scenes.
Discovering both movable and static objects relies on further joint optimization by 2D-3D cues and temporal cues .
In section 3.3.1, we introduce the speciﬁc process of joint iterative optimization.
Speciﬁcally, The 3D segments output by # \ 2 can project to the 2D image to train the ! \ 1 , and the 2D proposals output by ! \ 1 can lift back to 3D view to train the # \ 2 .
Temporal consistency ensures the objects appear continuously in 2D and 3D views, which is a critical constraint in optimization.
The joint optimization can be iterated several times since the 2D localization network and 3D ClusterNet can beneﬁt from each other.
In section 3.3.2, we will introduce the technical design for static object discovery.
3.3.1 Model training
In Eq.
6, our goal is to optimize the \ 1 and \ 2 without annotations.
C and % C denote the RGB image and point cloud at frame C .
It is challenging to optimize both parameters simultaneously, so we divide the optimization process into two iterative steps: 2D step and 3D step.
[Equation]
2D step.
In this step, the \ 2 is ﬁxed and optimized \ 1 .
Since the ClusterNet # \ 2 are able to generate 3D instance segments b 1 , ..., b = in 3D space, we can further project the 3D instance segments to 2D image plane by the transformation ) 2; (from the LiDAR sensor to the camera) and projection matrix % ? 2 (from camera to pixels) deﬁned by the camera intrinsic.
[Equation]
in which u denotes the pixel location in the 2D image plane, and x represents the 3D position of LiDAR points.
Hence we can obtain the object point sets { l 1 , .., l = } in the 2D image plane by projecting the LiDAR points of 3D instance segments { b 1 , ..., b = } .
The 2D bounding boxes { 1 ⇤ 1 , ..., 1 ⇤ = } derived from projected object point sets { l 1 , .., l = } , can use to optimize the weights of localization network !
3D step.
In this step, the \ 1 is ﬁxed and optimized \ 2 .
The localization network ! \ 1 can output 2D bounding box predictions { 1 0 1 , ..., 1 0: } based on the image appearance information.
It enables us to discover more objects in the scene (e.g., parked cars regarded as background by motion cues).
We can get the updated 2D object set 1 ⇤ by Eq.
8 (box IoU set to 0.3 in Non-Maximum Suppression).
[Equation]
Since many 3D instance segments may have been labelled as background by motion cues before, we later reﬁned the label with the help of the 2D object set 1 ⇤ .
Although the projection from LiDAR to the image is non-invertible without dense depth maps, we could still utilize the mapping between the LiDAR point and image pixels.
It suggests using the LiDAR points within the 2D bounding box to relabel the 3D instance segments.
However, the bounding box may contain many LiDAR points corresponding to different 3D instance segments.
Practically, we only consider the primary segment b 9 (with most points) inside the bounding box and relabel the primary segment as the foreground object.
With the reﬁned label, we obtain the updated 3D object set b ⇤ of 3D instance segments { b ⇤ 1 , ..., b ⇤ = } , which further utilize to optimize the weights for ClusterNet # \ 2 .
Temporal cues.
Temporal information can be integrated into the 2D step and 3D step as extra constraints.
As shown in Eq.
9, 1 C , b C represent the predicted 2D bounding box set and predicted 3D segments set for frame C by ! \ 1 and # \ 2 .
1 C ⇤ , b C ⇤ denote the pseudo annotation for 2D bounding boxes ( { 1 ⇤ 1 , ..., 1 ⇤ = } ) and 3D instance segments ( { b ⇤ 1 , ..., b ⇤ = } ) from previous 2D step and 3D step.
L 2 ⇡ is the loss for the localization network, and L 3 ⇡ is the loss for ClusterNet, which is introduced in Eq. 3.
L B<>>C⌘ encourages that the same object has consistent object labels across frames.
The constraint can be added to both 2D views and 3D views.
Therefore, it can help ﬁnd new potential objects and ﬁlter out wrong annotations across time.
More details are illustrated in Appendix C.
[Equation]
3.3.2 Static object discovery
Static object discovery is crucial in joint iterative training since the initialization by motion could handle movable objects well.
During the joint iterative training, two technical designs are important for static object discovery.
One is from the aspect of visual appearance, the other is from the aspect of temporal information.
Discover static objects by visual appearance.
2D localization network learns the object representation by visual appearance.
It indicates the good generalization ability for static objects since movable objects and static objects usually have similar visual appearances.
However, a critical design is the selection of positive and negative samples in model training.
Initially, the 2D pseudo annotations generated by motion cues mainly come from moving objects.
It is crucial to avoid static objects becoming negative samples so that the model can have better generalization ability to static objects.
Table 4 compares different sampling strategies for the training.
Discover static objects by temporal information.
Temporal information is also beneﬁcial for static object discovery.
Due to the occlusion in the 2D view, it is more applicable to discover potential new objects by tracking in the 3D view.
Practically, we used Kalman ﬁltering for 3D tracking, and rediscover new objects in the static tracklets (center offset between the start and end frames less than 3 meters).
Since we only focus on static objects, the mean center of the tracklet would be a good prediction for lost objects.
Experiments 4.1 Dataset and implementation details
We evaluate our method on the challenging Waymo Open Dataset (WOD) [36], which provides 3D point clouds and 2D RGB image data that is suitable for our task setting.
It is of great signiﬁcance to verify our unsupervised method under such a real and large-scale complex scene.
Dataset.
Waymo Open Dataset [36] is a recently released large-scale dataset for autonomous driving.
We utilize point clouds from the ‘top’ LiDAR (64 channels, a maximum distance of 75 meters), and video frames (at a resolution of 1280 ⇥ 1920 pixels) from the ‘front’ camera.
The training and validation sets contain around 158k and 40k frames, respectively.
All training frames and validation frames are manually annotated with 2D bounding boxes and 3D bounding boxes, which are capable of evaluating the performance of 2D object detection and 3D instance segmentation.
Furthermore,
WOD also provides the scene ﬂow annotation in the latest version [17], which can illustrate the upper potential of our method.
Evaluation protocol.
Evaluation is conducted on the annotated validation set of WOD.
We evaluate the performance of 2D object detection and 3D instance segmentation.
The dataset contains four annotated object categories (‘vehicles’, ‘pedestrians’, ‘cyclists’, and ‘sign’).
We test the class-agnostic average precision (AP) score for vehicles, pedestrians, and cyclists.
For 2D object detection, the
AP score is reported at the box intersection-over-union (IoU) threshold of 0.5.
For better analysis, results are also evaluated on small (area < 32 2 pixels), medium ( 32 2 pixels < area < 96 2 pixels) and large objects (area > 96 2 pixels).
We also calculated the average recall (AR) to measure the ability of object discovery.
For 3D instance segmentation, no previous metrics have been proposed on WOD.
Referring to the 2D AP metrics, we propose to compute 3D AP score based on the IoU between predicted instance point sets and the ground truth.
The ground truth for the instance segmentation can be obtained by labelling the point within 3D bounding boxes.
The 3D AP score is reported at the point sets IoU threshold of 0.7 and 0.9, denoted as AP 70 and AP 90 , respectively.
We also calculated the recall and foreground IoU for better analysis, which can measure the ability of object discovery from more perspectives.
Note here the AP 2 ⇡ denotes 2D object detection AP 50 score.
AP 3 ⇡ denotes the 3D instance segmentation AP 70 score.
Implementation details.
Our implementation is based on the open-sourced code of mmdetection3d [9] for 3D detection and detectron2 [44] for 2D detection.
For 2D localization network, we utilize Faster R-CNN [31] with FPN [20] by default, where ResNet-50 [16] is used as the backbone.
The network is trained on 8 GPUs (A100) with 2 images per GPU for 12k iterations.
The learning rate is initialized to 0.02 and is divided by 10 at the 6k and the 9k iterations.
The weight decay and the momentum parameters are set as 10  4 and 0 . 9 , respectively.
For 3D ClusterNet, the input raw point clouds removed ground points ﬁrst by [4] and remained the points that can only be seen on the front camera.
The cluster range is [ 0 < , 74 . 88 < ] for the X-axis, [  37 .
44 < , 37 .
44 < ] for the Y-axis and [  2 < , 4 < ] for the Z-axis.
The voxel size is ( 0 .
32 < , 0 .
32 < , 6 < ) .
The feature extractor for voxelized points is [11], and the embedding dim for ⇠ is 128.
In the focal loss for class prediction, we set W = 2 . 0 , U = 0 .
The balance weight _ for Eq.
3 is set to 5.
During inference, we set the minimum number of points to 5 for clustering.
The ClusterNet is trained on 8 GPUs (A100) with 2 point clouds per GPU for 12 epochs.
The learning rate is initialized to 10  5 and adopts the cyclic cosine strategy (target learning rate is 10  3 ).
For hyper-parameters in HDBSCAN [5], we set the min cluster size to 15, and the others follow the default.
For more implementation details, please refer to Appendix A.
4.2 Main results
2D object detection.
Table 1 compares the results between annotation from manual and annotaion from our method (termed as ClusterNet) for class-agnostic 2D object detection.
All the experiments in Table 1 utilized the same model (Faster R-CNN [31]) for fair evaluation.
For the 2D bounding box annotations, the distant boxes are LiDAR invisible.
The manual-annotated supervised baseline is trained with LiDAR visible 2D box annotations for a fair comparison.
Even though our method still has the gap 43.2 vs. 54.4 to supervised baseline using fully manual annotation (1137k bounding boxes), we can outperform the case when the annotation is limited.
Limited annotation is frequent in real-world applications.
Compared with 10% manual annotation (127k bounding boxes), our method could achieve 43.2 AP without any manual annotations and beat the 33.8 AP by a large margin.
Since our method relies on the motion cues estimated unsupervised, we proved that the performance could increase to an incredible 51.8 AP with ground truth scene ﬂow, which is very close to the performance of fully manual annotation but without bounding box annotation .
Since the previous unsupervised methods only focused on still 2D images and could not extract the objects from the background accurately, it is no surprise they could only achieve poor results in such challenging scenes.
LOST [35] can only extract one primary object from the background, which does not apply to the driving scenes.
Freesolo [42] often generates a large mask for a row of cars, which can not distinguish speciﬁc instances.
Felzenszwalb Segmentation [12] generate potential proposals by graph-based segmentation but lacks the ability to identify the foreground objects.
Our method has great performance advantages over these still-image methods.
[Table]
3D instance segmentation.
Table 2 illustrates the effectiveness of our ClusterNet on 3D instance segmentation.
Our ClusterNet achieved 26.2 AP 70 and 19.2 AP 90 without any annotation, superior to 10% supervised baseline 23.6 AP 70 and 15.5 AP 90 with 397k 3D bounding boxes annotation.
We proved that our method with accurate motion cues (ground truth scene ﬂow) could achieve 42.0 APand 33.2 AP 90 , even comparable to that supervised baseline with fully manual annotation (4268k 3D bounding boxes).
No previous method can achieve such high performance under an unsupervised setting.
Figure 3 illustrates the object prediction of our approach on the WOD validation set. [Table] ClusterNet
[Table]
w/
[Table]
sceneﬂow)
[Table]
ClusterNet
[Table]
2D instance segmentation.
We can also conduct instance segmentation by projecting the LiDAR points of 3D instance segments to the 2D image plane.
The key difference is that instance segmentation masks other than object bounding boxes deriving from 3D instance segments as pseudo annotations.
We utilized alpha shape [1] to generate the mask of object points (LiDAR points projected to the 2D image).
The localization network can change conveniently to Mask R-CNN [15] for instance segmentation without manual annotations.
Some predictions on validation set are illustrated in
Figure 4 and Appendix D. Because the Waymo Open Dataset did not provide the annotation for instance mask, the performance of instance segmentation cannot be evaluated quantitatively.
4.3 Ablation studies
Analysis of multi cues for training.
Table 3 analyze the contributions of multi cues in our approach to the WOD validation set.
The ﬁnal results are obtained after three iterations of joint optimization.
AP 2 ⇡ denotes the AP 50 score for 2D object detection, and AP 3 ⇡ denotes the AP 70 score for 3D instance segmentation.
The ClusterNet was trained with the pseudo-annotations obtained by HDBSCAN
Clustering [5] for the ﬁrst time.
So a simple baseline is directly using the HDBSCAN for 3D instance segmentation and project on 2D for localization network training.
In comparison, we demonstrate the effectiveness of using ClusterNet; the performance increased by 4.9 AP 2 ⇡ (from 25.1 to 30.0) and 2.2 AP 3 ⇡ (from 4.6 to 6.8).
Furthermore, the performance improved signiﬁcantly by joint optimizing for the ClusterNet and localization network, with 2D-3D cues and temporal cues.
[Table] [Table]
(a) IoU + >0.7, IoU  <0.
[Table]
(b) IoU + >0.6, IoU  <0.
[Table]
(c) IoU + >0.6, 0.1<IoU  <0.
[Table]
[Table] [Table]
Training strategy for localization network.
Training the localization network with the pseudo annotations generated by ClusterNet is quite different from training with manual annotations.
Speciﬁcally, the pseudo annotations will be noisy and incomplete before joint iterative training.
Therefore, two points we found crucial for the initial training: (1) sampling strategy in RPN (Region Proposal
Network) and (2) early stopping for training.
The following ablation experiments are conducted for the ﬁrst-time training of the localization network.
Table 4 compares three different strategies: (a) sample anchors with box IoU > 0 .
7 as positive example and box IoU < 0 .
3 as negative example, as in the standard Faster R-CNN; (b) sample anchors with box IoU > 0 .
6 as positive example and box
IoU < 0 .
4 as negative example; (c) sample anchors with box IoU > 0 .
6 as positive example, and box 0 .
1 < IoU < 0 .
4 as negative example.
In this way, the strategy considerably reduces the chance of sampling static objects as negative examples.
Table 5 compares different training iterations and shows that early stopping improves the generalization performance.
Since the pseudo annotations have noise, training for a long time may overﬁt the noise in the training set, leading to the degradation of generalization performance.
Joint iterative optimization.
Table 6 presents the effectiveness of our joint iterative optimization for the 2D localization network and 3D ClusterNet.
Iteration 0 represents the initial performance of ClusterNet trained by motion cues (estimated scene ﬂow).
Next, each iteration means a 2D step and a 3D step.
Even though the model did not perform well at the beginning, with joint iterative optimization, both AP 2 ⇡ and AP 3 ⇡ improved rapidly.
Applying more than one iteration improves the results, indicating that the 2D localization network and 3D ClusterNet can beneﬁt from each other.
We set the iteration number as 3 by default.
Minimum points for ClusterNet.
Minimum points determine the minimum number of LiDAR points for 3D instance segments.
Table 7 analyze the model performance under different parameters during the inference.
We set the min points to 5 by default.
[Table] [Table]
Discussion and conclusions
Discussion.
Unsupervised object discovery used to believe infeasible due to the ambiguity of objects and the complexity of scenarios.
However, 4D data with the sequence of image frames and point clouds provide enough cues to discover the movable objects, even without manual annotation.
The complementary information behind the 3D LiDAR points and 2D image and constraints from temporal are the critical factors for the success of unsupervised object discovery.
With 4D sensor data readily available onboard, our approach shows extraordinary potential for scenarios with limited or no annotation.
The only limitation is that our method is suitable for movable objects (vehicles, pedestrians); static things (never move) like beds or chairs can not be discovered.
Conclusions.
In this work, we propose a new task named 4D Unsupervised Object Discovery .
The task needs to discover objects both on the image and point clouds without manual annotations.
We present the ﬁrst practical approach for this task by proposing a ClusterNet for 3D instance segmentation and joint iterative optimization.
Extensive experiments on the large-scale Waymo Open
Dataset demonstrate the effectiveness of our approach.
So far as we know, we are the ﬁrst work to achieve such high performance for unsupervised 2D object detection and 3D instance segmentation, bridging the gap between unsupervised methods and supervised methods.
Our work sheds light on a new perspective on the future study of unsupervised object discovery.
Societal Impacts.
The development of unsupervised object discovery requires large datasets, introducing privacy issues.
The technology of unsupervised detection dramatically reduces the labelling cost; it may affect the people engaged in the labelling industry in the future.
The elimination of human intervention may also cause some data annotators to lose their current jobs.
Our approach only tests in driving scenes for effectiveness, which may lead to some wrong detection in other scenes.
Acknowledgments and Disclosure of Funding
The authors thank the anonymous reviewers for their constructive comments.
This work was supported in part by the Major Project for New Generation of AI (No.2018AAA0100400), the National Natural
Science Foundation of China (No. 61836014, No.
U21B2042, No. 62072457, No. 62006231), and the InnoHK program.
The authors would like to thank Xizhou Zhu and Jifeng Dai for conceiving an early idea of this work.
Also, our sincere and hearty appreciations go to Jiawei He, Lue Fan and Yuxi
Wang, who polishes our paper and offers many valuable suggestions.