OGC: Unsupervised 3D Object Segmentation from Rigid Dynamics of Point Clouds
Anonymous Author(s)
AffiliationAddress email
Abstract
In this paper, we study the problem of 3D object segmentation from raw pointclouds.
Unlike all existing methods which usually require a large amount of humanannotations for full supervision, we propose the first unsupervised method, called
OGC, to simultaneously identify multiple 3D objects in a single forward pass,without needing any type of human annotations.
The key to our approach is tofully leverage the dynamic motion patterns over sequential point clouds as thesupervision signals to automatically discover rigid objects.
Our method consists ofthree major components, 1) the object segmentation network to directly estimatemulti-object masks from a single point cloud frame, 2) the auxiliary self-supervisedscene flow estimator, and 3) our core object geometry consistency component.
Bycarefully designing a series of loss functions, we effectively take into account themulti-object rigid consistency and the object shape invariance in both temporal andspatial scales.
This allows our method to truly discover the object geometry evenin the absence of annotations.
We extensively evaluate our method on four datasets,demonstrating the superior performance for object part instance segmentation andgeneral object segmentation in both indoor and the challenging outdoor scenarios.
Introduction
Identifying 3D objects from point clouds is vital for machines to tackle high-level tasks such asautonomous planning and manipulation in real-world scenarios.
Inspired by the seminal work
PointNet [56], a plethora of sophisticated models [70, 76, 37] have been developed to accuratelydetect and segment individual objects from the sparse and irregular point clouds.
Although thesemethods have achieved excellent performance on a wide range of public datasets, they primarily relyon a huge amount of human annotations for full supervision.
However, it is extremely costly to fullyannotate every objects in point clouds due to the irregularity of data format.
Very recently, a few works start to address 3D object segmentation in the absence of human annotations.
By analysing 3D scene flows from sequential point clouds, Jiang et al .
[28] apply theconventional subspace clustering optimization technique to identify moving objects from raw pointcloud sequences.
With the self-supervised learning of 3D scene flow, SLIM [3] is the first learningbased work to showcase that the set of moving points can be effectively learned as an object againstthe stationary background.
Fundamentally, their design principle shares the key spirit of Gestalttheory [73, 69] developed exactly 100 years ago: the raw sensory data with similar motion are likelyto be organized into an object.
This is indeed true in our real world where solid objects usually havestrong correlation in rigid motions.
However, these methods cannot learn to simultaneously segmentmultiple interested 3D objects from a single point cloud in one go. Motivated by the potential of motion dynamics, this paper aims to design a general neural frameworkto simultaneously segment multiple 3D objects, without requiring any human annotations but the
[Figure]
inherent object dynamics in training.
To achieve this, a naïve approach is to train a neural networkto directly cluster motion vectors into groups from sequential point clouds, which is widely knownas motion segmentation in 2D images [77, 78].
However, such design requires that the input datapoints are sequential in both training and testing phases, and the trained model cannot infer objectsfrom a single point cloud.
Fundamentally, this is because the learned motion segmentation strategiessimply cluster similar motion vectors instead of discriminating the diverse geometry of objects, andtherefore such design is not general enough for real-world applications.
In this regard, we design a new pipeline which takes a single point cloud as input and directlyestimates multiple object masks in a single forward pass.
Without needing any human annotations,our pipeline instead leverages the underlying dynamics of sequential point clouds as supervisionsignals.
In particular, as shown in Figure 1, our architecture consists of three major components:1) an object segmentation network to extract per-point features and estimate all object masks froma single point cloud , as indicated by the orange block; 2) an auxiliary self-supervised network toestimate per-point motion vectors from a pair of point clouds, as indicated by the green block; 3)a series of loss functions to fully utilize the motion dynamics to supervise the object segmentationbackbone, as indicated by the blue block.
For the first two components, we are actually flexibleto adopt any of existing neural feature extractors [57] and self-supervised motion estimators [34].
Nevertheless, the third component is particularly challenging to design, primarily because we need totake into account not only the consistency of diverse dynamics of multiple objects in a sequence, butalso the invariance of object geometry irregardless of different moving patterns.
To tackle this challenge, we introduce three key losses to end-to-end train our object segmentationnetwork from scratch: 1) a multi-object dynamic rigid consistency loss, which aims to evaluate howcoherently all estimated object masks (shapes) can fit the motion via rigid transformations; 2) anobject shape smoothness prior, which regularizes all points of each estimated object to be spatiallycontinual instead of fragmented; 3) an object shape invariance loss, which drives multiple estimatedmasks of a particular object to be invariant given different (augmented) rigid transformations.
Theselosses together force all estimated o bjects’ g eometry to be c onsistent and represented by high-qualitymasks, purely from raw 3D point clouds without any human annotations.
Our method is called OGCand our contributions are:
• We introduce the first unsupervised multi-object segmentation pipeline on single point cloud frames,without needing any human annotations in training or multiple frames as input.
• We design a set of geometry consistency based losses to fully leverage the object rigid dynamics and shape invariance as effective supervision signals.
• We demonstrate promising object segmentation performance on four datasets, showing significantly better results than classical clustering and optimization baselines.
Difference from Scene Flow Estimation: We do not aim to design a new scene flow estimationmethod such as [74, 34, 20, 71].
Instead, we use unsupervised learning based per-point scene flow assupervision signals for single-frame multi-object segmentation.
Difference from Motion Segmentation: We neither aim to segment motion vectors such as [71, 63]which require multiple successive frames as input in both training and testing.
Instead, our networkdirectly estimates object masks from single frames, and therefore is more flexible and general.
Scope:
This paper does not intend to replace fully-supervised approaches because the never-seenstatic objects are unlikely to be discovered due to the lack of supervision signals.
In addition,estimating object categories or non-rigid objects such as articulated buses and semi-truck with trailersis also out of the scope of this paper. [Figure]
Related Works
Fully-supervised 3D Object Segmentation: To identify 3D objects from point clouds, existing fullysupervised solutions can be divided as 1) bounding box based object detection methods [83, 37, 60] or2) mask based instance segmentation pipelines [70, 76, 68].
Thanks to the dense human annotationsand the well developed backbones including projection-based [38, 8, 37], point-based [57, 65, 26]and voxel-based [22, 12] feature extractors, these methods achieve impressive performance on bothindoor and outdoor datasets.
However, manually labelling every object in large-scale point clouds iscostly.
To alleviate this burden, we aim to pioneer 3D object segmentation without human labels.3D Scene Flow Estimation and Motion Segmentation:
Given sequential point clouds, per-point3D motion vectors, also known as scene flow, can be accurately estimated.
Early works focus onfully-supervised scene flow estimation [42, 4, 24, 43, 55, 53, 72], whereas recent methods start toexplore self-supervised motion estimation [50, 74, 34, 46, 81, 39].
Taking the scene flow as input,a number of works [79, 27, 64, 3] aim to group similar motion vectors, and then obtain boundingboxes or masks only for dynamic objects.
Although achieving encouraging results, they either rely onground truth segmentation for supervision or can only segment simple foreground and backgroundobjects, without being able to simultaneously segment multiple objects.
In this paper, we leverage thesuccessful self-supervised scene flow estimator as our auxiliary neural network to provide valuablesupervision signals, so that multiple objects can be identified in a single forward pass.
Unsupervised 2D Object Segmentation: Inspired by the early work AIR [16], a large numberof generative models have been proposed to discover objects from single images without needinghuman annotations, including MONet [6], IODINE [23], Slot-Att [44], etc.
These methods arefurther extended to segment objects from video frames [35, 25, 49, 29, 84, 15].
However, all theseapproaches can only process simple synthetic datasets, and cannot discover objects from complexreal-world images or videos yet.
It is still elusive to apply these ideas on 3D point clouds where 3Dobjects are far more complicated and diverse in terms of geometry.2D Scene Flow Estimation and Motion Segmentation:
Given image sequences, pixel-level 2Dscene flow, also known as optical flow, have been extensively studied in literature [18, 82].
Theestimated flow field can be further grouped as objects [61, 62, 10, 45, 77, 41, 32].
Drawing insightsfrom these works, this paper aims to segment multiple diverse objects in the complex 3D space.
OGC
3.1 Overview
As shown in Figure 2, given a single point cloud P t with N points as input, i .
e ., P t ∈ R N × 3 , whereeach point only has a location { x, y, z } without color for simplicity, the object segmentation networkextracts per-point features and directly reasons a set of object masks, denoted as O t ∈ R N × K , where K is a predefined number of objects that is large enough for a specific dataset.
In particular, we firstlyadopt PointNet++ [57] to extract the per-point local features.
Then we employ Transformer decoders[67] to attend to the point features and yield all object masks in parallel. The whole architecture canbe regarded as a 3D extension of the recent MaskFormer [9] which shows excellent performance inobject segmentation in 2D images.
Thanks to the powerful Transformer module, each inferred objectmask is effectively modeled over the entire point cloud.
In the meantime, we have the corresponding sequence of point clouds for supervision, denoted as{ P t , P t +1 , · · · } .
For simplicity, we only use the first two frames { P t , P t +1 } and feed them intothe auxiliary self-supervised scene flow network , obtaining satisfactory motion vectors for every point in the first point cloud frame, denoted as M t ∈ R N × 3 , where each motion vector representspoint displacement { ∆ x, ∆ y, ∆ z } .
Among the existing self-supervised scene flow methods, wechoose the recent FlowStep3D [34] which shows excellent scene flow estimation in multiple datasets.
Implementation details of above components are in appendix.
To train the object segmentationnetwork from scratch, the key component is the supervision mechanism as discussed below.
3.2 Object Geometry Consistency Losses
Given the input cloud P t and its output object masks O t and motion M t , we introduce the followingobjectives to satisfy the geometry consistency on both frames P t and ( P t + M t ).
Note that, themasks O t are meaningless at the very beginning and need to be optimized appropriately.
(1) Geometry Consistency over Dynamic Object Transformations
From time t to t + 1 , the rigid objects in point cloud frame P t usually exhibit different dynamictransformations which can be described by matrices belonging to SE (3) group.
For the k th object,we firstly retrieve its (soft) binary mask
O tk , and then feed the tuple { P t , P t + M t , O tk } into thedifferentiable weighted-Kabsch algorithm [31, 21], estimating its transformation matrix T k ∈ R 4 × 4 .
In order to drive all raw object masks to be more and more accurate, so as to fully explain thecorresponding motion patterns within all masks, the following dynamic rigid loss is designed tominimize the discrepancy of per-point scene flow between time t and t + 1 .
Formally, it is defined as:
[Equation]
[Equation]
where the operation ◦ represents the rigid transformation of point clouds.
Intuitively, if one inferredobject mask happens to include two sets of points with two different moving directions, the transformed point cloud can be only in favor of one moving direction, thereby resulting in higher errors.
Therefore, the above constraint can push all object masks to fit the dynamic and diverse motionpatterns.
However, here arises a critical issue: a single rigid object may be assigned to multiple masks,i .
e .
oversegmentation.
We alleviate this issue by a simple smoothness regularizer discussed below.
We observe that such rigid constraint concept is also applied in recent scene flow estimation method[20].
However, their objective is to push the scene flow to be consistent given object masks (estimatedby DBSCAN clustering), while our objective is to learn high-quality masks.
[Equation]
[Figure]
Geometry Smoothness Regularization
The primary reason why a single object may be oversegmented is the lack of spatial connectivitybetween individual points.
However, our common observation is that physically neighbouring pointsusually belong to a single object.
In this regard, we simply introduce a geometry smoothnessregularizer.
Particularly, for a specific n th 3D point p n in the point cloud P t , we firstly search Hpoints from its neighbourhood using either KNN or spherical querying methods, and then force theirmask assignments to be consistent with the center point p n .
Mathmatically, it is defined as:
[Equation]
where o p n ∈ R 1 × K represents the object assignment of center point p n , and o p hn ∈ R 1 × K representsthe object assignment of its h th neighbouring point.
The distance function d () is flexible to choose thesimple L 1 and L 2 or a more aggressive cross-entropy function.
In fact, we may also consider the motion similarity between the centering point and its H neighbouring points as relative scores to weightour smoothness loss, e .
g ., ℓ ′ smooth =
[Equation] N (cid:80) [Equation] ) (cid:17)where m p n , m p
hn represent point motions.
However, we empirically find it is not robust due to theinaccuracy of estimated motions.
Note that, such local smoothness prior is successfully used for sceneflow estimation [42, 34].
Here, we instead demonstrate its effectiveness for object segmentation.
(3) Geometry Invariance over Scene Transformations
With the above geometry constraints designed in (1)(2), the shapes of dynamic objects can bereasonably segmented.
However, the learned object geometry may not be general enough.
For example, a moving car can be well segmented, yet another similar parked car may not be discovered.
To this end, we introduce an object geometry invariance constraint as follows:
• Firstly, given P t , we apply two transformations to get augmented point clouds P tv 1 and P tv 2 .
• Secondly, we feed P tv 1 and P tv 2 into our object segmentation network, obtaining two sets ofobject masks O tv 1 and O tv 2 .
Because the per-point locations in two point clouds are transformed
differently, the position sensitive PointNet++ [57] features generate two different sets of masks.
• Thirdly, we leverage the Hungarian algorithm [36] to one-one match the individual masks in O tv and O tv 2 according to the object pair-wise IoU scores.
Basically, this is to address the issue that
there is no fixed order for predicted object masks from the two augmented point clouds.
• At last, we reorder the masks in O tv 2 to align with O tv 1 , and design the invariance loss as follows.
[Equation]
where ˆ o nv 1 and ˆ o nv 2 are the reordered object assignments of the two augmented point clouds for aspecific n th point.
The distance function ˆ d () is flexible to use L 1 , L 2 or cross-entropy.
Ultimately,this loss drives the estimated object masks to be invariant with different views of input point clouds.
Notably, unlike existing self-supervised learning [11] which usually uses invariance prior for betterlatent representations, here we aim to generalize the segmentation strategy to similar yet static objects.
3.3 Iterative Optimization of Object Segmentation and Motion Estimation
With the designed geometry consistency loss functions, the object segmentation network is optimizedfrom scratch by the combined loss: ℓ seg =
ℓ dynamic + ℓ smooth + ℓ invariant .
For efficiency, theauxiliary self-supervised scene flow network FlowStep3D [34] is independently trained by its ownlosses until convergence.
Intuitively, with better and better object masks estimated, the estimatedscene flow is also expected to be improved further if we use the masks properly.
An ideal solution isto directly pass the estimated object masks into the scene flow network.
However, this is non-trivialbecause we need to either adapt the scene flow network layers or add extra loss functions, and thuswe leave it for our further exploration.
In this paper, we instead propose a following Algorithm 1 toiteratively improve object segmentation and motion estimation.
: Initial scene flow estimation. • Independently and fully train the self-supervised scene flow network on the whole training data split obtain scene
number
Stage 1: Object segmentation optimization.
Train the object segmentation network using ℓ seg for a total E epochs on the whole training split: { ( P t , P t +1 , M t ) 1 · · · ( P t , P t +1 , M t ) S } .
• Estimate reasonable object masks: { ( P t , P t +1 , M t , O t ) 1 · · · ( P t , P t +1 , M t , O t ) S } .
Stage 2: Scene flow improvement.
• For each pair of data ( P t , P t +1 , M t , O t ) , by drawing insights from the classical ICP [2], we propose an object aware ICP algorithm to estimate new scene flow ˆ M t for point cloud P t
• Update the new scene flow for next round training: { ( P t , P t +1 , M t ) 1 · · · ( P t , P t +1 , M t ) S } ← { ( P t , P t +1 , ˆ M t ) 1 · · · ( P t , P t +1 , ˆ M t ) S }
Empirically, setting the total number of rounds R to be 2 or 3 has a good trade off between accuracy and efficiency.
Due to the limited space, details of object-aware ICP algorithm is in appendix.
3.4 Implementation
Our object segmentation network is implemented in PyTorch and Adam optimizer with a learningrate of 0.001 is adopted.
All experiments are conducted on a single RTX3090 GPU.
For the geometrysmoothness regularization, the choice of neighbouring points H is specified in appendix for differentdatasets.
For the geometry invariance loss, the augmentation method is also detailed in appendix. [Table]
Experiments
Our method is evaluated on four different scenarios: 1) part instance segmentation of articulatedobjects on SAPIEN dataset [75], 2) object segmentation of indoor scenes on our own synthetic dataset,3) object segmentation of real-world outdoor scenes on KITTI datasets [48] and 4) the large-scale SemanticKITTI dataset [5].
For evaluation metrics, we follow [51] and report the F1-score , Precision ,and Recall with an IoU threshold of 0.5.
In addition, we also report the Average Precision ( AP )score following COCO dataset [40] and the Panoptic Quality ( PQ ) score defined in [33].
The mean Intersection over Union ( mIoU ) score and the Rand Index ( RI ) score implemented in [27] are alsoincluded.
Note that, all metrics are computed in a class-agnostic manner.
4.1 Evaluation on SAPIEN Dataset
The SAPIEN dataset [75] provides 720 simulated articulated objects with part instance level annotations.
Each object has 4 sequential scans.
The part instances have different articulating (moving)states.
We follow [27] to use the training data generated from [80].
In particular, there are 82092pairs of point clouds for training, 2880 single point cloud frames for testing.
Each point cloud isdownsampled to 512 points in both training and testing.
Since there is no existing unsupervised method for multi-object segmentation on 3D point clouds, wefirstly implement two classical clustering methods: WardLinkage [30] and DBSCAN [17] to directlygroup 3D points from single point clouds into objects.
Secondly, we implement two classical motionsegmentation methods: TrajAffn [52] and SSC [51].
Note that, these two methods take the sameestimated scene flows of FlowStep3D as input, while our method uses the estimated scene flowsduring training only but takes single point clouds during testing.
In addition, we also include theexcellent results of several fully-supervised methods (PointNet++ [57], MeteorNet [43], DeepPart[79]) reported in MBS [27].
Their experimental details can be found in MBS [27].
Lastly, we trainour object segmentation network using single point clouds with full annotations, denoted as OGC sup .
All implementation details are in appendix.
Analysis: As shown in Table 1, our OGC surpasses the classical clustering based and motionsegmentation methods by large margins on all metrics, showing the advantage of our method in fullyleveraging both the motion patterns and various types of geometry consistency.
Compared with thefully supervised baselines, our method is only inferior to the strong MBS [27] and OGC sup .
However,we observe that our OGC actually shows a higher precision score than OGC sup , primarily becauseour method tends to learn better objectness thanks to the additional motion pattern constraints andavoid dividing a single object into pieces.
Figure 3 shows qualitative results.
4.2 Evaluation on Indoor Scene Dataset
We further evaluate our method to segment objects in indoor 3D scenes.
Considering that the existingdataset FlyingThings3D [47] tends to have unrealistically cluttered scenes with severely fragmentedobjects and it is originally introduced for scene flow estimation, we turn to synthesize a new dynamicroom dataset, called OGC-DR , that suits both scene flow estimation and object segmentation.
Inparticular, we follow [54] to randomly place 4 ∼ 8 objects belonging to 7 classes of ShapeNet [7] {chair, table, lamp, sofa, cabinet, bench, display} into each room.
In total, we create 3750, 250, and1000 indoor rooms (scenes) for training/validation/test splits.
In each scene, we create rigid dynamicsby applying continuous random transformations to each object and record 4 sequential frames forevaluation.
Each point cloud frame is downsampled to 2048 points.
Note that, we follow [13] to splitdifferent object instances for train/val/test sets.
More details of the dataset are in appendix and thisdataset will be released.
Analysis: As shown in Table 2, our method outperforms all classical unsupervised methods includingthe clustering based and the motion segmentation based methods.
Since our synthetic rooms all havecomplete 3D objects, and the generated point cloud sequences are of high quality.
Therefore, theperformance gap between our OGC and the supervised OGC sup is nearly closed.
This shows that therigid dynamic motions can indeed provide sufficient supervision signals, if not more than the humanannotations, to identify objects.
Figure 3 shows qualitative results. [Table]
WardLinkage [
[Table]
[Table]
4.3 Evaluation on KITTI Benchmarks
We additionally evaluate our method on the challenging real-world outdoor dataset.
In particularly,we conduct the following two groups of experiments on two benchmarks of KITTI dataset.
• Group 1: Officially, KITTI-SF (Scene Flow) dataset [48] consists of 200 (training) pairs of pointclouds from real-world traffic scenes and an online hidden test for scene flow estimation.
Inour experiment, we train our pipeline on the first 100 pairs of point clouds, and then test on theremaining 100 pairs (200 single point clouds).
We observe that in the 100 training pairs, themoving objects are only cars and trucks.
Therefore, in the testing phase, we only keep the humanannotations [1] of cars and trucks in every single frame to compute the scores.
All other objectsare treated as part of background.
Note that, the whole background is not ignored, but counted asone object in our evaluation, and the cars and trucks can be static or moving.
We find KITTI-SF istoo challeging for the classical unsupervised methods, due to the extreme imbalance of 3D pointsbetween objects and background.
Besides, the background and objects in KITTI-SF are alwaysconnected because of the Earth’s gravity, while clustering-based WardLinkage and DBSCAN favorspatially separated objects. Therefore, we leverage the prior about ground planes in KITTI-SF toassist these methods. We detect and specially handle the ground planes, leaving above-groundpoints only for these methods to handle. Implementation details are in appendix.• Group 2:
Given our well trained model in Group 1, we directly test it on the popular single framebased KITTI 3D Object Detection benchmark [19].
In this benchmark, there are officially 3712point clouds for training, 3769 for validation.
Again, we only keep the ground truth object masksobtained from the bounding boxes for the car category in each frame.
All other objects are treatedas part of background.
For comparison, we download the official pretrained models of threefully-supervised methods PointRCNN [59], PV-RCNN [
] and Voxel-RCNN [14] to directly teston the validation split using the same settings as ours.
In addition, we also directly use the welltrained OGC sup in Group 1 to directly test for comparison, denoted as OGC* sup .
At last, we trainthe OGC sup on the training split (3712 frames) from scratch and test it on the remaining 3769frames using the same evaluation settings, denoted as OGC sup .
Analysis: As shown in Table 3, our method obtains superior segmentation scores on the KITTI-SFdataset, being very close to our fully-supervised counterpart OGC sup .
As shown in Table 4, ourmethod can directly generalize to the single frame based 3D object segmentation with satisfactoryresults, also being close to the fully-supervised counterpart OGC* sup .
It is understandable that the
[Table] [Table]
other three fully-supervised models have a clear advantage over ours, because they are fully trainedon the similar training split while ours does not.
We hope that our method can serve the first baselineand inspire more advanced unsupervised methods in the future to close the gap.
Figure 3 showsqualitative results.
4.4 Evaluation on SemanticKITTI Dataset
We additionally evaluate our method on SemanticKITTI [5], another challenging large-scale outdoor
Lidar dataset.
Similar to the evaluation on KITTI-Det in Group 2 of Section 4.3, we directly use thewell-trained model of KITTI-SF to test single frames in SemanticKITTI.
Only ground truth objects ofcar and truck categories are kept.
The total 11 training sequences with 23201 point cloud frames areused.
Compared with KITTI-Det, SemanticKITTI holds ∼ 6 × larger testing frames and covers morediverse scenes.
As shown in Table 5, our method generalizes to SemanticKITTI with satisfactoryresults, also being close to the fully-supervised baseline.
Figure 3 shows qualitative results.
[Table]
4.5 Ablation Study
[Table]
{+Full OGC 49.
+}
[Table]
[Table] [Table] (1) Geometry Consistency Losses: Tovalidate the choice of our design, wefirstly conduct three groups of ablativeexperiments on the SAPIEN dataset [75]: 1) only remove the dynamic rigid lossℓ dynamic , 2) only remove the smoothness loss ℓ smooth ,
and only
remove the invariance loss ℓ invariant As shownin Table 6, combining the proposed three losses together gives the highest segmentation scores.
Basically, the dynamic rigid loss serves to discriminate multiple objects from different motion patterns.
Without it, the network tends to assign all points to a single object as the shortcut to minimize theother two losses.
However, we observe in Table 6 that without ℓ dynamic , the network still works to some extend.
This is because the synthetic SAPIEN dataset tends to have a number of point cloudframes with only 2 or 3 objects, thus assigning all points to a single object can still get plausiblescores.
This issue is further validated by conducting additional ablation experiments on curated
SAPIEN dataset.
More details are in appendix.
In addition, we evaluate the robustness of our object segmentation method with regard to differenttypes of motion estimations, and different choices of the hyperparameter H in our smoothness loss. More results are in appendix.
[Table]
) Iterative Optimization Algorithm:
We also conduct ablative experiments tovalidate the effectiveness of our proposed
Algorithm 1.
We set the number of iterative rounds R as { 1 , 2 , 3 } .
As shownin Table 7, after 2 rounds, satisfactorysegmentation results can be achieved,although we expect better results aftermore rounds with longer training time.
This shows that our iterative optimization algorithm canindeed fully leverage the mutual benefits between object segmentation and motion {+estimation.4.
+}
Pushing the Boundaries of Unsupervised Scene Flow Estimation and Segmentation
[Table]
In addition to the improvement of object segmentation from our iterative optimization algorithm, the scene flow estimation can be naturally further improvedfrom our estimated object masks as well.
Given our well trained model on the
KITTI-SF dataset in Section 4.3, we usethe estimated object masks to further improved the scene flow estimation.
As shown in Table 8, following the exact evaluation settingsof FlowStep3D [34], our method, not surprisingly, significantly boosts the scene flow accuracy,surpassing the state-of-the-art unsupervised FlowStep3D [34] and other baselines in all metrics.
[Table]
In fact, our object segmentation backbone is also flexible to take the sceneflow as input instead of point xyz to segment objects.
This is commonly calledmotion segmentation.
We replace thesingle point clouds by (estimated) sceneflow vectors as our network inputs, and train the network from scratch using the same settings on the
KITTI-SF dataset.
As shown in Table 9, we can see that our network can still achieve superior resultsregardless of the modality of inputs, demonstrating the generality of our framework.
Conclusion
In this paper, we demonstrate for the first time that 3D objects can be accurately segmented usingan unsupervised method from raw point clouds.
Unlike the existing approaches which usuallyrely on a large amount of human annotations of every 3D object for training networks, we insteadturn to leverage the diverse motion patterns over sequential point clouds as supervision signalsto automatically discover the objectness from single point clouds.
A series of loss functions aredesigned to preserve the object geometry consistency over spatial and temporal scales.
Extensiveexperiments over multiple datasets including the extremely challenging outdoor scenes demonstratethe effectiveness of our method.
One limitation of our method is the lack of capability to discoverunseen object types, and we leave it for our future exploration.
Broader Impact:
The proposed OGC learns 3D objects from raw point clouds without requiringhuman annotations for supervision.
We showcase the effectiveness for some basic applicationsincluding object part instance segmentation, indoor object segmentation and outdoor moving vehicleidentification.
We also believe that our method can be general for other domains such as AR/VR.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeSection 5(d)
Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running exper-
iments multiple times)?
[No] Error bars are not reported because it would be too
computationally expensive
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 3.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
[Table] (e) Did you discuss whether the data you are using/curating contains personally identifiable [Table]
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table] (a) Did you include the full text of instructions given to participants and screenshots, if [Table]
applicable?
[N/A]
[Table] (b) Did you describe any potential participant risks, with links to Institutional Review [Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table] (c) Did you include the estimated hourly wage paid to participants and the total amount [Table]
spent on participant compensation?
[N/A]
[Table]