Towards Scale Balanced 6-DoF Grasp Detection in Cluttered Scenes
Abstract:
In this paper, we focus on the problem of feature learning in the presence of scale imbalance for 6-DoF grasp detection and propose a novel approach to especially address the difﬁculty in dealing with small-scale samples.
A Multiscale Cylinder Grouping (MsCG) module is presented to enhance local geometry representation by combining multi-scale cylinder features and global context.
Moreover, a Scale Balanced Learning (SBL) loss and an Object Balanced
Sampling (OBS) strategy are designed, where SBL enlarges the gradients of the samples whose scales are in low frequency by apriori weights while OBS captures more points on small-scale objects with the help of an auxiliary segmentation network.
They alleviate the inﬂuence of the uneven distribution of grasp scales in training and inference respectively.
In addition, Noisy-clean Mix (NcM) data augmentation is introduced to facilitate training, aiming to bridge the domain gap between synthetic and raw scenes in an efﬁcient way by generating more data which mix them into single ones at instance-level.
Extensive experiments are conducted on the GraspNet-1Billion benchmark and competitive results are reached with signiﬁcant gains on small-scale cases.
Besides, the performance of real-world grasping highlights its generalization ability.
Our code is available at https://github.com/mahaoxiang822/Scale-Balanced-Grasp .
Introduction
Grasping is a fundamental task in robotic manipulation, and grasp detection aims to generate rich gripper conﬁgurations on objects for stable lift.
It is expected to deal with objects varying in shape, size, appearance, material, pose etc .
Traditional methods usually work with 3D models of objects and policies are manually designed [1, 2, 3].
However, these methods highly rely on the accuracy of pose estimation and are not applicable to unseen objects, limiting their use in real-world scenarios.
With the innovation of depth sensors and the development of deep learning, data-driven methods have been greatly advanced [4, 5, 6], offering the advantage in generalization.
More recently, several studies have extended 6-DoF grasp detection to a more difﬁcult case, investigating diverse grasping poses for objects in clutter [7, 8, 9].
Although such attempts deliver promising results, they incline to large- and medium-scale objects or parts.
There exist quite a number of hard examples.
As shown in Fig.
1 (a), there are many qualiﬁed grasps detected on the large red box in (1), but very few for the bolt in (2) and the fork in (3).
For the bottle in (4), there exist several valid grasps on the body while none is on the red head.
Indeed, grasping on small objects ( e.g. the bolt in (2) and the fork in (3) in Fig.
1 (a)) or small parts of large- and medium-scale objects ( e.g. the red head of the bottle in (4) in Fig.
1 (a)) is still in low precision and recall.
Here, we formally deﬁne the scale of a grasp as the minimal width between two ﬁngers of the gripper, which is determined by the pose of the grasp and the shape of the region.
To illustrate the correlation between the grasp scale and quality, we visualize the scale distribution of the objects and their corresponding grasp performance by the baseline method on the GraspNet-
1Billion benchmark [8].
As depicted in Fig.
1 (b), current grasp detection method performs well on the objects which seldom have small-scale ground truth grasps.
On the contrary, for the objects with a large proportion of small-scale ground truth grasps, the baseline method tends to fail.
[Figure]
It is really challenging to improve the results of such hard cases towards scale-balanced grasp detection.
Firstly , for a local region, the ambiguity of its geometry increases when its size becomes smaller since information conveyed reduces.
[8, 10, 11] extract local geometry features through speciﬁc modules, but due to ﬁxed receptive ﬁelds, they are not so competent to distinguish similar but different shapes.
Secondly , the scale distribution of grasps for daily objects in cluttered scenes is imbalanced.
Large- and medium-scale grasps dominate and thus suppress the learning on small-scale ones.
To the best of our knowledge, this issue has not been investigated in 6-DoF grasp detection before.
Thirdly , the noise of mainstream depth sensors ( e.g. Intel RealSense and MS Kinect) is serious, which degrades shape description, in particular for small regions.
[12, 13] provide tentative solutions.
Unfortunately, the former applies point completion, mainly working on seen objects because completion of missing regions of unknown objects is confusing without prior knowledge [14], while the latter conducts denoising, which requires rendering raw data by adding noise of different patterns to clean point-clouds for training models, struggling in generalizing to the complicated real-world noise from RGB-D sensors.
This paper proposes a novel approach to 6-DoF grasp detection, which addresses the difﬁculties aforementioned in a uniﬁed framework.
Speciﬁcally, a Multi-scale Cylinder Grouping (MsCG) module is introduced, where multi-scale cylinder features are combined to a global feature with additional context embedded for more comprehensive description of local shapes.
Furthermore, a
Scale Balanced Learning (SBL) loss and an Object Balanced Sampling (OBS) strategy are designed for the training and inference phases respectively.
SBL enlarges the gradients of the samples whose scales are in low frequency ( i.e. small) by apriori weights while OBS captures more points on smallscale objects with the help of an auxiliary segmentation network.
They both alleviate the inﬂuence of the uneven distribution of grasp scales.
Additionally, a data augmentation method, namely Noisyclean Mix (NcM), is presented to facilitate training.
It aims to bridge the domain gap between synthetic and raw scenes in an efﬁcient way, by generating more data which mix them into single scenes at instance-level.
In summary, our contributions are as follows: (1) we introduce an MsCG module to enhance the representation of local shapes for grasp detection; (2) we design an SBL training loss and an OBS inference strategy to mitigate the imbalanced scale distribution of grasps in cluttered scenes; (3) we present NcM data augmentation by blending point-clouds with and without noise at instance-level to improve model training; and (4) we carry out extensive simulation and real-world experiments and reach competitive results with signiﬁcant gains on small-scale grasps and a strong generalization ability.
Related Work
Grasping is a long-standing task in robotic manipulation.
Recently, there has been a trend on 6-DoF grasp detection in cluttered scenes.
[7] proposes a pioneering framework with point-cloud input, which ﬁrst samples grasp candidates and then scores them with a Convolutional Neural Network (CNN).
[9] employs a point-based neural network to score grasp candidates for improvements.
Due to the fact that grasp candidate sampling is time-consuming and prone to objects (or parts) of thin shapes, a number of studies [15, 16, 8, 10, 17] handle this task in an end-to-end manner.
[15] introduces a variational auto-encoder to generate grasp poses from point-clouds of single objects.
To directly regress grasping in clutter, [16] builds a single-shot grasp proposal network.
[8] generates diverse grasps for objects in clutter by constructing a discrete grasp pose space and predicting the gripper conﬁgurations progressively.
For more accurate and robust grasps, [17] deﬁnes graspness as the quality metric to determine where to grasp and [10] focuses on capturing local features from points inside grasp regions.
Although the methods above have made large progress, they generally work well on objects (or parts) of large- and medium-scales and have difﬁculty in tackling smallscale ones, leading to the imbalanced grasping detection performance.
As the overwhelming majority of the studies on 6-DoF grasp detection take point-clouds as input and such data often contain noise of depth sensors, there exists another line to ameliorate scene representation.
[18, 19] bring additional clues to complement description where [18] uses multiple frames to map raw input to a Truncated Signed Distance Function (TSDF) to smooth noise and [19] integrates texture maps to deliver multi-modal features.
Besides, [20] introduces implicit representations to model scenes and employs 3D reconstruction as an auxiliary task to reﬁne data.
Such solutions either require more information in inference or highly correlate to the speciﬁc model, suggesting the necessity of a plug and play alternative.
Methodology
The pipeline of the scale balanced grasp detection approach is demonstrated in Fig.
During training, we ﬁrst conduct Noisy-clean Mix (NcM) augmentation, which synthesizes clean pointclouds from CAD models and mixes them with the raw noisy ones at instance-level.
The new pointcloud ∈ R N × 3 is employed as input, where N is the number of points.
After that, M candidate points are sampled by Farthest Point Sampling (FPS) from the input and encoded as seed features ∈ R M × C by a transformer-based point encoder [21].
We predict whether these candidates can be grasped and the approach directions of positive candidates using an approach head.
To regress other gripper conﬁgurations, Multi-scale Cylinder Grouping (MsCG) is built to extract local features for positive candidates from original point-clouds.
An operation head is employed to predict the plane rotation, width and score of the grasp.
During inference, Object Balanced Sampling (OBS) is used to explore more positive samples at different scales, where a 3D instance segmentation network is integrated to predict the 3D masks ∈ (0 , 1) N × K ( K is the number of instances) and new positions ∈ R M × 3 are evenly sampled in each mask.
The seed features of these new samples are interpolated from the initial ones in scene and used for successive grasp conﬁguration prediction.
3.1 Multi-scale Cylinder Grouping
Considering that grasps correspond to detailed local geometries, it is rather rough to choose a ﬁxed
Receptive Field (RF) to represent the grasp region.
As Fig.
3 (a) displays, a single RF tends to incur ambiguity for the small-scale grasp in position A and miss crucial cues for the large-scale one in position B.
To enhance geometric representations for shapes in different sizes, we propose the
Multi-scale Cylinder Grouping (MsCG) module, which captures local features of a position with varying RFs and combines them to deliver a comprehensive description.
Speciﬁcally, we set up several cylinders with different radii and crop related points.
Radii are evenly distributed within the maximum width of the gripper.
Multi-layer Perceptrons (MLPs) and max-pooling are employed to encode the points from different cylinders.
To further improve the semantics, we introduce the corresponding seed features as guidance for local feature fusion at different scales, and a gate module is applied to screen information in them.
Fig.
3 (b) shows the structure of the MsCG module.
With the help of this module, our network achieves more accurate local geometric representations for grasps at different
[Figure]
3.2 Grasp Scale Balancing
Firstly, we give an analysis on the scale distribution of grasps in cluttered scenes on the training set of the GraspNet-1Billion benchmark [8].
1,024 points are sampled by FPS in each scene and for each sample, we search for the grasp with the highest score in the SO (3) space, whose gripper width is considered as the scale of the sample.
Through this, for all scenes in the training set, we draw the grasp scale distribution, as shown in Fig.
It is observed that medium- and large-scale grasps are much more than small-scale ones.
This imbalance issue degrades grasp detection in two aspects.
First, in training, the model is prone to ﬁt medium- and large-scale grasps whose percentage is high and thus suppress the learning of small-scale grasps.
Second, in inference, few small-scale grasps can be sampled by current evenly sampling techniques in the 3D space such as FPS and voxel sampling.
Simply enlarging the number of samples can indeed ease this problem, but the memory cost and run-time both sharply increase, making it impractical.
To address the imbalance issue in training, we design a Scale Balanced Learning (SBL) loss to weight the errors of the samples at different scales inspired by Cost-Sensitive Learning [22], where the weights are calculated based on the frequencies of the classes of samples.
Different from the imbalance in the discrete classiﬁcation problem, that on grasping scales is continuous and we thus divide the max width of the gripper into T bins.
For sample i in the scene, its weight W i is calculated by:
[Equation]
where C i is the number of grasps of the same scale with sample i and C max is the maximum of the
T scales.
For negative samples where no successful grasp is in its SO (3) space, we assign W n = 1 .
With the loss in [8], we add the weight W to the approach and operation heads:
[Equation]
where α is a trade-off hyper-parameter.
Please refer to the supplementary material for more details.
To handle the imbalance issue in inference, we propose an Object Balanced Sampling (OBS) strategy, which adjusts the number of samples on different objects so that sufﬁcient samples are obtained on them.
In this case, we ﬁrst train a 3D instance segmentation network in advance, i.e. a modiﬁed version of [23], only with point-clouds as input to generate masks for objects.
We then uniformly carry out FPS on each foreground mask and each object has MK samples, where M and K are previously deﬁned as the number of total samples and the number of instances respectively.
The features of new sampled positions are linearly interpolated by three nearest neighbors from the original seed features in scene.
With the OBS strategy, we are able to explore more grasp candidates on smallscale objects, beneﬁting small-scale grasp detection.
on GraspNet-1Billion.
3.3 Noisy-clean Mix
The noise of point-clouds affects the accuracy of grasping detection, especially for small-scale cases.
On the one hand, the noise destroys the geometry structure of the grasp, which makes it hard to detect.
The attempts on point-cloud completion or denoising partially solve this problem during inference but numerous unseen objects and complicated noise patterns in the real-world prevent their further use.
On the other hand, the noise degrades the training of grasp detection where the network struggles in learning the mapping from ambiguous shapes to grasp labels.
Compared to larger geometries, the distortions due to noise on small ones are more serious, decreasing the contribution in training.
To overcome this downside, we propose a Noisy-clean Mix (NcM) data augmentation method.
We introduce the clean version of the input point-clouds to increase the number of valid grasp samples.
However, Sim-to-Real raises another challenge when only using synthetic clean point-clouds during training.
Therefore, we mix noisy and clean data into single scenes, which reduces the domain gap between training and inference.
Fig.
5 shows the process of NcM. With 3D models and poses of objects, we synthesize corresponding clean scenes by composing them to full point-clouds.
We use the original point-clouds captured by the depth sensor as noisy scenes.
For the missing parts in noisy scenes caused by camera poses and occlusions, we also remove the points in the same region of the clean point-clouds to keep the consistency.
The mixed input is constructed by randomly replacing the objects in noisy scenes with the clean counterparts, leading to more valid samples to facilitate the learning of grasps.
Experiments 4.1 Benchmark, Baseline and Metric
We conduct all the simulation experiments on GraspNet-1Billion [8], which includes 190 cluttered scenes captured in the real-world by Realsense/Kinect cameras in 256 views.
For each point on the object surface, grasps are annotated densely in its SO (3) space by force closure estimation.
It has nearly 1 billion annotations, offering rich grasps to explore the scale distribution in scene.
The baseline method is an enhanced version of GraspNet-baseline [8], where a transformer-based point encoder and a new graspable loss with a higher threshold are applied for better performance.
For more details about the baseline, please refer to the supplementary material.
To evaluate the grasps detected in cluttered scenes, the precision of the top- k ranked grasps is used.
As in [8], AP µ is computed to represent the average Precision@k for k ranging from 1 to 50 with friction µ , and AP is obtained by the average of AP µ , where µ varies from 0.2 to 1.2.
The original metric does not directly reﬂect the quality of grasps at different scales because the top ranked grasps mostly distribute on objects (or parts) with simple and large geometries.
We thus give a new metric to better evaluate the scale-aware grasping quality in scene.
The max width of the gripper (10cm in benchmark) is divided to three intervals, where widths in 0cm-4cm, 4cm-7cm, and 7cm-10cm are small-scale, medium-scale and large-scale.
With such width masks, we evaluate
AP S , AP M and AP L for grasps in small-, medium- and large-scale respectively.
4.2 Ablation Study
First, we validate the effectiveness of the proposed MsCG module, SBL loss, OBS strategy, and
NcM augmentation method, and the results are listed in Table 1.
All the studies are conducted on the real-world scenes captured by Intel RealSense D435.
We evaluate AP S , AP M and AP L , and calculate the mean to reﬂect the grasp quality at all the scales.
According to Table 1, MsCG achieves signiﬁcant improvements in all the metrics, which shows the importance of enhanced shape representations.
For SBL, the performance of small-scale grasps is boosted by increasing the weights for such samples in training, comparable to that on medium- and large-scale grasps.
NcM augmentation brings additional gains for almost all the settings, which demonstrates its necessity in reducing ambiguities during training.
For fair validation of the advantage of OBS, we keep the same number of samples (1,024 in our experiment) and take Foreground Sampling (FS) as comparison, where we do FPS in foreground point-clouds.
Compared to FS, OBS delivers an improvement of 2.37% for small-scale grasps in the seen set with stable results of medium- and large-scale grasps.
We also note that OBS does not show an improvement on small-scale grasps in the novel set (but still comparable), and it is mainly caused by the inferior performance of the segmentation network for unseen objects.
[Table]
4.3 Analysis on Different Scales
We discuss how the proposed approach inﬂuences the number and success rate of grasps at different scales.
For each object in scene, we select the top-10 ranked grasps for statistics and set the friction µ = 0 .
8 for grasp successful checking.
Results on seen objects are shown in Fig.
For small-scale samples, MsCG, SBL, and OBS all increase the number and success rate of detected grasps.
Note that NcM augmentation reduces the number of small-scale grasps but improves the success rate.
By introducing more valid small-scale samples in learning, our network avoids to generate grasps in the region where the shape is seriously corrupted by noise.
For medium- and large-scale samples, MsCG still shows a signiﬁcant improvement in terms of the success rate.
4.4 Visualization
We visualize some grasps generated by our approach on the objects that the baseline does not perform well.
The results are shown in Fig.
Gripper poses in red are good grasps while those in purple and blue are collision and bad ones respectively.
Objects in (1-3) are small-scale and our approach detects more successful grasps on them.
The scissors in (4) and the bottle in (5) consist of several geometric parts in different sizes.
Beneﬁting from the ability to detect grasps at various scales, our approach delivers diverse grasps for individual parts, helpful to the downstream manipulation task.
4.5 Comparison with the State-of-the-art
We make fair comparison with the state-of-the-art counterparts on the data acquired by the RealSense camera, including [7, 6, 24, 9, 8, 19, 25, 17] and the results are displayed in Table 2.
Model
Seen
Similar
Novel
AP
[Table]
AP
[Table]
AP AP [Table]
AP
[Table] AP
AP
[Table]
AP [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] PointnetGPD
[Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
Although we focus on scale balanced grasp detection in particular tackling small-scale cases, the proposed approach achieves an AP of 63.83% on seen objects, 58.46% on similar objects and 24.63% on novel objects, which outperforms most counterparts.
GSNet [17] shows better performance on seen objects mainly because they design an efﬁcient cascaded graspness model to detect grasps in the position which has more successful grasps in its SO (3) space.
However, GSNet tends to incur negative impact on the diversity of grasps to some extent since grasps in low graspness positions are not taken into consideration.
4.6 Real-world Evaluation
To validate the generalization ability of the proposed approach, we conduct additional real-world experiments.
As shown in Figure 8 (a), the grasping system is built on a 7-DoF Agile Diana-7 robot arm.
30 objects are randomly chosen from the YCB dataset [26], including 10 objects only with grasps of small scales (GSS) (Fig.
8 (b)) and 20 objects with grasps of diverse scales (GDS) (Fig.(c)).
[Figure]
We compare our model to the baseline in two settings: isolated object grasping and cluttered object grasping.
For isolated object grasping, only the GSS objects are considered and each object is placed in three different poses.
Success Rate (SR) is used as the metric.
As shown in Table 3, our model delivers a large improvement, which demonstrates its effectiveness in dealing with grasps of smallscales.
For cluttered object grasping, 5 ∼ 6 objects from both GSS and GDS compose a scene and we make the robot remove them all with a maximum number of operations at 8.
SR and Scene
Completion Rate (SCR) are employed as the metrics.
According to Table 3, our model performs better in both SR and SCR, also indicating its superiority (as the baseline often fails on the GSS objects).
[Table]
[Figure]
Limitations
Despite the advantages, our approach still has two limitations.
First, as in Fig.
9 (a), our model fails on a very complicated object, which is composed of some small geometrical components.
Second, for small-scale ﬂat objects laying in the plane, few grasps are detected, as in Fig.
9 (b) shows.
We speculate that the noise is the major reason in the two cases, which requires more discussion on high-precision sensor selection and efﬁcient data reﬁnement.
Conclusion
In this paper, we work towards scale balanced 6-DoF grasp detection.
An MsCG module is proposed to capture comprehensive local description for grasps at different scales.
To mitigate the imbalanced scale distribution of grasp labels, we design an SBL loss to adjust the weights of grasp samples in training and an OBS strategy to sample more points on small-scale objects in inference.
We also present NcM data augmentation to reduce the inﬂuence of sensor noise.
Extensive experiments show the advantages of our approach in grasp detection, especially for small-scale samples which are not well handled in the previous work.
Acknowledgments
This work is partly supported by the National Natural Science Foundation of China (No. 62022011), the Research Program of State Key Laboratory of Software Development Environment (SKLSDE2021ZX-04), and the Fundamental Research Funds for the Central Universities.