Fully Self-Supervised Class Awareness in Dense Object Descriptors
Anonymous Author(s)
AfﬁliationAddress email
Abstract:
In this paper we address the problem of inferring self-supervised densesemantic correspondences between objects in multi-object scenes.
The methodintroduces learning of class-aware dense object descriptors by providing eitherunsupervised discrete labels or conﬁdence in object similarities.
We quantitatively and qualitatively show that the introduced method outperforms previoustechniques with more robust pixel-to-pixel matches.
We also show an examplerobotic application - grasping of objects in clutter based on corresponding points.
Introduction
Self-supervised methods allow models to train with no manual labelling, thus improving the abilityto scale and learn from more data.
In robotic applications, they could potentially allow a robotto learn continuously ‘in the wild’.
Robots of the future would beneﬁt from the ability to learnhow to interact with many objects, including unseen novel ones.
One particular problem is ﬁndingaccurate pixel correspondences between similar objects in such a way that a robot agent recognisesthe corresponding parts between two items and manipulates them in similar ways.
A recent work producing self-supervised object descriptors, namely Dense Object Nets [1] (DONs),has been applied as a representation learner for visuomotor policies [2, 3] improving onto previousend-to-end methods [4].
While DONs produce excellent point correspondences between similar objects, without access to ground truth classes they fail to ﬁnd robust matches in multi-object scenesand produce false positives.
In this work, we propose a method for class-aware Dense Object Network and show that it leads to more informative descriptors that are better suited for correspondencegeneration in multi-object scenes.
We present two variants exploring different ways of representingthe object classes: “hard” discrete labels and “soft” labels, reﬂecting our conﬁdence in the similaritybetween the objects.
We demonstrate that our method is better than the state-of-the-art in ﬁnding correspondences in multi-object images, while simultaneously improving performance of on single- object images.
A real example application is shown using a mobile manipulator.
This method could enable future work on more complex tasks such as unsupervised learning of category-level affor- dances and manipulation of objects in clutter
Related Work
In this section we brieﬂy overview recent advances in descriptor learning.
We then review methods for unsupervised classiﬁcation and class disentanglement.
Our work focuses on self-supervisedlearning of descriptor representations, such that the descriptors of similar points belonging to similarobject are consistent across time, view, and object instance.
Our method builds upon Dense Object
Nets (DONs) [1] which makes use of 2D pixel to 3D point projections as a supervisory signal.
Themethod is described in more detail in Sec. 3.
Most other works on descriptor learning such as [5, 6, 7]make use of fully- or weakly-supervised learning requiring large amounts of human-annotated dataand do not make efﬁcient use of depth data, which robots usually have access to.
Several methods relieve the need for annotations by using 3D CAD models of the observed objects [8, 9], however thisis not applicable when encountering novel objects.
Many works make use of spatio-temporal consistency across video frames, using tracking trajectories or multiple modalities: [10] learns uniquedescriptors via optical ﬂow contrasive learning from dynamic data.
Grasp2Vec [11] learn similarfeatures through autonomous robot intereaction.
The semi-supervised S3K [12] uses multi-viewconsistency to produce keypoints with high accuracy.
With only few ground truth examples, it islearns 3D keypoints by estimating the projections of 2D keypoints.
It proves to be useful for applications such as cable plug-in, where keypoints do not have to be bound to a physical location on theobject, as long as they are consistent across views.
However, it does not produce dense correspondences.
We are not aware of any work that produces pixel-pixel correspondences across or betweenmulti-object scenes, based on unsupervised or self-supervised training without ground truth modelsor per-image labels, which is introduced in here.
For our method to work, we require a way of assessing the similarity or dissimilarity between objects and disentangling their representations.
This can be done by assigning some discrete classlabels to each object or measuring our conﬁdence in the similarity of object pairs.
Most recentworks use unsupervised clustering techniques such as mean-shift and K-Means in combination withembeddings [13, 14, 15].
Patten et al.
[16] improve on previous approaches by considering geometric similarity and experience queries.
Object-Contrastive Network [17] ﬁnds object candidates via
Faster-RCNN, embeds them and then samples triplets based on nearest neighbours in embeddingspace.
Recently, Tan et al.
[18] showed that clustering of object embeddings alone is not enough.
Instead, they propose a method for similarity measurement and triplet sampling via random walkergraphs.
Their method demonstrates state-of-the-art results on various benchmarks.
We use it as aninspiration to our method and provide further details in Sec. 4.2.1.
Dense Object Nets
Dense Object Nets [1] which are largely based on Schmidt et al.
[19] are the foundation of ourmethod.
In this section we explain how they work and identify their current limitations.
DONslearn a dense descriptor representation of RGB inputs that is applicable to both rigid and non-rigidobjects.
It uses the strong prior of depth information taken from an RGB-D camera, and knowledgeof a robot manipulator’s pose to project 2D image pixels onto a reconstructed 3D model, thus learning correspondences between different views of the same 3D points.
In particular, this is done byhaving a robot point a camera towards an object and move around to capture it from different views.
In Siamese fashion, the model is fed pairs of augmented images from which pixels are randomlysampled.
Augmentations consist of random background, color jitter, crops, and scale.
Pixels corresponding to the same 3D point are considered ‘matches’ while any other random pair of pixels areconsidered ‘non-matches’.
The loss is the sum of match and non-match losses deﬁned as follows:
[Equation]
[Equation]
where d a and d b are the output descriptors at the given pixel pair and N + is the total number of matchpairs.
Similarly, N − is the total number of non-matches.
M is a margin hyperparameter determiningthe minimum distance between non-match pairs - if a non-match pair is closer than M , the loss isincreased.
While L match encourages true pairs to be similar, L non − match ensures the pixels don’t all converge to the same descriptor.
Since the robot has no knowledge of the object’s class, the learneddescriptors tend to share the same descriptor space for all objects.
While the original work showsthat when the ground truth class is known, the descriptor subsets can be distinctly separated, it doesnot propose how to implement this and remove the use of ground truth.
Method
The method presented in this section tries to solve the problem of disentangling the descriptor spacesshared by DON descriptors by including class awareness.
[Figure]
4.1 Problem Setup
[Figure]
The inputs used for training are a set of n sequences S
= { S 1 , S 2 ,..., S n } , where each sequencecontains timed data for a single object S i
=
{ O ( i ) 1 , O ( i ) 2 ,..., O ( i ) t } .
The data O ( i ) t for timepoint t andobject ( i ) , consists of a camera pose, full RGB and Depth image and object mask.
A 3D TSDFreconstruction is also generated for the whole sequence.
See Fig.
In this paper, each trainingsequence strictly contains a single instance of a single object, while during inference we can workwith both single and multi-object sequences.
We discuss how the method could be adapted to trainfrom multi-object sequences in Sec. 6.
We refer to each distinct object as an instance class and itsgrouping as a category , e.g. different hats would belong to different instance classes but the samecategory.
Given S we want to learn visual descriptors that are robust over different objects
[Figure]
To disentangle the descriptor subsets used by different objects, most straightforward is to ﬁnd nonmatch pairs between them and learn to repel them.
We considered two variants: (i) DON+Hardproducing discrete “hard” labels by assigning each instance to a cluster or (ii) DON+Soft usingsimilarity-based triplet sampling and continuous “soft” conﬁdence scores.
Both variants use thesame DON model for dense correspondence, and similarity graph for object disentanglement, differing mainly in data ﬂow, sampling and loss generation, as described below (see Fig. 3).
4.2.1 Similarity Graph
Since we assume access object sequences but not their class labels, the method requires a way ofunderstanding each object’s identity or class grouping in a different way.
Here we propose objectdisentanglement based on a recent work [18] which consists of a similarity graph (SG) and randomwalker sampling (RWS).
SG is an undirected graph with each node representing a training sequenceand each edge the similarity between the node pair.
Thus, given any node in the SG, we can ﬁndwhich other nodes are most probably similar to it.
RWS is the sampling of node pairs by “walking”on edges with transitional probability proportional to the edge weights.
The similarity graph is undirected, with each training sequence representing a node S i .
The edgesbetween each pair of nodes is determined by similarity weights W ( S i , S j ) which are calculated usinga similarity and dissimilarity measure, W + and W − , respectively.
Each RGB frame from eachsequence is pre-processed via a pre-trained ResNet before being used for edge weight calculation.
Other inputs were also considered in Sec. 5.
The frames of each sequence are clustered into N l local intra-sequence groupings via K-Means.
Since the sequence contains only a single object, these clusters tend to contain different view pointsof that object.
This solves the over-representation issue that some sequences may suffer from, e.g. ifone the views is prevalent over others.
Each sequence S i then has l corresponding cluster centroids
C l , i .
The centroids have the same dimensions as the input features.
Through minimum linear sumassignment [20] with assignment matrix X , the minimal cluster distance sum is found and used as adissimilarity measure (See Eq. 3).
In our version of this weight, it is normalized by the number ofpairs N p and the number of dimensions N d .
[Equation]
s.t.
Each cluster being assigned to at most 1 pair
[Equation]
Similarly, all frames from all sequences are clustered together in N g global inter-sequence clustersvia K-Means.
By using a large N g , the frequency at which two objects share the same clusters isindicative of how similar those objects are, shaping the similarity measure W + , as in [18].
Finally, the two measures are combined along with a tuning parameter λ which determines theimportance of each measure over the other (see Eq. 4).
The ﬁnal weight is strictly non-negative.
[Equation]
Given a starting node in the graph, a transition probability distribution p t is formed proportional tothe connected edge weights.
By sampling a transition from p t , a ‘random walk’ is done to a node thatis more likely similar.
Similarly, sampling from ( 1 − p t ) is used to reach more probably dissimilarnodes.
We propose a dissimilarity conﬁdence c in the pairing equal to the min-max normalized W − .
This is a replacement of the conﬁdence proposed in [18] which was found to be very sensitive to λ .
4.2.2 Dense Object Net
For DONs we use a 34 layer, 8-stride ResNet architecture.
Its output descriptor image is upsampledto the original input size and has shape ( H , W , D desc ) , where H and W are the input image heightand width, and D desc is the number of descriptor dimensions.
Hard Classiﬁcation: As in [18], for DON+Hard the triplets (anchor A , positive P , and negative
N ) of sample frames are generated using the similarity graph and fed into a projection networkcomprised of two fully connected layers with ReLU activations.
The projection network is trainedvia triplet loss scaled by our conﬁdence c and margin M .
The loss is made non-negative as shown:
[Equation]
The resulting projection output is a continuous latent class representation.
Finally, the representations of the training data are grouped into K clusters, determining the discrete classes.
In inferencetime, assuming the input belongs to one of the existing classes, it can be projected and matched tothe closest cluster.
We denote this as training phase one.
In the second phase, the DON iteratively accepts two kinds of image pairs as inputs with equal probability: (i) same sequence pairs that consist of two different frames of the same object from the same sequence and (ii) different sequencepairs that consist of two frames from random different sequences.
In the different sequence scenario,the objects’ discrete classes are inferred by assigning them to one of the K clusters - if their classesare the same, the pair is ignored, otherwise - non-match pixel pairs are generated and used for DONloss.
In the same sequence, both match and non-match pairs are generated.
Soft Classiﬁcation: For the DON+Soft variant, the similarity graph is generated with all availablesequences (training phase one).
A random sequence is sampled and a negative sequence pairing israndomly chosen based on the transition probabilities of the similarity graph.
Two frame samplesare taken from the initial sequence (anchor A and positive P ) and one from the negative sequence,
N , forming a triplet.
The triplet is passed through the DON.
Only non-matches are generated forthe negative pair.
A combination of the DON pixelwise match and non-match losses is used (Seedeﬁnitions in Eq.
1 and Eq. 2).
The negative pair does not have any matches, instead its L non − matchis scaled by the conﬁdence c in the negative pairing, thus if there is strong belief that the pair is fromthe same class, the loss is smaller.
The total loss is deﬁned as follows:
[Equation]
Experiments
With our experiments we want to answer how effective our method is at separating object descriptorsbased on object categories.
In this section, we investigate this both qualitatively and qualitatively.
5.1 Experimental Setup
Training Data: For training we use (i) real data from [1] and (ii) newly produced simulated data(see Fig. 4).
For the simulated data we placed a Gazebo-simulated Franka Emika Panda manipulator equipped with a RealSense RS200 RGB-D camera, and captured objects from the 3D GEMdataset [21] in randomised poses.
A summary of data is presented in Table 1.
Samples of thedatasets are shown in Fig.
Note, that because of the use of background augmentation in DON, theunrealistic scene in the simulation does not affect training.
[Table]
Hyperparameters: Unless otherwise speciﬁed, all trainings are done for 3 , 500 iterations with
D desc = 5 descriptor dimensions.
The learning rate is 10 − 4 , with a decay of 0 .
DON margin parameter M is set to 0 .
For the similarity graph, we used N l = 5 and N g = 300 clusters, and λ = 0 . 1.
For DON+Hard , the projection network is trained for 500 steps (phase one) before proceeding.
5.2 Object Disentanglement Experiments
Setup:
For our method to work, object representations need to be effectively disentangled.
We testthe ability of the full classiﬁcation model (i.e., consisting of the Similarity Graph and Projection
Network, as in DON+Hard ) to work with DON-compatible object sequences.
We consider threeinput types: (i) raw RGB; (ii) output of pre-trained frozen ResNet50; and (iii) masked Descriptor
Image produced by a pre-trained DON.
All inputs have been cropped to ﬁt only the object andinterpolated to a ﬁxed size.
We test both on simulated (C&B) and real (H&S) objects, where eachobject category contains 4 separate instance classes.
We take 600 random images for training of the model and then test its cluster assignment on 2 , 000 random test images of known object instancesfrom unseen sequences.
Inference into clusters is done via K-Means with K = 2 and K = 8 clusterscorresponding to the number of object categories and instance classes respectively.
Note that Kdoes not affect training.
Cluster assignment to ground truth classes is done by minimum linear sumassignment [20].
The model accuracy is then the fraction of images that are correctly assigned.
We also aim to show whether the descriptors produced by our methods are effective in separating theclasses and how they compare to DONs trained with ground truth labels and without any labels.
Themodels are trained on C&B.
The descriptors of 50 random images are produced and 400 on-objectpixels are sampled from each.
These pixels are then projected via TSNE.
For DON+Hard , K is setto 5 class clusters to highlight the difﬁculty of not knowing the exact number of classes.
Results: As can be seen in Table 2, the ResNet features appear far superior over other inputs.
Notonly are they effective in separating the features into categories but also into object instance classes.
Moreover, as it can be seen in Fig.
5, even without any parameter informing the DON+Hard modelthat there are two object categories, we observe a clear separation of two clusters containing onlyobjects from that category.
Our qualitative evaluation demonstrates that the method is effective inseparating objects, having similar objects share similar descriptor spaces.
Comparing it with the
DON+Soft , there is no clear discrete separation between the classes, but there’s a gradual transitionbetween them.
We want to note that this is closer to how we interpret objects ourselves - if twoobjects have some similar features such as a colour, a handle, or other, we notice that similarity,while still detecting the overall difference between the objects.
[Table]
C&B @ K =52.93% 99.20% 96.31%
C&B @
K =22.13% 95.25% 76.23%
H&S @ K =51.00% 97.80 % 85.26%
H&S @ K =
[Table]
5.3 Descriptor Match Accuracy Experiments
Setup:
Given the descriptor at a random point from a query image, its best match in a target descriptor image is found as the one with minimal L 2-norm.
When working in single object scenarios,the original DON ﬁnds good semantic correspondences between semantically corresponding objects.
However, as soon as there are multiple objects in the target image, the quality of the matchesfalls.
To test this, we create new images, with unseen object instances in visually cluttered scenes,either through image collages or real photos (see Fig. 6).
We manually label points on a set of40 single-object images out of H&S (not used for training) and our 20 multi-object images, eachhaving labelled semantically signiﬁcant locations (when not occluded).
For example, this includesdistinctive points such as tip, top and back of a hat.
This results in 432 pixel pairs for multi-objectmatches and 1218 for single-object matches.
The closest descriptors of the labelled points in thesingle-object images are found in query single- or multi-object images.
The distance between thebest match and the labelled point is the error.
We plot the normalised error versus the fraction of images as a cumulative distribution (CDF).
Comparison is done between our models and DON.
Asupper baseline - models trained with ground truth DON+GT (as per [1]) and DON+Soft+GT, while DAISY [22] provide a non-semantic lower baseline.
All trained on H&S.
A DON+Soft trained onthree extra classes (on H&S+Bots) is also compared with DON+Soft.
The quality of matches is evaluated by picking random points on an image and comparing the correspondences on a test image with novel instances of known classes.
Models trained on H&S.
Results: The CDFs is shown in Fig.
It can be seen that the change in performance with increaseof classes is minimal, while there is an even smaller error in single object matches.
As it can also beseen in Fig.
8, DON+Soft outperforms the other methods.
Discussion: As it can be seen both qualitatively and quantitatively, DON+Soft produces more robust correspondences in multi-object scenarios.
Surprisingly there was a slight improvement insingle-object matches.
We believe this could be due to the contrast between objects encouraging thedescriptors to be more speciﬁc and informative.
5.4 Real Robot Experiment
Setup: To showcase the applicability of the model on a real robot we set up the following experiment.
We use a Franka Emika Panda robot arm equipped with a RealSense D435i RGB-D camera.
Firstly, a point is selected on an image of an object from the training set, in this case, a shoe or a hat.
Then the robot ﬁnds the best match within its environment, by querying a DON+Soft trained on
H&S.
The correspondent pixel is projected into 3D space via the depth image and camera-to-worldtransformation.
Finally, the robot produces a grasp for said point.
Note that the objects used in theexperiment are not part of the training set, and the camera used by the robot is different from theones used for the data gathering.
The clutter observed by the robot contains both known objects onwhich the model is trained and novel ones - to check that the model does not confuse the correspondences with either.
The source object, source point on object as well as the clutter conﬁgurationswere varied to ensure robustness.
A total of 20 trials were done.
Results: See Fig.
The correspondences are correctly placed on the queried object during all trials.
One failure occurred when the point found was at the edge of the object - it couldn’t be picked.
Conclusions
Limitations: Our method shows more robust matches than its predecessor in multi-object scenarios.
One failure case we observed is how our method deals with occlusion.
Matches to occluded pointsare found in completely wrong locations which indicates that although the descriptor subsets forsimilar object are similar, the individual distances between descriptors within the subsets are big.
Querying for best-matches does not consider neighbouring descriptors, geometric consistency orother metrics which likely causes some of the other errors in our best-match correspondences.
The
DON in its current form is unable to work with dynamic training scenes.
This could potentially bealleviated with the use of multiple viewpoints or tracking pixels with optical ﬂow.
Moreover, sincethe current system relies on masks of the input sequence objects, it could be extended to multi-objecttraining scenes by adding a semantic segmentation network.
The use of K-Means in DON+Hardleads to computational complexity scaling with number of sequences, as well as dealing with aconstant number of clusters K when the real number of classes is unknown.
In our exploratorywork, to alleviate these issues we tried Mean Shift Clustering, Bayesian Gaussian Mixture Modelsand DBSCAN [23].
However, it was found that the classiﬁcation accuracy on training data droppedsigniﬁcantly, likely because the latent classes are not described by normal distributions.
DON+Softwould be unaffected by this since it purely relies on similarity measures.
Applications: This work enables applications that might treat different classes of objects differently.
Here we provide three pick-and-place examples: (i) given some knowledge base of different objectsand graspable points on them, the method can be used to automatically detect the closest matchpoints even in cluttered multi-object scenes; (ii) or the opposite way, the model could see an object,infer its latent class and match it to the closest of its class clusters, thus generate the best graspingpoints for that class; (iii) alternatively, the model could be used to observe how people interact withobjects, and learn which descriptor-points are suitable grasping points for each self-supervised class.
In terms of more complex applications, we believe this representation could pave the way towardmethods such as self-supervised robot affordance learning and action imitation.
Future Work:
In this paper, we extended an existing method for self-supervised object descriptorsto perform more accurately in multi-object queries and cluttered scenes.
We showcase its improvement over its predecessor both qualitatively and quantitatively.
Further analysis of the method couldexplore how having novel objects vs known objects in the background of a scene affects the performance, i.e. would it be better for the model to be trained on everything it sees, or does performancedegrade once an object has been ‘learned’.
Future work will be focused on making the descriptorsmore useful by grounding them to actionable representations, and extending the training towardtraining in the wild from dynamic unstructured scenes.