Original: Learning Dense Visual Descriptors using Image Augmentations for Robot Manipulation Tasks
Revised: Learning Dense Visual Descriptors using Image Augmentations for Robot Manipulation Tasks

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract: We propose a self-supervised training approach for learning viewinvariant dense visual descriptors using image augmentations.
Revised: Abstract: We propose a self-supervised training approach for learning viewinvariant dense visual descriptors using image augmentations.

Original: Unlike existingworks, which often require complex datasets, such as registered RGBD sequences,we train on an unordered set of RGB images.
Revised: Unlike existing works, which often require complex datasets, such as registered RGBD sequences, we train on an unordered set of RGB images.

Original: This allows for learning from a single camera view, e.g., in an existing robotic cell with a ﬁx-mounted camera.
Revised: This allows for learning from a single camera view, e.g., in an existing robotic cell with a ﬁx-mounted camera.

Original: Wecreate synthetic views and dense pixel correspondences using data augmentations.
Revised: We create synthetic views and dense pixel correspondences using data augmentations.

Original: We ﬁnd our descriptors are competitive  tothe existing methods, despite the simpler data recording and setup requirements.
Revised: We ﬁnd our approach to be competitive compared to existing methods, despite the simpler data recording and setup requirements.

Original: We show that training on syntheticcorrespondences provides descriptor consistency across a broad range of cameraviews.
Revised: We show that training on synthetic correspondences provides descriptor consistency across a broad range of camera views.

Original: We compare against training with geometric correspondence from multiple views and provide ablation studies.
Revised: We compare against training with geometric correspondence from multiple views and provide ablation studies.

Original: We also show a robotic bin-picking experiment using descriptors learned from a ﬁx-mounted camera for deﬁning grasppreferences.
Revised: We also show a robotic bin-picking experiment using descriptors learned from a ﬁx-mounted camera for deﬁning grasp preferences.

Original:  Introduction
Revised:  Introduction

Original: Scene and object understanding is essential for robot manipulationtasks, including assembly or bin picking.
Revised: Scene and object understanding is essential for robot manipulation tasks, including assembly or bin picking.

Original: Often, the representation ofchoice is task-speciﬁc segmentation or pose estimation, trained in asupervised manner with labeled data.
Revised: Often, the representation of choice is task-speciﬁc segmentation or pose estimation, trained in a supervised manner with labeled data.

Original: Labeling, however, is expensive and time-consuming, which is why self-supervised learning ofdense visual descriptors has recently gained substantial attention inthe robotics community, inspired by the works of Schmidt et al. [1]and Florence et al.
Revised: Labeling, however, is expensive and time-consuming, which is why self-supervised learning of dense visual descriptors has recently gained substantial attention in the robotics community, inspired by the works of [1] and [2].

Original: Dense Object Nets (DONs)proposed by Florence et al. [2] learn densevisual descriptors of objects fully self-supervised in a robotic environment.
Revised: Dense Object Nets (DONs) learn dense visual descriptors of objects fully self-supervised in a robotic environment [2].

Original: The learned descriptors are view-invariant, show potential forwithin-class generalization and they naturally apply to non-rigid objects.
Revised: The learned descriptors are view-invariant, show potential for category level generalization and they naturally apply to non-rigid objects.

Original: The dense descriptor representation can be ﬂexibly used forvarious downstream robotic tasks, such as, grasping (Florence et al.[2], Kupcsik et al. [3], Adrian et al. [4]), rope manipulation (Sundaresan et al. [5]) and learning control (Manuelli et al.
Revised: The dense descriptor representation can be ﬂexibly used for various downstream robotic tasks, such as, grasping [2, 3, 4], rope manipulation [5] and learning control [6, 7].

Original: [6], Florence et al.[7]).

Original: Self-supervised training of DONs, however, relies on pixel correspondences across multiple camera views provided by a registered RGBDimage sequence, which requires accurate camera calibration and poserecording.
Revised: Self-supervised training of DONs, however, relies on pixel correspondences across multiple camera views provided by a registered RGBD image sequence, which requires accurate camera calibration and pose recording.

Original: Furthermore, pixel correspondence tends to be inaccuratewith inexpensive depth cameras, even with data preprocessing.
Revised: Furthermore, pixel correspondence tends to be inaccurate with inexpensive depth cameras, even with data preprocessing.

Original: Finally, data collection is constrained by robot kinematics and the needfor an expert setting up and supervising the procedure.
Revised: Finally, data collection is constrained by robot kinematics and the need for an expert setting up and supervising the procedure.

Original: In this paper we relax these assumptions fundamentally and instead of a complex setup with asingle, robot-mounted moving camera, or multiple static ones, we solely rely on an unordered setof RGB images to learn object descriptors, for example, recorded by a single ﬁxed camera.
Revised: In this paper we relax these assumptions fundamentally and instead of a complex setup with a single, robot-mounted moving camera, or multiple static ones, we solely rely on an unordered set of RGB images to learn object descriptors, for example, recorded by a single ﬁxed camera.

Original: In ourwork, instead of relying on multi-view, geometric correspondence , we use augmentations of singleimages to obtain alternative views and synthetic correspondence .
Revised: In our work, instead of relying on multi-view, geometric correspondence , we use augmentations of single images to obtain alternative views and synthetic correspondence .

Original: This idea was already explored incomputer vision by Thewlis et al. [8] and Novotny et al. [9], in the context of learning geometricallyconsistent pixel-level descriptors across multiple object classes.
Revised: This idea was already explored in computer vision [8, 9], in the context of learning geometrically consistent pixel-level descriptors across multiple object classes.

Original: While our approach does not requirea robot, depth camera, or scene registration, we show that it still achieves competitive performance interms of keypoint tracking accuracy compared to a network trained with geometric correspondence.
Revised: In this paper we show that relying on synthetic image augmentations achieves competitive performance in terms of keypoint tracking accuracy compared to a network trained with geometric correspondence.

Original: Our approach can easily be adopted to existing industrial setups with ﬁx-mounted cameras, or withcameras too heavy to be mounted on a robotic arm, without additional engineering effort.
Revised: Importantly, this approach can easily be adopted to existing industrial setups with ﬁx-mounted cameras, or with cameras too heavy to be mounted on a robotic arm, without additional engineering effort.

Original: We showsuch a robotic bin-picking setup in Fig.
Revised: We show such a robotic bin-picking setup in Fig.

Original: 1, with an overhead, ﬁx-mounted camera.
Revised: 1, with an overhead, ﬁx-mounted camera.

Original: Our contributions are as follows: (i) we combine the approach of learning visual descriptors fromsingle images by Novotny et al.
Revised: Our contributions are as follows: (i) we adapt existing work on training self-supervised pixel embeddings [9, 10] to robotic grasping downstream tasks.

Original: [9] with the image augmentation technique and loss formulation by Chen et al. [10], (ii) we compare learning with synthetic and geometric correspondence on keypointtracking accuracy, and ﬁnally (iii) we show a  robotic bin-picking experiment where human preference on grasp conﬁguration is encoded with dense visual descriptors, with the constraint of using asingle, ﬁxed-mounted camera.
Revised: (ii) We show that for robotic grasping tasks our approach is en par with state-of-the-art [4] in terms of keypoint tracking accuracy while drastically simplifying the data collection. Finally, (iii) we show a real-world robotic bin-picking experiment where human preference on grasp conﬁguration is encoded with dense visual descriptors, with the constraint of using a single, ﬁxed-mounted camera.

Original:  Related work
Revised:  Related work

Original: In the following, we review recent work on self-supervised dense visual descriptor learning forrobotic manipulation in more detail.
Revised: In the following, we review recent work on self-supervised dense visual descriptor learning for robotic manipulation in more detail.

Original: We also discuss related work on self-supervised representationlearning and learning from a set of single images, which are core concepts in our work.
Revised: We also discuss related work on self-supervised representation learning and learning from a set of single images, which are core concepts in our work.

Original: Dense visual descriptors in robotic manipulation.
Revised: Dense visual descriptors in robotic manipulation.

Original: Inspired by the work of Schmidt et al. Florence et al.
Revised: Inspired by [1], Florence et al.

Original: [2] proposed self-supervised training of dense visual descriptors by and for roboticmanipulation.
Revised: [2] proposed self-supervised training of dense visual descriptors by and for robotic manipulation.

Original: Their approach was later adoptedby Florence et al. [7] to learn from multi-viewcorrespondence in dynamic scenes and showed an application for policy learning.
Revised: Their approach was later adopted to learn from multi-view correspondence in dynamic scenes with an application for policy learning [7].

Original: Sundaresan et al.[5] applied the descriptor space representation  to learn challenging rope manipulation in simulations.
Revised: In [5] the descriptor space representation was applied to learn challenging rope manipulation in simulations.

Original: Applyingthe dense descriptor representation to learn model predictive controllers was shown by Manuelli et al.
Revised: Applying dense descriptors for state representation in model predictive control learning was shown in [6].

Original: Vecerik et al. [11]use multi-view consistency for keypoint detection and showan application for  reinforcement learning.
Revised: In [11] multi-view consistency for keypoint learning was proposed with an application for state representation in reinforcement learning.

Original: Another line of work investigated improved training strategies of dense visual descriptors.
Revised: Another line of work investigated improved training strategies of dense visual descriptors.

Original: There aremultiple contributions focusing on learning multi-object and multi-class descriptors by Yang et al.[12], Hadjivelichkov and Kanoulas [13] and Adrian et al.
Revised: There are multiple contributions focusing on learning multi-object and multi-class descriptors [12, 13, 4].

Original: The work by Kupcsik et al.

Original:  [3] exploitsknown object geometry to compute optimal descriptor embeddings.
Revised: The work in [3] exploits known object geometry to compute optimal descriptor embeddings.

Original: Finally, Yen-Chen et al.

Original: [14]utilize NeRF  to generate dense correspondence datasets from RGB images.
Revised: In [14] NeRF is utilized to generate dense correspondence datasets from RGB images.

Original: This alleviates problemswith noisy depth data and proves especially helpful for thin and reﬂective objects.
Revised: This alleviates problems with noisy depth data and proves especially helpful for thin and reﬂective objects.

Revised: There are several works that proposed to learn dense visual object representations directly from synthetic images composed of random backgrounds and randomly sampled, masked objects distributed over the image, see [2, 15, 12].

Revised: Learning from such synthetic images can be more efﬁcient due to higher object density and ground truth correspondence, however, they rely on labeled datasets with object masks.

Original: Our proposedsolution also relies only on RGB images, but without the need for registration, thus allowing for anunordered RGB set, or for images taken by  a single, ﬁx-mounted camera.
Revised: Masking is either achieved by 3D reconstruction with a robot wrist-mounted camera [2, 15], or a labelled RGBD dataset [12]. As opposed to image composition via mask-labeled datasets, the image augmentation technique, as in this paper, only requires an unordered, unlabelled RGB dataset.

Revised: This signiﬁcantly simpliﬁes data collection and opens up the possibilities to learn dense visual descriptors where no 3D reconstruction is possible, or where object masks are not available.

Original: Self-supervised descriptor learning from single images.
Revised: Self-supervised descriptor learning from RGB images .

Original: An intuitive way to generate geometriccorrespondence is to estimate the optical ﬂow of subsequent frames from a video.
Revised: An intuitive way to generate geometric correspondence is to estimate the optical ﬂow of subsequent frames from a video.

Original: Deekshith et al.[15] adopted this technique  to train DONs using contrastive learning.
Revised: In [16] this technique was adapted to train DONs using contrastive learning.

Original: Thewlis et al.

Original: [8] proposed touse optical ﬂow from videos, or image augmentations to embed pixels of objects in view-invariantcoordinate frames.
Revised: [8] proposed to use optical ﬂow from videos, or image augmentations to embed pixels of objects in view-invariant coordinate frames.

Original: Novotny et al.

Original: [9] adopted this method  for pretraining of geometry-orientedtasks, such as object speciﬁc part detection in images.
Revised: This method was adopted in [9] for pretraining of geometry-oriented tasks, such as object speciﬁc part detection in images.

Original: Zhang and Maire [16] propose to learn pixelwise descriptors from single images with augmentations by using hierarchical visual grouping ofimage patches based on contour.
Revised: The work in [17] learns pixel-wise descriptors from single images with augmentations by using hierarchical visual grouping of image patches based on contour.

Original: Our work follows the image augmentation technique of single RGB images to generate alternative views and synthetic correspondence.
Revised: Our work follows the image augmentation technique of single RGB images to generate alternative views and synthetic correspondence.

Revised: Equivariant network architectures (e.g., [18, 19]) could replace certain augmentations (e.g. rotation) during training and improve sample efﬁciency.

Revised: In our setup we can apply afﬁne transformation on training data and we rely on a vanilla ResNet architecture for our experiments, which achieves good SE (2) equivariance, as shown in our experiments.

Original: Self-supervised visual representations learning.
Revised: Self-supervised visual representations learning.

Original: Instead of training on large supervised datasets,self-supervised methods have become a popular way to obtain visual representations, which can beﬁne-tuned to speciﬁc downstream tasks.
Revised: Instead of training on large supervised datasets, self-supervised methods have become a popular way to obtain visual representations, which can be ﬁne-tuned to speciﬁc downstream tasks.

Original: A recent and very successful approach using contrastivelearning is SimCLR by Chen et al.
Revised: A recent and very successful approach using contrastive learning is SimCLR [10].

Original: It aims to maximize agreement between two augmentedversions of the same image, while considering all other images in the batch as negative samples.
Revised: It aims to maximize agreement between two augmented versions of the same image, while considering all other images in the batch as negative samples.

Original: Grill et al.

Original: [17] proposed BYOL (Bootstrap your own latent) that, in comparison to contrastivemethods, does not rely on the sampling of negatives.
Revised: BYOL (Bootstrap your own latent) [20], in comparison to contrastive methods, does not rely on the sampling of negatives.

Original: With Barlow Twins Zbontar et al. [18] also forgo negative samples by optimizing the cross-correlation matrix between embeddings from twoaugmented versions of the same image to be close to identity.
Revised: Barlow Twins [21] also forgo negative samples by optimizing the cross-correlation matrix between embeddings from two augmented versions of the same image to be close to identity.

Original: Our approach is most similar to SimCLR as we employ the same loss formulation, but with the important difference that our batchis constituted by individual pixel descriptors instead of full image embeddings.
Revised: Our approach is most similar to SimCLR as we employ the same loss formulation, but with the important difference that our batch is constituted by individual pixel descriptors instead of full image embeddings.

Original:  Method
Revised:  Method

Original: In this section we discuss our proposed training approach using image augmentations.
Revised: In this section we discuss our proposed training approach using image augmentations.

Original: We ﬁrst givean overview of the whole training pipeline, then discuss image augmentation techniques and ﬁnallypresent the loss formulation and dataset requirements.
Revised: We ﬁrst give an overview of the whole training pipeline, then discuss image augmentation techniques and ﬁnally present the loss formulation and dataset requirements.

Original: For an illustration of the training pipeline werefer to Fig.2.
Revised: For an illustration of the training pipeline we refer to Fig.

Original: Finally, we compute the contrastive loss and backpropagate the error using the embedded images I (cid:48)

Original: d and I (cid:48)(cid:48)

Original: d along with the correspondence information.

Original: 3.1 Dense Descriptor Training with Synthetic Correspondence
Revised: 3.1 Dense Descriptor Training with Synthetic Correspondence

Original: Inspired by the work ofNovotny et al. [9] we rely on training on an unordered set of images anduse image augmentations to arrive at alternative views of each image.
Revised: Inspired by the work of [9] we rely on training on an unordered set of images and use image augmentations to arrive at alternative views of each image.

Original: First, we sample a minibatchof N RGB images from the training data set consisting of independent RGB images.
Revised: First, we sample a minibatch of N RGB images from the training data set consisting of independent RGB images.

Original: For everyimage I in the minibatch we sample two augmented views I (cid:48) ∼ g ( I ) and I (cid:48)(cid:48) ∼ g ( I ) by applyingrandomized augmentations g : R H × W × 3 (cid:55)→ R H
Revised: For every image I in the minibatch we sample two augmented views I (cid:48) ∼ g ( I ) and I (cid:48)(cid:48) ∼ g ( I ) by applying randomized augmentations g : R H × W × 3 (

Original:  × W × 3 (described in more detail below).
Revised: cid:55)→ R H × W × 3 (described in more detail below).

Original: A learnedfully-convolutional network [19] model f ( · ; θ ) , f : R H × W × 3 (cid:55)→ R H
Revised: A learned fully-convolutional network [22] model f ( · ; θ ) , f : R H × W × 3 (cid:55)→ R H

Original: × W × D maps the augmentedimages I (cid:48) and I (cid:48)(cid:48) to their descriptor space embeddings I (cid:48)
Revised: × W × D maps the augmented images I (cid:48) and I (cid:48)(cid:48) to their descriptor space embeddings I (cid:48)

Original: d and I (cid:48)(cid:48) d .
Revised: d and I (cid:48)(cid:48) d .

Original: The user deﬁned parameter D ∈ N + controls the resolution of the descriptor space.
Revised: The user deﬁned parameter D ∈ N + controls the resolution of the descriptor space.

Original: By keeping track of the origin of pixels before the augmentations, we sample pairs of pixel locationsbetween I (cid:48) and I (cid:48)(cid:48) that have the same origin in I .
Revised: By keeping track of the position of each pixel in the original image I during the augmentations, we sample pairs of pixel locations between I (cid:48) and I (cid:48)(cid:48) that share the same position in I .

Original: We refer to these as synthetic correspondences,emphasizing the use of synthetic image augmentations as opposed to geometric correspondencescoming from the 3D geometry of multiple camera views, as  in [1, 2].
Revised: We refer to these as synthetic correspondences, emphasizing the use of synthetic image augmentations as opposed to geometric correspondences coming from the 3D geometry of multiple camera views, as [Figure] in [1, 2].

Original: The descriptor values at thesampled pixel correspondence locations serve as positive pairs for contrastive learning.
Revised: The descriptor values at the sampled pixel correspondence locations serve as positive pairs for contrastive learning.

Original: 3.2 Image Augmentations
Revised: 3.2 Image Augmentations

Original: For robotic applications we require descriptors that are invariant to translations, rotations and perspective changes of the objects, as well as changes of lighting conditions.
Revised: For robotic applications we require descriptors that are invariant to translations, rotations and perspective changes of the objects, as well as changes of lighting conditions.

Original: In vanilla DONs training,cf.
Revised: In vanilla DONs training, cf.

Original: Florence et al.

Original: [2], this is achieved by recording a diverse training set of registered RGBD image sequences, which contain sufﬁcient variance in camera and object poses, and to some extent inlighting conditions.
Revised: [2], this is achieved by recording a diverse training set of registered RGBD image sequences, which contain sufﬁcient variance in camera and object poses, and to some extent in lighting conditions.

Original: In our work, we achieve a similar effect purely by imposing data augmentationson single RGB images from an unordered set.
Revised: In our work, we achieve a similar effect purely by imposing data augmentations on single RGB images from an unordered set.

Original: [Figure] We carefully select augmentations, which reﬂect the desired invariance properties stated above, asfollows: afﬁne transformation induce rotations and scale changes (zoom out), perspective distortionsmimic view changes, resize&crop implies scale changes (zoom in), and lastly, color jitter affectsbrightness, contrast, hue, and saturation of the image, hence the lighting conditions of the scene.
Revised: We carefully select augmentations, which reﬂect the desired invariance properties stated above, as follows: afﬁne transformation induce rotations and scale changes (zoom out), perspective distortions mimic view changes, resize&crop implies scale changes (zoom in), and lastly, color jitter affects brightness, contrast, hue, and saturation of the image, hence the lighting conditions of the scene.

Original: Weshow an illustration of these augmentations in Fig.
Revised: We show an illustration of these augmentations in Fig.

Original: We utilize torchvision library [20] of Pytorchas reference implementations.
Revised: We utilize torchvision library [23] of Pytorch as reference implementations.

Original: Adrian et al. [4] already demonstrated the impact  on improved performance for training of DONson geometric datasets.
Revised: The positive impact of image augmentations on the training of dense visual descriptors was already demonstrated in [4] for datasets utilizing geometric correspondences.

Original: In our work, each augmentation is not only helpful, but relevant to the abilityof the model to successfully learn an invariant descriptor space.
Revised: In our work, each augmentation is not only helpful, but relevant to the ability of the model to successfully learn an invariant descriptor space.

Original: See Sec. 4.3 for an ablation on therespective impact of each augmentation on the overall performance.
Revised: See Appendix F.1 for an ablation on the respective impact of each augmentation on the overall performance.

Original: 3.3 Loss Function
Revised: 3.3 Loss Function

Original: FollowingChen et al. [10] we adopt the NT-Xent loss, for learning the dense descriptor representations.
Revised: Following [10] we adopt the NT-Xent loss, for learning the dense descriptor representations.

Original: For a pair of corresponding descriptors { d i , d j } , obtained from images I (cid:48) and I (cid:48)(cid:48) , we deﬁnetheir individual loss term as
Revised: For a pair of corresponding descriptors { d i , d j } , obtained from images I (cid:48) and I (cid:48)(cid:48) in a minibatch of size

Revised: M , we compare their distance to the distance of d i to all other sampled descriptors in the given minibatch arriving at the following individual loss term:

Original: [Equation]
Revised: [Equation]

Revised: with temperature parameter τ which we ﬁx to τ = 0 .

Revised: 07 throughout this paper following [4].

Original: where we choose the metric dist ( d i , d j ) to be the cosine similarity between descriptors
Revised: We choose the metric dist ( · ) to be the cosine similarity between descriptors

Original: d i and d j .
Revised: d i and d j .

Revised: Due to the cosine similarity we normalize the descriptors d .

Original: The total loss is given as the mean over all individual loss terms, cf.
Revised: The total loss is given as the mean over all individual loss terms, cf.

Original: Chen et al. Note that this loss, together with our correspondence sampling, does not distinguish backgroundfrom objects, nor does it explicitly address multiple object classes and instances, as opposed to theapproach by Florence et al.
Revised: Note that this loss, together with our correspondence sampling, does not distinguish background from objects, nor does it explicitly address multiple object classes and instances, as opposed to [2].

Original: Instead, we followthe method of Adrian et al. [4] and samplecorrespondences uniformly in image plane and assume that every pixel is unique.
Revised: Instead, we follow [4] and sample correspondences uniformly in image plane assuming that every pixel is unique.

Original: This method istailored to datasets depicting densely packed scenes with single object instances, for example, aheap of objects in a bin-picking scenario.
Revised: This method is tailored to datasets depicting densely packed scenes with single object instances, for example, a heap of objects in a bin-picking scenario.

Original: The learned descriptor space does not imply semanticinformation on object classes or background, but still provides consistent keypoint detection androbust tracking performance, which is essential for downstream tasks.
Revised: The learned descriptor space does not imply semantic information on object classes or background, but still provides consistent keypoint detection and robust tracking performance, which is essential for downstream tasks.

Original:  Comparison of Training with Synthetic and Geometric Correspondence
Revised:  Comparison of Training with Synthetic and Geometric Correspondence

Original: In this section we show an in-depth comparison between training with geometric and syntheticcorrespondence.
Revised: In this section we show an in-depth comparison between training with geometric and synthetic correspondence.

Original: We also investigate the invariance of descriptors obtained from a network trainedwith synthetic correspondence with respect to object-camera relative transformations.
Revised: We also investigate the invariance of descriptors obtained from a network trained with synthetic correspondence with respect to object-camera relative transformations.

Original: In all our evaluations we utilize a pretrained ResNet-34 with 8-stride output as used by Florence et al.[2], which yields an upsampled output matching the resolution of the input.
Revised: Similar to [2] we use a pretrained ResNet-34 with 8-stride output for all experiments. The upsampled output matches the resolution of the input.

Original: Note that each descriptord is normalized since we utilize the cosine similarity in the loss formulation (see Section 3.3).

Original: In thesupplementary material we give a brief introduction into the baseline training method we use in ourcomparisons relying on geometric correspondence by Adrian et al.
Revised: In Appendix B we give more details on the baseline training method using geometric correspondence.

Original: We recorded a datasetconsisting of a set of registered RGBD sequences, to enable comparisonbetween both approaches.
Revised: We recorded a dataset of registered RGBD sequences.

Original: Despite the availability for registered image pairs, the synthetic viewtraining only uses single RGB images for both training and validation.
Revised: Despite the availability for registered image pairs, the synthetic training only uses single RGB images for both training and validation.

Original: However, the camera poseshelp with the evaluation as they allow us to generate ground truth pixel matches across any two images of the same static scene without the need for manual labeling.
Revised: However, the camera poses help with the evaluation as they allow us to generate ground truth pixel matches across any two images of the same static scene without the need for manual labeling.

Original: The dataset consists of eightscenes with various object conﬁgurations, with every scene containing only one instance per object,and every object is visible to some extent in every frame.
Revised: The dataset consists of eight scenes with various object conﬁgurations. Every scene contains only one instance per object.

Original: The scenes are recorded with a robot wristmounted camera while the robot arm follows a predeﬁned trajectory keeping the objects in view.
Revised: The scenes are recorded with a robot wrist-mounted camera while the robot arm follows a predeﬁned trajectory keeping the camera view on the objects.

Original: Both approaches are evaluated on the same ground-truth image pairs and correspondences.
Revised: Both approaches are evaluated on the same ground-truth image pairs and correspondences.

Original: Forrobustness, we perform a k-fold cross-validation, that is, each scene from our total dataset wasonce used as test set.
Revised: For robustness, we perform a k-fold cross-validation, that is, each scene from our total dataset was once used as test set.

Original: One scene is chosen as validation set, with the remaining 6 scenes used fortraining.
Revised: One scene is chosen as validation set, with the remaining six scenes used for training.

Original: The averaged results are reported.
Revised: We report the averaged results.

Original: We use the same loss function and training parametersfor both approaches, with the exception that augmentations are chosen with 50% probability for thegeometric training, as it yields better results.
Revised: We use the same loss function, training parameters and augmentations for both approaches, with the exception that augmentations are chosen with 50% probability for the geometric training, as it yields better results.

Original: For synthetic training, each augmentation is alwaysused.
Revised: For synthetic training, each augmentation is applied.

Original: Further training details are listed in supplementary materials.
Revised: An ablation study of using different augmentation probabilities is given in Appendix F.2.

Original: Given a common dataset, measuring keypoint tracking accuracy provides a task-agnostic comparison between both training methods. We deﬁne a keypoint as a location in image plane k i = ( u, v ) .
Revised: For brevity, we report the median and 75% quantile pixel errors.

Original: Each keypoint is associated with a unique descriptor d ∈ R D and the keypoint tracking problem is deﬁned asﬁnding the pixel location k ∗ i closest to d in the descriptor image I d such thatk ∗ i = arg min k i dist ( I d ( k i ) , d ) .
Revised: Extended results, including the mean, 90%, and 95% quantile pixel errors, as well as the percentage-of-correct keypoints@k (PCK@k) for k ∈ { 3 , 5 , 10 , 25 , 50 } , are reported in the respective Appendix sections.

Original: Evaluation. From the test set we sample 1000 image pairs { A, B } representing alternative views ofthe same scene. For each image pair we sample 200 keypoints { k Ai , d A } , where d A = f ( A ; θ )( k Ai )is located on one of the objects in image A . We also recorded keypoint locations in image B ,{ k Bi } , such that every pair { k Ai , k Bi } unprojects to the same location in world coordinates.
Revised: 4.1 Keypoint Tracking Performance

Original: Then,we solve the keypoint tracking problem for every descriptor d A in image B and record the pixelerror e = (cid:107) k
Revised: Given a common dataset, measuring keypoint trackingThen, wery descriptor d A in image B and record the pixel error e = (cid:107) k

Original: Bi − k ∗ i (cid:107) 2 .
Revised: Bi − k ∗ i (cid:107) 2 .

Original: The network parameters θ are either obtained by geometric [4] or syntheticcorrespondence training proposed in Sec. 3.1.
Revised: The network parameters θ are either obtained by geometric [4] or synthetic correspondence training proposed in Sec. 3.1.

Original: Results.
Revised: Results.

Original: Fig.
Revised: Fig.

Original: 4a shows the distribution of pixel errors for the two training approaches.
Revised: 4a shows the distribution of pixel errors for the two training approaches.

Original: The medianof the corresponding distribution is highlighted as a dashed line.
Revised: The median of the corresponding distribution is highlighted as a dashed line.

Original: With a difference of only 1 . 9 px inmedian pixel error, the synthetic correspondence performs competitively to the geometric training.
Revised: With a difference of only 1 . 9 px in median pixel error, the synthetic correspondence performs competitively to the geometric training.

Original: The percentage of pixel errors that are larger than 50 pixels are 10 .
Revised: The percentage of pixel errors that are larger than 50 pixels are 10 .

Original: 1% and 5 .
Revised: 1% and 5 .

Original: 4% respectively, asindicated in the bottom right part of Fig. 4a.
Revised: 4% respectively, as indicated in the bottom right part of Fig. 4a.

Original: [Figure]
Revised: [Figure]

Original: In Fig.
Revised: In Fig.

Original: 4b we compare the median and the 75% quantile of the two training methods with respectto the descriptor dimensions D .
Revised: 4b we compare the median and the 75% quantile of the two training methods with respect to the descriptor dimensions D .

Original: The results were obtained without k-fold cross-validation.
Revised: These results were obtained without k-fold cross-validation.

Original: Forboth training approaches the median pixel error decreases for larger descriptor dimensions.
Revised: For both training approaches the median pixel error decreases for larger descriptor dimensions.

Original: Fordimensions larger than 9 the median pixel error decreases only marginally.
Revised: For dimensions larger than 9 the median pixel error decreases only marginally.

Original: In contrast, the 75%quantile error still increases further until D = 64 , before improvements saturate.
Revised: In contrast, the 75% quantile error still increases further until D = 64 , before improvements saturate.

Original: For additionalinsights into these results we refer to the supplementary material.
Revised: For additional insights into these results we refer to the supplementary material.

Original: 4.2 Invariance Tests
Revised: 4.2 Invariance Tests

Original: In the following, we wish to answer the question: how well does training with image augmentationsproposed in Sec.
Revised: In the following, we discuss the question: how well does training with image augmentations proposed in Sec.

Original: 3.2 generalize to physical camera transformations?
Revised: 3.2 generalize to physical camera transformations?

Original: For this purpose, we recordeddifferent test scenes with a wrist-mounted camera and the following speciﬁc camera movements:(i) changing camera perspective (camera tilting), (ii) translation along the camera z-axis (zoomingin and out), and (iii) camera rotation along the camera z-axis.
Revised: We recorded different test scenes with a wrist-mounted camera and the following speciﬁc camera movements: (i) changing camera perspective (camera tilting), (ii) translation along the camera z-axis (zooming in and out), and (iii) camera rotation along the camera z-axis (see Appendix D.2 for details).

Original: We compare the performance ofa network trained with synthetic and one with geometric correspondence both with 64 descriptordimensions and afﬁne, perspective and resize&crop augmentations.
Revised: We compare the performance of a network trained with synthetic and one with geometric correspondence both trained on the same data as described in Sec. 4.1. We use 64 descriptor dimensions and afﬁne, perspective and resize&crop augmentations.

Original: As in the previous section wecompute the 75% quantile pixel error and use 1000 keypoints per image pair, ﬁx the base image Aand only vary image B , which shows the changing camera views.
Revised: As in the previous section we compute the 75% quantile pixel error and use 1000 keypoints per image pair, ﬁx the base image A and only vary image B , which shows the changing camera views.

Original: The results for three differenttypes of transformations are compiled in Fig.
Revised: The results for three different types of transformations are compiled in Fig.

Original: It can be seen that training with synthetic correspondence generalizes well to a large range of cameratransformations, especially to those parallel to the object plane (Fig.
Revised: It can be seen that training with synthetic correspondence generalizes well to a large range of camera transformations, especially to those parallel to the object plane (Fig.

Original: 5a, Fig. 5b).
Revised: 5a, Fig. 5b).

Original: These physicalcamera transformations are very similar to the afﬁne and resize&crop augmentations used duringtraining.
Revised: These physical camera transformations are very similar to the afﬁne and resize&crop augmentations used during training.

Original: For generalizing to the perspective transformations with angles above 45 ◦ degrees, asshown in Fig.
Revised: For generalizing to the perspective transformations with angles above 45 ◦ degrees, as shown in Fig.

Original: 5c, the synthetic correspondence training shows a clear deterioration in performance.
Revised: 5c, the synthetic correspondence training shows a clear deterioration in performance.

Original: As the perspective changes, occluded parts become visible and vice versa.
Revised: As the perspective changes, occluded parts become visible and vice versa.

Original: This physical transformation is not well captured by the synthetic augmentations for larger angles.
Revised: This physical transformation is not well captured by the synthetic augmentations for larger angles.

Original: [Figure]
Revised: [Figure]

Original: 4.3 Augmentations

Original: Complementing the ﬁndings in section 4.2 we study the inﬂuence of different augmentations forthe synthetic correspondence training on the ability of the network to generalize to different cameratransformations.

Original: Fig.

Original: 6 shows the 75% quantile of the pixel error distribution for the syntheticcorrespondence training with different augmentations.

Original: We ﬁnd that afﬁne transformations are mostcritical, as only a model trained with it shows invariance to rotations, see Fig.

Original: 6a.

Original: This matchesthe expectations as standard CNN are by default not invariant to rotations.

Original: Nevertheless, we ﬁndthat both resize&crop and perspective distortion both further improve the performance of just afﬁnetransformations.

Original: In particular, for camera movements that induce perspective distortions and scalechanges, see Fig.

Original: 6c and Fig.

Original: 6b, the error decreases considerably.

Original: Lastly, we ﬁnd that color jitterfurther reduces the overall mean pixel error from 19 . 4 to 17 .

Original: 1 pixel.

Original: The improvement appearsmodest, but we note that our test dataset was recorded at the same time as train and validation,and the lighting conditions of the scene were not explicitly altered.

Original: For a complete table of all thecombinations of augmentations, see the supplementary material.

Original:  Grasping Experiment with Fix-mounted Camera
Revised:  Grasping Experiment with Fix-mounted Camera

Original: We demonstrate a robotic bin-picking experiment that relies on dense visual descriptors for deﬁninggrasp preferences.
Revised: We demonstrate a robotic bin-picking experiment that relies on dense visual descriptors for deﬁning grasp preferences.

Original: We use a 7-DoF Franka Emika Panda arm with a suction gripper mounted on the[Figure] end-effector, see Fig. 
Revised: We use a 7-DoF Franka Emika Panda arm with a suction gripper mounted on the end-effector, see Fig. 1.

Original: Our setup uses a ﬁx-mounted Zivid One+ camera above the bin in a roboticcell.
Revised: Our setup uses a ﬁx-mounted Zivid One+ camera above the bin in a robotic cell.

Revised: Training a descriptor network using this setup prevents the use of geometric correspondences.

Revised: Instead, we show that our proposed method can be trained on a setup as it is often present in real world applications and prove that the keypoint tracking accuracy is good enough for guiding a generic grasping method by human annotated grasp preferences.

Original: In the experiment  we consider picking ten different types of objects from one bin.
Revised: In the experiment [Figure] we consider picking ten different types of objects from one bin.

Original: However,using a suction gripper the objects are difﬁcult to grasp at certain locations: some have cutouts onthe packaging, transparent or foldable parts, and uneven surfaces (see supplementary material).
Revised: However, using a suction gripper the objects are difﬁcult to grasp at certain locations: some have cutouts on the packaging, transparent or foldable parts, and uneven surfaces (see Appendix G for higher resolution images).

Original: Inour experiments, a purely model-free grasp pose generator often predicts poses at these challengingparts of objects, ultimately reducing picking performance.
Revised: In our experiments, a purely model-free grasp pose generator often predicts poses at these locations, ultimately reducing picking performance.

Original: [Figure]
Revised: [Figure]

Original: The predicted grasp poses overlayed on the heatmaps. We hypothesize that introducing human domain knowledge to aid grasp pose selection will leadto higher grasp success rates.
Revised: We want to utilize human domain knowledge to aid grasp pose selection using dense visual descriptors to increase grasp success rates for these objects.

Original: Human domain knowledge is considered by highlighting parts ofthe RGB image where grasp poses are preferred.

Original: To do so, we show a small number of RGBimages depicting the objects in different conﬁgurations in the bin and ask the human to click onpixel locations corresponding to a preferred grasp location.
Revised: To do so, we show a small number of RGB images depicting the objects in different conﬁgurations in the bin and ask the human to click on pixel locations corresponding to a preferred grasp location.

Original: We track these descriptors to generatea preference heatmap , as shown in Fig.
Revised: We track these descriptors to generate a preference heatmap , as shown in Fig.

Original: 7c.
Revised: 6c.

Revised: The heatmap, in contrast to discrete single-pixel correspondences, enables intuitive consideration of matching uncertainty in the form of distance in descriptor space.

Revised: Furthermore, it provides a more ﬂexible and quantiﬁable basis for combination with grasp detectors.

Original: To generate the ﬁnal set of grasp pose candidates weintersect the grasp preference heatmap with the detected graspable areas identiﬁed by the modelfree grasp detector (Fig. 7b).
Revised: To generate the ﬁnal set of grasp pose candidates we intersect the grasp preference heatmap with the detected graspable areas identiﬁed by the model-free grasp detector (Fig. 6b).

Revised: While we make no speciﬁc assumption regarding which grasp detection method to use, we employ a dense pixel-wise graspability estimation in this experiment.

Revised: It is based on a fully convolutional neural network with

Revised: RGB-D input, speciﬁcally UNet [24] trained on annotated pixel-wise labels of expected graspability for a wide range of different bin picking scenes.

Original: For the example image the ﬁnal set of poses are shown in Fig.
Revised: For the example image, the resulting poses are shown in Fig.

Original: 7a and Fig.
Revised: 6a and Fig.

Original: 7d.
Revised: 6d.

Original: Note that the descriptor network is trained purely from the set of RGB images of the bin,including the  objects in random conﬁgurations, recorded with the overhead camera.
Revised: Note that the descriptor network is trained purely from the set of RGB images showing the bin with objects in random conﬁgurations, recorded with the overhead camera.

Original: We require asingle instance of the objects to be present in the bin.
Revised: See Sec.

Original: For more details on the experiment setup andheatmap generation we refer to the supplementary material.
Revised: G for more details on the experimental setup and heatmap generation.

Original: 5.1 Quantitative Evaluation
Revised: 5.1 Quantitative Evaluation

Original: We evaluate the beneﬁt of the proposed method for robotic bin picking based on a set of sixteenmanually annotated scenes.
Revised: We evaluate the beneﬁt of the proposed method for robotic bin picking based on a set of sixteen manually annotated scenes.

Original: The scene images include the ten known objects, but are not containedin the training set.
Revised: The scene images include the ten known objects, but in different conﬁgurations compared to the training set.

Original: For each  image, we manually annotated object instances, pixel-wise graspable areas, and those areas that correspond to selected descriptors.
Revised: For each evaluation image, we manually annotated object instances, pixel-wise graspable areas, and those areas that correspond to selected descriptors.

Original: In total, the evaluation dataset contains131 graspable objects of which 103 have visible descriptor spots.
Revised: In total, the evaluation dataset contains 131 graspable objects of which 103 have visible descriptors.

Original: Table 1 shows a summary of the results.

Original: We compare our descriptor-based grasping approach with apurely model-free approach that directly uses the graspable areas without accounting for descriptors.
Revised: We compare our descriptor-based grasping approach with a purely model-free approach that directly uses the graspable areas without accounting for descriptors.

Revised: With this study, we investigate two questions.

Revised: First, how effective is our approach to re-identify descriptor spots compared to grasping at these spots by chance?

Revised: And second, is there a considerable negative effect on the amount of objects that can be grasped, e.g., due to non-detected grasp pose annotations?

Revised: Table 1 shows a summary of the results.

Original: Success Rate denotes the number of successful grasps compared to all grasps attempted.
Revised: Success Rate denotes the number of successful grasps compared to all grasps attempted.

Original: Descriptor Success denotes the number of grasps at those spots marked as desired grasping points compared toall grasps attempted.
Revised: Descriptor Success denotes the number of grasps at those spots marked as desired grasping points compared to all grasps attempted.

Original: Object Hits denotes the percentage of all graspable objects for which at leastone feasible grasp pose has been found irrespective of descriptors, including objects without visibledescriptor spots.
Revised: Object Hits denotes the percentage of all graspable objects for which at least one feasible grasp pose has been found irrespective of descriptors, including objects without visible descriptor spots.

Original: Descriptor Hits denotes the percentage of objects for which a grasping pose hasbeen found at the respective descriptor spot, only including objects with visible descriptors.
Revised: Descriptor Hits denotes the per- centage of objects for which a grasping pose has been found at the respective descriptor spot, only including objects with visible descriptors.

Revised: We consider descriptor spots with a tolerance of around 1cm, which corresponds to the radius of the suction gripper.

Original: [Table]
Revised: [Table]

Original: It can be concluded from Table 1 that using the proposed method to encode grasp preferences iseffective as it signiﬁcantly increases the amount of grasps at desired spots from 50 .
Revised: It can be concluded from Table 1 that using the proposed method to encode grasp preferences is effective as it signiﬁcantly increases the amount of grasps at desired spots from 50 .

Original: 4% to 91 .
Revised: 4% to [Table]

Original: Due to the challenging object geometries, this helps to raise the overall grasp success rate from79 .
Revised: Due to the challenging object geometries, this helps to raise the overall grasp success rate from 79 .

Original: 9% to 98 .
Revised: 9% to 98 .

Original: The  downside, however, is that the proposed method ﬁnds grasp poses only for asmaller amount of objects, 63 .
Revised: The expected downside, however, is that the proposed method ﬁnds grasp poses only for a smaller amount of objects, 63 .

Original: 4% instead of 78 .
Revised: 4% instead of 78 .

Original: Still, this includes objects that have no desiredgrasping spot visible.
Revised: Still, this includes objects that have no desired grasping spot visible.

Original: In some applications it can be the desired behavior to not propose a grasppose for objects if they cannot be grasped at the preferred location.
Revised: In some applications it can be the desired behavior to not propose a grasp pose for objects if they cannot be grasped at the preferred location.

Original: In consequence, when onlyconsidering which preferred grasping spots have been covered by grasp poses, our method managesto ﬁnd grasps for 78 .
Revised: When only considering which preferred grasping spots have been covered by grasp poses, our method manages to ﬁnd grasps for 78 .

Original: 6% instead of 68 . 0% of the visible spots.
Revised: 6% instead of 68 . 0% of the visible spots.

Revised: In consequence, we conclude that there is only a moderate negative effect due to limiting grasping to descriptor spots which can indeed be beneﬁcial for some applications.

Original:  Limitations
Revised:  Limitations

Original: In the following we share more insights on the limitations of our method and discuss future research.

Original: Variance in camera poses.
Revised: Variance in camera poses.

Original: As seen in Fig.
Revised: As seen in Fig.

Original: 5, the synthetic training ensures that descriptors are stable within a limited margin of camera transformations relative to the object.
Revised: 5, the synthetic training ensures that descriptors are stable within a limited margin of camera transformations relative to the object.

Original: For example, in Fig.
Revised: For example, in Fig.

Original: 5c,for viewing angles steeper than 45 ◦ , accuracy deteriorates.
Revised: 5c, for viewing angles steeper than 45 ◦ , accuracy deteriorates.

Original: This imposes a limit on descriptor consistency across images with large differences in object poses.
Revised: This imposes a limit on descriptor consistency across images with large differences in object poses.

Original: Changing Environment.
Revised: Changing Environment.

Original: A change in environment (background, lighting) between training andinference time may have a negative inﬂuence on keypoint tracking performance.
Revised: A change in environment (background, lighting) between training and inference time may have a negative inﬂuence on keypoint tracking performance.

Original: It is expected thataugmentations such as color jitter and, if masks are available, background randomization can mitigate these effects.
Revised: It is expected that augmentations such as color jitter and, if masks are available, background randomization can mitigate these effects.

Original: We will investigate these aspects in future research.
Revised: We will investigate these aspects in future research.

Original: Generalization to unseen objects.
Revised: Generalization to unseen objects.

Original: AlthoughFlorence et al. [2] demonstrated capabilities to generalize intra-class instances, our work focuses on instance speciﬁc descriptors.
Revised: Although [2] demonstrated capabilities to generalize intra-class instances, our work focuses on instance speciﬁc descriptors.

Original: Given our outlinedtraining setup, it works best on known objects from the training set.
Revised: Given our outlined training setup, it works best on known objects from the training set.

Original: Object Edges.
Revised: Object Edges.

Original: We observe worse descriptor consistency across images for keypoints located at theedges of objects or close to parts that are occluded by other objects.
Revised: We observe worse descriptor consistency across images for keypoints located at the edges of objects or close to parts that are occluded by other objects.

Original: In setups where object masksare present background randomization could reduce this effect.
Revised: In setups where object masks are present background randomization could reduce this effect.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper we proposed a novel training method for learning dense visual descriptors based onimage augmentations for robotic manipulation.
Revised: In this paper we proposed a novel training method for learning dense visual descriptors based on image augmentations for robotic manipulation.

Original: The evaluation shows that overall our proposedmethod is competitive to the existing geometric training approach.
Revised: The evaluation shows that overall our proposed method is competitive to the existing geometric training approach.

Original: For physical transformationslike changing the camera perspective on the scene, which are harder to mimic by augmentations,the training method using geometric correspondences shows superior performance.
Revised: For physical transformations like changing the camera perspective on the scene, which are harder to mimic by augmentations, the training method using geometric correspondences shows superior performance.

Original: Being awareof these limitations our proposed method is expected to perform well for setups where objects aremostly altered by translations, rotations parallel to the camera plane, or are slightly tilted.
Revised: Being aware of these limitations our proposed method is expected to perform well for setups where objects are mostly altered by translations, rotations parallel to the camera plane, or are slightly tilted.

Original: As this isoften the case for random heaps of objects in a bin, our method is especially suitable for such setupsthat are constrained by a ﬁx-mounted camera.
Revised: As this is often the case for random heaps of objects in a bin, our method is especially suitable for such setups that are constrained by a ﬁx-mounted camera.

Original: Finally, we demonstrated the use of our method in arealistic grasping experiment to increase grasp success rates by human annotated grasp preferences.
Revised: Finally, we demonstrated the use of our method in a realistic grasping experiment to increase grasp success rates by human annotated grasp preferences.