Original: One-Shot Transfer of Affordance Regions? AffCorrs!
Revised: One-Shot Transfer of Affordance Regions? AffCorrs!

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract:
Revised: Abstract:

Original: In this work, we tackle one-shot visual search of object parts.
Revised: In this work, we tackle one-shot visual search of object parts.

Original: Given asingle reference image of an object with annotated affordance regions, we segmentsemantically corresponding parts within a target scene.
Revised: Given a single reference image of an object with annotated affordance regions, we segment semantically corresponding parts within a target scene.

Original: We propose AffCorrs, anunsupervised model that combines the properties of pre-trained DINO-ViT’s image descriptors and cyclic correspondences.
Revised: We propose AffCorrs, an unsupervised model that combines the properties of pre-trained DINO-ViT’s image descriptors and cyclic correspondences.

Original: We use AffCorrs to find corresponding affordances both for intra- and inter-class one-shot part segmentation.
Revised: We use AffCorrs to ﬁnd corresponding affordances both for intra- and inter-class one-shot part segmentation.

Original: Thistask is more difficult than supervised alternatives but enables future work such aslearning affordances via imitation and assisted teleoperation.
Revised: This task is more difﬁcult than supervised alternatives, but enables future work such as learning affordances via imitation and assisted teleoperation.

Revised: Project page with code and dataset: https://sites.google.com/view/affcorrs

Original:  Introduction
Revised:  Introduction

Original: Robot agents can significantly benefit from perceiving and understanding what the environmentaffords them to do.
Revised: Robot agents can signiﬁcantly beneﬁt from perceiving and understanding what the environment affords them to do.

Original: Affordances [1] are representations of how a part of the environment can beused, e.g., a spoon affords to be grasped, and to contain liquid.
Revised: Affordances [1] are representations of how a part of the environment can be used, e.g., a spoon affords to be grasped and to contain liquid.

Original: Being grounded on human semantics,they are intuitive and explainable.
Revised: Being grounded on human semantics, they are intuitive and explainable.

Original: Part-based affordance representations can be efficiently used asan intermediate representation that reduces the dimensionality of many robot learning problemssignificantly [2].
Revised: Part-based affordance representations can be efﬁciently used as an intermediate representation that reduces the dimensionality of many robot learning problems signiﬁcantly [2].

Original: Just as people are able to transfer the knowledge of an object’s functionality toother objects from only a few examples, e.g., the graspable handles of jugs to mugs, it would bebeneficial for robots to understand such correspondences, too.
Revised: Just as people are able to transfer the knowledge of an object’s functionality to other objects from only a few examples, e.g., the graspable handles of jugs to mugs, it would be beneﬁcial for robots to understand such correspondences, too.

Original: We motivate the topic of recognizingknown affordance regions in unseen novel objects as a useful step toward more autonomous robots,assisted teleoperation, visual inspection, and scene understanding.
Revised: We motivate the topic of recognizing known affordance regions in unseen novel objects as a useful step toward more autonomous robots, assisted teleoperation, visual inspection and scene understanding.

Original: Evidently, semantic part correspondence can be achieved with fully supervised methods [3, 4].
Revised: Evidently, semantic part correspondence can be achieved with fully supervised methods [3, 4].

Original: However, they are limited to objects similar to the ones present in the dataset, each requiring multipleinstances for better generalization of the object category.
Revised: However, they are limited to objects similar to the ones present in the dataset, each requiring multiple instances for better generalization of the object category.

Original: Meanwhile, the self-supervision and unsupervised learning paradigms, present an alternative direction, which alleviates the data annotationbottleneck [5], and could enable robots to learn continually by themselves [6].
Revised: Meanwhile, the self-supervision and unsupervised learning paradigms, present an alternative direction, which alleviates the data annotation bottleneck [5], and could enable robots to learn continually by themselves [6].

Original: In this work, we demonstrate how the pre-trained DINO-ViT model [7], which is shown to producegood co-segmentation and point correspondences [8], can be used for part querying and findingsemantically corresponding parts in one-shot – a novel formulation of the One-Shot Instance Segmentation (OSIS) problem extended to parts, which we call One-Shot Affordance Part Segmentation(OSAPS).
Revised: In this work, we demonstrate how the pre-trained DINO-ViT model [7], which is shown to produce good co-segmentation and point correspondences [8], can be used for part querying and ﬁnding semantically corresponding parts in one-shot – a novel formulation of the One-Shot Instance Segmentation (OSIS) problem extended to parts, which we call One-Shot Affordance Part Segmentation (OSAPS).

Original: The ability to query on particular parts leverages the semantic prior given by the user, oralternatively by a preceding system.
Revised: The method enables us to query on any segmentation mask leveraging the semantic prior given by a user, or a preceding system.

Original: We demonstrate how these parts can be associated with predefined skills.
Revised: In particular, we demonstrate how querying on affordance part regions can be associated with predeﬁned skills.

Original: As far as we are aware, this work is one of the first to extend one-shot instancesegmentation to part masks, and affordance parts in particular.
Revised: As far as we are aware, this work is one of the ﬁrst to extend one-shot instance segmentation to part masks, and affordance parts in particular.

Original: Our proposed method presents several contributions: (i) One-Shotaffordance transfer  instead of supervised learning; (ii) Decoupledrepresentation that allows the stack of affordance skills to be separate from the affordance discovery; (iii) Query-based transfer instead of co-part segmentation, which allows us to transfer specificsemantic parts rather than parts based on visual or geometric features only; (iv) Benchmark subsetof an affordance dataset curated for one-shot part transfer.
Revised: Our proposed method presents several contributions: (i) One-Shot transfer of affordance regions instead of learning through supervision; (ii) Decoupled skills that allow the separation of affordance discovery and execution; (iii) Query-based transfer instead of co-part segmentation, which allows us to transfer speciﬁc semantic parts rather than parts based on visual or geometric features only; (iv) Benchmark subset of an affordance dataset curated for one-shot part transfer.

Original:  Related Work
Revised:  Related Work

Original: Most affordance learning work focuses on fully supervised methods that learn a particular affordance, predefined in the training set as image segments, contact points or interaction regions [9, 10, 11].
Revised: Most affordance learning work focuses on fully supervised methods that learn a particular affordance, predeﬁned in the training set as image segments, contact points or interaction regions [9, 10, 11].

Original: Some works focus on learning from interaction combined with geometric andperceptual features [12, 13].
Revised: Some works focus on learning from interaction combined with geometric and perceptual features [12, 13].

Original: Model-based approaches are a potential path towards efficient knowledge transfer.
Revised: Model-based approaches are a potential path towards efﬁcient knowledge transfer.

Original: However, they still rely on expensive annotations, such as full object template models [14].
Revised: However, they still rely on expensive annotations, such as full object template models [14].

Original: One-shot affordance detection was recently demonstrated [15], but this approach is limitedto instance masks and bounding boxes, which are insufficient to transfer knowledge to a robot.
Revised: One-shot affordance detection was recently demonstrated [15], but this approach is limited to instance masks and bounding boxes, which are insufﬁcient to transfer knowledge to a robot.

Original: Models requiring little to no supervision have great potential for robot applications that benefit fromadapting quickly and learning continuously.
Revised: Models requiring little to no supervision have great potential for robot applications that beneﬁt from adapting quickly and learning continuously.

Original: It is of particular interest to understand how previousworks in computer and robot vision solve the semantic correspondence problem in the domains ofpoint- and part-correspondence with limited supervision.
Revised: It is of particular interest to understand how previous works in computer and robot vision solve the semantic correspondence problem in the domains of point- and part-correspondence with limited supervision.

Original: Many works enforce the semantic relationship between two inputs through cyclic matching, either through a cyclic loss or a proxy representation.
Revised: Many works enforce the semantic relationship between two inputs through cyclic matching, either through a cyclic loss or a proxy representation.

Original: Recent examples include the use of uniformcategory-level representations of objects as template 3D objects [16] or unit spheres [17].
Revised: Recent examples include the use of uniform category-level representations of objects as template 3D objects [16] or unit spheres [17].

Original: Dense
Revised: Dense

Original: Object Nets [18] extended previous work [19] and presented a method trained with self-supervisionby projecting points across multiple viewpoints of the same objects, and showing how the learnedobject descriptors effectively generalize over other semantically similar objects.
Revised: Object Nets [18] extended previous work [19] and presented a method trained with self-supervision by projecting points across multiple viewpoints of the same objects, and showing how the learned object descriptors effectively generalize over other semantically similar objects.

Original: Recently, they havebeen extended to learn from even less data and work with more objects through optical flow ofmonocular videos [20], neural radiance fields [21], and unsupervised object classification [22].
Revised: Recently, they have been extended to learn from even less data and work with more objects through optical ﬂow of monocular videos [20], neural radiance ﬁelds [21], and unsupervised object classiﬁcation [22].

Original: Partbased methods also often use latent representations that encode each part’s appearance, shape, orpose [23, 24].
Revised: Partbased methods also often use latent representations that encode each part’s appearance, shape, or pose [23, 24].

Original: Very close to our method is [25] in that it is using cycle-consistency over transformerdescriptors to produce one-shot instance segmentation.
Revised: Very close to our method is [25] in that it is using cycle-consistency over transformer descriptors to produce one-shot instance segmentation.

Original: Other methods rely on large image datasets, such as ImageNet, for pre-training of vision models as means of encoding semantic and perceptual information.
Revised: Other methods rely on large image datasets, such as ImageNet, for pre-training of vision models as means of encoding semantic and perceptual information.

Original: Such works in point correspondence [26, 27, 28, 29] and part correspondence [23, 24] show state-of-the-art performance on severalcomputer vision benchmarks.
Revised: Such works in point correspondence [26, 27, 28, 29] and part correspondence [23, 24] show state-of-the-art performance on several computer vision benchmarks.

Original: However, these methods often struggle with highly occluded inputswith large viewpoint variance.
Revised: However, these methods often struggle with highly occluded inputs with large viewpoint variance.

Original: Caron et al.
Revised: Caron et al.

Original: [7] show that DINO-ViT, a variant of ViT [30] trainedvia self-supervised knowledge distillation, can produce descriptors that contain explicit information about the underlying semantic content, with properties suitable for k-nearest neighbors (kNN)search.
Revised: [7] show that DINO-ViT, a variant of ViT [30] trained via self-supervised knowledge distillation, can produce descriptors that contain explicit information about the underlying semantic content, with properties suitable for k-nearest neighbours (kNN) search.

Original: These properties were recently used in unison with cyclic correspondence for unparalleledco-part segmentation without fine-tuning [8].
Revised: These properties were recently used in unison with cyclic correspondence for unparalleled co-part segmentation without ﬁne-tuning [8].

Original: However, the co-segmentation task does not solve ourproblem, since it often results in segmentation that is consistent but semantically meaningless.
Revised: However, the co-segmentation task does not solve our problem, since it often results in segmentation that is consistent but semantically meaningless.

Original: The literature review leads to several conclusions: (i) for any local descriptor to be suitable forsemantic correspondence, it needs to be ‘aware’ of the full semantic context of the object that it ispart of; (ii) in matching any two objects belonging to the same semantic category, not all points willalways have a true correspondence, either due to occlusion or the lack or addition of parts (such ashandle on a cup, or a switch on a lamp); (iii) the currently best performing models combine semanticpriors provided by pre-trained models and cyclicity.
Revised: The literature review leads to several conclusions: (i) for any local descriptor to be suitable for semantic correspondence, it needs to be ‘aware’ of the full semantic context of the object that it is part of; (ii) in matching any two objects belonging to the same semantic category, not all points will always have a true correspondence, either due to occlusion or the lack or addition of parts (such as handle on a cup, or a switch on a lamp); (iii) the currently best performing models combine semantic priors provided by pre-trained models and cyclicity.

Original:  Method
Revised:  Method

Original: The one-shot semantic instance segmentation (OSIS) problem [31] is defined as finding and segmenting a previously unseen object in a novel scene, based on a single instruction example.
Revised: The one-shot semantic instance segmentation (OSIS) problem [31] is deﬁned as ﬁnding and segmenting a previously unseen object in a novel scene, based on a single instruction example.

Original: Similarly, in this work, we aim to solve one-shot affordance part segmentation (OSAPS): given a supportreference image I R ∈ R 3 × H R × W R and query mask region M Q ∈ { 0 , 1 } H R × W R , the task is to find thesemantically corresponding region in the target image I T ∈ R 3 ×
Revised: Similarly, in this work, we aim to solve one-shot affordance part segmentation (OSAPS): given a support reference image I R ∈ R 3 × H R × W R and query mask region M Q ∈ { 0 , 1 } H R × W R , the task is to ﬁnd the semantically corresponding region in the target image I T ∈ R 3 ×

Original: H T × W T .
Revised: H T × W T .

Original: The variables H R , W R denotethe height and width of the support, while H T , W T denote those of the target, respectively.
Revised: The variables H R , W R denote the height and width of the support, while H T , W T denote those of the target, respectively.

Original: 3.1 Query Region Correspondence
Revised: 3.1 Query Region Correspondence

Original: [Figure] The key takeaway from successful attempts in related works is that cyclicity plays an importantrole in correspondence—a matching region corresponds to the query region and vice versa (i.e., thequery region corresponds to the matching region).
Revised: Pre-processing: The key takeaway from successful attempts in related works is that cyclicity plays an important role in correspondence—a matching region corresponds to the query region and vice versa (i.e., the query region corresponds to the matching region).

Original: Given the kNN-like properties of DINO-ViT, wepropose to use its descriptors to find correspondences.
Revised: Given the kNN-like properties of DINO-ViT, we propose to use its descriptors to ﬁnd correspondences.

Original: Thus, we get descriptor images F R and F Tcorresponding to I R and I T , respectively.
Revised: Thus, we get descriptor images F R and F T corresponding to I R and I T , respectively.

Original: Since the descriptors retain their spatial locations, we canuse the query mask M Q and target mask M T (e.g., one generated through saliency or user-provided)to only select the foreground pixels (Fig. 1-Step 1).
Revised: Since dense descriptors retain their spatial locations, we can use the query mask M Q and target mask M T (e.g., one generated through saliency or user-provided) to only select the foreground pixels (Fig. 1-Step 1).

Original: A conventional pixel correspondence method, such as [32], can then be applied to find matchesbetween the two images.
Revised: A conventional pixel correspondence method, such as [32], can then be applied to ﬁnd matches between the two images.

Original: However, the method isfaced with several issues due to dealing with partsinstead of points: (i) occluded or missing parts/objects in the target (e.g., a cup that does not havea handle); (ii) some parts are more distinctive than others (e.g., the tip of aknife’s blade versus itscentre); (iii)single descriptors could be noisy and Euclidean distance between them is not sufficientfor best-matching (see [18, 22]).
Revised: However, it is dealing with several issues: (i) occluded or missing parts/objects in the target (e.g., a cup that does not have a handle); (ii) some parts are more distinctive than others (e.g., the tip of a blade versus its centre); (iii) descriptors could be noisy and Euclidean distance between them is not sufﬁcient for best-matching (see [18, 22]).

Original: Here, as shown in Fig. 1-Step 2, we counteract these issues byfirst grouping the query descriptorsinto K Q query groups andthe foreground target pixels into K T target groups. Each group is thendescribed by its centroid descriptors F Q and F T , fg , respectively.
Revised: We start by grouping the descriptors into K Q query groups and K T target groups (see Fig. 1-Step 2), we get mean descriptors F Q and F T , fg , respectively, which are less noisy than single descriptors.

Original:  Through pairwise cosine similarity, we can then find how close each of the F Q centroids is to each F T , fg centroid descriptors, assim ( F Q , F T , fg )
Revised: Forward matching: Through pairwise cosine similarity, we can ﬁnd how close each of the F Q centroids is to each F T , fg centroid descriptors, as sim ( F Q , F T , fg )

Original: ∈ R K Q × K T .
Revised: ∈ R K Q × K T .

Original: Through the softmax operation along the target axis, we can find, for eachquery centroid, we determine the probability of matching each target centroid as
Revised: For each query centroid, we determine the probability that it matches the target centroids as

Original: [Equation]
Revised: [Equation]

Original: where τ QT is the softmax temperature.
Revised: where τ QT is the softmax temperature and T , fg is the target centroid axis.

Original: Similarly, we could find the backward pass, i.e., matchesfrom the target to the support.
Revised: We want to enable one-tomany matching, as well as matching of regions with different scales (e.g., matching a small handle to a large one).

Original: However,in this case, we are not interested only in the matcheswithin the query, rather in all matches anywhere in the support.
Revised: However, we ﬁnd that the matching probability A QT in larger less distinctive areas is spread more and thus – much smaller.

Original: With this information, we couldexclude any matches that correspond to another area of the support image. Thus, our backwardmatching affinity is computed with  the full F R as
Revised: By summing over the query centroids as in Eq. (2), we get ‘votes’ V QT which deal with this issue by summing over the query centroids as

Original: [Equation]
Revised: [Equation]

Original: Summing the A TR over the query pixels, gives us a probability P TQ of whether each centroid corresponds to the query. Also, each centroid with P TQ > 0 .
Revised: The votes balance well between matches to distinctive areas (which are fewer but more conﬁdent) and matches to less distinctive ones (which are more numerous but less conﬁdent).

Original: 5 is likely a correspondence. Meanwhile,we also want to determine the target centroids which are the likely matches of thequery centroids in a way that allows one-to-many matching and does not only find the  matches  to the distinctiveregions of the query.
Revised: Backward matching: Similarly, we could ﬁnd the matches from the target to the support. We match to the full support descriptor image F R rather than the centroids of the selected area F Q – with this information, any matches that correspond to another area of the support image can be excluded.

Original: By summing over the query centroids asin Eq.
Revised: Thus, our backward matching afﬁnity is computed with the full F R as

Original: (3), we get ‘votes’ V QT .
Revised: [Equation]

Original: Sincedistinctive areas are typically smaller but produce more confident probabilities (in the affinity A QT ),while less distinctive ones are typically wider and have many, but lower probability, matches. Thissummation serves well tobalance the matching scores:
Revised: while the probability that each target centroid corresponds to the query region of the support image is computed by summing over the mask query pixels M Q :

Original: [Equation]
Revised: [Equation]

Original: A score for eachforeground target centroid based on both passes can be obtained by S T , fg =
Revised: Post-processing: By multiplying P TQ and V QT , we get a ‘score’ S T , fg for each target centroid, as S T , fg =

Original: V QT · P TQ .
Revised: V QT · P TQ .

Original: We map each descriptor to the score value that corresponds to its clustering.
Revised: We map each pixel to the score value of the centroid which represents it (See Fig.

Original: Since any centroidwith V QT > K Q / K T is more likely than not to be a part of a match, while any centroid with P TQ > 0 .
Revised: 1-Step 3). Each target centroid with P TQ > 0 . 5 is likely a correspondence. Meanwhile, V QT is not a probability, but rather a sum of K Q probabilities, each of which represents a match that is more likely than average to represent a true correspondence if it’s larger than 1 / K T .

Original: 5corresponds confidently back to the query, we determine K Q /2 K T as  a score threshold for goodcorrespondences.
Revised: Hence, we deem the heuristic threshold V QT > K Q / K T as representative of a likely match for each target centroid.

Original: Finally, as shown in Fig.
Revised: Finally, the two thresholds can be multiplied to get a ‘score’ threshold for each centroid.

Original: 1-Step 3, to get a smoother result with the original image dimensions, thescore image is passed along with the  K Q / 2 K T constant background energy term through a Conditional Random Field (CRF) that determines foreground and background pixels.
Revised: Using a Conditional Random Field (CRF), a smooth binary mask can be produced which loosely follows an energy boundary - we set the foreground energy term to be the scores image, while the background energy – a constant of K Q / 2 K T which is the score threshold.

Original: Design choices: Ourparticular choice of descriptor model is the DINO-ViT-S with patch size 8, pretrained on ImageNet, due toits superior properties in the similar task of co-part segmentation [8].
Revised: Design choices: Our choice of descriptor model is the DINO-ViT-S with patch size 8, pre-trained on ImageNet, due to superior properties in the similar task of co-part segmentation [8].

Original: The support and target descriptors filtering is done by applying the support query and saliency masks,respectively.
Revised: The support and target descriptors ﬁltering is done by applying the support query and saliency masks, respectively.

Original: Clustering is achieved via Fast K-Means [33] into an over-segmented image[-
 Here,we use Fast K-Means [33]-] with K Q = K T = 10 2  .
Revised: Clustering is achieved via Fast K-Means [33] into an over-segmented image[-
 Here,we use Fast K-Means [33]-] with K Q = K T = 10 2 3 .

Original: We choose τ QT = 0 .
Revised: We choose τ QT = 0 .

Original: 2, τ TR = 0 .
Revised: 2, τ TR = 0 .

Original: 02 empirically.
Revised: 02 empirically.

Original: Having a large τ QT means the forward-pass is lenient and selects many potential candidates, whilethe low τ TR filters in only the parts of those candidates that are confidently matching to the querypart rather than the rest of the reference.
Revised: Having a large τ QT means the forward matching is lenient and selects many potential candidates, while the low τ TR ﬁlters in only the parts of those candidates that are conﬁdently matching to the query part rather than the rest of the reference.

Original: This setup allows the model to find one-to-many matches,unlike other correspondence works, which assume exactly one match exists [18, 20].

Original: This comes asa trade-off of more false positives over false negatives.

Original: 3.2 Affordance Transfer
Revised: 3.2 Affordance Transfer

Original: We propose one-shot affordance skill transfer by defining a stack of skills S , i.e., affordance functions
Revised: We propose one-shot affordance skill transfer by deﬁning a stack of skills S , i.e., affordance functions

Original: S i ( o af f ) associated with manually annotated parts (and their descriptors) o
Revised: S i ( o af f ) associated with manually annotated parts (and their descriptors) o

Original: af f sensor inputs.
Revised: af f sensor inputs.

Original: Thoseskills would then be transferred to the corresponding parts
Revised: Those skills would then be transferred to the corresponding parts

Original: ˆ o af f of other objects by applying S i ( ˆ o af f ) .
Revised: ˆ o af f of other objects by applying S i ( ˆ o af f ) .

Original: For example, a skill might be to grasp at the centroid of a graspable region, or place a ball in acontainment region.
Revised: For example, a skill might be to grasp at the centroid of a graspable region, or place a ball in a containment region.

Original: In more general terms, the skill can be formulated as S i ( ˆ o af f , o class , X ) , whereo class is the (either categorical or latent) class of the support object and X is the robot’s sensory state.
Revised: In more general terms, the skill can be formulated as S i ( ˆ o af f , o class , X ) , where o class is the (either categorical or latent) class of the support object and X is the robot’s sensory state.

Original: The skills can be either robot-specific or robot-agnostic, since the chosen affordance representationdoes not make assumptions.
Revised: The skills can be either robot-speciﬁc or robot-agnostic, since the chosen affordance representation does not make assumptions.

Original: In this work, we show this system with simple first-order affordances(grasping and containing) in simple scenes, and leave multi-object manipulations for the future.
Revised: In this work, we show this system with simple ﬁrst-order affordances (grasping and containing) in simple scenes, and leave multi-object manipulations for the future.

Original:  Experiments 
Revised:  Experiments 4.1 Quality of Part Transfer

Original: 4.1 Quality of Part Transfer

Original: Metrics: We use metrics standard in affordance learning literature [34, 35]: per-affordance class Intersection over Union (also called Jaccard index), and F wb -measure [36] which is a weighted versionof IoU that accounts for pixel location and mask interpolation.
Revised: Metrics: We use metrics standard in affordance learning literature [34, 35]: per-affordance class Intersection over Union (also called Jaccard index), and F wb -measure [36] which is a weighted version of IoU that accounts for pixel location and mask interpolation.

Original: For qualitative evaluation, we showvisual comparisons.

Original: Datasets: As our main benchmark, we use the UMD Affordance Dataset [37] due to its variety ofobjects, allowing us to evaluate affordance transfer in both inter- and intra-class pairs.
Revised: Datasets: As our main benchmark, we use the UMD Affordance Dataset [37] due to its variety of objects, allowing us to evaluate affordance transfer in both inter- and intra-class pairs.

Original: Similarly tosuperpixelization how works in one-shot instance transfer use modified folds of PASCAL-5 i [38] and COCO-20 i [39],we present UMD i3 – a one/few-shot correspondence variant of UMD, which is composed of a singleinstance of each object in the dataset with both RGB image and affordance ground truth annotation.
Revised: Similarly to how works in one-shot instance transfer use modiﬁed folds of PASCAL-5 i [38] and COCO-20 i [39], we present UMD i – a one-shot correspondence variant of UMD, which is composed of a single instance of each object in the dataset with both RGB image and affordance ground truth annotation.

Original: The originalUMD annotations are kept to highlight the difficulties of semantic transfer, as no twohuman annotations are the same.
Revised: The original annotations are kept to highlight the difﬁculties of semantic transfer, as no two human annotations are the same.

Original: The classes include common objects such as bowls, ladles, scissors,and knives, with manually labelledannotations of grasp, scoop, wrap-grasp, support, contain, cut,and pound affordances.
Revised: The classes include common objects such as bowls, ladles, and knives, with manually labelled grasp, scoop, wrap-grasp, support, contain, cut, and pound affordances.

Original: Experimental Setup: For each object in UMD i , we attempt to transfer its ground truth part masksto each other object of the same class ( intra-class ) and to each other object of classes that possessthe same affordance ( inter-class ) from the dataset.
Revised: Experimental Setup: For each object in UMD i , we attempt to transfer its ground truth part masks to each other object of the same class ( intra-class ) and to each other object of classes that possess the same affordance ( inter-class ) from the dataset.

Original: The quantitative metrics are averaged over eachaffordance type.
Revised: The quantitative metrics are averaged over each affordance type.

Original: The qualitative comparison is shown as well.
Revised: The qualitative comparison is shown as well.

Original: Baselines: As an upper baseline, we include  fully supervisedmodel reported scores [34, 35] on the
Revised: Baselines: As an upper baseline, we include the reported metrics of fully supervised [34, 35] on the

Original: UMD dataset (standard split for intra- and novel split for inter-class results).
Revised: UMD dataset (standard split for intra- and novel split for inter-class results).

Original: These are not directlycomparable with our method, since they could learn the way affordances are labelled within thedataset through supervision and are evaluated on the test subset of UMD, instead of UMD i .
Revised: These are not directly comparable with our method, since they could learn the way affordances are labelled within the dataset through supervision and are evaluated on the test subset of UMD, instead of UMD i .

Original: As baselines, we have included two variants of the current SotA in one-shot instance transfer–BAM [40].
Revised: As baselines, we have included two variants of the current SotA in one-shot instance transfer–BAM [40].

Original: The unsupervised co-segmentation method [8], which inspired our work, was included by takingall parts that have at least 50% overlap with theground truth parts of the support image and usingtheir correspondences as the ‘estimated parts’ in the target image. It is given the support and targetimages, and searches for between 2 and 10 parts with K-Means elbow point as in [8].
Revised: We also include the unsupervised co-segmentation method [8], which inspired our work. Since this method produces 2 to 10 unsupervised segments (by using K-Means elbow point), we take all segments in the support that have at least 50% overlap with the support ground truth, and use the aggregate of their correspondences as the ‘estimated parts’. This estimate is then compared with the target ground truth.

Original: Finally, to showcase the difficulty of semantic labelling we also included a ‘human level’: a personis shown two objects from the same class, they are given the task of one-shot transferring labels byobserving the ground truth annotation of one image and annotating the second.
Revised: Finally, to showcase the difﬁculty of semantic labelling we also included a ‘human level’: a person is shown two objects from the same class, they are given the task of one-shot transferring labels by observing the ground truth annotation of one image and annotating the second.

Original: This is done oncefor each UMD i object as the target, after which the metric score is scaled by the number of objectsbelonging to that class before computing the per-affordance means.
Revised: This is done once for each UMD i object as the target, after which the metric score is scaled by the number of objects belonging to that class before computing the per-affordance means.

Revised: support image are transferred

Original:  Results: A visualization of parts transferred with AffCorrs in Figure 2 shows that the masks are relatively robust to viewpoint variance (see mugs), and missing correspondences (see knife-to-hammer,and mug-to-cup), while being surprisingly capable of transferring affordance across dissimilar regions.
Revised: AffCorrs to [Table] [Table] Results: A visualization of parts transferred with AffCorrs in Figure 2 shows that the masks are relatively robust to viewpoint variance (see mugs), and missing correspondences (see knife-to-hammer, and mug-to-cup), while being surprisingly capable of transferring affordance across dissimilar regions.

Original: In Figure 3 we show a comparison with the unsupervised baselines.
Revised: In Figure 3 we show a comparison with the unsupervised baselines.

Original: Both the qualitative andquantitative comparisons (in Tables 1 and 2) affirm that AffCorrs performs better on UMD i 4 .
Revised: Both the qualitative and quantitative comparisons (in Tables 1 and 2) afﬁrm that AffCorrs performs better on UMD i 4 .

Original: The
Revised: The

Original: BAM baseline under-performs, likely due to being tailored for whole object instance transfer ratherthan parts.
Revised: BAM baseline under-performs, likely due to being tailored for whole object instance transfer rather than parts.

Original: Meanwhile, the co-part segmentation, which uses the same saliency masks and backboneas AffCorrs, appears to often ignore the foreground when it doesn’t deem it common enough acrossthe objects, and produce parts that don’t align with what we would consider semantically significant. Grasp Cut Scoop
Revised: Meanwhile, the co-part segmentation, which uses the same saliency masks and backbone as AffCorrs, appears to often ignore the foreground when it doesn’t deem it common enough across the objects, and produce parts that don’t align with what we would consider semantically signiﬁcant.

Original: Contain

Original: Wrap

Original: [Figure]

Original: grasp

Original: Pound

Original: Support

Original: IoU

Original: F

Original: [Figure]

Original: IoU

Original: F

Original: [Figure]

Original: β

Original: IoU

Original: F w β

Original: IoU

Original: F w β

Original: IoU

Original: F

Original: [Figure]

Original: IoU

Original: F

Original: [Figure]

Original: IoU

Original: F

Original: [Figure]

Original: Supervised

Original: ResNet [34] 0.

Original: ADNet

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: AffNet [3]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: Unsupervised / One

Original: [Figure]

Original: Shot Transfer

Original: BAM

Original: [Figure]

Original: ResNet [40]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Figure]

Original: VGG

Original: [Table]

Original: [Table]

Original: DINO

Original: [Figure]

Original: ViT [8]

Original: [Table]

Original: AffCorrs (ours)

Original: [Table]

Original: Human level

Original: [Table]

Original: [Table]

Original: 4.2 Affordance Transfer in the Real World
Revised: 4.2 Affordance Transfer in the Real World

Original: Experimental Setup: To showcase the applicationof AffCorrs to affordance transfer and evaluatethe current limitations in realistic scenes, we present the following evaluation setup: we use a Franka Emika Panda robot with an arm-mounted RGB-D camera.
Revised: Experimental Setup: To showcase the application to affordance transfer and evaluate the current limitations in realistic scenes, we present the following evaluation setup: we use a Franka Emika manipulator with an arm-mounted RGB-D camera.

Original: As a query, we use a single image of ascrewdriver toy withthe annotated grasping area, and a mug withthe annotated containment.
Revised: As a query, we use a single image of a screwdriver toy with annotated grasping area, and a mug with annotated containment.

Original: Thequery is one-shot transferred to the robot’s unseen environment.
Revised: The query is one-shot transferred to the robot’s unseen environment.

Original: Finally, the robot attempts to usethe affordance skill.
Revised: Finally, the robot attempts to use the affordance skill.

Original: We test the method in two settings (See Figure 4) – single object and multiple objects.
Revised: We test the method in two settings (See Figure 4) – single object and multiple objects.

Original: Single objectscenes contain one object that belongs to the same class as the query (intra-class), or an object from adifferent class (inter-class) that has an affordance equivalent to the queried one (e.g., graspable part).
Revised: Single object scenes contain one object that belongs to the same class as the query (intra-class), or an object from a different class (inter-class) that has an affordance equivalent to the queried one (e.g., graspable part).

Original: In the scenes containing multiple objects, there are several objects that are ‘true’ correspondences, along with distractor objects.
Revised: In the scenes containing multiple objects, there are several objects that are ‘true’ correspondences (See Appx. C) , along with distractor objects.

Original: In both cases, the background was varied with different surfaces (table,carton, textile, cloths).
Revised: In both cases, the background was varied with different surfaces (table, carton, textile, whiteboard).

Original: The objects used are relatively spaced out, with occasional occlusion.
Revised: The objects used are relatively spaced out, with occasional occlusion.

Original: Tenexamples of either setting were used for the following robot experiments.
Revised: Ten examples of either setting were used for the following robot experiments.

Original: Two simple first-order affordance skills are shown: grasping and containment.

Original: More complex multiobject interactions are left for future work.

Original: The grasping skill is defined as picking at the 3D centroidof a part, with grasp orientation along the largest PCA axis in XY-space (i.e., we assume a top grasp).

Original: The containment skill is defined as opening the robot gripper above the XY centroid of the affordance region.

Original: Each affordance is attempted ten times for each setting.

Original: An attempt is deemedsuccessful if the robot successfully uses the affordance of all objects that correspond to the query,e.g., grasps all objects with tool handles and doesn’t attempt to grasp a non-corresponding object.

Original: Results: An example grasp in multi-object scene is shown in Figure 5.
Revised: Results: An example grasp in multi-object scene is shown in Figure 5.

Original: In the one-object setting,we observed a 100% success rate for both grasping and containment.
Revised: In the one-object setting, we observed a 100% success rate for both grasping and containment.

Original: In the multi-object setting,the grasping success rate dropped to 70% and the containment – to 80%.
Revised: In the multi-object setting, the grasping success rate dropped to 70% and the containment – to 80%.

Revised: Refer to Appendix B for grasping location examples and baseline comparison.

Original: As noted by previous OSIS approaches [31, 25], AffCorrs too shows that cluttered scenes are more challenging, whichunderscores the importance of extending the model in order to process such scenes robustly.
Revised: As noted by previous OSIS approaches [31, 25], AffCorrs too shows that cluttered scenes are more challenging, which underscores the importance of extending the model in order to process such scenes robustly.

Original: 4.3 Transfer across Less Common Objects
Revised: 4.3 Transfer across Less Common Objects

Original: Finally, we briefly motivate the practical usefulness of the one-shot property of AffCorrs over supervised methods: AffCorrs can produce good semantic part correspondences across less commonobjects, allowing it to work with very specific affordances that likely are not represented in any big dataset.
Revised: Finally, we brieﬂy motivate the practical usefulness of the one-shot property of AffCorrs over supervised methods: AffCorrs can produce good semantic part correspondences across less common objects, allowing it to work with very speciﬁc affordances that likely are not represented in any big dataset.

Original: Moreover, transferring a skill from simulation to reality could be made easy by using suchan affordance representation; see examples in Figure 6.
Revised: Moreover, transferring a skill from simulation to reality could be made easy by using such an affordance representation; see examples in Figure 6.

Original:  Limitations
Revised:  Limitations

Original: We highlight several important limitations of the current method (see Figure 7): AffCorrs searchesfor correspondences with probability-based cycle-consistency, which means that it can find correspondences between vastly different objects provided that they have similar descriptors.
Revised: We highlight several important limitations of the current method (see Figure 7): AffCorrs searches for correspondences with probability-based cycle-consistency, which means that it can ﬁnd correspondences between vastly different objects provided that they have similar descriptors.

Original: This is inthe method’s favour when the support image is suitable - e.g., when transferring from a screwdriverto a hammer’s handle, but not as much when the support image is different, such as a bowl to ahammer.
Revised: This is in the method’s favour when the support image is suitable - e.g., when transferring from a screwdriver to a hammer’s handle, but not as much when the support image is different, such as a bowl to a hammer.

Original: We have seen that the descriptor similarity alone may not be enough to always preventincorrect matches from appearing, which limits the current model to simpler scenes without severeclutter.
Revised: We have seen that the descriptor similarity alone may not be enough to always prevent incorrect matches from appearing, which limits the current model to simpler scenes without severe clutter.

Original: This is also limiting the method’s ability to perform well in clutter.
Revised: This is also limiting the method’s ability to perform well in clutter.

Original: Meanwhile, the alternative of having more conservative matching would limit the transfer-ability across inter-classpairs.
Revised: Meanwhile, the alternative of having a conservative matching would limit the transfer-ability across inter-class pairs.

Original: Another issue stems from the descriptors themselves – the DINO-ViT descriptors confuse betweenan object’s texture (e.g., a print of a dog on a mug) and an actual object (e.g., a real dog).
Revised: Another issue stems from the descriptors themselves – the DINO-ViT descriptors confuse between an object’s texture (e.g., a print of a dog on a mug) and an actual object (e.g., a real dog).

Original: This wouldmean that two same-class objects with and without textures sometimes result in different parts.
Revised: This would mean that two same-class objects with and without textures sometimes result in different parts.

Original: Thetransformer’s positional encoding makes the model biased toward picking a correspondence thatis similarly located within the image instead of the actual correspondence.
Revised: The transformer’s positional encoding makes the model biased toward picking a correspondence that is similarly located within the image instead of the actual correspondence.

Original: This limitation couldpotentially be addressed by performing flipping and cropping augmentations before the clusteringsteps.
Revised: This limitation could potentially be addressed by performing ﬂipping and cropping augmentations before the clustering steps.

Original: In terms of affordances, this work shows some simple affordance interactions, however themethod could, in theory, be applied to multi-object interactions.
Revised: In terms of affordances, this work shows some simple affordance interactions, however the method could, in theory, be applied to multi-object interactions.

Original: The limitations of that direction areyet to be assessed.
Revised: The limitations of that direction are yet to be assessed.

Original:  Conclusion and Future Work
Revised:  Conclusion and Future Work

Original: [Figure] In this work, we showed how pre-trained DINO-ViT’s descriptors can be adapted for part-basedtransfer of affordances.
Revised: In this work, we showed how pre-trained DINO-ViT’s descriptors can be adapted for part-based transfer of affordances.

Original: We have shown that AffCorrs is better suited for this task than the current best one-shot instance segmentation baseline and has an impressive cross-object transfer of partsegmentation.
Revised: We have shown that AffCorrs is better suited for this task than the current best one-shot instance segmentation baseline and has an impressive cross-object transfer of part segmentation.

Original: However, it’s limitations in working with clutter need to be addressed before anyreal-world application.
Revised: However, it’s limitations in working with clutter need to be addressed before any real-world application.

Original: Potential candidates to solve this problem include using an object detectionmodel (such as DetCo [41]), more sophisticated matching, or some form of unsupervised latent classification.
Revised: Potential candidates to solve this problem include using an object detection model (such as DetCo [41]), more sophisticated matching, or some form of unsupervised latent classiﬁcation.

Original: Solving this issue would make the method suitable for more complex dataset benchmarks,such as the IIT’s affordance dataset [10].
Revised: Solving this issue would make the method suitable for more complex dataset benchmarks, such as the IIT’s affordance dataset [10].

Original: While this work is the first tackling the one-shot affordancetransfer problem, it opens the door for many future directions such as learning to both discover anduse affordances from one-shot observations by looking at how people interact with objects; usingaffordance regions for transfer of more complex multi-object interactions; assisting teleoperationusing affordance-guided object manipulation rather than hand-to-robot movement transfer. 
Revised: While this work is the ﬁrst tackling the one-shot affordance transfer problem, it opens the door for many future directions such as learning to both discover and use affordances from one-shot observations by looking at how people interact with objects; using affordance regions for transfer of more complex multi-object interactions; assisting teleoperation using affordance-guided object manipulation rather than hand-to-robot movement transfer. Acknowledgement

Revised: This work was supported by the UKRI Future Leaders Fellowship [MR/V025333/1] (RoboHike) and the CDT for Foundational Artiﬁcial Intelligence [EP/S021566/1].

Revised: For the purpose of Open Access, the author has applied a CC BY public copyright licence to any Author Accepted Manuscript version arising from this submission.