Original: TAX-Pose: Task-Speciﬁc Cross-Pose Estimation for Robot Manipulation
Revised: TAX-Pose: Task-Speciﬁc Cross-Pose Estimation for Robot Manipulation

Original: Abstract: How do we imbue robots with the ability to efﬁciently manipulate unseen objects and transfer relevant skills based on demonstrations?
Revised: Abstract: How do we imbue robots with the ability to efﬁciently manipulate unseen objects and transfer relevant skills based on demonstrations?

Original: End-to-end learning methods often fail to generalize to novel objects or unseen conﬁgurations.
Revised: End-to-end learning methods often fail to generalize to novel objects or unseen conﬁgurations.

Original: Instead, we focus on the task-speciﬁc pose relationship between relevant parts of interacting objects.
Revised: Instead, we focus on the task-speciﬁc pose relationship between relevant parts of interacting objects.

Original: We conjecture that this relationship is a generalizable notion of a manipulation task that can transfer to new objects in the same category; examples include the relationship between the pose of a pan relative to an oven or the pose of a mug relative to a mug rack.
Revised: We conjecture that this relationship is a generalizable notion of a manipulation task that can transfer to new objects in the same category; examples include the relationship between the pose of a pan relative to an oven or the pose of a mug relative to a mug rack.

Original: We call this task-speciﬁc pose relationship “cross-pose” and provide a mathematical deﬁnition of this concept.
Revised: We call this task-speciﬁc pose relationship “cross-pose” and provide a mathematical deﬁnition of this concept.

Original: We propose a vision-based system that learns to estimate the cross-pose between two objects for a given manipulation task using learned cross-object correspondences.
Revised: We propose a vision-based system that learns to estimate the cross-pose between two objects for a given manipulation task using learned cross-object correspondences.

Original: The estimated cross-pose is then used to guide a downstream motion planner to manipulate the objects into the desired pose relationship (placing a pan into the oven or the mug onto the mug rack).
Revised: The estimated cross-pose is then used to guide a downstream motion planner to manipulate the objects into the desired pose relationship (placing a pan into the oven or the mug onto the mug rack).

Original: We demonstrate our method’s capability to generalize to unseen objects, in some cases after training on only 10 demonstrations in the real world.
Revised: We demonstrate our method’s capability to generalize to unseen objects, in some cases after training on only 10 demonstrations in the real world.

Original: Results show that our system achieves state-of-the-art performance in both simulated and real-world experiments across a number of tasks.
Revised: Results show that our system achieves state-of-the-art performance in both simulated and real-world experiments across a number of tasks.

Original: Supplementary information and videos can be found on our project website.
Revised: Supplementary information and videos can be found on our project website.

Original: [Figure]
Revised: [Figure]

Original:  Introduction
Revised:  Introduction

Original: Many manipulation tasks require a robot to move an object to a location relative to another object.
Revised: Many manipulation tasks require a robot to move an object to a location relative to another object.

Original: For example, a cooking robot may need to place a lasagna in an oven, place a pot on a stove, place a plate in a microwave, place a mug onto a mug rack, or place a cup onto a shelf.
Revised: For example, a cooking robot may need to place a lasagna in an oven, place a pot on a stove, place a plate in a microwave, place a mug onto a mug rack, or place a cup onto a shelf.

Original: Understanding and placing objects in task-speciﬁc locations is a key skill for robots operating in human environments.
Revised: Understanding and placing objects in task-speciﬁc locations is a key skill for robots operating in human environments.

Original: Further, this skill should generalize to novel objects within the training categories, such as placing new trays into the oven or new mugs onto a mug rack.
Revised: Further, this skill should generalize to novel objects within the training categories, such as placing new trays into the oven or new mugs onto a mug rack.

Original: A common approach in robot learning is to train a policy “end-to-end,” mapping from pixel observations to low-level robot actions.
Revised: A common approach in robot learning is to train a policy “end-to-end,” mapping from pixel observations to low-level robot actions.

Original: However, end-to-end trained policies cannot easily reason about complex pose relationships such as the ones described above, and they have difﬁculty generalizing to unseen object conﬁgurations.
Revised: However, end-to-end trained policies cannot easily reason about complex pose relationships such as the ones described above, and they have difﬁculty generalizing to unseen object conﬁgurations.

Original: In contrast, we propose a method that learns to reason about the 3D geometric relationship between a pair of objects.
Revised: In contrast, we propose a method that learns to reason about the 3D geometric relationship between a pair of objects.

Original: For the type of tasks deﬁned above, the robot needs to reason about the relationship between key parts on one object with respect to key parts on another object.
Revised: For the type of tasks deﬁned above, the robot needs to reason about the relationship between key parts on one object with respect to key parts on another object.

Original: For example, to place a mug on a mug rack, the robot must reason about the relationship between the pose of the mug handle and the pose of the mug rack; if the mug rack changes its pose, then the pose of the mug must change accordingly in order to still be placed on the rack (see Figure 3).
Revised: For example, to place a mug on a mug rack, the robot must reason about the relationship between the pose of the mug handle and the pose of the mug rack; if the mug rack changes its pose, then the pose of the mug must change accordingly in order to still be placed on the rack (see Figure 3).

Original: We name this taskspeciﬁc notion of the pose relationship between a pair of objects as “cross-pose” and we formally deﬁne it mathematically.
Revised: We name this taskspeciﬁc notion of the pose relationship between a pair of objects as “cross-pose” and we formally deﬁne it mathematically.

Original: Further, we propose a vision system that can efﬁciently estimate the crosspose from a small number of demonstrations of a given task, generalizing to novel objects within the training categories.
Revised: Further, we propose a vision system that can efﬁciently estimate the crosspose from a small number of demonstrations of a given task, generalizing to novel objects within the training categories.

Original: To complete the manipulation task, we use the estimated cross-pose as the target of a motion planning algorithm, which will move the objects into the desired conﬁguration (e.g. placing the mug onto the rack, placing the lasagna into the oven, etc).
Revised: To complete the manipulation task, we use the estimated cross-pose as the target of a motion planning algorithm, which will move the objects into the desired conﬁguration (e.g. placing the mug onto the rack, placing the lasagna into the oven, etc).

Original: [Figure]
Revised: [Figure]

Original:  In this paper, we present TAX-Pose (TAsk-speciﬁc CrossPose), a deep 3D vision-based method that learns to predict a task-speciﬁc pose relationship between a pair of objects from a set of demonstrations.
Revised: the mug rack. In this paper, we present TAX-Pose (TAsk-speciﬁc CrossPose), a deep 3D vision-based method that learns to predict a task-speciﬁc pose relationship between a pair of objects from a set of demonstrations.

Original: Our cross-pose estimation system is provably translation equivariant and can generalize from a small number of real-world demonstrations (in some cases as few as 10) to new objects in unseen poses.
Revised: Our cross-pose estimation system is provably translation equivariant and can generalize from a small number of real-world demonstrations (in some cases as few as 10) to new objects in unseen poses.

Original: The contributions of this paper include:
Revised: The contributions of this paper include:

Original: A precise deﬁnition of “cross-pose,” which deﬁnes a task-speciﬁc pose relationship between two objects.
Revised: A precise deﬁnition of “cross-pose,” which deﬁnes a task-speciﬁc pose relationship between two objects.

Original: A novel method that estimates soft-correspondences between two objects, from which the cross-pose between the objects can be estimated (see Figure 1); this method is provably translation equivariant and can learn from a small number of real-world demonstrations.
Revised: A novel method that estimates soft-correspondences between two objects, from which the cross-pose between the objects can be estimated (see Figure 1); this method is provably translation equivariant and can learn from a small number of real-world demonstrations.

Original: A robot system to manipulate objects into the desired cross-pose to achieve a given manipulation task.
Revised: A robot system to manipulate objects into the desired cross-pose to achieve a given manipulation task.

Original: We present simulated and real-world experiments to test the performance of our system in achieving a variety of relative placement manipulation tasks.
Revised: We present simulated and real-world experiments to test the performance of our system in achieving a variety of relative placement manipulation tasks.

Original: We demonstrate our method on a semantic placement task, in which the robot must place an object in, on, or around a novel object (Figure 2, top).
Revised: We demonstrate our method on a semantic placement task, in which the robot must place an object in, on, or around a novel object (Figure 2, top).

Original: We also demonstrate our method on precise placement tasks, such as hanging a mug on a rack (Figure 2, bottom) or placing a bottle or bowl on a shelf; in both cases our method generalizes to new object conﬁgurations and new objects within the training categories.
Revised: We also demonstrate our method on precise placement tasks, such as hanging a mug on a rack (Figure 2, bottom) or placing a bottle or bowl on a shelf; in both cases our method generalizes to new object conﬁgurations and new objects within the training categories.

Original:  Related Work
Revised:  Related Work

Original: Object Pose Estimation :
Revised: Object Pose Estimation :

Original: Pose estimation is the task of detecting and inferring the 6DoF pose of an object, which includes its position and orientation, with respect to some previously deﬁned object reference frame [1, 2, 3, 4, 5, 6].
Revised: Pose estimation is the task of detecting and inferring the 6DoF pose of an object, which includes its position and orientation, with respect to some previously deﬁned object reference frame [1, 2, 3, 4, 5, 6].

Original: Recent work [7, 8, 9, 10] proposed to use 3D semantic keypoints as an alternative form of object representation.
Revised: Recent work [7, 8, 9, 10] proposed to use 3D semantic keypoints as an alternative form of object representation.

Original: While keypoint-based methods can generalize within an object class, they require a signiﬁcant amount of hand annotated data or access to a simulated version of the task to learn to estimate the keypoint locations.
Revised: While keypoint-based methods can generalize within an object class, they require a signiﬁcant amount of hand annotated data or access to a simulated version of the task to learn to estimate the keypoint locations.

Original: In contrast, our method is able to learn from just 10 real-world demonstrations.
Revised: In contrast, our method is able to learn from just 10 real-world demonstrations.

Original: Another approach is to use dense embeddings, such as Dense Object Nets (DON) [11] and Neural Descriptor Fields (NDF) [12], which achieve generalization across classes by predicting dense embeddings in the observation and matching them to embeddings of the demonstration objects.
Revised: Another approach is to use dense embeddings, such as Dense Object Nets (DON) [11] and Neural Descriptor Fields (NDF) [12], which achieve generalization across classes by predicting dense embeddings in the observation and matching them to embeddings of the demonstration objects.

Original: However, DON [11] and NDF [12] assume that the target object is moved relative to a static reference object in a “known canonical conﬁguration” (e.g. the pose of the mug rack in NDF [12] is assumed to be known and ﬁxed).
Revised: However, DON [11] and NDF [12] assume that the target object is moved relative to a static reference object in a “known canonical conﬁguration” (e.g. the pose of the mug rack in NDF [12] is assumed to be known and ﬁxed).

Original: In contrast, our method reasons about the geometric relationship between a pair of objects and hence does not need to assume a static environment.
Revised: In contrast, our method reasons about the geometric relationship between a pair of objects and hence does not need to assume a static environment.

Original: Thus, for example, our method is able to perform the mug hanging task while varying the pose of the mug rack (see our project website), whereas the baselines (DON [11],
Revised: Thus, for example, our method is able to perform the mug hanging task while varying the pose of the mug rack (see our project website), whereas the baselines (DON [11],

Original: NDF [12]) cannot.
Revised: NDF [12]) cannot.

Original: Further, we show that our method signiﬁcantly outperforms both the DON [11] and NDF [12] baselines, especially when given a very small number of demonstrations.
Revised: Further, we show that our method signiﬁcantly outperforms both the DON [11] and NDF [12] baselines, especially when given a very small number of demonstrations.

Original: Point Cloud Registration : Our method for estimating the cross-pose between two objects builds upon previous work in point cloud registration.
Revised: Point Cloud Registration : Our method for estimating the cross-pose between two objects builds upon previous work in point cloud registration.

Original: The typical objective in point cloud registration is to ﬁnd the optimal rigid alignment between two point clouds, to minimize the sum of squared distances between two sets of points.
Revised: The typical objective in point cloud registration is to ﬁnd the optimal rigid alignment between two point clouds, to minimize the sum of squared distances between two sets of points.

Original: Traditionally, Iterative Closest Point (ICP) [13] and its variants [14, 15, 16, 17, 18, 19] have been used to compute the optimal rigid alignment between two point clouds.
Revised: Traditionally, Iterative Closest Point (ICP) [13] and its variants [14, 15, 16, 17, 18, 19] have been used to compute the optimal rigid alignment between two point clouds.

Original: Deep Closest Point (DCP) [20] avoids local minima common for ICP by seeking to approximate correspondence in a high-dimensional learned feature space.
Revised: Deep Closest Point (DCP) [20] avoids local minima common for ICP by seeking to approximate correspondence in a high-dimensional learned feature space.

Original: Our method builds upon the architecture of DCP for cross-pose estimation; however, in contrast to point cloud registration, in which the objective is to minimize the sum of squared distances between two sets of points on the same object in two different poses, our objective is to estimate a task-speciﬁc pose relationship between two different objects.
Revised: Our method builds upon the architecture of DCP for cross-pose estimation; however, in contrast to point cloud registration, in which the objective is to minimize the sum of squared distances between two sets of points on the same object in two different poses, our objective is to estimate a task-speciﬁc pose relationship between two different objects.

Original: Extending the framework from DCP, we learn a residual to the soft correspondences, allowing for points to match outside the convex hull of each object.
Revised: Extending the framework from DCP, we learn a residual to the soft correspondences, allowing for points to match outside the convex hull of each object.

Original: This component is necessary when computing soft correspondences between objects of drastically different morphologies (such as a mug and a mug rack).
Revised: This component is necessary when computing soft correspondences between objects of drastically different morphologies (such as a mug and a mug rack).

Original:  Problem Statement
Revised:  Problem Statement

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Relative placement tasks:
Revised: Relative placement tasks:

Original: In this paper, we are specifically interested in “relative placement tasks.”
Revised: In this paper, we are specifically interested in “relative placement tasks.”

Original: Given two objects, A and B , a “relative placement task” is the task of placing object A at a pose relative to object B .
Revised: Given two objects, A and B , a “relative placement task” is the task of placing object A at a pose relative to object B .

Original: For example, consider the task of placing a lasagna in an oven, placing a mug on a rack, or placing a robot gripper on the rim of a mug.
Revised: For ex- ample, consider the task of placing a lasagna in an oven, placing a mug on a rack, or placing a robot gripper on the rim of a mug.

Original: All of these tasks involve placing one object (which we call the “action” object A ) at a semantically meaningful location relative to another object (which we call the “anchor” object B ) 2 .
Revised: All of these tasks involve placing one object (which we call the “action” object A ) at a semantically meaningful location relative to another object (which we call the “anchor” object B ) 2 .

Original: Speciﬁcally, suppose that T ∗A and T ∗B are SE (3) poses for objects A and B respectively (in a shared world reference frame 3 ) for which a desired task is considered complete (lasagna is in the oven; mug is on the rack, etc).
Revised: Speciﬁcally, suppose that T ⇤ A and T ⇤ B are SE (3) poses for objects A and B respectively (in a shared world refer- ence frame 3 ) for which a desired task is considered complete (lasagna is in the oven; mug is on the rack, etc).

Original: Then for a relative placement task, if objects A and B are in poses T · T ∗A and T · T ∗B (respectively) for any transform T , then the task will also be considered to be complete, as seen in Figure 3.
Revised: Then for a relative placement task, if objects A and B are in poses T · T ⇤ A and T · T ⇤ B (respectively) for any trans- form T , then the task will also be considered to be complete, as seen in Figure 3.

Original: In other words, if T ∗B represents the pose of the rack and T ∗A represents the pose of the mug on the rack (at task completion); then if we transform the both the mug and rack poses by T , then the mug will still be located on the rack.
Revised: In other words, if T ⇤ B represents the pose of the rack and T ⇤ A represents the pose of the mug on the rack (at task completion); then if we transform the both the mug and rack poses by T , then the mug will still be located on the rack.

Original: Formally, this property can be deﬁned with the following Boolean function,
Revised: Formally, this property can be deﬁned with the following Boolean function,

Original: RelPlace [Equation]
Revised: RelPlace ( T A , T B ) =

Original: A
Revised: S UCCESS iff 9 T 2 SE (3) s.t.

Original: [Equation]
Revised: [Equation]

Original: For many real semantic placement tasks, there are actually sets of valid solutions which solve each task (i.e., there are many potential locations to place an object on a table to achieve a semantic “object-on-table” relationship).
Revised: For many real semantic placement tasks, there are actually sets of valid solutions which solve each task (i.e., there are many potential locations to place an object on a table to achieve a semantic “object-on-table” relationship).

Original: However, for this work, we consider precise placement tasks under the simplifying assumption that, for a given pose of object B , there is a single, unambiguous pose of object A needed to achieve the task.
Revised: However, for this work, we consider precise placement tasks under the simplifying assumption that, for a given pose of object B , there is a single, unambiguous pose of object A needed to achieve the task.

Original: Deﬁnition of Cross-Pose:
Revised: Deﬁnition of Cross-Pose:

Original: Given the above deﬁnition of a relative placement task, our goal will be to determine how to move object A so that it will be in the “goal pose,” which, as described above, is deﬁned relative to the pose of object B .
Revised: Given the above deﬁnition of a relative placement task, our goal will be to determine how to move object A so that it will be in the “goal pose,” which, as described above, is deﬁned relative to the pose of object B .

Original: To achieve this, one option is to estimate the poses of objects
Revised: To achieve this, one option is to estimate the poses of objects

Original: A and B separately and then compute the transformation needed to move object A into the goal pose.
Revised: A and B separately and then compute the transformation needed to move object A into the goal pose.

Original: However, the pose estimate of each object will have errors, and these errors will accumulate when the poses are combined into the single relative pose needed to reach the goal conﬁguration.
Revised: However, the pose estimate of each object will have errors, and these errors will accumulate when the poses are combined into the single relative pose needed to reach the goal conﬁguration.

Original: Instead of estimating the pose of each object independently, we aim to learn a function f
Revised: Instead of estimating the pose of each object independently, we aim to learn a function f

Original: ( P A , P B ) , which takes as input the point clouds P A and P B for both objects A and B , where P A ∈ R 3 × N A and P B ∈ R 3 × N B are 3D point clouds of sizes N A and N B , respectively.
Revised: ( P A , P B ) , which takes as input the point clouds P A and P B for both objects A and B , where P A 2 R 3 ⇥ N A and P B 2 R 3 ⇥ N B are 3D point clouds of sizes N A and N B , respectively.

Original: This function outputs an SE(3) rigid transformation, f ( P A , P B ) 
Revised: This function outputs an SE(3) rigid transformation, f ( P A , P B ) =

Original: = T AB , where we refer to T AB as the “cross-pose” between object A and object B .
Revised: T AB , where we refer to T AB as the “cross-pose” between object A and object B .

Original: For notational convenience, we occasional write f as a function of the poses T A , T B of point clouds P A and P B respectively (with respect to a global reference frame) such that f ( T A , T B ) :
Revised: For notational convenience, we occasional write f as a function of the poses T A , T B of point clouds P A and P B respectively (with respect to a global reference frame) such that f ( T A , T B ) :

Original: = f ( P A , P B ) .
Revised: = f ( P A , P B ) .

Original: This notational change is to make the transformation math more intuitive; in practice, this function only ever receives point clouds as input.
Revised: This notational change is to make the transformation math more intuitive; in practice, this function only ever receives point clouds as input.

Original: We will deﬁne the cross-pose T AB (below) such that, if we transform object A by T AB , then object A will be in the goal pose relative to object B for the relative placement task.
Revised: We will deﬁne the cross-pose T AB (below) such that, if we transform object A by T AB , then object A will be in the goal pose relative to object B for the relative placement task.

Original: For example, suppose that T ∗A and T ∗B are poses for objects A and B , respectively, for which a desired relative placement task is considered complete.
Revised: For example, suppose that T ⇤ A and T ⇤ B are poses for objects A and B , respectively, for which a desired relative placement task is considered complete.

Original: In this conﬁguration, the cross-pose of these objects would be f ( T ∗A , T ∗B )
Revised: In this conﬁguration, the cross-pose of these objects would be f ( T ⇤ A , T ⇤ B )

Original: =
Revised: =

Original: I where I is the identity, as object A does not need to be moved to complete the task.
Revised: I where I is the identity, as object A does not need to be moved to complete the task.

Original: Further, based on the deﬁnition of a relative placement task given above, if both objects are transformed by the same transform T , then the objects will still be in the desired relative pose,
Revised: Further, based on the deﬁnition of a relative placement task given above, if both objects are transformed by the same transform T , then the objects will still be in the desired relative pose,

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original:  for any transform T ∈ SE (3) .
Revised: ) for any transform T 2 SE (3) .

Original: Now, let us assume that objects A and B are not in the goal conﬁguration and have pose T A
Revised: Now, let us assume that objects A and B are not in the goal conﬁguration and have pose T A

Original: = T α · T ∗A and T B = T β · T ∗B , respectively, for arbitrary transforms T α and T β ∈ SE (3) .
Revised: = T ↵ · T ⇤ A and T B = T  · T ⇤ B , respectively, for arbitrary transforms T ↵ and T  2 SE (3) .

Original: We then deﬁne the “cross-pose” of objects A and B as:
Revised: We then deﬁne the “cross-pose” of objects A and B as:

Original: [Equation]
Revised: [Equation]

Original: Note that this deﬁnition is equivalent to Equation 2 for the special case of T α = T β .
Revised: Note that this deﬁnition is equivalent to Equation 2 for the special case of T ↵ = T  .

Original: This deﬁnition of cross-pose allows us to move object A into the goal conﬁguration, relative to object B :
Revised: This deﬁnition of cross-pose allows us to move object A into the goal conﬁguration, relative to object B :

Original: [Equation]
Revised: [Equation]

Original: satisfying the relative placement condition deﬁned in Equation 1 with T = T β .
Revised: satisfying the relative placement condition deﬁned in Equation 1 with T = T  .

Original: Alternatively, we could have instead transformed object B by the inverse of the cross-pose to achieve the task.
Revised: Alternatively, we could have instead transformed object B by the inverse of the cross-pose to achieve the task.

Original:  Method
Revised:  Method

Original: Overview: We frame the task of cross-pose estimation as a soft correspondence-prediction task between a pair of point clouds, followed by an analytical least-squares optimization to ﬁnd the optimal cross-pose for the predicted correspondences.
Revised: Overview: We frame the task of cross-pose estimation as a soft correspondence-prediction task between a pair of point clouds, followed by an analytical least-squares optimization to ﬁnd the optimal cross-pose for the predicted correspondences.

Original: As described in Appendix B, this correspondencebased approach allows our method to be translation-equivariant: translating either object ( A or B ) will lead to a translated cross-pose prediction.
Revised: As described in Appendix B, this correspondencebased approach allows our method to be translation-equivariant: translating either object ( A or B ) will lead to a translated cross-pose prediction.

Original: This allows our method to automatically adapt to novel positions of both the anchor and action objects, unlike previous work which assumes a static anchor [12].
Revised: This allows our method to automatically adapt to novel positions of both the anchor and action objects, unlike previous work which assumes a static anchor [12].

Original: Our method for task-speciﬁc cross-pose estimation, known as TAX-Pose, consists of the following steps, as shown in Figure 4:
Revised: Our method for task-speciﬁc cross-pose estimation, known as TAX-Pose, consists of the following steps, as shown in Figure 4:

Original: Soft Correspondence Prediction : For a pair of objects A , B , a neural network learns to predict a per-point embedding to establish a (soft) correspondence between A and B , which are called
Revised: Soft Correspondence Prediction : For a pair of objects A , B , a neural network learns to predict a per-point embedding to establish a (soft) correspondence between A and B , which are called

Original: “virtual soft correspondences.”
Revised: “virtual soft correspondences.”

Original: The corresponding points are constrained to be within the convex hulls of B and A respectively.
Revised: The corresponding points are constrained to be within the convex hulls of B and A respectively.

Original: Adjustment via Correspondence Residuals :
Revised: Adjustment via Correspondence Residuals :

Original: For most estimation tasks, some points in object A may not be within the convex hull of object B ; for instance, when a mug is placed on a mug rack, most points on the mug will be outside of the convex hull of the mug rack.
Revised: For most estimation tasks, some points in object A may not be within the convex hull of object B ; for instance, when a mug is placed on a mug rack, most points on the mug will be outside of the convex hull of the mug rack.

Original: To accommodate these cases, we apply a pointwise residual vector to displace each of the predicted soft correspondences.
Revised: To accommodate these cases, we apply a pointwise residual vector to displace each of the predicted soft correspondences.

Original: These “corrected virtual correspondences” allow points in A to correspond to locations in free space near B .
Revised: These “corrected virtual correspondences” allow points in A to correspond to locations in free space near B .

Original: Find the Optimal Transform : Because the cross-pose is deﬁned as a rigid transformation of object A , we use a differentiable weighted SVD to ﬁnd the transformation that minimizes the weighted least squares difference to the corrected virtual correspondences.
Revised: Find the Optimal Transform : Because the cross-pose is deﬁned as a rigid transformation of object A , we use a differentiable weighted SVD to ﬁnd the transformation that minimizes the weighted least squares difference to the corrected virtual correspondences.

Original: Because each step above is differentiable, the whole model can be optimized end-to-end, despite having an interpretable internal structure which we describe below.
Revised: Because each step above is differentiable, the whole model can be optimized end-to-end, despite having an interpretable internal structure which we describe below.

Original: Our method is heavily inspired by Deep Closest Point (DCP) [20].
Revised: Our method is heavily inspired by Deep Closest Point (DCP) [20].

Original: The key difference between our pose alignment model and DCP is that we predict the cross-pose between two different objects for a given task instead of registering two point clouds of an identical object.
Revised: The key difference between our pose alignment model and DCP is that we predict the cross-pose between two different objects for a given task instead of registering two point clouds of an identical object.

Original: Additionally, TAX-Pose can predict relationships where these clouds may not have any points of contact or overlap.
Revised: Additionally, TAX-Pose can predict relationships where these clouds may not have any points of contact or overlap.

Original: We now describe our cross-pose estimation algorithm in detail.
Revised: We now describe our cross-pose estimation algorithm in detail.

Original: To recap the problem statement, given objects A and B with point cloud observations P A ∈ R 3 × N A , P B ∈ R 3 × N B respectively, our objective is to estimate the task-speciﬁc cross-pose TAB =
Revised: To recap the problem statement, given objects A and B with point cloud observations P A 2 R 3 ⇥ N A , P B 2 R 3 ⇥ N B respectively, our objective is to estimate the task-speciﬁc cross-pose T

Original:  f ( P A , P B ) ∈ SE (3) .
Revised: AB = f ( P A , P B ) 2 SE (3) .

Original: Note that the cross-pose between object A and B is deﬁned with respect to a given task (e.g. putting a lasagna in the oven, putting a mug on the rack, etc).
Revised: Note that the cross-pose between object A and B is deﬁned with respect to a given task (e.g. putting a lasagna in the oven, putting a mug on the rack, etc).

Original: 4.1 Cross-Pose Estimation via Soft Correspondence Prediction
Revised: 4.1 Cross-Pose Estimation via Soft Correspondence Prediction

Original: Soft Correspondence Prediction :
Revised: Soft Correspondence Prediction :

Original: The ﬁrst step of the method is to compute two sets of correspondences between A and B , one which maps from points in A to B , and one which maps from points in B to A .
Revised: The ﬁrst step of the method is to compute two sets of correspondences between A and B , one which maps from points in A to B , and one which maps from points in B to A .

Original: These need not be a bijection, and can be asymmetric.
Revised: These need not be a bijection, and can be asymmetric.

Original: As we want each step to be differentiable, we follow DCP’s conventions and estimate a soft correspondence .
Revised: As we want each step to be differentiable, we follow DCP’s conventions and estimate a soft correspondence .

Original: This assigns a virtual soft corresponding point v A i ∈ V A to every point p
Revised: This assigns a virtual soft corresponding point v A i 2 V A to every point p

Original: A i ∈ P
Revised: A i 2 P

Original: A by computing a convex combination of points in P B , and vice versa.
Revised: A by computing a convex combination of points in P B , and vice versa.

Original: Formally:
Revised: Formally:

Original: [Equation]
Revised: [Equation]

Revised: with normalized weight vectors w A !

Original: 5b
Revised: B i

Original: [Equation]
Revised: [Equation]

Original: with normalized weight vectors w A→B i ∈ W A→B and w B→A i ∈ W B→A .
Revised: A .

Original: Importantly, these virtual corresponding points are not constrained to the surfaces of A or B ; instead, they are constrained to the convex hulls of P B and P A , respectively.
Revised: Importantly, these virtual corresponding points are not constrained to the surfaces of A or B ; instead, they are constrained to the convex hulls of P B and P A , respectively.

Original: To compute the weights w A→B
Revised: To compute the weights w A !

Original:  i , w B→A i in Equations 5a and 5b, we ﬁrst encode each point cloud
Revised: B i , w B ! A i in Equations 5a and 5b, we ﬁrst encode each point cloud

Original: P A and P B into a latent space using a neural network encoder, DGCNN [21].
Revised: P A and P B into a latent space using a neural network encoder, DGCNN [21].

Original: This encoder head is comprised of two distinct encoders g A and g B , each of which receives point cloud P A and P B , respectively, zero-centers them, and outputs a dense, point-wise embedding for each object (see
Revised: This encoder head is comprised of two distinct encoders g A and g B , each of which receives point cloud P A and P B , respectively, zero-centers them, and outputs a dense, point-wise embedding for each object (see

Original: Figure 4):
Revised: Figure 4):

Original: [Equation]
Revised: [Equation]

Original: Ψ B = g B (¯ P B ) ∈ R N B × d where ψ K i ∈ Ψ K is the ddimensional embedding of the i -th point in object K , and ¯ P K is the zero-centered point cloud for object K .
Revised: K is the d - dimensional embedding of the i -th point in object K , and ¯ P K is the zero-centered point cloud for object K .

Original: Because we want the cross-correspondence to incorporate information about both point clouds, we then employ a cross-object attention module between the two dense feature sets to obtain cross-object point embeddings , Φ A ∈ R N A × d and Φ B ∈ R N B × d , deﬁned as:
Revised: Because we want the cross-correspondence to incorporate information about both point clouds, we then employ a cross-object attention module between the two dense feature sets to obtain cross-object point embeddings ,  A 2 R N A ⇥ d and  B 2 R N B ⇥ d , deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where g T A , g T B are Transformers [22].
Revised: where g T A , g T B are Transformers [22].

Original: Finally, recall that our goal was to compute a set of normalized weight vectors W A→B , W B→A .
Revised: Finally, recall that our goal was to compute a set of normalized weight vectors W A ! B , W B !

Original:  To compute the virtual corresponding point v A i assigned to any point p A i ∈ P A , we can extract the desired normalized weight vector w A→B i from intermediate attention features of the cross-object attention module as:
Revised: A . To compute the virtual corresponding point v A i assigned to any point p A i 2 P A , we can extract the desired normalized weight vector w A ! B i from intermediate attention features of the cross-object attention module as:

Original: [Equation]
Revised: [Equation]

Original: where q K i ∈ Q K , and Q K , K K ∈ R N K × d are the query and key values (respectively) for object K associated with cross-object attention Transformer module g T K (see Appendix C for details).
Revised: where q K i 2 Q K , and Q K , K K 2 R N K ⇥ d are the query and key values (respectively) for object K associated with cross-object attention Transformer module g T K (see Appendix C for details).

Original: These weights are then used to compute the virtual soft correspondences V A , V B using Equation 5.
Revised: These weights are then used to compute the virtual soft correspondences V A , V B using Equation 5.

Original: Adjustment via Correspondence Residuals: As previously stated, the virtual soft correspondences
Revised: Adjustment via Correspondence Residuals: As previously stated, the virtual soft correspondences

Original: V A , V B given by Equations 5a and 5b are constrained to be within the convex hull of each object.
Revised: V A , V B given by Equations 5a and 5b are constrained to be within the convex hull of each object.

Original: However, many relative placement tasks cannot be solved perfectly with this constraint.
Revised: However, many relative placement tasks cannot be solved perfectly with this constraint.

Original: For instance, we might want a point on the handle of a teapot to correspond to some point above a stovetop (which lies outside the convex hull of the points on the stovetop).
Revised: For instance, we might want a point on the handle of a teapot to correspond to some point above a stovetop (which lies outside the convex hull of the points on the stovetop).

Original: To allow for such off-object correspondences, we further learn a residual vector , δ A i ∈ ∆ A for each point i that corrects each virtual corresponding point v A i .
Revised: To allow for such off-object correspondences, we further learn a residual vector ,  A i 2  A for each point i that corrects each virtual corresponding point v A i .

Original: This allows us to displace each virtual corresponding point to any arbitrary location that might be suitable for the task.
Revised: This allows us to displace each virtual corresponding point to any arbitrary location that might be suitable for the task.

Original: To compute these residual vectors, we use a point-wise neural network g R A , g R B to map each point’s embedding into a 3D residual vector:
Revised: To compute these residual vectors, we use a point-wise neural network g R A , g R B to map each point’s embedding into a 3D residual vector:

Original: [Equation]
Revised: [Equation]

Original: Applying these residual offsets to the virtual points, we get a set of corrected virtual correspondences , ˜ v A i ∈ ˜V A and ˜ v B i ∈ ˜V B , deﬁned as
Revised: Applying these residual offsets to the virtual points, we get a set of corrected virtual correspondences , ˜ v A i 2 ˜V A and ˜ v B i 2 ˜V B , deﬁned as

Original: [Equation]
Revised: [Equation]

Original: These corrected virtual correspondences ˜ v A i deﬁne the estimated goal location relative to object B for each point p i ∈ P A of object A , and likewise for each point in object B (see visualization in Appendix A.1).
Revised: These corrected virtual correspondences ˜ v A i deﬁne the estimated goal location relative to object B for each point p i 2 P A of object A , and likewise for each point in object B (see visualization in Appendix A.1).

Original: Least-Squares Cross-Pose Optimization with Weighted SVD:
Revised: Least-Squares Cross-Pose Optimization with Weighted SVD:

Revised: Given the sets of dense correspondences, ⇣

Revised: P

Revised: [Equation]

Original: Given the sets of dense correspondences, (cid:16) P A , ˜V A (cid:17) and (cid:16) P B , ˜V B (cid:17) , we would like to compute a single rigid transformation for object A .
Revised: , ˜V [Equation] ⌘ and ⇣ P B , ˜V B ⌘ , we would like to compute a single rigid transformation for object A .

Original: To do so, we solve for the transformation T AB (the cross-pose) that minimizes the weighted distance between each point and its corrected virtual correspondence.
Revised: To do so, we solve for the transformation T AB (the cross-pose) that minimizes the weighted distance between each point and its corrected virtual correspondence.

Original: Formally, this leads to the following weighted least squares optimization:
Revised: Formally, this leads to the following weighted least squares optimization:

Original: [Equation]
Revised: [Equation]

Original: where the weights α A i ∈ α A , α B i ∈ α B signify the importance of each correspondence and are predicted by a point-wise MLP as shown in Figure 4.
Revised: where the weights ↵ A i 2 ↵ A , ↵ B i 2 ↵ B signify the importance of each correspondence and are predicted by a point-wise MLP as shown in Figure 4.

Original: These weights are learned end-to-end as parameters of our network; they are visualized in Appendix A.2, which shows that the network has learned to assign more weight to the parts of the object that are most important for the task, such as the region around the mug handle (on the mug) and the region around the peg (on the rack).
Revised: These weights are learned end-to-end as parameters of our network; they are visualized in Appendix A.2, which shows that the network has learned to assign more weight to the parts of the object that are most important for the task, such as the region around the mug handle (on the mug) and the region around the peg (on the rack).

Original: Equation 9 is the well-known weighted Procrustes problem, for which there exists an analytical solution.
Revised: Equation 9 is the well-known weighted Procrustes problem, for which there exists an analytical solution.

Original: To maintain the differentiablity of the system, we use a weighted differentiable SVD operation [23] to compute the cross-pose T AB that minimizes this objective (see Appendix D for details).
Revised: To maintain the differentiablity of the system, we use a weighted differentiable SVD operation [23] to compute the cross-pose T AB that minimizes this objective (see Appendix D for details).

Original: This allows us to train the system end-to-end as described below.
Revised: This allows us to train the system end-to-end as described below.

Original: 4.2 TAX-Pose Training Pipeline
Revised: 4.2 TAX-Pose Training Pipeline

Original: To train our model, we use a segmented set of demonstration point clouds of a pair of objects in the goal conﬁguration.
Revised: To train our model, we use a segmented set of demonstration point clouds of a pair of objects in the goal conﬁguration.

Original: For each demonstration point cloud, we generate multiple training examples by transforming each object’s point cloud, P A and P B by random SE(3) transformations T α and T β , respectively.
Revised: For each demonstration point cloud, we generate multiple training examples by transforming each object’s point cloud, P A and P B by random SE(3) transformations T ↵ and T  , respectively.

Original: The predicted cross-pose, T AB , is then compared with the ground truth cross-pose,
Revised: The predicted cross-pose, T AB , is then compared with the ground truth cross-pose,

Original: T GT AB := T β T − 1 α , using an average distance loss [24] with dense regularization (see more details on our training losses in Appendix E.1).
Revised: T GT AB := T  T  1 ↵ , using an average distance loss [24] with dense regularization (see more details on our training losses in Appendix E.1).

Original:  Experiments
Revised:  Experiments

Original: To evaluate TAX-Pose, we conduct a wide range of simulated and real-world experiments on two classes of relative placement tasks: NDF [12] Tasks and PartNet-Mobility Placement Tasks.
Revised: To evaluate TAX-Pose, we conduct a wide range of simulated and real-world experiments on two classes of relative placement tasks: NDF [12] Tasks and PartNet-Mobility Placement Tasks.

Original: All tasks involve placing an “action” object at a speciﬁc location relative to an anchor object, in which the relative pose is speciﬁed by a set of demonstrations.
Revised: All tasks involve placing an “action” object at a speciﬁc location relative to an anchor object, in which the relative pose is speciﬁed by a set of demonstrations.

Original: Our method then generalizes to perform this task on novel objects in unseen conﬁgurations.
Revised: Our method then generalizes to perform this task on novel objects in unseen conﬁgurations.

Original: We refer the reader to our project website for additional results and videos.
Revised: We refer the reader to our project website for additional results and videos.

Original: [Figure]
Revised: [Figure]

Original: 5.1 NDF Tasks
Revised: 5.1 NDF Tasks

Original: We evaluate our method on all three NDF [12] tasks ( mug hanging, bottle placement, and bowl placement); see Appendix F.1.3 for results on bottle and bowl placement.
Revised: We evaluate our method on all three NDF [12] tasks ( mug hanging, bottle placement, and bowl placement); see Appendix F.1.3 for results on bottle and bowl placement.

Original: Results on mug hanging are described in more detail below.
Revised: Results on mug hanging are described in more detail below.

Original: Simulation Experiments: For our simulation experiments, we perform the task of hanging a mug on a rack as two sequential cross-pose estimation steps: grasping the mug (estimating the crosspose between the gripper and the mug) and hanging the mug on the rack (estimating the crosspose between the mug and the rack).
Revised: Simulation Experiments: For our simulation experiments, we perform the task of hanging a mug on a rack as two sequential cross-pose estimation steps: grasping the mug (estimating the crosspose between the gripper and the mug) and hanging the mug on the rack (estimating the crosspose between the mug and the rack).

Original: In Pybullet [25], we simulate a Franka Panda above a table with 4 depth cameras placed on the corners of the table.
Revised: In Pybullet [25], we simulate a Franka Panda above a table with 4 depth cameras placed on the corners of the table.

Original: The model is trained on 10 simulated demonstrations of mug hanging.
Revised: The model is trained on 10 simulated demonstrations of mug hanging.

Original: We evaluate task execution success on unseen mug instances in randomly generated initial conﬁgurations.
Revised: We evaluate task execution success on unseen mug instances in randomly generated initial conﬁgurations.

Original: We measure task success rates of 1) Grasping , where success is achieved when the object is grasped stably; 2) Placing , where success is achieved when the mug is placed stably on the rack; 3) Overall , when the predicted transforms enable both grasp and place success in sequence.
Revised: We measure task success rates of 1) Grasping , where success is achieved when the object is grasped stably; 2) Placing , where success is achieved when the mug is placed stably on the rack; 3) Overall , when the predicted transforms enable both grasp and place success in sequence.

Original: We compare our method to Neural Descriptor Field (NDF) [12] and
Revised: We compare our method to Neural Descriptor Field (NDF) [12] and

Original: Dense Object Nets (DON) [11].
Revised: Dense Object Nets (DON) [11].

Original: Details of these methods can be found in prior work [12].
Revised: Details of these methods can be found in prior work [12].

Original: Simulation Results: We evaluate our method in simulation in 100 trials consisting of unseen mug instances in random initial and goal conﬁgurations for both Upright and Arbitrary poses.
Revised: Simulation Results: We evaluate our method in simulation in 100 trials consisting of unseen mug instances in random initial and goal conﬁgurations for both Upright and Arbitrary poses.

Original: As shown in Table 1, our method signiﬁcantly outperforms the baselines for simulated mug hanging.
Revised: As shown in Table 1, our method signiﬁcantly outperforms the baselines for simulated mug hanging.

Original: We report additional results for simulated bottle and bowl placement tasks in Table S4 in Appendix F.1.3.
Revised: We report additional results for simulated bottle and bowl placement tasks in Table 8 in Appendix F.1.3.

Original: Ablation Analysis: Effects of Number of Demonstrations.
Revised: Ablation Analysis: Effects of Number of Demonstrations.

Original: To study how the number of demonstrations observed affects our method’s performance, we train our model on { 10, 5, 1 } demonstrations of upright pose mug hanging.
Revised: To study how the number of demonstrations observed affects our method’s performance, we train our model on { 10, 5, 1 } demonstrations of upright pose mug hanging.

Original: Results are found in Table 2.
Revised: Results are found in Table 2.

Original: Our method outperforms the baselines for all number of demonstrations; TAX-Pose can perform well even with as few as 5 demonstrations.
Revised: Our method outperforms the baselines for all number of demonstrations; TAX-Pose can perform well even with as few as 5 demonstrations.

Original: Cross-Pose Estimation Design Choices.
Revised: Cross-Pose Estimation Design Choices.

Original: We analyze the effects of design choices made in our CrossPose estimation algorithm for the upright pose mug hanging task.
Revised: We analyze the effects of design choices made in our CrossPose estimation algorithm for the upright pose mug hanging task.

Original: Speciﬁcally, we analyze the effects of 1) computing residual correspondence; 2) the use of weighted SVD over non-weighted in computing cross-pose; 3) using a transformer as the cross-object attention, as opposed to simpler model such as a 3-layer MLP.
Revised: Speciﬁcally, we analyze the effects of 1) computing residual correspondence; 2) the use of weighted SVD over non-weighted in computing cross-pose; 3) using a transformer as the cross-object attention, as opposed to simpler model such as a 3-layer MLP.

Original: Table 3 shows that each major component of our system is important for task success.
Revised: Table 3 shows that each major component of our system is important for task success.

Original: See more ablation experiments in Appendix F.1.1.
Revised: See more ablation experiments in Appendix F.1.1.

Original:  [Table] [Table]
Revised: [Figure] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Real-World Experiments: We explore the hanging component of the mug on a rack task in a real world environment, which requires estimating the cross-pose between the mug and the rack.
Revised: Real-World Experiments: We explore the hanging component of the mug on a rack task in a real world environment, which requires estimating the cross-pose between the mug and the rack.

Original: We train TAX-Pose using real demonstrations of 10 different mugs hung on a rack (1 demonstration each, for a total of only 10 real-world demonstrations for training).
Revised: We train TAX-Pose using real demonstrations of 10 different mugs hung on a rack (1 demonstration each, for a total of only 10 real-world demonstrations for training).

Original: A motion primitive is used to grasp each mug, after which the robot plans a trajectory to apply the predicted cross-pose to the grasped mug.
Revised: A motion primitive is used to grasp each mug, after which the robot plans a trajectory to apply the predicted cross-pose to the grasped mug.

Original: We evaluate the model on the 10 training mugs in novel poses, as well as onunseen mugs.
Revised: We evaluate the model on the 10 training mugs in novel poses, as well as onunseen mugs.

Original: For each of the 20 mugs, we conduct 5 trials, varying the mug’s and rack’s starting poses in each trial.
Revised: For each of the 20 mugs, we conduct 5 trials, varying the mug’s and rack’s starting poses in each trial.

Original: Success is recorded if a peg penetrates the mug handle at the end of the trial.
Revised: Success is recorded if a peg penetrates the mug handle at the end of the trial.

Original: Our method achieves a success rate of 62% on training mugs in novel poses and 54% on unseen mugs.
Revised: Our method achieves a success rate of 62% on training mugs in novel poses and 54% on unseen mugs.

Original: A visualization of the results can be seen in Figure 5 and on the project website.
Revised: A visualization of the results can be seen in Figure 5 and on the project website.

Original: Note that our method is able to perform the mug hanging task while varying the pose of the mug rack (see our project website), whereas the baselines (NDF [12], DON [11]) cannot because they assume a ﬁxed, known rack position (see NDF [12] for baseline details).
Revised: Note that our method is able to perform the mug hanging task while varying the pose of the mug rack (see our project website), whereas the baselines (NDF [12], DON [11]) cannot because they assume a ﬁxed, known rack position (see NDF [12] for baseline details).

Original: 5.2 PartNet-Mobility Placement Tasks
Revised: 5.2 PartNet-Mobility Placement Tasks

Original: Task Description : We also deﬁne a PartNet-Mobility Placement task as placing a given action object relative to an anchor object based on a semantic goal position.
Revised: Task Description : We also deﬁne a PartNet-Mobility Placement task as placing a given action object relative to an anchor object based on a semantic goal position.

Original: We select a set of household furniture objects from the PartNet-Mobility dataset [26] as the anchor objects, and a set of small rigid objects released with the Ravens simulation environment [27] as the action objects.
Revised: We select a set of household furniture objects from the PartNet-Mobility dataset [26] as the anchor objects, and a set of small rigid objects released with the Ravens simulation environment [27] as the action objects.

Original: For each anchor object, we deﬁne a set of semantic goal positions (i.e. ‘top’, ‘left’, ‘right’, ‘in’), where action objects should be placed relative to each anchor.
Revised: For each anchor object, we deﬁne a set of semantic goal positions (i.e. ‘top’, ‘left’, ‘right’, ‘in’), where action objects should be placed relative to each anchor.

Original: Each semantic goal position deﬁnes a unique task in our cross-pose prediction framework.
Revised: Each semantic goal position deﬁnes a unique task in our cross-pose prediction framework.

Original: Given a synthetic point cloud observation of both objects, the task is to predict a cross-pose that places the object at the speciﬁc semantic goal.
Revised: Given a synthetic point cloud observation of both objects, the task is to predict a cross-pose that places the object at the speciﬁc semantic goal.

Original: We evaluate both a goal-conditioned variant ( TAX-Pose GC ), which is trained across all goals, and a task-speciﬁc variant ( TAX-Pose ) of our model, which trains a separate model per goal type (see Appendix F.2.
Revised: We evaluate both a goal-conditioned variant ( TAX-Pose GC ), which is trained across all goals, and a task-speciﬁc variant ( TAX-Pose ) of our model, which trains a separate model per goal type (see Appendix F.2.

Original: for details).
Revised: for details).

Original: In both cases we train only 1 model across all action and anchor objects.
Revised: In both cases we train only 1 model across all action and anchor objects.

Original: All models are trained entirely on simulated data and transfer directly to real-world with no ﬁnetuning.
Revised: All models are trained entirely on simulated data and transfer directly to real-world with no ﬁnetuning.

Original: Further task details can be found in the Appendix G.2.
Revised: Further task details can be found in the Appendix G.2.

Original: Baselines : We compare our method to a variety of end-to-end imitation-learning-based methods trained from a motion planner expert in simulation (see Appendix G.2.4 for details).
Revised: Baselines : We compare our method to a variety of end-to-end imitation-learning-based methods trained from a motion planner expert in simulation (see Appendix G.2.4 for details).

Original: Note that in the PartNet-Mobility Placement experiments, the pose of the anchor object poses are randomly varied.
Revised: Note that in the PartNet-Mobility Placement experiments, the pose of the anchor object poses are randomly varied.

Original: As such, we omit a comparison to methods that assume a static anchor, such as the Neural
Revised: As such, we omit a comparison to methods that assume a static anchor, such as the Neural

Original: Descriptor Field (NDF) [12] and Dense Object Nets (DON) [11] baselines used in the mug hanging task (Section 5.1), as both methods assume that the anchor objects are in a ﬁxed, known position.
Revised: Descriptor Field (NDF) [12] and Dense Object Nets (DON) [11] baselines used in the mug hanging task (Section 5.1), as both methods assume that the anchor objects are in a ﬁxed, known position.

Original: [Table]
Revised: [Table]

Original: Results : We report rotation ( E R ) and translation ( E t ) error between our predicted transform and the ground truth as geodesic rotational distance [28, 29] and L 2 distance, respectively.
Revised: Results : We report rotation ( E R ) and translation ( E t ) error between our predicted transform and the ground truth as geodesic rotational distance [28, 29] and L 2 distance, respectively.

Original: In both our simulated experiments (Table 4 Top) and our real-world experiments (Table 4 Bottom), we ﬁnd that TAX-Pose outperforms the baseline end-to-end imitation learning methods.
Revised: In both our simulated experiments (Table 4 Top) and our real-world experiments (Table 4 Bottom), we ﬁnd that TAX-Pose outperforms the baseline end-to-end imitation learning methods, with the goal-conditioned variant, TAX-Pose

Revised: GC, performing the best.

Original: In real-world experiments, our method generalizes to novel distributions of starting poses better than the Goal Flow baseline, placing action objects into the goal regions with a 92% success rate (see Appendix F.2 for more detailed results and Appendix G.2.4 for baseline details.).
Revised: In real-world experiments, our method generalizes to novel distributions of starting poses better than the Goal Flow baseline, placing action objects into the goal regions with a 92% success rate (see Appendix F.2 for more detailed results and Appendix G.2.4 for baseline details.).

Original:  Conclusion and Limitations
Revised:  Conclusion and Limitations

Original: In this paper, we show that dense soft correspondence can be used to learn task speciﬁc object relationships that generalize to novel object instances.
Revised: In this paper, we show that dense soft correspondence can be used to learn task speciﬁc object relationships that generalize to novel object instances.

Original: Correspondence residuals allow our method to estimate correspondences to virtual points, outside of the objects convex hull, drastically increasing the number of tasks this method can complete.
Revised: Correspondence residuals allow our method to estimate correspondences to virtual points, outside of the objects convex hull, drastically increasing the number of tasks this method can complete.

Original: We further show that this “cross-pose” can be learned for a task, using a small number of demonstrations.
Revised: We further show that this “cross-pose” can be learned for a task, using a small number of demonstrations.

Original: Finally, we show that our method far outperforms the baselines on two challenging tasks in both real and simulated experiments.
Revised: Finally, we show that our method far outperforms the baselines on two challenging tasks in both real and simulated experiments.

Original: While our method is able to predict relative pose relationships with high precision, it has several limitations:
Revised: While our method is able to predict relative pose relationships with high precision, it has several limitations:

Original: • Requires segmentation : Our method requires an accurate segmentation of two objects in order to predict their relative goal pose.
Revised: • Requires segmentation : Our method requires an accurate segmentation of two objects in order to predict their relative goal pose.

Original: • Performance degrades under occlusion : Our method performs best when complete point clouds are provided, captured via multiple cameras or by repeatedly reorienting the objects.
Revised: • Performance degrades under occlusion : Our method performs best when complete point clouds are provided, captured via multiple cameras or by repeatedly reorienting the objects.

Original: • Poorly deﬁned for multimodal relationships : Because our method extracts a single global estimate of relative pose from a ﬁxed set of correspondences, performance on objects with multiple valid goals is not well-deﬁned.
Revised: • Poorly deﬁned for multimodal relationships : Because our method extracts a single global estimate of relative pose from a ﬁxed set of correspondences, performance on objects with multiple valid goals is not well-deﬁned.

Original: Our method might be augmented with a consensus-based or sampling-based approach to capture the multimodality of the solution space in these cases.
Revised: Our method might be augmented with a consensus-based or sampling-based approach to capture the multimodality of the solution space in these cases.

Original: We leave this for future work.
Revised: We leave this for future work.

Original: Acknowledgements
Revised: Acknowledgements

Original: This material is based upon work supported by the National Science Foundation under Grant No. IIS-1849154.
Revised: This material is based upon work supported by the National Science Foundation under Grant No. IIS-1849154.

Original: This work was also supported by LG Electronics.
Revised: This work was also supported by LG Electronics.

Original: We are grateful to Daniel Seita and Jenny Wang for their helpful feedback and discussion on the paper.
Revised: We are grateful to Daniel Seita and Jenny Wang for their helpful feedback and discussion on the paper.