Original: TAX-Pose: Task-Speciﬁc Cross-Pose Estimation for Robot Manipulation
Revised: TAX-Pose: Task-Speciﬁc Cross-Pose Estimation for Robot Manipulation

Original: Abstract: How do we imbue robots with the ability to efﬁciently manipulate unseen objects and transfer relevant skills based on demonstrations?
Revised: Abstract: How do we imbue robots with the ability to efﬁciently manipulate unseen objects and transfer relevant skills based on demonstrations?

Original: End-to-end learning methods often fail to generalize to novel objects or unseen conﬁgurations.
Revised: End-to-end learning methods often fail to generalize to novel objects or unseen conﬁgurations.

Original: Instead, we focus on the task-speciﬁc pose relationship between relevant parts of interacting objects.
Revised: Instead, we focus on the task-speciﬁc pose relationship between relevant parts of interacting objects.

Original: We conjecture that this relationship is a generalizable notion of a manipulation task that can transfer to new objects in the same category; examples include the relationship between the pose of a pan relative to an oven or the pose of a mug relative to a mug rack.
Revised: We conjecture that this relationship is a generalizable notion of a manipulation task that can transfer to new objects in the same category; examples include the relationship between the pose of a pan relative to an oven or the pose of a mug relative to a mug rack.

Original: We call this task-speciﬁc pose relationship “cross-pose” and provide a mathematical deﬁnition of this concept.
Revised: We call this task-speciﬁc pose relationship “cross-pose” and provide a mathematical deﬁnition of this concept.

Original: We propose a vision-based system that learns to estimate the cross-pose between two objects for a given manipulation task using learned cross-object correspondences.
Revised: We propose a vision-based system that learns to estimate the cross-pose between two objects for a given manipulation task using learned cross-object correspondences.

Original: The estimated cross-pose is then used to guide a downstream motion planner to manipulate the objects into the desired pose relationship (placing a pan into the oven or the mug onto the mug rack).
Revised: The estimated cross-pose is then used to guide a downstream motion planner to manipulate the objects into the desired pose relationship (placing a pan into the oven or the mug onto the mug rack).

Original: We demonstrate our method’s capability to generalize to unseen objects, in some cases after training on only 10 demonstrations in the real world.
Revised: We demonstrate our method’s capability to generalize to unseen objects, in some cases after training on only 10 demonstrations in the real world.

Original: Results show that our system achieves state-of-the-art performance in both simulated and real-world experiments across a number of tasks.
Revised: Results show that our system achieves state-of-the-art performance in both simulated and real-world experiments across a number of tasks.

Original: Supplementary information and videos can be found on our project website.
Revised: Supplementary information and videos can be found on our project website.

Original: [Figure]
Revised: [Figure]

Original:  Introduction
Revised:  Introduction

Original: Many manipulation tasks require a robot to move an object to a location relative to another object.
Revised: Many manipulation tasks require a robot to move an object to a location relative to another object.

Original: For example, a cooking robot may need to place a lasagna in an oven, place a pot on a stove, place a plate in a microwave, place a mug onto a mug rack, or place a cup onto a shelf.
Revised: For example, a cooking robot may need to place a lasagna in an oven, place a pot on a stove, place a plate in a microwave, place a mug onto a mug rack, or place a cup onto a shelf.

Original: Understanding and placing objects in task-speciﬁc locations is a key skill for robots operating in human environments.
Revised: Understanding and placing objects in task-speciﬁc locations is a key skill for robots operating in human environments.

Original: Further, this skill should generalize to novel objects within the training categories, such as placing new trays into the oven or new mugs onto a mug rack.
Revised: Further, this skill should generalize to novel objects within the training categories, such as placing new trays into the oven or new mugs onto a mug rack.

Original: A common approach in robot learning is to train a policy “end-to-end,” mapping from pixel observations to low-level robot actions.
Revised: A common approach in robot learning is to train a policy “end-to-end,” mapping from pixel observations to low-level robot actions.

Original: However, end-to-end trained policies cannot easily reason about complex pose relationships such as the ones described above, and they have difﬁculty generalizing to unseen object conﬁgurations.[Table]
Revised: However, end-to-end trained policies cannot easily reason about complex pose relationships such as the ones described above, and they have difﬁculty generalizing to unseen object conﬁgurations.

Original: In contrast, we propose a method that learns to reason about the 3D geometric relationship between a pair of objects.
Revised: In contrast, we propose a method that learns to reason about the 3D geometric relationship between a pair of objects.

Original: For the type of tasks deﬁned above, the robot needs to reason about the relationship between key parts on one object with respect to key parts on another object.
Revised: For the type of tasks deﬁned above, the robot needs to reason about the relationship between key parts on one object with respect to key parts on another object.

Original: For example, to place a mug on a mug rack, the robot must reason about the relationship between the pose of the mug handle and the pose of the mug rack; if the mug rack changes its pose, then the pose of the mug must change accordingly in order to still be placed on the rack (see Figure 3).
Revised: For example, to place a mug on a mug rack, the robot must reason about the relationship between the pose of the mug handle and the pose of the mug rack; if the mug rack changes its pose, then the pose of the mug must change accordingly in order to still be placed on the rack (see Figure 3).

Original: We name this taskspeciﬁc notion of the pose relationship between a pair of objects as “cross-pose” and we formally deﬁne it mathematically.
Revised: We name this taskspeciﬁc notion of the pose relationship between a pair of objects as “cross-pose” and we formally deﬁne it mathematically.

Original: Further, we propose a vision system that can efﬁciently estimate the crosspose from a small number of demonstrations of a given task, generalizing to novel objects within the training categories.
Revised: Further, we propose a vision system that can efﬁciently estimate the crosspose from a small number of demonstrations of a given task, generalizing to novel objects within the training categories.

Original: To complete the manipulation task, we use the estimated cross-pose as the target of a motion planning algorithm, which will move the objects into the desired conﬁguration (e.g. placing the mug onto the rack, placing the lasagna into the oven, etc).
Revised: To complete the manipulation task, we use the estimated cross-pose as the target of a motion planning algorithm, which will move the objects into the desired conﬁguration (e.g. placing the mug onto the rack, placing the lasagna into the oven, etc).

Original: [Figure]
Revised: [Figure]

Original: In this paper, we present TAX-Pose (TAsk-speciﬁc CrossPose), a deep 3D vision-based method that learns to predict a task-speciﬁc pose relationship between a pair of objects from a set of demonstrations.
Revised: In this paper, we present TAX-Pose (TAsk-speciﬁc CrossPose), a deep 3D vision-based method that learns to predict a task-speciﬁc pose relationship between a pair of objects from a set of demonstrations.

Original: Our cross-pose estimation system is provably translation equivariant and can generalize from a small number of real-world demonstrations (in some cases as few as 10) to new objects in unseen poses.
Revised: Our cross-pose estimation system is provably translation equivariant and can generalize from a small number of real-world demonstrations (in some cases as few as 10) to new objects in unseen poses.

Original: The contributions of this paper include:
Revised: The contributions of this paper include:

Original: A precise deﬁnition of “cross-pose,” which deﬁnes a task-speciﬁc pose relationship between two objects.
Revised: A precise deﬁnition of “cross-pose,” which deﬁnes a task-speciﬁc pose relationship between two objects.

Original: A novel method that estimates soft-correspondences between two objects, from which the cross-pose between the objects can be estimated (see Figure 1); this method is provably translation equivariant and can learn from a small number of real-world demonstrations.
Revised: A novel method that estimates soft-correspondences between two objects, from which the cross-pose between the objects can be estimated (see Figure 1); this method is provably translation equivariant and can learn from a small number of real-world demonstrations.

Original: A robot system to manipulate objects into the desired cross-pose to achieve a given manipulation task.
Revised: A robot system to manipulate objects into the desired cross-pose to achieve a given manipulation task.

Original: We present simulated and real-world experiments to test the performance of our system in achieving a variety of relative placement manipulation tasks.
Revised: We present simulated and real-world experiments to test the performance of our system in achieving a variety of relative placement manipulation tasks.

Original: We demonstrate our method on a semantic placement task, in which the robot must place an object in, on, or around a novel object (Figure 2, top).
Revised: We demonstrate our method on a semantic placement task, in which the robot must place an object in, on, or around a novel object (Figure 2, top).

Original: We also demonstrate our method on precise placement tasks, such as hanging a mug on a rack (Figure 2, bottom) or placing a bottle or bowl on a shelf; in both cases our method generalizes to new object conﬁgurations and new objects within the training categories.
Revised: We also demonstrate our method on precise placement tasks, such as hanging a mug on a rack (Figure 2, bottom) or placing a bottle or bowl on a shelf; in both cases our method generalizes to new object conﬁgurations and new objects within the training categories.

Original:  Related Work
Revised:  Related Work

Original: Object Pose Estimation :
Revised: Object Pose Estimation :

Original: Pose estimation is the task of detecting and inferring the 6DoF pose of an object, which includes its position and orientation, with respect to some previously deﬁned object reference frame [1, 2, 3, 4, 5, 6].
Revised: Pose estimation is the task of detecting and inferring the 6DoF pose of an object, which includes its position and orientation, with respect to some previously deﬁned object reference frame [1, 2, 3, 4, 5, 6].

Original: Recent work [7, 8, 9, 10] proposed to use 3D semantic keypoints as an alternative form of object representation.
Revised: Recent work [7, 8, 9, 10] proposed to use 3D semantic keypoints as an alternative form of object representation.

Original: While keypoint-based methods can generalize within an object class, they require a signiﬁcant amount of hand annotated data or access to a simulated version of the task to learn to estimate the keypoint locations.
Revised: While keypoint-based methods can generalize within an object class, they require a signiﬁcant amount of hand annotated data or access to a simulated version of the task to learn to estimate the keypoint locations.

Original: In contrast, our method is able to learn from just 10 real-world demonstrations.
Revised: In contrast, our method is able to learn from just 10 real-world demonstrations.

Original: Another approach is to use dense embeddings, such as Dense Object Nets (DON) [11] and Neural Descriptor Fields (NDF) [12], which achieve generalization across classes by predicting dense embeddings in the observation and matching them to embeddings of the demonstration objects.
Revised: Another approach is to use dense embeddings, such as Dense Object Nets (DON) [11] and Neural Descriptor Fields (NDF) [12], which achieve generalization across classes by predicting dense embeddings in the observation and matching them to embeddings of the demonstration objects.

Original: However, DON [11] and NDF [12] assume that the target object is moved relative to a static reference object in a “known canonical conﬁguration” (e.g. the pose of the mug rack in NDF [12] is assumed to be known and ﬁxed).
Revised: However, DON [11] and NDF [12] assume that the target object is moved relative to a static reference object in a “known canonical conﬁguration” (e.g. the pose of the mug rack in NDF [12] is assumed to be known and ﬁxed).

Original: In contrast, our method reasons about the geometric relationship between a pair of objects and hence does not need to assume a static environment.
Revised: In contrast, our method reasons about the geometric relationship between a pair of objects and hence does not need to assume a static environment.

Original: Thus, for example, our method is able to perform the mug hanging task while varying the pose of the mug rack (see our project website), whereas the baselines (DON [11],
Revised: Thus, for example, our method is able to perform the mug hanging task while varying the pose of the mug rack (see our project website), whereas the baselines (DON [11],

Original: NDF [12]) cannot.
Revised: NDF [12]) cannot.

Original: Further, we show that our method signiﬁcantly outperforms both the DON [11] and NDF [12] baselines, especially when given a very small number of demonstrations.
Revised: Further, we show that our method signiﬁcantly outperforms both the DON [11] and NDF [12] baselines, especially when given a very small number of demonstrations.

Original: Point Cloud Registration : Our method for estimating the cross-pose between two objects builds upon previous work in point cloud registration.
Revised: Point Cloud Registration : Our method for estimating the cross-pose between two objects builds upon previous work in point cloud registration.

Original: The typical objective in point cloud registration is to ﬁnd the optimal rigid alignment between two point clouds, to minimize the sum of squared distances between two sets of points.
Revised: The typical objective in point cloud registration is to ﬁnd the optimal rigid alignment between two point clouds, to minimize the sum of squared distances between two sets of points.

Original: Traditionally, Iterative Closest Point (ICP) [13] and its variants [14, 15, 16, 17, 18, 19] have been used to compute the optimal rigid alignment between two point clouds.
Revised: Traditionally, Iterative Closest Point (ICP) [13] and its variants [14, 15, 16, 17, 18, 19] have been used to compute the optimal rigid alignment between two point clouds.

Original: Deep Closest Point (DCP) [20] avoids local minima common for ICP by seeking to approximate correspondence in a high-dimensional learned feature space.
Revised: Deep Closest Point (DCP) [20] avoids local minima common for ICP by seeking to approximate correspondence in a high-dimensional learned feature space.

Original: Our method builds upon the architecture of DCP for cross-pose estimation; however, in contrast to point cloud registration, in which the objective is to minimize the sum of squared distances between two sets of points on the same object in two different poses, our objective is to estimate a task-speciﬁc pose relationship between two different objects.
Revised: Our method builds upon the architecture of DCP for cross-pose estimation; however, in contrast to point cloud registration, in which the objective is to minimize the sum of squared distances between two sets of points on the same object in two different poses, our objective is to estimate a task-speciﬁc pose relationship between two different objects.

Original: Extending the framework from DCP, we learn a residual to the soft correspondences, allowing for points to match outside the convex hull of each object.
Revised: Extending the framework from DCP, we learn a residual to the soft correspondences, allowing for points to match outside the convex hull of each object.

Original: This component is necessary when computing soft correspondences between objects of drastically different morphologies (such as a mug and a mug rack).
Revised: This component is necessary when computing soft correspondences between objects of drastically different morphologies (such as a mug and a mug rack).

Original:  Problem Statement
Revised:  Problem Statement

Original: [Figure]
Revised: [Figure]

Original: Relative placement tasks:
Revised: Relative placement tasks:

Original: In this paper, we are specifically interested in “relative placement tasks.”
Revised: In this paper, we are specifically interested in “relative placement tasks.”

Original: Given two objects, A and B , a “relative placement task” is the task of placing object A at a pose relative to object B .
Revised: Given two objects, A and B , a “relative placement task” is the task of placing object A at a pose relative to object B .

Original: For example, consider the task of placing a lasagna in an oven, placing a mug on a rack, or placing a robot gripper on the rim of a mug.
Revised: For example, consider the task of placing a lasagna in an oven, placing a mug on a rack, or placing a robot gripper on the rim of a mug.

Original: All of these tasks involve placing one object (which we call the “action” object A ) at a semantically meaningful location relative to another object (which we call the “anchor” object B ) 2 .
Revised: All of these tasks involve placing one object (which we call the “action” object A ) at a semantically meaningful location relative to another object (which we call the “anchor” object B ) 2 .

Original: Speciﬁcally, suppose that T ∗A and T ∗B are SE (3) poses for objects A and B respectively (in a shared world reference frame 3 ) for which a desired task is considered complete (lasagna is in the oven; mug is on the rack, etc).
Revised: Speciﬁcally, suppose that T ∗A and T ∗B are SE (3) poses for objects A and B respectively (in a shared world reference frame 3 ) for which a desired task is considered complete (lasagna is in the oven; mug is on the rack, etc).

Original: Then for a relative placement task, if objects A and B are in poses T · T ∗A and T · T ∗B (respectively) for any transform T , then the task will also be considered to be complete, as seen in Figure 3.
Revised: Then for a relative placement task, if objects A and B are in poses T · T ∗A and T · T ∗B (respectively) for any transform T , then the task will also be considered to be complete, as seen in Figure 3.

Original: In other words, if T ∗B represents the pose of the rack and T ∗A represents the pose of the mug on the rack (at task completion); then if we transform the both the mug and rack poses by T , then the mug will still be located on the rack.
Revised: In other words, if T ∗B represents the pose of the rack and T ∗A represents the pose of the mug on the rack (at task completion); then if we transform the both the mug and rack poses by T , then the mug will still be located on the rack.

Original: Formally, this property can be deﬁned with the following Boolean function,
Revised: Formally, this property can be deﬁned with the following Boolean function,

Original: RelPlace
Revised: RelPlace

Original: [Equation]
Revised: [Equation]

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: For many real semantic placement tasks, there are actually sets of valid solutions which solve each task (i.e., there are many potential locations to place an object on a table to achieve a semantic “object-on-table” relationship).
Revised: For many real semantic placement tasks, there are actually sets of valid solutions which solve each task (i.e., there are many potential locations to place an object on a table to achieve a semantic “object-on-table” relationship).

Original: However, for this work, we consider precise placement tasks under the simplifying assumption that, for a given pose of object B , there is a single, unambiguous pose of object A needed to achieve the task.
Revised: However, for this work, we consider precise placement tasks under the simplifying assumption that, for a given pose of object B , there is a single, unambiguous pose of object A needed to achieve the task.

Original: Deﬁnition of Cross-Pose:
Revised: Deﬁnition of Cross-Pose:

Original: Given the above deﬁnition of a relative placement task, our goal will be to determine how to move object A so that it will be in the “goal pose,” which, as described above, is deﬁned relative to the pose of object B .
Revised: Given the above deﬁnition of a relative placement task, our goal will be to determine how to move object A so that it will be in the “goal pose,” which, as described above, is deﬁned relative to the pose of object B .

Original: To achieve this, one option is to estimate the poses of objects
Revised: To achieve this, one option is to estimate the poses of objects

Original: A and B separately and then compute the transformation needed to move object A into the goal pose.
Revised: A and B separately and then compute the transformation needed to move object A into the goal pose.

Original: However, the pose estimate of each object will have errors, and these errors will accumulate when the poses are combined into the single relative pose needed to reach the goal conﬁguration.
Revised: However, the pose estimate of each object will have errors, and these errors will accumulate when the poses are combined into the single relative pose needed to reach the goal conﬁguration.

Original: Instead of estimating the pose of each object independently, we aim to learn a function f
Revised: Instead of estimating the pose of each object independently, we aim to learn a function f

Original: ( P A , P B ) , which takes as input the point clouds P A and P B for both objects A and B , where P A ∈ R 3 × N A and P B ∈ R 3 × N B are 3D point clouds of sizes N A and N B , respectively.
Revised: ( P A , P B ) , which takes as input the point clouds P A and P B for both objects A and B , where P A ∈ R 3 × N A and P B ∈ R 3 × N B are 3D point clouds of sizes N A and N B , respectively.

Original: This function outputs an SE(3) rigid transformation, f ( P A , P B )
Revised: This function outputs an SE(3) rigid transformation, f ( P A , P B )

Original: = T AB , where we refer to T AB as the “cross-pose” between object A and object B .
Revised: = T AB , where we refer to T AB as the “cross-pose” between object A and object B .

Original: For notational convenience, we occasional write f as a function of the poses T A , T B of point clouds P A and P B respectively (with respect to a global reference frame) such that f ( T A , T B ) :
Revised: For notational convenience, we occasional write f as a function of the poses T A , T B of point clouds P A and P B respectively (with respect to a global reference frame) such that f ( T A , T B ) :

Original: = f ( P A , P B ) .
Revised: = f ( P A , P B ) .

Original: This notational change is to make the transformation math more intuitive; in practice, this function only ever receives point clouds as input.
Revised: This notational change is to make the transformation math more intuitive; in practice, this function only ever receives point clouds as input.

Original: We will deﬁne the cross-pose T AB (below) such that, if we transform object A by T AB , then object A will be in the goal pose relative to object B for the relative placement task.
Revised: We will deﬁne the cross-pose T AB (below) such that, if we transform object A by T AB , then object A will be in the goal pose relative to object B for the relative placement task.

Original: For example, suppose that T ∗A and T ∗B are poses for objects A and B , respectively, for which a desired relative placement task is considered complete.
Revised: For example, suppose that T ∗A and T ∗B are poses for objects A and B , respectively, for which a desired relative placement task is considered complete.

Original: In this conﬁguration, the cross-pose of these objects would be f ( T ∗A , T ∗B )
Revised: In this conﬁguration, the cross-pose of these objects would be f ( T ∗A , T ∗B )

Original: =
Revised: =

Original: I where I is the identity, as object A does not need to be moved to complete the task.
Revised: I where I is the identity, as object A does not need to be moved to complete the task.

Original: Further, based on the deﬁnition of a relative placement task given above, if both objects are transformed by the same transform T , then the objects will still be in the desired relative pose,
Revised: Further, based on the deﬁnition of a relative placement task given above, if both objects are transformed by the same transform T , then the objects will still be in the desired relative pose,

Original: [Equation]
Revised: [Equation]

Original: for any transform T ∈ SE (3) .
Revised: for any transform T ∈ SE (3) .

Original: Now, let us assume that objects A and B are not in the goal conﬁguration and have pose T A
Revised: Now, let us assume that objects A and B are not in the goal conﬁguration and have pose T A

Original: = T α · T ∗A and T B = T β · T ∗B , respectively, for arbitrary transforms
Revised: = T α · T ∗A and T B = T β · T ∗B , respectively, for arbitrary transforms

Original: T α and T β ∈ SE (3) .
Revised: T α and T β ∈ SE (3) .

Original: We then deﬁne the “cross-pose” of objects A and B as:
Revised: We then deﬁne the “cross-pose” of objects A and B as:

Original: [Equation]
Revised: [Equation]

Original: Note that this deﬁnition is equivalent to Equation 2 for the special case of T α = T β .
Revised: Note that this deﬁnition is equivalent to Equation 2 for the special case of T α = T β .

Original: This deﬁnition of cross-pose allows us to move object A into the goal conﬁguration, relative to object B :
Revised: This deﬁnition of cross-pose allows us to move object A into the goal conﬁguration, relative to object B :

Original: [Equation]
Revised: [Equation]

Original: satisfying the relative placement condition deﬁned in Equation 1 with T = T β .
Revised: satisfying the relative placement condition deﬁned in Equation 1 with T = T β .

Original: Alternatively, we could have instead transformed object B by the inverse of the cross-pose to achieve the task.
Revised: Alternatively, we could have instead transformed object B by the inverse of the cross-pose to achieve the task.

Original:  Method
Revised:  Method

Original: Overview: We frame the task of cross-pose estimation as a soft correspondence-prediction task between a pair of point clouds, followed by an analytical least-squares optimization to ﬁnd the optimal cross-pose for the predicted correspondences.
Revised: Overview: We frame the task of cross-pose estimation as a soft correspondence-prediction task between a pair of point clouds, followed by an analytical least-squares optimization to ﬁnd the optimal cross-pose for the predicted correspondences.

Original: As described in Appendix B, this correspondencebased approach allows our method to be translation-equivariant: translating either object ( A or B ) will lead to a translated cross-pose prediction.
Revised: As described in Appendix B, this correspondencebased approach allows our method to be translation-equivariant: translating either object ( A or B ) will lead to a translated cross-pose prediction.

Original: This allows our method to automatically adapt to novel positions of both the anchor and action objects, unlike previous work which assumes a static anchor [12].
Revised: This allows our method to automatically adapt to novel positions of both the anchor and action objects, unlike previous work which assumes a static anchor [12].

Original: Our method for task-speciﬁc cross-pose estimation, known as TAX-Pose, consists of the following steps, as shown in Figure 4:
Revised: Our method for task-speciﬁc cross-pose estimation, known as TAX-Pose, consists of the following steps, as shown in Figure 4:

Original: Soft Correspondence Prediction : For a pair of objects A , B , a neural network learns to predict a per-point embedding to establish a (soft) correspondence between A and B , which are called
Revised: Soft Correspondence Prediction : For a pair of objects A , B , a neural network learns to predict a per-point embedding to establish a (soft) correspondence between A and B , which are called

Original: “virtual soft correspondences.”
Revised: “virtual soft correspondences.”

Original: The corresponding points are constrained to be within the convex hulls of B and A respectively.
Revised: The corresponding points are constrained to be within the convex hulls of B and A respectively.

Original: Adjustment via Correspondence Residuals :
Revised: Adjustment via Correspondence Residuals :

Original: For most estimation tasks, some points in object A may not be within the convex hull of object B ; for instance, when a mug is placed on a mug rack, most points on the mug will be outside of the convex hull of the mug rack.
Revised: For most estimation tasks, some points in object A may not be within the convex hull of object B ; for instance, when a mug is placed on a mug rack, most points on the mug will be outside of the convex hull of the mug rack.

Original: To accommodate these cases, we apply a pointwise residual vector to displace each of the predicted soft correspondences.
Revised: To accommodate these cases, we apply a pointwise residual vector to displace each of the predicted soft correspondences.

Original: These “corrected virtual correspondences” allow points in A to correspond to locations in free space near B .
Revised: These “corrected virtual correspondences” allow points in A to correspond to locations in free space near B .

Original: Find the Optimal Transform : Because the cross-pose is deﬁned as a rigid transformation of object A , we use a differentiable weighted SVD to ﬁnd the transformation that minimizes the weighted least squares difference to the corrected virtual correspondences.
Revised: Find the Optimal Transform : Because the cross-pose is deﬁned as a rigid transformation of object A , we use a differentiable weighted SVD to ﬁnd the transformation that minimizes the weighted least squares difference to the corrected virtual correspondences.

Original: Because each step above is differentiable, the whole model can be optimized end-to-end, despite having an interpretable internal structure which we describe below.
Revised: Because each step above is differentiable, the whole model can be optimized end-to-end, despite having an interpretable internal structure which we describe below.

Original: Our method is heavily inspired by Deep Closest Point (DCP) [20].
Revised: Our method is heavily inspired by Deep Closest Point (DCP) [20].

Original: The key difference between our pose alignment model and DCP is that we predict the cross-pose between two different objects for a given task instead of registering two point clouds of an identical object.
Revised: The key difference between our pose alignment model and DCP is that we predict the cross-pose between two different objects for a given task instead of registering two point clouds of an identical object.

Original: Additionally, TAX-Pose can predict relationships where these clouds may not have any points of contact or overlap.
Revised: Additionally, TAX-Pose can predict relationships where these clouds may not have any points of contact or overlap.

Original: We now describe our cross-pose estimation algorithm in detail.
Revised: We now describe our cross-pose estimation algorithm in detail.

Original: To recap the problem statement, given objects A and B with point cloud observations P A ∈ R 3 × N A , P B ∈ R 3 × N B respectively, our objective is to estimate the task-speciﬁc cross-pose T AB =
Revised: To recap the problem statement, given objects A and B with point cloud observations P A ∈ R 3 × N A , P B ∈ R 3 × N B respectively, our objective is to estimate the task-speciﬁc cross-pose T AB =

Original: f ( P A , P B ) ∈ SE (3) .
Revised: f ( P A , P B ) ∈ SE (3) .

Original: Note that the cross-pose between object A and B is deﬁned with respect to a given task (e.g. putting a lasagna in the oven, putting a mug on the rack, etc).
Revised: Note that the cross-pose between object A and B is deﬁned with respect to a given task (e.g. putting a lasagna in the oven, putting a mug on the rack, etc).

Original: 4.1 Cross-Pose Estimation via Soft Correspondence Prediction
Revised: 4.1 Cross-Pose Estimation via Soft Correspondence Prediction

Original: Soft Correspondence Prediction :
Revised: Soft Correspondence Prediction :

Original: The ﬁrst step of the method is to compute two sets of correspondences between A and B , one which maps from points in A to B , and one which maps from points in B to A .
Revised: The ﬁrst step of the method is to compute two sets of correspondences between A and B , one which maps from points in A to B , and one which maps from points in B to A .

Original: These need not be a bijection, and can be asymmetric.
Revised: These need not be a bijection, and can be asymmetric.

Original: As we want each step to be differentiable, we follow DCP’s conventions and estimate a soft correspondence .
Revised: As we want each step to be differentiable, we follow DCP’s conventions and estimate a soft correspondence .

Original: This assigns a virtual soft corresponding point v A i ∈ V A to every point p
Revised: This assigns a virtual soft corresponding point v A i ∈ V A to every point p

Original: A i ∈ P
Revised: A i ∈ P

Original: A by computing a convex combination of points in P B , and vice versa.
Revised: A by computing a convex combination of points in P B , and vice versa.

Original: Formally:
Revised: Formally:

Original: [Equation]
Revised: [Equation]

Original: 5b
Revised: 5b

Original: [Equation]
Revised: [Equation]

Original: with normalized weight vectors w A→B i ∈ W A→B and w B→A i ∈ W B→A .
Revised: with normalized weight vectors w A→B i ∈ W A→B and w B→A i ∈ W B→A .

Original: Importantly, these virtual corresponding points are not constrained to the surfaces of A or B ; instead, they are constrained to the convex hulls of P B and P A , respectively.
Revised: Importantly, these virtual corresponding points are not constrained to the surfaces of A or B ; instead, they are constrained to the convex hulls of P B and P A , respectively.

Original: To compute the weights w A→B
Revised: To compute the weights w A→B

Original: i , w B→A i in Equations 5a and 5b, we ﬁrst encode each point cloud
Revised: i , w B→A i in Equations 5a and 5b, we ﬁrst encode each point cloud

Original: P A and P B into a latent space using a neural network encoder, DGCNN [21].
Revised: P A and P B into a latent space using a neural network encoder, DGCNN [21].

Original: This encoder head is comprised of two distinct encoders g A and g B , each of which receives point cloud P A and P B , respectively, zero-centers them, and outputs a dense, point-wise embedding for each object (see
Revised: This encoder head is comprised of two distinct encoders g A and g B , each of which receives point cloud P A and P B , respectively, zero-centers them, and outputs a dense, point-wise embedding for each object (see

Original: Figure 4):
Revised: Figure 4):

Original: [Equation]
Revised: [Equation]

Original: Ψ B = g B (¯ P B )
Revised: Ψ B = g B (¯ P B )

Original: ∈
Revised: ∈

Original: R N B × d where ψ K i ∈
Revised: R N B × d where ψ K i ∈

Original: Ψ K is the ddimensional embedding of the i -th point in object K , and ¯ P K is the zero-centered point cloud for object K .
Revised: Ψ K is the ddimensional embedding of the i -th point in object K , and ¯ P K is the zero-centered point cloud for object K .

Original: Because we want the cross-correspondence to incorporate information about both point clouds, we then employ a cross-object attention module between the two dense feature sets to obtain cross-object point embeddings , Φ A ∈ R N A × d and Φ B ∈ R N B × d , deﬁned as:
Revised: Because we want the cross-correspondence to incorporate information about both point clouds, we then employ a cross-object attention module between the two dense feature sets to obtain cross-object point embeddings , Φ A ∈ R N A × d and Φ B ∈ R N B × d , deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where g T A , g T B are Transformers [22].
Revised: where g T A , g T B are Transformers [22].

Original: Finally, recall that our goal was to compute a set of normalized weight vectors W A→B , W B→A .
Revised: Finally, recall that our goal was to compute a set of normalized weight vectors W A→B , W B→A .

Original: To compute the virtual corresponding point v A i assigned to any point p A i ∈ P A , we can extract the desired normalized weight vector w A→B i from intermediate attention features of the cross-object attention module as:
Revised: To compute the virtual corresponding point v A i assigned to any point p A i ∈ P A , we can extract the desired normalized weight vector w A→B i from intermediate attention features of the cross-object attention module as:

Original: [Equation]
Revised: [Equation]

Original: where q K i ∈ Q K , and Q K , K K ∈ R N K ×
Revised: where q K i ∈ Q K , and Q K , K K ∈ R N K ×

Original: d are the query and key values (respectively) for object K associated with cross-object attention Transformer module g T K (see Appendix C for details).
Revised: d are the query and key values (respectively) for object K associated with cross-object attention Transformer module g T K (see Appendix C for details).

Original: These weights are then used to compute the virtual soft correspondences V A , V B using Equation 5.
Revised: These weights are then used to compute the virtual soft correspondences V A , V B using Equation 5.

Original: Adjustment via Correspondence Residuals: As previously stated, the virtual soft correspondences
Revised: Adjustment via Correspondence Residuals: As previously stated, the virtual soft correspondences

Original: V A , V B given by Equations 5a and 5b are constrained to be within the convex hull of each object.
Revised: V A , V B given by Equations 5a and 5b are constrained to be within the convex hull of each object.

Original: However, many relative placement tasks cannot be solved perfectly with this constraint.
Revised: However, many relative placement tasks cannot be solved perfectly with this constraint.

Original: For instance, we might want a point on the handle of a teapot to correspond to some point above a stovetop (which lies outside the convex hull of the points on the stovetop).
Revised: For instance, we might want a point on the handle of a teapot to correspond to some point above a stovetop (which lies outside the convex hull of the points on the stovetop).

Original: To allow for such off-object correspondences, we further learn a residual vector , δ A i ∈
Revised: To allow for such off-object correspondences, we further learn a residual vector , δ A i ∈

Original: ∆
Revised: ∆

Original: A for each point i that corrects each virtual corresponding point v A i .
Revised: A for each point i that corrects each virtual corresponding point v A i .

Original: This allows us to displace each virtual corresponding point to any arbitrary location that might be suitable for the task.
Revised: This allows us to displace each virtual corresponding point to any arbitrary location that might be suitable for the task.

Original: To compute these residual vectors, we use a point-wise neural network g R A , g R B to map each point’s embedding into a 3D residual vector:
Revised: To compute these residual vectors, we use a point-wise neural network g R A , g R B to map each point’s embedding into a 3D residual vector:

Original: [Equation]
Revised: [Equation]

Original: Applying these residual offsets to the virtual points, we get a set of corrected virtual correspondences , ˜ v A i ∈ ˜V A and ˜ v B i ∈ ˜V B , deﬁned as
Revised: Applying these residual offsets to the virtual points, we get a set of corrected virtual correspondences , ˜ v A i ∈ ˜V A and ˜ v B i ∈ ˜V B , deﬁned as

Original: [Equation]
Revised: [Equation]

Original: These corrected virtual correspondences ˜ v A i deﬁne the estimated goal location relative to object B for each point p
Revised: These corrected virtual correspondences ˜ v A i deﬁne the estimated goal location relative to object B for each point p

Original: i ∈
Revised: i ∈

Original: P A of object A , and likewise for each point in object B (see visualization in
Revised: P A of object A , and likewise for each point in object B (see visualization in

Original: Appendix A.1).
Revised: Appendix A.1).

Original: Least-Squares Cross-Pose Optimization with Weighted SVD:
Revised: Least-Squares Cross-Pose Optimization with Weighted SVD:

Original: Given the sets of dense correspondences, (cid:16) P A , ˜V A (cid:17) and (cid:16) P B , ˜V B (cid:17) , we would like to compute a single rigid transformation for object A .
Revised: Given the sets of dense correspondences, (cid:16) P A , ˜V A (cid:17) and (cid:16) P B , ˜V B (cid:17) , we would like to compute a single rigid transformation for object A .

Original: To do so, we solve for the transformation T AB (the cross-pose) that minimizes the weighted distance between each point and its corrected virtual correspondence.
Revised: To do so, we solve for the transformation T AB (the cross-pose) that minimizes the weighted distance between each point and its corrected virtual correspondence.

Original: Formally, this leads to the following weighted least squares optimization:
Revised: Formally, this leads to the following weighted least squares optimization:

Original: [Equation]
Revised: [Equation]

Original: where the weights α A i ∈ α A , α B i ∈ α B signify the importance of each correspondence and are predicted by a point-wise MLP as shown in Figure 4.
Revised: where the weights α A i ∈ α A , α B i ∈ α B signify the importance of each correspondence and are predicted by a point-wise MLP as shown in Figure 4.

Original: These weights are learned end-to-end as parameters of our network; they are visualized in Appendix A.2, which shows that the network has learned to assign more weight to the parts of the object that are most important for the task, such as the region around the mug handle (on the mug) and the region around the peg (on the rack).
Revised: These weights are learned end-to-end as parameters of our network; they are visualized in Appendix A.2, which shows that the network has learned to assign more weight to the parts of the object that are most important for the task, such as the region around the mug handle (on the mug) and the region around the peg (on the rack).

Original: Equation 9 is the well-known weighted Procrustes problem, for which there exists an analytical solution.
Revised: Equation 9 is the well-known weighted Procrustes problem, for which there exists an analytical solution.

Original: To maintain the differentiablity of the system, we use a weighted differentiable SVD operation [23] to compute the cross-pose T AB that minimizes this objective (see Appendix D for details).
Revised: To maintain the differentiablity of the system, we use a weighted differentiable SVD operation [23] to compute the cross-pose T AB that minimizes this objective (see Appendix D for details).

Original: This allows us to train the system end-to-end as described below.
Revised: This allows us to train the system end-to-end as described below.

Original: 4.2 TAX-Pose Training Pipeline
Revised: 4.2 TAX-Pose Training Pipeline

Original: To train our model, we use a segmented set of demonstration point clouds of a pair of objects in the goal conﬁguration.
Revised: To train our model, we use a segmented set of demonstration point clouds of a pair of objects in the goal conﬁguration.

Original: For each demonstration point cloud, we generate multiple training examples by transforming each object’s point cloud, P A and P B by random SE(3) transformations T α and T β , respectively.
Revised: For each demonstration point cloud, we generate multiple training examples by transforming each object’s point cloud, P A and P B by random SE(3) transformations T α and T β , respectively.

Original: The predicted cross-pose, T AB , is then compared with the ground truth cross-pose,
Revised: The predicted cross-pose, T AB , is then compared with the ground truth cross-pose,

Original: T GT AB := T β T − 1 α , using an average distance loss [24] with dense regularization (see more details on our training losses in Appendix E.1).
Revised: T GT AB := T β T − 1 α , using an average distance loss [24] with dense regularization (see more details on our training losses in Appendix E.1).

Original:  Experiments
Revised:  Experiments

Original: To evaluate TAX-Pose, we conduct a wide range of simulated and real-world experiments on two classes of relative placement tasks: NDF [12] Tasks and PartNet-Mobility Placement Tasks.
Revised: To evaluate TAX-Pose, we conduct a wide range of simulated and real-world experiments on two classes of relative placement tasks: NDF [12] Tasks and PartNet-Mobility Placement Tasks.

Original: All tasks involve placing an “action” object at a speciﬁc location relative to an anchor object, in which the relative pose is speciﬁed by a set of demonstrations.
Revised: All tasks involve placing an “action” object at a speciﬁc location relative to an anchor object, in which the relative pose is speciﬁed by a set of demonstrations.

Original: Our method then generalizes to perform this task on novel objects in unseen conﬁgurations.
Revised: Our method then generalizes to perform this task on novel objects in unseen conﬁgurations.

Original: We refer the reader to our project website for additional results and videos.
Revised: We refer the reader to our project website for additional results and videos.

Original: [Figure]
Revised: [Figure]

Original: 5.1 NDF Tasks
Revised: 5.1 NDF Tasks

Original: We evaluate our method on all three NDF [12] tasks ( mug hanging, bottle placement, and bowl placement); see Appendix F.1.3 for results on bottle and bowl placement.
Revised: We evaluate our method on all three NDF [12] tasks ( mug hanging, bottle placement, and bowl placement); see Appendix F.1.3 for results on bottle and bowl placement.

Original: Results on mug hanging are described in more detail below.
Revised: Results on mug hanging are described in more detail below.

Original: Simulation Experiments: For our simulation experiments, we perform the task of hanging a mug on a rack as two sequential cross-pose estimation steps: grasping the mug (estimating the crosspose between the gripper and the mug) and hanging the mug on the rack (estimating the crosspose between the mug and the rack).
Revised: Simulation Experiments: For our simulation experiments, we perform the task of hanging a mug on a rack as two sequential cross-pose estimation steps: grasping the mug (estimating the crosspose between the gripper and the mug) and hanging the mug on the rack (estimating the crosspose between the mug and the rack).

Original: In Pybullet [25], we simulate a Franka Panda above a table with 4 depth cameras placed on the corners of the table.
Revised: In Pybullet [25], we simulate a Franka Panda above a table with 4 depth cameras placed on the corners of the table.

Original: The model is trained on 10 simulated demonstrations of mug hanging.
Revised: The model is trained on 10 simulated demonstrations of mug hanging.

Original: We evaluate task execution success on unseen mug instances in randomly generated initial conﬁgurations.
Revised: We evaluate task execution success on unseen mug instances in randomly generated initial conﬁgurations.

Original: We measure task success rates of 1) Grasping , where success is achieved when the object is grasped stably; 2) Placing , where success is achieved when the mug is placed stably on the rack; 3) Overall , when the predicted transforms enable both grasp and place success in sequence.
Revised: We measure task success rates of 1) Grasping , where success is achieved when the object is grasped stably; 2) Placing , where success is achieved when the mug is placed stably on the rack; 3) Overall , when the predicted transforms enable both grasp and place success in sequence.

Original: We compare our method to Neural Descriptor Field (NDF) [12] and
Revised: We compare our method to Neural Descriptor Field (NDF) [12] and

Original: Dense Object Nets (DON) [11].
Revised: Dense Object Nets (DON) [11].

Original: Details of these methods can be found in prior work [12].
Revised: Details of these methods can be found in prior work [12].

Original: Simulation Results: We evaluate our method in simulation in 100 trials consisting of unseen mug instances in random initial and goal conﬁgurations for both Upright and Arbitrary poses.
Revised: Simulation Results: We evaluate our method in simulation in 100 trials consisting of unseen mug instances in random initial and goal conﬁgurations for both Upright and Arbitrary poses.

Original: As shown in Table 1, our method signiﬁcantly outperforms the baselines for simulated mug hanging.
Revised: As shown in Table 1, our method signiﬁcantly outperforms the baselines for simulated mug hanging.

Original: We report additional results for simulated bottle and bowl placement tasks in Table S4 in Appendix F.1.3.
Revised: We report additional results for simulated bottle and bowl placement tasks in Table S4 in Appendix F.1.3.

Original: Ablation Analysis: Effects of Number of Demonstrations.
Revised: Ablation Analysis: Effects of Number of Demonstrations.

Original: To study how the number of demonstrations observed affects our method’s performance, we train our model on { 10, 5, 1 } demonstrations of upright pose mug hanging.
Revised: To study how the number of demonstrations observed affects our method’s performance, we train our model on { 10, 5, 1 } demonstrations of upright pose mug hanging.

Original: Results are found in Table 2.
Revised: Results are found in Table 2.

Original: Our method outperforms the baselines for all number of demonstrations; TAX-Pose can perform well even with as few as 5 demonstrations.
Revised: Our method outperforms the baselines for all number of demonstrations; TAX-Pose can perform well even with as few as 5 demonstrations.

Original: Cross-Pose Estimation Design Choices.
Revised: Cross-Pose Estimation Design Choices.

Original: We analyze the effects of design choices made in our CrossPose estimation algorithm for the upright pose mug hanging task.
Revised: We analyze the effects of design choices made in our CrossPose estimation algorithm for the upright pose mug hanging task.

Original: Speciﬁcally, we analyze the effects of 1) computing residual correspondence; 2) the use of weighted SVD over non-weighted in computing cross-pose; 3) using a transformer as the cross-object attention, as opposed to simpler model such as a 3-layer MLP.
Revised: Speciﬁcally, we analyze the effects of 1) computing residual correspondence; 2) the use of weighted SVD over non-weighted in computing cross-pose; 3) using a transformer as the cross-object attention, as opposed to simpler model such as a 3-layer MLP.

Original: Table 3 shows that each major component of our system is important for task success.
Revised: Table 3 shows that each major component of our system is important for task success.

Original: See more ablation experiments in Appendix F.1.1.
Revised: See more ablation experiments in Appendix F.1.1.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Real-World Experiments: We explore the hanging component of the mug on a rack task in a real world environment, which requires estimating the cross-pose between the mug and the rack.
Revised: Real-World Experiments: We explore the hanging component of the mug on a rack task in a real world environment, which requires estimating the cross-pose between the mug and the rack.

Original: We train TAX-Pose using real demonstrations of 10 different mugs hung on a rack (1 demonstration each, for a total of only 10 real-world demonstrations for training).
Revised: We train TAX-Pose using real demonstrations of 10 different mugs hung on a rack (1 demonstration each, for a total of only 10 real-world demonstrations for training).

Original: A motion primitive is used to grasp each mug, after which the robot plans a trajectory to apply the predicted cross-pose to the grasped mug.
Revised: A motion primitive is used to grasp each mug, after which the robot plans a trajectory to apply the predicted cross-pose to the grasped mug.

Original: We evaluate the model on the 10 training mugs in novel poses, as well as onunseen mugs.
Revised: We evaluate the model on the 10 training mugs in novel poses, as well as onunseen mugs.

Original: For each of the 20 mugs, we conduct 5 trials, varying the mug’s and rack’s starting poses in each trial.
Revised: For each of the 20 mugs, we conduct 5 trials, varying the mug’s and rack’s starting poses in each trial.

Original: Success is recorded if a peg penetrates the mug handle at the end of the trial.
Revised: Success is recorded if a peg penetrates the mug handle at the end of the trial.

Original: Our method achieves a success rate of 62% on training mugs in novel poses and 54% on unseen mugs.
Revised: Our method achieves a success rate of 62% on training mugs in novel poses and 54% on unseen mugs.

Original: A visualization of the results can be seen in Figure 5 and on the project website.
Revised: A visualization of the results can be seen in Figure 5 and on the project website.

Original: Note that our method is able to perform the mug hanging task while varying the pose of the mug rack (see our project website), whereas the baselines (NDF [12], DON [11]) cannot because they assume a ﬁxed, known rack position (see NDF [12] for baseline details).
Revised: Note that our method is able to perform the mug hanging task while varying the pose of the mug rack (see our project website), whereas the baselines (NDF [12], DON [11]) cannot because they assume a ﬁxed, known rack position (see NDF [12] for baseline details).

Original: 5.2 PartNet-Mobility Placement Tasks
Revised: 5.2 PartNet-Mobility Placement Tasks

Original: Task Description : We also deﬁne a PartNet-Mobility Placement task as placing a given action object relative to an anchor object based on a semantic goal position.
Revised: Task Description : We also deﬁne a PartNet-Mobility Placement task as placing a given action object relative to an anchor object based on a semantic goal position.

Original: We select a set of household furniture objects from the PartNet-Mobility dataset [26] as the anchor objects, and a set of small rigid objects released with the Ravens simulation environment [27] as the action objects.
Revised: We select a set of household furniture objects from the PartNet-Mobility dataset [26] as the anchor objects, and a set of small rigid objects released with the Ravens simulation environment [27] as the action objects.

Original: For each anchor object, we deﬁne a set of semantic goal positions (i.e. ‘top’, ‘left’, ‘right’, ‘in’), where action objects should be placed relative to each anchor.
Revised: For each anchor object, we deﬁne a set of semantic goal positions (i.e. ‘top’, ‘left’, ‘right’, ‘in’), where action objects should be placed relative to each anchor.

Original: Each semantic goal position deﬁnes a unique task in our cross-pose prediction framework.
Revised: Each semantic goal position deﬁnes a unique task in our cross-pose prediction framework.

Original: Given a synthetic point cloud observation of both objects, the task is to predict a cross-pose that places the object at the speciﬁc semantic goal.
Revised: Given a synthetic point cloud observation of both objects, the task is to predict a cross-pose that places the object at the speciﬁc semantic goal.

Original: We evaluate both a goal-conditioned variant ( TAX-Pose GC ), which is trained across all goals, and a task-speciﬁc variant ( TAX-Pose ) of our model, which trains a separate model per goal type (see Appendix F.2.
Revised: We evaluate both a goal-conditioned variant ( TAX-Pose GC ), which is trained across all goals, and a task-speciﬁc variant ( TAX-Pose ) of our model, which trains a separate model per goal type (see Appendix F.2.

Original: for details).
Revised: for details).

Original: In both cases we train only 1 model across all action and anchor objects.
Revised: In both cases we train only 1 model across all action and anchor objects.

Original: All models are trained entirely on simulated data and transfer directly to real-world with no ﬁnetuning.
Revised: All models are trained entirely on simulated data and transfer directly to real-world with no ﬁnetuning.

Original: Further task details can be found in the Appendix G.2.
Revised: Further task details can be found in the Appendix G.2.

Original: Baselines : We compare our method to a variety of end-to-end imitation-learning-based methods trained from a motion planner expert in simulation (see Appendix G.2.4 for details).
Revised: Baselines : We compare our method to a variety of end-to-end imitation-learning-based methods trained from a motion planner expert in simulation (see Appendix G.2.4 for details).

Original: Note that in the PartNet-Mobility Placement experiments, the pose of the anchor object poses are randomly varied.
Revised: Note that in the PartNet-Mobility Placement experiments, the pose of the anchor object poses are randomly varied.

Original: As such, we omit a comparison to methods that assume a static anchor, such as the Neural
Revised: As such, we omit a comparison to methods that assume a static anchor, such as the Neural

Original: Descriptor Field (NDF) [12] and Dense Object Nets (DON) [11] baselines used in the mug hanging task (Section 5.1), as both methods assume that the anchor objects are in a ﬁxed, known position.
Revised: Descriptor Field (NDF) [12] and Dense Object Nets (DON) [11] baselines used in the mug hanging task (Section 5.1), as both methods assume that the anchor objects are in a ﬁxed, known position.

Original: [Table]
Revised: [Table]

Original: Results : We report rotation ( E R ) and translation ( E t ) error between our predicted transform and the ground truth as geodesic rotational distance [28, 29] and L 2 distance, respectively.
Revised: Results : We report rotation ( E R ) and translation ( E t ) error between our predicted transform and the ground truth as geodesic rotational distance [28, 29] and L 2 distance, respectively.

Original: In both our simulated experiments (Table 4 Top) and our real-world experiments (Table 4 Bottom), we ﬁnd that TAX-Pose outperforms the baseline end-to-end imitation learning methods.
Revised: In both our simulated experiments (Table 4 Top) and our real-world experiments (Table 4 Bottom), we ﬁnd that TAX-Pose outperforms the baseline end-to-end imitation learning methods.

Original: In real-world experiments, our method generalizes to novel distributions of starting poses better than the Goal Flow baseline, placing action objects into the goal regions with a 92% success rate (see Appendix F.2 for more detailed results and Appendix G.2.4 for baseline details.).
Revised: In real-world experiments, our method generalizes to novel distributions of starting poses better than the Goal Flow baseline, placing action objects into the goal regions with a 92% success rate (see Appendix F.2 for more detailed results and Appendix G.2.4 for baseline details.).

Original:  Conclusion and Limitations
Revised:  Conclusion and Limitations

Original: In this paper, we show that dense soft correspondence can be used to learn task speciﬁc object relationships that generalize to novel object instances.
Revised: In this paper, we show that dense soft correspondence can be used to learn task speciﬁc object relationships that generalize to novel object instances.

Original: Correspondence residuals allow our method to estimate correspondences to virtual points, outside of the objects convex hull, drastically increasing the number of tasks this method can complete.
Revised: Correspondence residuals allow our method to estimate correspondences to virtual points, outside of the objects convex hull, drastically increasing the number of tasks this method can complete.

Original: We further show that this “cross-pose” can be learned for a task, using a small number of demonstrations.
Revised: We further show that this “cross-pose” can be learned for a task, using a small number of demonstrations.

Original: Finally, we show that our method far outperforms the baselines on two challenging tasks in both real and simulated experiments.
Revised: Finally, we show that our method far outperforms the baselines on two challenging tasks in both real and simulated experiments.

Original: While our method is able to predict relative pose relationships with high precision, it has several limitations:
Revised: While our method is able to predict relative pose relationships with high precision, it has several limitations:

Original: • Requires segmentation : Our method requires an accurate segmentation of two objects in order to predict their relative goal pose.
Revised: • Requires segmentation : Our method requires an accurate segmentation of two objects in order to predict their relative goal pose.

Original: • Performance degrades under occlusion : Our method performs best when complete point clouds are provided, captured via multiple cameras or by repeatedly reorienting the objects.
Revised: • Performance degrades under occlusion : Our method performs best when complete point clouds are provided, captured via multiple cameras or by repeatedly reorienting the objects.

Original: • Poorly deﬁned for multimodal relationships : Because our method extracts a single global estimate of relative pose from a ﬁxed set of correspondences, performance on objects with multiple valid goals is not well-deﬁned.
Revised: • Poorly deﬁned for multimodal relationships : Because our method extracts a single global estimate of relative pose from a ﬁxed set of correspondences, performance on objects with multiple valid goals is not well-deﬁned.

Original: Our method might be augmented with a consensus-based or sampling-based approach to capture the multimodality of the solution space in these cases.
Revised: Our method might be augmented with a consensus-based or sampling-based approach to capture the multimodality of the solution space in these cases.

Original: We leave this for future work.
Revised: We leave this for future work.

Original: Acknowledgements
Revised: Acknowledgements

Original: This material is based upon work supported by the National Science Foundation under Grant No. IIS-1849154.This work was also supported by LG Electronics. We are grateful to Daniel Seita and Jenny Wang for their helpful feedback and discussion on the paper. G Task Details
Revised: This material is based upon work supported by the National Science Foundation under Grant No. IIS-1849154.

Original: G.1 NDF Task Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: G.1.1 Baseline Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: G.1.2 Training Data . . .

Original: G.1.3 Training and Inference . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: G.1.4 Motion Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: G.1.5 Real-World Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: G.2 PartNet-Mobility Object Placement Task Details . . . . . . . . . . . . . . . . . . .

Original: G.2.1 Dataset Preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: G.2.2 Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: G.2.3 Motion Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: G.2.4 Baselines Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: H Author Contributions

Original: A Visual Explanations of TAX-Pose A.1 Illustration of Corrected Virtual Correspondence

Original: The virtual corresponding points, V A , V B given by Equation 3 in the main text, are constrained to be within the convex hull of each object.

Original: However, correspondences which are constrained to the convex hull are insufﬁcient to express a large class of desired tasks.

Original: For instance, we might want a point on the handle of a teapot to correspond to some point above a stovetop, which lies outside the convex hull of the points on the stovetop.

Original: To allow for such placements, for each point-wise embedding φ i , we further learn a residual vector , δ A i ∈

Original: ∆

Original: A that corrects each virtual corresponding point, allowing us to displace each virtual corresponding point to any arbitrary location that might be suitable for the task.

Original: Concretely, we use a point-wise neural network g R which maps each embedding into a 3D residual vector:

Original: [Equation]

Original: Applying these to the virtual points, we get a set of corrected virtual correspondences , ˜ v A i ∈ ˜V A and ˜ v B i ∈ ˜V B , deﬁned as

Original: [Equation]

Original: These corrected virtual correspondences ˜ v A i deﬁne the estimated goal location relative to object B for each point p i ∈

Original: P A in object A , and likewise for each point in object B , as shown in Figure S.1.

Original: A.2 Learned Importance Weights

Original: A visualization of the learned importance weights, α A and α B for the mug and rack are visualized by both color scheme and point size in Figure S.

Original: B Proof of TAX-Pose Translational Equivariance

Original: One beneﬁt of our method is that it is translationally equivariant by construction.

Original: This mean that if the object point clouds, P A and P B , are translated by random translation t α and t β , respectively,

Original: i.e.

Original: P A (cid:48)

Original: =

Original: P A + t α and P B (cid:48)

Original: =

Original: P B + t β , then the resulting corrected virtual correspondences, ˜V B and ˜V A , respectively, are transformed accordingly, i.e. ˜V B + t β and ˜V A + t α , respectively, as we will show below.

Original: This results in an estimated cross-pose transformation that is also equivariant to translation by construction.

Original: This is achieved because our learned features and correspondence residuals are invariant to translation, and our virtual correspondence points are equivariant to translation.

Original: First, our point features are a function of centered point clouds.

Original: That is, given point clouds P A and

Original: P B , the mean of each point cloud is computed as

Original: [Equation]

Original: This mean is then subtracted from the clouds,

Original: [Equation]

Original: which centers the cloud at the origin.

Original: The features are then computed on the centered point clouds:

Original: Φ k = g k

Original: [Equation]

Original: ¯P k ) .

Original: Since the point clouds are centered before features are computed, the features Φ k are invariant to an arbitrary translation P k (cid:48)

Original: =

Original: P k + t κ .

Original: These translationally invariant features are then used, along with the original point clouds, to compute “corrected virtual points” as a combination of virtual correspondence points, v k (cid:48)

Original: i and the correspondence residuals, δ k (cid:48) i .

Original: As we will see below, the “corrected virtual points” will be translationally equivariant by construction.

Original: The virtual correspondence points, v k (cid:48)

Original: i , are computed using weights that are a function of only the translationally invariant query and key values from the cross-object attention transformer g T K , Q K and K K , which are in turn functions of only the translationally invariant features, Φ k :

Original: [Equation]

Original: thus the weights are also translationally invariant.

Original: These translationally invariant weights are applied to the translated cloud

Original: [Equation]

Original: t

Original: [Equation]

Original: j w A→B

Original: i

Original: [Equation]

Original: j = P B w A→B

Original: i + t β

Original: [Equation]

Original: since (cid:80) N B j =1 w A→B ij = 1 .

Original: Thus the virtual correspondence points v A (cid:48)

Original: i are equivalently translated.

Original: The same logic follows for the virtual correspondence points v B (cid:48) i .

Original: This gives us a set of translationally equivaraint virtual correspondence points v A (cid:48)

Original: i and v B (cid:48) i .

Original: The correspondence residuals, δ k (cid:48)

Original: i , are a direct function of only the translationally invariant features Φ k ,

Original: [Equation]

Original: therefore they are also translationally invariant.

Original: Since the virtual correspondence points are translationally equivariant, v A (cid:48)

Original: i = v A

Original: i + t β and the correspondence residuals are translationally invariant, δ k (cid:48)

Original: i =

Original: δ ki , the ﬁnal corrected virtual correspondence points, ˜ v A (cid:48)

Original: i , are translationally equivariant, i.e. ˜ v A (cid:48)

Original: i = v A i + δ ki + t β . This  also holds for ˜ v B (cid:48)
Revised: This work was also supported by LG Electronics.

Original: i , giving us the ﬁnal translationally equivariant correspondences between the translated object clouds as (cid:16)

Original: P A

Original: + t α , ˜V B + t β (cid:17) and (cid:16) P B + t β , ˜V A + t α (cid:17) ,

Original: where ˜V B = (cid:2) ˜ v A 1 . . .

Original: ˜ v A N A (cid:3) (cid:62) .

Original: As a result, the ﬁnal computed transformation will be automatically adjusted accordingly.

Original: Given that we use weighted SVD to compute the optimal transform, T AB , with rotational component

Original: R AB and translational component t AB , the optimal rotation remains unchanged if the point cloud is translated, R A (cid:48) B (cid:48)

Original: = R AB , since the rotation is computed as a function of the centered point clouds.

Original: The optimal translation is deﬁned as

Original: [Equation]

Original: where ¯˜ v A and ¯ p

Original: A are the means of the corrected virtual correspondence points, ˜V B , and the object cloud P A , respectively, for object A .

Original: Therefore, the optimal translation between the translated point cloud P A (cid:48) and corrected virtual correspondence points ˜V A (cid:48) is

Original: [Equation]

Original: To simplify the analysis, if we assume that, for a given example, R AB

Original: =

Original: I , then we get t A (cid:48) B (cid:48)

Original: =

Original: t AB + t β − t α , demonstrating that the computed transformation is translation-equivariant by construction.

Original: C Description of Cross-Object Attention Weight Computation

Original: To map our estimated features Ψ A and Ψ B obtained from object-speciﬁc embedding networks (DGCNN), g A and g B respectively, to a set of normalized weight vectors W A→B and W B→A , we use the cross attention mechanism of our cross-object attention Transformer module [1].

Original: Following

Original: Equations 5a and 5b from the paper, we can extract the desired normalized weight vector w A→B i for  the virtual corresponding point v A i assigned to any point p A i ∈ P A using the intermediate attention embeddings of cross-object attention module as:
Revised: We are grateful to Daniel Seita and Jenny Wang for their helpful feedback and discussion on the paper.

Original: [Equation]

Original: where q K i ∈ Q K , and Q K , K K ∈ R N K ×

Original: d are the query and key (respectively) for object K associated with cross-object attention Transformer module g T K , as shown in Figure S.3.

Original: These weights are then used to compute the virtual corresponding points V A , V B using Equations 5a and 5b in the main paper.

Original: C.1 Ablation

Original: To explore the importance of this weight computation design choice described in Equation 2, we conducted an ablation experiment on this design choice against an alternative, arguably simpler method for cross-object attention weight computation that was used in prior work [2].

Original: Since the point embeddings φ A i and φ B i have the same dimension d , we can select the inner product of the space as a similarity metric between two embeddings.

Original: To compute the virtual corresponding point v A i assigned to any point p A i ∈ P A , we can extract the desired normalized weight vector w A→B i with the softmax function:

Original: [Equation]

Original: This is the approach used in the prior work of Deep Closest Point (DCP) [2].

Original: In the experiments below, we refer to this approach as point embedding dot-product .

Original: We conducted an ablation experiment on the weight computation method used in TAX-Pose (Equation 2) against the simpler approach from DCP [2] (Equation 3), on the upright mug hanging task in simulation.

Original: The models are trained from 10 demonstrations and tested on 100 trials over the test mug set.

Original: As seen in Table S1, the TAX-Pose approach (Equation 2) outperforms point embedding dot-product (Equation 3) in all three evaluation categories on grasp , place , and overall in terms of test success rate.

Original: Attention Weight Ablation

Original: Grasp

Original: Place

Original: Overall

Original: Point

Original: Embedding Dot-Product (Eqn. 3) 0.

Original: TAX-Pose (Ours) (Eqn. 2)

Original: [Table]

Original: D Description of Weighted SVD

Original: The objective function for computing the optimal rotation and translation given a set of correspondences for object K , { p ki → ˜ v ki } N k i and weights { α ki } N k

Original: i , is as follows:

Original: [Equation]

Original: First we center (denoted with ∗ ) the point clouds and virtual points independently, with respect to the learned weights, and stack them into frame-speciﬁc matrices (along with weights) retaining their relative position and correspondence:

Original: [Equation]

Original: Then the minimizing rotation R AB is given by:

Original: [Equation]

Original: where Σ ∗ = diag( (cid:2) 1 , 1 , ... det( UV (cid:62) ) (cid:3) and svd is a differentiable SVD operation [3].

Original: The optimal translation can be computed as:

Original: [Equation]

Original: 5a

Original: [Equation]

Original: with N = N A + N B .

Original: In the special translation-only case, the optimal translation and be computed by setting R AB to identity in above equations.

Original: The ﬁnal transform can be assembled:

Original: [Equation]

Original: E Training Details E.1 Supervision

Original: To train the encoders g A (¯ P A ) , g B (¯ P B ) as well as the residual networks g R A (cid:16)

Original: φ A i (cid:17) , g R B (cid:16) φ B i (cid:17) , we use a set of losses deﬁned below.

Original: We assume we have access to a set of demonstrations of the task, in which the action and anchor objects are in the target relative pose such that T AB = I .

Original: Point Displacement Loss [4, 5]: Instead of directly supervising the rotation and translation (as is done in DCP), we supervise the predicted transformation using its effect on the points.

Original: For this loss, we take the point clouds of the objects in the demonstration conﬁguration, and transform each cloud by a random transform, ˆP A

Original: = T α P A , and ˆP B = T β P B .

Original: This would give us a ground truth transform of T GT AB = T β T − 1 α ; the inverse of this transform would move object B to the correct position relative to object A .

Original: Using this ground truth transform, we compute the MSE loss between the correctly transformed points and the points transformed using our prediction.

Original: [Equation]

Original: Direct Correspondence Loss.

Original: While the Point Displacement Loss best describes errors seen at inference time, it can lead to correspondences that are inaccurate but whose errors average to the correct pose.

Original: To improve these errors we directly supervise the learned correspondences ˜

Original: V A and ˜ V B :

Original: [Equation]

Original: Correspondence Consistency Loss.

Original: Furthermore, a consistency loss can be used.

Original: This loss penalizes correspondences that deviate from the ﬁnal predicted transform.

Original: A beneﬁt of this loss is that it can help the network learn to respect the rigidity of the object, while it is still learning to accurately place the object.

Original: Note, that this is similar to the Direct Correspondence Loss, but uses the predicted transform as opposed to the ground truth one.

Original: As such, this loss requires no ground truth:

Original: [Equation]

Original: Overall Training Procedure.

Original: We train with a combined loss L net =

Original: L disp + λ 1 L corr + λ 2 L cons , where λ 1 and λ 2 are hyperparameters.

Original: We use a similar network architecture as DCP [2], which consists of DGCNN [6] and a Transformer [1].

Original: In order to quickly adapt to new tasks, we optionally pre-train the DGCNN embedding networks over a large set of individual objects using the InfoNCE loss [7] with a geometric distance weighting and random transformations, to learn SE (3) invariant embeddings, see Appendix E.2 for details.

Original: E.2 Pretraining

Original: We utilize pretraining for the embedding network for the mug hanging task, and describe the details below.

Original: We pretrain embedding network for each object category (mug, rack, gripper), such that the embedding network is SE (3) invariant with respect to the point clouds of that speciﬁc object category.

Original: Speciﬁcally, the mug-speciﬁc embedding network is pretrained on 200 ShapeNet [8] mug instances, while the rack-speciﬁc and gripper-speciﬁc embedding network is trained on the same rack and Franka gripper used at test time, respectively.

Original: Note that before our pretraining, the network is randomly initialized with the Kaiming initialization scheme [9]; we don’t adopt any third-party pretrained models.

Original: For the network to be trained to be SE (3) invariant, we pre-train with InfoNCE loss [7] with a geometric distance weighting and random SE (3) transformations.

Original: Speciﬁcally, given a point cloud of an object instance, P A , of a speciﬁc object category A , and an embedding network g A , we deﬁne the point-wise embedding for P A as Φ A

Original: = g A

Original: ( P A ) , where φ A i ∈

Original: Φ A is a d -dimensional vector for each point p

Original: A i ∈ P A .

Original: Given a random SE (3) transformation, T , we deﬁne Ψ

Original: A =

Original: g A ( TP A ) , where ψ A i ∈

Original: Ψ

Original: A is the d -dimensional vector for the i th point p A i ∈ P A .

Original: The weighted contrastive loss used for pretraining, L wc , is deﬁned as

Original: [Equation]

Original: For this pretraining, we use λ := 10 .

Original: E.3 Architectural Variants

Original: Goal-Conditioned TAX-Pose : To enable a single TAX-Pose model to scale to multiple related placement sub-tasks for a pair of action and anchor objects, we design a goal-conditioned variant ( TAX-Pose GC ), which receives a one-hot encoding of the desired semantic goal position (e.g.

Original: ‘top’, ‘left’, ...) for the task.

Original: This contextual encoding is incorporated into each DGCNN module in the same way as proposed in the original DGCNN paper.

Original: This encoding can be used to provide an embedding of the speciﬁc placement relationship that is desired in a scene (e g. selecting a “top” vs.

Original: ) and thus enable conditioned placement.

Original: Vector Neurons : We brieﬂy experimented with Vector Neurons [10] and found that this led to worse performance on this task.

Original: F Additional Results F.1 NDF Placement Tasks F.1.1 Further Ablations on Mug Hanging Task

Original: In order to examine the effects of different design choices in the training pipeline, we conduct ablation experiments with ﬁnal task-success ( grasp , place , overall ) as evaluation metrics for Mug

Original: Hanging task with upright pose initialization for the following components of our method, see Table

Original: S2 for full ablation results along six ablated dimensions as detailed below.

Original: We also performed an ablation experiment on alternative cross-object attention weight computation, as explained in

Original: Appendix C and results can be found in Table S1.

Original: For consistency, all ablated models are trained for 15K steps.

Original: Loss.

Original: In the full pipeline reported, we use a weighted sum of the three types of losses described in Section 4.2 of the paper.

Original: Speciﬁcally, the loss used L net is given by

Original: [Equation]

Original: where we chose λ 1 = 0 .

Original: 1 , λ 2 = 1 after hyperparameter search.

Original: We ablate usage of all three types of losses, by reporting the ﬁnal task performance in simulation for all experiments, speciﬁcally, we report task success on the following L net variants.

Original: (a) Remove the point displacement loss term, L disp , after which we are left with L (cid:48) net = (0 . 1) L cons + L corr (b) Remove the direct correspondence loss term, L corr , after which we are left with L (cid:48)

Original: net =

Original: L disp + (0 . 1)

Original: L cons (c) Remove the correspondence consistency loss term, L cons , after which we are left with L (cid:48)

Original: net =

Original: L disp + L corr (d) From testing loss variants above, we found that the point displacement loss is a vital contributing factor for task success, where removing this loss term results in no overall task success, as shown in Table S2.

Original: However, in practice, we have found that adding the correspondence consistency loss and direct correspondence loss generally help to lower the rotational error of predicted placement pose compared to the ground truth of collected demos.

Original: To further investigate the effects of the combination of these two loss terms,

Original: we

Original: used a scaled weighted combination of L cons and L corr , such that the former weight of the displacement loss term is transferred to consistency loss term, with the new λ 1 = 1 . 1 , and with λ 2 = 1 stays unchanged.

Original: Note that this is different from variant (a) above, as now the consistency loss given a comparable weight with dense correspondence loss term, which intuitively makes sense as the consistency loss is a function of the predicted transform T AB to be used, while the dense correspondence loss is instead a function of the ground truth transform, T GT AB , which provides a less direct supervision on the predicted transforms.

Original: Thus we are left with

Original: [Equation]

Original: Usage of Correspondence Residuals.

Original: After predicting a per-point soft correspondence between objects A and B , we adjust the location of the predicted corresponding points by further predicting a point-wise correspondence residual vector to displace each of the predicted corresponding point.

Original: This allows the predicted corresponding point to get mapped to free space outside of the convex hulls of points in object A and B .

Original: This is a desirable adjustment for mug hanging task, as the desirable cross-pose usually require points on the mug handle to be placed somewhere near but not in contact with the mug rack, which can be outside of the convex hull of rack points.

Original: We ablate correspondence residuals by directly using the soft correspondence prediction to ﬁnd the cross-pose transform through weighted SVD, without any correspondence adjustment via correspondence residual.

Original: Weighted SVD vs Non-weighted SVD.

Original: We leverage weighted SVD as described in Section 4.1 of the paper as we leverage predicted per-point weight to signify the importance of speciﬁc correspondence.

Original: We ablate the use of weighted SVD, and we use an un-weighted

Original: SVD, where instead of using the predicted weights, each correspondence is assign equal weights of 1 N , where N is the number of points in the point cloud P used.

Original: Pretraining.

Original: In our full pipeline, we pretrain the point cloud embedding network such that the embedding network is SE (3) invariant.

Original: Speciﬁcally, the mug-speciﬁc embedding network is pretrained on 200 ShapeNet mug objects, while the rack-speciﬁc and gripper speciﬁc embedding network is trained on the same rack and Franka gripper used at test time, respectively.

Original: We conduct ablation experiments where

Original: (a) We omit the pretraining phase of embedding network (b) We do not ﬁnetune the embedding network during downstream training with taskspeciﬁc demonstrations.

Original: Note that in practice, we ﬁnd that pretraining helps speed up the downstream training by about a factor of 3, while models with or without pretraining both reach a similar ﬁnal performance in terms of task success after both models converge.

Original: Usage of Transformer as Cross-object Attention Module.

Original: In the full pipeline, we use transformer as the cross-object attention module, and we ablate this design choice by replacing the transformer architecture with a simple 3-layer MLP with ReLU activation and hidden dimension of 256, and found that this leads to worse place and grasp success.

Original: Dimension of Embedding.

Original: In the full pipeline, the embedding is chosen to be of dimension 512.

Original: We conduct experiment on much lower dimension of 16, and found that with dimension =16, the place success is much lower, dropped from 0.97 to 0.59.

Original: Ablation Experiment

Original: [Table]

Original: 3-Layer MLP In Place of Transformer

Original: [Table]

Original: F.1.2 Effects of Pretraining on Mug Hanging Task

Original: We explore the effects of pretraining on the ﬁnal task performance

Original: [Table]

Original: as well as training convergence speed.

Original: We have found that pretraining the point cloud embedding network as described in E.2, is a helpful but not necessary component in our training pipeline.

Original: Speciﬁcally, we ﬁnd that while utilizing pretraining reduces training time, allowing the model to reach similar task performance and train rotation/translation error with much fewer training steps, this component is not necessary if training time is not of concern.

Original: In fact, as see in Table S3, we ﬁnd that for mug hanging tasks, by training the models from scratch without our pretraining, the models are able to reach similar level of task performance of 0 .

Original: 99 grasp, 0 . 92 for place and 0 .

Original: 92 for overall success rate.

Original: Furthermore, it is able to achieve similar level of train rotation error of 4 . 91 ◦ and translation error of 0 .

Original: 01 m , compared to the models with pretraining.

Original: However, without pre-trainig, the model needs to be trained for about 2 times longer (26K steps compared to 15K steps) to reach the similar level of performance.

Original: Thus we adopt our object-level pretraining in our overall pipeline to allow lower training time.

Original: Another beneﬁt of pretraining is that the pretraining for each object category is done in a taskagnostic way, so the network can be more quickly adapted to new tasks after the pretraining is performed.

Original: For example, we use the same pre-trained mug embeddings for both the gripper-mug cross-pose estimation for grasping as well as the mug-rack cross-pose estimation for mug hanging.

Original: [Table]

Original: F.1.3 Additional Simulation Experiments on Bowl and Bottle Placement Task

Original: [Table]

Original: Additional results on Grasp , Place and Overall success rate in simulation for Bowl and Bottle are shown in Table S4.

Original: For bottle and bowl experiment, we follow the same experimentation setup as in [12], where the successful grasp is considered if a stable grasp of the object is obtained, and a successful place is considered when the bottle or bowl is stably placed upright on the elevated ﬂat slab over the table without falling on the table.

Original: Reported task success results in are for both Upright

Original: Pose and Arbitrary Pose run over 100 trials each.

Original: F.1.4 Failure Cases

Original: Some failure cases for TAX-Pose occur when the predicted gripper misses the rim of the mug by a xy-plane translation error, thus resulting in failure of grasp, as seen in Figure S.4a.

Original: A common failure mode for the mug placement subtask is characterized by an erroneous transform prediction that results in the mug’s handle completely missing the rack hanger, thus resulting in placement failure, as seen in Figure S.4b.

Original: [Figure]

Original: F.2 PartNet-Mobility Tasks F.2.1 Expanded Results Tables

Original: In the main text, we presented aggregated results of the performance of each method by averaging the quantitative metrics for each sub-task for each object (“In”, “On”, “Left”, and “Right” in simulation and “In”, “On” and “Left” in real-world)

Original: [Table]

Original: and then averaged across object classes to arrive at a single metric per method.

Original: Here, we present the per-class breakdown of performance.

Original: See Table S for simulated results

Original: [Table]

Original: and Table S6 for real-world results

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Table]

Original: We further provide results per-sub-task in simulation.

Original: For each category of anchor objects, sub-tasks may or may not all be well-deﬁned.

Original: For example, the doors of safes might occlude the action object completely in a demonstration for “Left” and “Right” tasks due to the handedness of the door; and a table’s height might be too tall for the camera to see the action object placed during the “Top” task.

Original: To avoid these ill-deﬁned cases

Original: [Table]

Original: we omit object-category / sub-task pairings which cannot be consistently deﬁned from training and evaluation.

Original: We show visualizations of each deﬁned task for each object category in Figure S.5.

Original: Results for each sub-task can be found in Tables S7 1 , S8, S9, and S10 respectively.

Original: N/A

Original: N/A

Original: N/A

Original: N/A

Original: N/A

Original: N/A

Original: N/A

Original: In

Original: On

Original: Left

Original: Right

Original: [Figure]

Original: [Table]

Original: [Table]

Original: t

Original: E R

Original: E t

Original: E R

Original: E t

Original: E R

Original: E t

Original: E R

Original: E t

Original: E

Original: [Table]

Original: [Table]

Original: [Table]

Original: E R

Original: E t

Original: E R

Original: E t

Original: E R

Original: E t

Original: E R

Original: E t

Original: E R

Original: E t

Original: E R

Original: E t

Original: [Table]

Original: F.2.2 Goal-Conditioned Variant

Original: We train our goal-conditioned variant TAX-Pose GC (Appendix E.3) to predict the correct crosspose across sub-tasks

Original: [Table]

Original: incorporating a one-hot encoding of each sub-task (i

Original: [Table]

Original: e. ‘top’, ‘in’, ‘left’, ’right‘) so the model can infer the desired semantic goal location.

Original: Importantly, as with the taskspeciﬁc model ( TAX-Pose ), the TAX-Pose GC model is trained across all PartNet-Mobility object categories .

Original: We report the performance of the variants in Table S

Original: [Table]

Original: F.2.3 Failure Cases

Original: Some failure cases for TAX-Pose occur when the predicted cross-pose does not respect the physical constraints in the scene.

Original: For example, as seen in Fig.

Original: S.6.

Original: TAX-Pose would fail when the prediction violates the physical constraints of the objects, which in this case the action object collides with the anchor object.

Original: In the real world, this would yield the robot unable to plan a correct path.

Original: [Figure]

Original: G Task Details G.1 NDF Task Details

Original: In this section, we describe the Mug Hanging task of the NDF Tasks and experiments in detail.

Original: The

Original: Mug Hanging task is consisted of two sub tasks: grasp and place .

Original: A grasp success is achieved when the mug is grasped stably by the gripper, while a place success is achieved when the mug is hung stably on the hanger of the rack.

Original: Overall mug hanging success is achieved when the predicted transforms enable both grasp and place success for the same trial.

Original: See Figure S.7 for a detailed breakdown of the mug hanging task in stages.

Original: G.1.1 Baseline Description

Original: In simulation, we compare our method to the results described in [12].

Original: • Dense Object Nets (DON) [11]: Using manually labeled semantic keypoints on the demonstration clouds, DON is used to compute sparse correspondences with the test objects.

Original: These correspondences are converted to a pose using SVD.

Original: A full description of usage of DON for the mug hanging task can be found in [12].

Original: • Neural Descriptor Field (NDF) [12]: Using the learned descriptor ﬁeld for the mug, the positions of a constellation of task speciﬁc query points are optimized to best match the demonstration using gradient descent.

Original: G.1.2 Training Data

Original: To be directly comparable with the baselines we compared to, we use the exact same sets of demonstration data used to train the network in NDF [12], where the data are generated via teleportation in

Original: PyBullet, collected on 10 mug instances with random pose initialization.

Original: G.1.3 Training and Inference

Original: Using the pretrained embedding network for mug and gripper, we train a grasping model for the grasping task to predict a transformation T g → m in gripper’s frame from gripper to mug to complete the grasp stage of the task.

Original: Similarly, using the pretrained embedding network for rack and mug, we train a placement model for the placing task to predict a transformation T m → r in mug’s frame from mug to rack to complete the place stage of the task.

Original: Both models are trained with the same combined loss L net as described in the main paper.

Original: During inference, we simply use grasping model to predict the T g → m at test time, and placement model to predict T m → r at test time.

Original: G.1.4 Motion Planning

Original: After the model predicts a transformation T g → m and T m → r , using the known gripper’s world frame pose, we calculate the desired gripper end effector pose at grasping and placement, and pass the end effector to IKFast to get the desired joint positions of Franka at grasping and placement.

Original: Next we pass the desired joint positions at gripper’s initial pose, and desired grasping joint positions to

Original: OpenRAVE motion planning library to solve for trajectory from gripper’s initial pose to grasp pose, and then grasp pose to placement pose for the gripper’s end effector.

Original: G.1.5 Real-World Experiments

Original: We pre-train the DGCNN embedding network with rotation-equivariant loss on ShapeNet mugs’ simulated point clouds in simulation.

Original: Using the pre-trained embedding, we then train the full TAXPose model with the 10 collected real-world point clouds.

Original: G.2 PartNet-Mobility Object Placement Task Details

Original: In this section, we describe the PartNet-Mobility Object Placement experiments in detail.

Original: We select a set of household furniture objects from the PartNet-Mobility dataset as the anchor objects, and a set of small rigid objects released with the Ravens simulation environment as the action objects.

Original: For each anchor object, we deﬁne a set of semantic goal positions (i.e. ‘top’, ‘left’, ‘right’, ‘in’), where action objects should be placed relative to each anchor.

Original: Each semantic goal position deﬁnes a unique task in our cross-pose prediction framework.

Original: G.2.1 Dataset Preparation

Original: Simulation Setup.

Original: We leverage the PartNet-Mobility dataset [15] to ﬁnd common household objects as the anchor object for TAX-Pose prediction.

Original: The selected subset of the dataset containscategories of objects.

Original: We split the objects into 54 seen and 14 unseen instances.

Original: During training, for a speciﬁc task of each of the seen objects, we generate an action-anchor objects pair by randomly sampling transformations from SE (3) as cross-poses.

Original: The action object is chosen from the Ravens simulator’s rigid body objects dataset [16].

Original: We deﬁne a subset of four tasks (“In”, “On”, “Left” and “Right”) for each selected anchor object.

Original: Thus, there exists a ground-truth cross-pose (deﬁned by human manually) associated with each deﬁned speciﬁc task.

Original: We then use the ground-truth TAXPoses to supervise each task’s TAX-Pose prediction model.

Original: For each observation action-anchor objects pair, we sample 100 times using the aforementioned procedure for the training and testing datasets.

Original: Real-World Setup.

Original: In real-world, we select a set of anchor objects: Drawer, Fridge, and Oven and a set of action objects: Block and Bowl.

Original: We test 3 (“In”, “On”, and “Left”) TAX-Pose models in real-world without retraining or ﬁnetuning.

Original: The point here is to show the method capability of generalizing to unseen real-world objects.

Original: G.2.2 Metrics

Original: Simulation Metrics.

Original: In simulation, with access to the object’s ground-truth pose, we are able to quantitatively calculate translational and rotation error of the TAX-Pose prediction models.

Original: Thus, we report the following metrics on a held-out set of anchor objects in simulation:

Original: Translational Error :

Original: The L2 distance between the inferred

Original: cross-pose translation ( t pred AB ) and the ground-truth pose translation ( t GT AB ).

Original: Rotational Error :

Original: The geodesic SO

Original: 3) distance [17, 18] between the predicted cross pose rotation ( R pred AB ) and the ground-truth rotation ( R GT AB ).

Original: [Equation]

Original: Real-World Metrics.

Original: In real-world, due to the difﬁculty of deﬁning ground-truth TAX-Pose, we instead manually, qualitatively deﬁne goal “regions” for each of the anchor-action pairs.

Original: The goalregion should have the following properties:

Original: • The predicted TAX-Pose of the action object should appear visually correct.

Original: For example, if the speciﬁed task is “In”, then the action object should be indeed contained within the anchor object after being transformed by predicted TAX-Pose.

Original: • The predicted TAX-Pose of the action object should not violate physical constraints of the workspace and of the relation between the action and anchor objects.

Original: Speciﬁcally, the action object should not interfere with/collide with the anchor object after being transformed by the predicted TAX-Pose.

Original: See Figure S.6 for an illustration of TAX-Pose predictions that fail to meet this criterion.

Original: G.2.3 Motion Planning

Original: In both simulated and real-world experiments, we use off-the-shelf motion-planning tools to ﬁnd a path between the starting pose and goal pose of the action object.

Original: Simulation.

Original: To actuate the action object from its starting pose T 0 to its goal pose transformed by the predicted TAX-Pose ˆ T AB T 0 , we plan a path free of collision.

Original: Learning-based methods such as [19] deal with collision checking with point clouds by training a collision classiﬁer.

Original: A more dataefﬁcient method is by leveraging computer graphics techniques, transforming the point clouds into

Original: [Figure]

Original: marching cubes [20], which can then be used to efﬁciently reconstruct meshes.

Original: Once the triangular meshes are reconstructed, we can deploy off-the-shelf collision checking methods such as FCL [21] to detect collisions in the planned path.

Original: Thus, in our case, we use position control to plan a trajectory of the action object A to move it from its starting pose to the predicted goal pose.

Original: We use OMPL [22] as the motion planning tool and the constraint function passed into the motion planner is from the output of FCL after converting the point clouds to meshes via marching cubes.

Original: Real World.

Original: In real-world experiments, we need to resolve several practical issues to make TAXPose prediction model viable.

Original: First, we do not have access to a mask that labels action and anchor objects.

Original: Thus, we manually deﬁne a mask by using a threshold value of y -coordinate to automatically detect discontinuity in y -coordinates, representing the gap of spacing between action and anchor objects upon placement.

Original: Next, grasping action objects is a non-trivial task.

Original: Since, we are only using 2 action objects (a cube and a bowl), we manually deﬁne a grasping primitive for each action object.

Original: This is done by hand-picking an offset from the centroid of the action object before grasping, and an approach direction after the robot reaches the pre-grasp pose to make contacts with the object of interest.

Original: The offsets are chosen via kinesthetic teaching on the robot when the action object is under identity rotation (canonical pose).

Original: Finally, we need to make an estimation of the action’s starting pose for motion planning.

Original: This is done by ﬁrst statistically cleaning the point cloud [14] of the action object, and then calculating the centroid of the action object point cloud as the starting position.

Original: For starting rotation, we make sure the range of the rotation is not too large for the pre-deﬁned grasping primitive to handle.

Original: Another implementation choice here is to use ICP [23] calculate a transformation between the current point cloud to a pre-scanned point cloud in canonical (identity) pose.

Original: We use the estimated starting pose to guide the pre-deﬁned grasp primitive.

Original: Once a successful grasp is made, the robot end-effector is rigidly attached to the action object, and we can then use the same predicted TAX-Pose to calculate the end pose of the robot end effector, and thus feed the two poses into MoveIt!

Original: to get a full trajectory in joint space.

Original: Note here that the collision function in motion planning is comprised of two parts: workspace and anchor object.

Original: That is, we ﬁrst reconstruct the workspace using boxes to avoid collision with the table top and camera mount, and we then reconstruct the anchor object in RViz using Octomap [24] using the cleaned anchor object point cloud.

Original: In this way, the robot is able to avoid collision with the anchor object as well.

Original: See Figure S.8 for the workspace.

Original: G.2.4 Baselines Description

Original: In simulation, we compare our method to a variety of baseline methods.

Original: • E2E Behavioral Cloning : Generate motion-planned trajectories using OMPL that take the action object from start to goal.

Original: These serve as “expert” trajectories for Behavioral Cloning (BC).

Original: Our policy is represented as a PointNet++ network that, at each time step, takes as input the point cloud observation of the action and anchor objects and outputs an incremental 6-DOF transformation that imitates the expert trajectory.

Original: The 6-DoF transformation is expressed using Euclidean xyz translation and rotation quaternion.

Original: The ﬁnal achieved pose of the action object at the terminal state is used for computing the evaluation metrics.

Original: • E2E DAgger : Using the same BC dataset and the same PointNet++ architecture as above, we train a policy that outputs the same transformation representation as in BC using DAg-

Original: ger [13].

Original: The ﬁnal achieved pose of the action object at the terminal state is used for computing the evaluation metrics.

Original: • Trajectory Flow : Using the same BC dataset with DAgger, we train a dense policy using PointNet++ to predict a dense per-point 3D ﬂow vector at each time step instead of a single 6-DOF transformation.

Original: Given this dense per-point ﬂow, we add the per-point ﬂow to each point of the current time-step’s point cloud, and we are able to extract a rigid transformation between the current point cloud and the point cloud transformed by adding per-point ﬂow vectors using SVD, yielding the next pose.

Original: The ﬁnal achieved pose of the action object at the terminal state is used for computing the evaluation metrics.

Original: • Goal Flow : Instead of training a multi-step policy to reach the goal, we train a PointNet++ network to output a single dense ﬂow prediction which assigns a per-point 3D ﬂow vector that points from each action object point from its starting pose directly to its corresponding goal location.

Original: Given this dense per-point ﬂow, we add the per-point ﬂow to each point of the start point cloud, and we are able to extract a rigid transformation between the start point cloud and the point cloud transformed by adding per-point ﬂow vectors using SVD, yielding goal pose.

Original: We pass the start and goal pose into a motion planner (OMPL) and execute the planned trajectory.

Original: The ﬁnal achieved pose of the action object at the terminal state is used for computing the evaluation metrics.

Original: H Author Contributions

Original: Chuer Pan designed and implemented the cross-correspondence TAX-Pose model, correspondence residuals, and the bi-directional weighted SVD method.

Original: She also designed, implemented, and conducted the simulation experiments of the NDF tasks as well as the study on varying the number of demos and the various ablation experiments.

Original: She also designed, implemented, and trained the models for mug hanging task in the real-world.

Original: Brian Okorn proposed, designed, and implemented the cross-correspondence model, correspondence residuals, and the bi-directional weighted SVD method, as well as analyzed the invariances of the current framework.

Original: Harry Zhang wrote early prototypes of TAX-Pose using residual ﬂows without cross-attention, which later became a baseline of TAX-Pose.

Original: He also wrote the infrastructure of the PartNet-Mobility object placement task’s simulated data collection and training.

Original: He also wrote all the baselines in the

Original: PartNet-Mobility placement task.

Original: He designed and conducted all real-world robot experiments for both tasks in the real-world.

Original: Ben Eisner designed the PartNet-Mobility tasks, implemented the Pybullet physics environment required for their rendering and simulation, and conducted training and evaluation of TAX-Pose on the

Original: PartNet-Mobility dataset.

Original: He also designed and evaluated the goal-conditioned variant of TAX-Pose.