Original: A Simple Method for Complex In-Hand Manipulation
Revised: A System for General In-Hand Object Re-Orientation

Original: Abstract: In-hand object reorientation has been a challenging problem in robotics due to high dimensional actuation space and the frequent change in contact state between the ﬁngers and the objects.
Revised: Abstract: In-hand object reorientation has been a challenging problem in robotics due to high dimensional actuation space and the frequent change in contact state between the ﬁngers and the objects.

Original: We present a simple model-free framework that can learn to reorient objects with both the hand facing upwards and downwards.
Revised: We present a simple model-free framework that can learn to reorient objects with both the hand facing upwards and downwards.

Original: We demonstrate the capability of reorienting over 2000 geometrically different objects in both cases.
Revised: We demonstrate the capability of reorienting over 2000 geometrically different objects in both cases.

Original: The learned policies show strong zero-shot transfer performance on new objects.
Revised: The learned policies show strong zero-shot transfer performance on new objects.

Original: We provide evidence that these policies are amenable to realworld operation by distilling them to use observations easily available in the real world.
Revised: We provide evidence that these policies are amenable to realworld operation by distilling them to use observations easily available in the real world.

Original: The videos of the learned policies are available at: https://taochenshh.
Revised: The videos of the learned policies are available at: https://taochenshh.

Original: github
Revised: github

Original: io/projects/in-hand-reorientation .
Revised: .io/ projects hand reorientation

Original:  Introduction
Revised:  Introduction

Original: A common maneuver in many tasks of daily living is to pick an object, reorient it in hand and then either place it or use it as a tool.
Revised: A common maneuver in many tasks of daily living is to pick an object, reorient it in hand and then either place it or use it as a tool.

Original: Consider three simpliﬁed variants of this maneuver shown in Figure1.
Revised: Consider three simpliﬁed variants of this maneuver shown in Figure

Original: The task in the top row requires an upward-facing multi-ﬁnger hand to manipulate an arbitrary object in a random orientation to a goal conﬁguration shown in the rightmost column.
Revised: The task in the top row requires an upward-facing multi-ﬁnger hand to manipulate an arbitrary object in a random orientation to a goal conﬁguration shown in the rightmost column.

Original: The next two rows show tasks where the hand is facing downward and is required to reorient the object either using the table as a support or without the aid of any support surface respectively.
Revised: The next two rows show tasks where the hand is facing downward and is required to reorient the object either using the table as a support or without the aid of any support surface respectively.

Original: The last task is the hardest because the object is in an intrinsically unstable conﬁguration owing to the downward gravitational force and lack of support from the palm.
Revised: The last task is the hardest because the object is in an intrinsically unstable conﬁguration owing to the downward gravitational force and lack of support from the palm.

Original: Additional challenges in performing such manipulation with a multi-ﬁnger robotic hand stem from the control space being high-dimensional and reasoning about the multiple transitions in the contact state between the ﬁnger and the object.
Revised: Additional challenges in performing such manipulation with a multi-ﬁnger robotic hand stem from the control space being high-dimensional and reasoning about the multiple transitions in the contact state between the ﬁnger and the object.

Original: Due to its practical utility and several unsolved issues, in-hand object reorientation remains an active area of research.
Revised: Due to its practical utility and several unsolved issues, in-hand object reorientation remains an active area of research.

Original: Past work has tackled the in-hand reorientation problem via several approaches: (i)
Revised: Past work has tackled the in-hand reorientation problem via several approaches: (i)

Original: The use of analytical models with powerful trajectory optimization methods [1, 2, 3].
Revised: The use of analytical models with powerful trajectory optimization methods [1, 2, 3].

Original: While these methods demonstrated remarkable performance, the results were largely in simulation with simple object geometries and required detailed knowledge of the object model and physical parameters.
Revised: While these methods demonstrated remarkable performance, the results were largely in simulation with simple object geometries and required detailed knowledge of the object model and physical parameters.

Original: As such, it remains unclear how to scale these methods to real-world and generalize to new objects.
Revised: As such, it remains unclear how to scale these methods to real-world and generalize to new objects.

Original: Another line of work has employed (ii) model-based reinforcement learning [4, 5]; or (iii) model-free reinforcement learning with [6, 7, 8, 9] and without expert demonstrations [10, 11, 12, 13].
Revised: Another line of work has employed (ii) model-based reinforcement learning [4, 5]; or (iii) model-free reinforcement learning with [6, 7, 8, 9] and without expert demonstrations [10, 11, 12, 13].

Original: While some of these works demonstrated learned skills on real robots, it required use of additional sensory apparatus not readily available in the real-world (e.g., motion capture system) to infer the object state, and the learned policies did not generalize to diverse objects.
Revised: While some of these works demonstrated learned skills on real robots, it required use of additional sensory apparatus not readily available in the real-world (e.g., motion capture system) to infer the object state, and the learned policies did not generalize to diverse objects.

Original: Furthermore, most prior methods operate in the simpliﬁed setting of the hand facing upwards.
Revised: Furthermore, most prior methods operate in the simpliﬁed setting of the hand facing upwards.

Original: The only exception is pick-and-place, but it does not involve any in-hand re-orientation.
Revised: The only exception is pick-and-place, but it does not involve any in-hand re-orientation.

Original: A detailed discussion of prior research is provided in Section 5.
Revised: A detailed discussion of prior research is provided in Section 5.

Original: In this paper, our goal is to study the object reorientation problem with a multi-ﬁnger hand in its general form.
Revised: In this paper, our goal is to study the object reorientation problem with a multi-ﬁnger hand in its general form.

Original: We desire (a) manipulation with hand facing upward or downward; (b) the ability of using external surfaces to aid manipulation; (c) the ability to reorient objects of novel shapes to arbitrary orientations; (d) operation from sensory data that can be easily obtained in the real world such as RGBD images and joint positions of the hand.
Revised: We desire (a) manipulation with hand facing upward or downward; (b) the ability of using external surfaces to aid manipulation; (c) the ability to reorient objects of novel shapes to arbitrary orientations; (d) operation from sensory data that can be easily obtained in the real world such as RGBD images and joint positions of the hand.

Original: While some of these aspects have been individually demonstrated in prior work, we are unaware of any published method that realizes all four.
Revised: While some of these aspects have been individually demonstrated in prior work, we are unaware of any published method that realizes all four.

Original: Our main contribution is building a simple framework that achieves the desiderata.
Revised: Our main contribution is building a system that achieves the desiderata.

Original: Our method is simple in the following aspects.

Original: First, we do not make use of explicit models of the object, the manipulator, or the contact dynamics.

Original: Our method builds upon awell-established model-free RL algorithm. The three key components: teacher-student learning, gravity curriculum, and stable initialization of objects are general ideas that are likely to be useful even outside the context of in-hand re-orientation tasks.
Revised: The core of our framework is a model-free reinforcement learning with three key components: teacher-student learning, gravity curriculum, and stable initialization of objects.

Original: Second, we do not require any special pre-processing of sensory observations.For example, our vision-based policies can work directly from RGBD point clouds captured by cameras.
Revised: Our system requires no knowledge of object or manipulator models, contact dynamics or any special pre-processing of sensory observations.

Original: We experimentally test our framework using a simulated Shadow hand.
Revised: We experimentally test our framework using a simulated Shadow hand.

Original: Due to the scope of the problem and the ongoing pandemic, we limit our experiments to be in simulation.
Revised: Due to the scope of the problem and the ongoing pandemic, we limit our experiments to be in simulation.

Original: However, we provide evidence indicating that the learned policies can be transferred to the real world in the future.
Revised: However, we provide evidence indicating that the learned policies can be transferred to the real world in the future.

Original: A Surprising Finding : While seemingly counterintuitive, we found that a single policy that had no access to shape information can manipulate a large number of previously unseen objects in all the three settings mentioned above.
Revised: A Surprising Finding : While seemingly counterintuitive, we found that policies that have no access to shape information can manipulate a large number of previously unseen objects in all the three settings mentioned above.

Original: At the start of the project, we hypothesized that developing visual processing architecture for inferring shape while the robot manipulates the object would be the primary research challenge.
Revised: At the start of the project, we hypothesized that developing visual processing architecture for inferring shape while the robot manipulates the object would be the primary research challenge.

Original: On the contrary, our results show that it is possible to learn control strategies for general in-hand object re-orientation that are shape-agnostic.
Revised: On the contrary, our results show that it is possible to learn control strategies for general in-hand object re-orientation that are shape-agnostic.

Original: Our results, therefore, suggest that visual perception may be less important for in-hand manipulation than previously thought.
Revised: Our results, therefore, suggest that visual perception may be less important for in-hand manipulation than previously thought.

Original: Of course, we still believe that the performance of our system can be improved by incorporating shape information.
Revised: Of course, we still believe that the performance of our system can be improved by incorporating shape information.

Original: However, our ﬁndings suggest a different framework of thinking: a lot can be achieved without vision, and that vision might be the icing on the cake instead of the cake itself.
Revised: However, our ﬁndings suggest a different framework of thinking: a lot can be achieved without vision, and that vision might be the icing on the cake instead of the cake itself.

Original:  Method
Revised:  Method

Original: Against our expectation, we found that simple extensions to existing techniques in robot learning can be used to construct adexterous system for general object reorientation.
Revised: We found that simple extensions to existing techniques in robot learning can be used to construct a system for general object reorientation.

Original: First, to avoid explicit modeling ofthe highly non-linear and frequent changes in the contact state between the object and the hand, we use model-free reinforcement learning (RL).
Revised: First, to avoid explicit modeling of non-linear and frequent changes in the contact state between the object and the hand, we use model-free reinforcement learning (RL).

Original: An added advantage is that model-free RL is amenable to directly operating from raw point cloud observations.
Revised: An added advantage is that model-free RL is amenable to direct operation from raw point cloud observations, which is preferred for real-world deployment.

Original: Second, we make use of privileged state information such as the positions/orientations/velocities of the object/ﬁngertips that is easily available in the simulator but not in the real world.Such state information can simplify policy learning and increase data efﬁciency.
Revised: We found that better policies can be trained faster using privileged state information such as the velocities of the object/ﬁngertips that is easily available in the simulator but not in the real world.

Original: We train an RL policy with privileged state information to reorient more than two thousand objects of diverse shapes (see Section 2.1). However, such a policy cannot be transferred to the real robot due to the challenges in obtaining such privileged information in real world.
Revised: To demonstrate the possibility of transferring learned policies to the real world in the future, we overcome the need for privileged information using the idea of teacher-student training [14, 15].

Original: We address this issue via teacher-student learning. The idea of ﬁrst learning an expert or teacher policy ( π E ) from privileged information, and then using it to guide the learning of a student policy ( π S ) that only uses sensory inputs available in the real world, is known as teacher-student training [14, 15]. We leverage this idea to demonstrate the possibility of transferring the learned policies to the real world in the future.
Revised: In this framework, ﬁrst, an expert or teacher policy ( π E ) is trained using privileged information. Next, the teacher policy guides the learning of a student policy ( π S ) that only uses sensory inputs available in the real world.

Original: Let the state space corresponding to π E and π S be S E and S S respectively.
Revised: Let the state space corresponding to π E and π S be S E and S S respectively.

Original: In general, S E (cid:54) =
Revised: In general, S E (cid:54) =

Original: S S .
Revised: S S .

Revised: We ﬁrst trained the teacher policy to reorient more than two thousand objects of diverse shapes (see

Original: We ﬁrst detail the method for distilling π E to a student policy using a reduced state space consisting of only the joint positions of the hand, the object position, and the difference in orientation from the goal conﬁguration (see Section 2.2.1).
Revised: Section 2.1). Next, we detail the method for distilling π E to a student policy using a reduced state space consisting of only the joint positions of the hand, the object position, and the difference in orientation from the goal conﬁguration (see Section 2.2.1).

Original: However, in the real world,  the object position and relative orientation must be inferred from sensory observation.
Revised: However, in the real world, even the object position and relative orientation must be inferred from sensory observation.

Original: Not only does this process require substantial engineering effort (e.g., a motion capture or a pose estimation system), but also inferring the pose of a symmetric object is prone to errors.
Revised: Not only does this process require substantial engineering effort (e.g., a motion capture or a pose estimation system), but also inferring the pose of a symmetric object is prone to errors.

Original: This is because a symmetric object at multiple orientations looks exactly the same in sensory space such as RGBD images.
Revised: This is because a symmetric object at multiple orientations looks exactly the same in sensory space such as RGBD images.

Original: To mitigate these issues, we further distill π E to operate directly from the point cloud and position of all the hand joints (see Section 2.2.2).
Revised: To mitigate these issues, we further distill π E to operate directly from the point cloud and position of all the hand joints (see Section 2.2.2).

Original: We propose a simple modiﬁcation that generalizes an existing 2D CNN architecture [16] to make this possible.
Revised: We propose a simple modiﬁcation that generalizes an existing 2D CNN architecture [16] to make this possible.

Original: The procedure described above works well for manipulation with the hand facing upwards and downwards when a table is available asa support.
Revised: The procedure described above works well for manipulation with the hand facing upwards and downwards when a table is available as support.

Original: However, when the hand faces downward without an underlying support surface,  itis important to have the objectinitialized in a stable conﬁguration.
Revised: However, when the hand faces downward without an underlying support surface, we found it important to initialize the object in a stable conﬁguration.

Original: In practical scenarios, an agent would ﬁrst lift the object and then re-orient it.

Original: The act of lifting results in  a stable initialization for further re-orientation. We, therefore, train a lifting policy thatprecedes in-hand manipulation. Finally, training in a curriculum where gravity is slowly introduced (i.e., gravity curriculum ) substantially improves performance. 
Revised: Finally, because gravity presents the primary challenge in learning policies with a downwardfacing hand, we found that training in a curriculum where gravity is slowly introduced (i.e., gravity curriculum ) substantially improves performance. These are discussed in Section 4.

Original: 2.1 Learning the teacher policy
Revised: 2.1 Learning the teacher policy

Original: We use model-free RL to learn the teacher policy ( π E ) for reorienting an object ( { O i | i = 1 , ..., N } ) from an initial orientation α o 0 to a target orientation α g ( α o 0 (cid:54) =
Revised: We use model-free RL to learn the teacher policy ( π E ) for reorienting an object ( { O i | i = 1 , ..., N } ) from an initial orientation α o 0 to a target orientation α g ( α o 0 (cid:54) =

Original: α g ).
Revised: α g ).

Original: At every time step t , the agent observes the state s t , executes the action a t sampled from the policy π E , and receives a reward r t .
Revised: At every time step t , the agent observes the state s t , executes the action a t sampled from the policy π E , and receives a reward r t .

Original: π E is optimized to maximize the expected discounted return: π = arg max π E E (cid:2) (cid:80) T − 1 t =

Revised: π E is optimized to maximize the expected discounted return: π = arg max π E E (cid:2) (cid:80) T − 1 t =

Original: γ t r t (cid:3) , where γ is the discount factor.
Revised: γ t r t (cid:3) , where γ is the discount factor.

Original: The task is successful if the angle difference ∆ θ between the object’s current ( α o t ) and the goal orientation ( α g ) is smaller than the threshold value ¯ θ , i.e. , ∆ θ ≤ ¯ θ .
Revised: The task is successful if the angle difference ∆ θ between the object’s current ( α ot ) and the goal orientation ( α g ) is smaller than the threshold value ¯ θ , i.e. , ∆ θ ≤ ¯ θ .

Original: To encourage the policy to be smooth, the previous action is appended to the inputs to the policy ( i.e. , a t = π E ( s t , a t − 1 ) ) and large actions are penalized in the reward function.
Revised: To encourage the policy to be smooth, the previous action is appended to the inputs to the policy ( i.e. , a t = π E ( s t , a t − 1 ) ) and large actions are penalized in the reward function.

Original: We experiment with two architectures for π E : (1) an MLP policy π M , (2) an RNN policy π R .
Revised: We experiment with two architectures for π E : (1) an MLP policy π M , (2) an RNN policy π R .

Original: We use PPO [17] to optimize π E .
Revised: We use PPO [17] to optimize π E .

Original: More details about the training are in Section C.1 and Section C.2 in the appendix.
Revised: More details about the training are in Section C.1 and Section C.2 in the appendix.

Original: Observation and action space : We deﬁne S E to include joint, ﬁngertip, object, and goal information as detailed in Table B.1 in the appendix.
Revised: Observation and action space : We deﬁne S E to include joint, ﬁngertip, object, and goal information as detailed in Table B.1 in the appendix.

Original: Note that S E does not include object shape or information about friction, damping, contact states between the ﬁngers and the object, etc.
Revised: Note that S E does not include object shape or information about friction, damping, contact states between the ﬁngers and the object, etc.

Original: We control the joint movements by commanding the relative change in the target joint angle ( q targett ) on each actuated joint (action a t ∈ R 20 ): q targett + = q targett + a t × ∆ t , where ∆ t is the control time step.
Revised: We control the joint movements by commanding the relative change in the target joint angle ( q targett ) on each actuated joint (action a t ∈ R 20 ): q targett + [Equation] where ∆ t is the control time step.

Original: We clamp the action command if necessary to make sure | ∆ q targett | ≤ 0 .
Revised: We clamp the action command if necessary to make sure | ∆ q targett | ≤ 0 .

Original: 33 rad .
Revised: 33 rad .

Original: The control frequency is 60 Hz .
Revised: The control frequency is 60 Hz .

Original: Dynamics randomization : Even though we do not test our policies on a real robot, we train and evaluate policies with domain randomization [18] to provide evidence that our work has the potential to be transferred to a real robotic system in the future.
Revised: Dynamics randomization : Even though we do not test our policies on a real robot, we train and evaluate policies with domain randomization [18] to provide evidence that our work has the potential to be transferred to a real robotic system in the future.

Original: We randomize the object mass, friction coefﬁcient, joint damping and add noise to the state observation s t and the action a t .
Revised: We randomize the object mass, friction coefﬁcient, joint damping and add noise to the state observation s t and the action a t .

Original: More details about domain randomization are provided in Table C.4 in the appendix.
Revised: More details about domain randomization are provided in Table C.4 in the appendix.

Original: 2.2 Learning the student policy
Revised: 2.2 Learning the student policy

Original: We distill the teacher π E into the student policy π S using Dagger [19], a learning-from-demonstration method that overcomes the covariate shift problem.
Revised: We distill the teacher π E into the student policy π S using Dagger [19], a learning-from-demonstration method that overcomes the covariate shift problem.

Original: We optimize π S by minimizing the KL-divergence between π S and π E : π S = arg min π S D KL (cid:0) π E ( s E t , a t − 1 )
Revised: We optimize π S by minimizing the KL-divergence between π S and π E : π S = arg min π S D KL (cid:0) π E ( s E t , a t − 1 )

Original: || π S ( s S t , a t − 1 )) (cid:1) .
Revised: || π S ( s S t , a t − 1 )) (cid:1) .

Original: Based on observation data available in real-world settings, we investigate two different choices of S S .
Revised: Based on observation data available in real-world settings, we investigate two different choices of S S .

Original: 2.2.1 Training student policy from low-dimensional state
Revised: 2.2.1 Training student policy from low-dimensional state

Original: In the ﬁrst case, we consider a non-vision student policy π S ( s S t , a t − 1 ) .
Revised: In the ﬁrst case, we consider a non-vision student policy π S ( s S t , a t − 1 ) .

Original: s S t ∈ R 31 includes the joint positions q t ∈ R 24 , object position p ot ∈ R 3 , quaternion difference between the object’s current and target orientation β t ∈ R 4 .
Revised: s S t ∈ R 31 includes the joint positions q t ∈ R 24 , object position p ot ∈ R 3 , quaternion difference between the object’s current and target orientation β t ∈ R 4 .

Original: In this case, S S ⊂ S E , and we assume the availability of object pose information, but do not require velocity information.
Revised: In this case, S S ⊂ S E , and we assume the availability of object pose information, but do not require velocity information.

Original: We use the same MLP and RNN network architectures used for π E on π S except the input dimension changes as the state dimension is different.
Revised: We use the same MLP and RNN network architectures used for π E on π S except the input dimension changes as the state dimension is different.

Original: 2.2.2 Training student policy from vision
Revised: 2.2.2 Training student policy from vision

Original: In the second case, S S only uses direct observations from RGBD cameras and the joint position ( q t ) of the robotic hand.
Revised: In the second case, S S only uses direct observations from RGBD cameras and the joint position ( q t ) of the robotic hand.

Original: We convert the RGB and Depth data into a colored point cloud using a pinhole camera model [20].
Revised: We convert the RGB and Depth data into a colored point cloud using a pinhole camera model [20].

Original: Our vision policy takes as input the voxelized point cloud of the scene W t , q t , and previous action command a t − 1 , and outputs the action a t , i.e. , a t = π S
Revised: Our vision policy takes as input the voxelized point cloud of the scene W t , q t , and previous action command a t − 1 , and outputs the action a t , i.e. , a t = π S

Original: ( W t , q t , a t − 1 ) .
Revised: ( W t , q t , a t − 1 ) .

Original: [Figure] Goal speciﬁcation : To avoid manually deﬁning per-object coordinate frame for specifying the goal quaternion, we provide the goal to the policy as an object point cloud rotated to the desired orientation
Revised: Goal speciﬁcation : To avoid manually deﬁning per-object coordinate frame for specifying the goal quaternion, we provide the goal to the policy as an object point cloud rotated to the desired orientation

Original: W g , i.e. , we only show the policy how the object should look like in the end (see the top left of
Revised: W g , i.e. , we only show the policy how the object should look like in the end (see the top left of

Original: Figure C.5).
Revised: Figure C.5).

Original: The input to π S is the point cloud W t =
Revised: The input to π S is the point cloud W t =

Original: W st ∪ W g where W st is the actual point cloud of the current scene obtained from the cameras.
Revised: W st ∪ W g where W st is the actual point cloud of the current scene obtained from the cameras.

Original: Details of obtaining W g are in Section C.2.
Revised: Details of obtaining W g are in Section C.2.

Original: Sparse3D-IMPALA-Net : To convert a voxelized point cloud into a lower-dimensional feature representation, we use a sparse convolutional neural network.
Revised: Sparse3D-IMPALA-Net : To convert a voxelized point cloud into a lower-dimensional feature representation, we use a sparse convolutional neural network.

Original: We extend the IMPALA policy architecture [16] for processing RGB images to process colored point cloud data using 3D convolution.
Revised: We extend the IMPALA policy architecture [16] for processing RGB images to process colored point cloud data using 3D convolution.

Original: Since  many voxels are unoccupied, the use of regular 3D convolution substantially increases computation time.
Revised: Since [Figure] many voxels are unoccupied, the use of regular 3D convolution substantially increases computation time.

Original: Hence, we use Minkowski Engine [21], a PyTorch library for sparse tensors, to design a 3D version of IMPALA-Net with sparse convolutions ( Sparse3D-IMPALA-Net ) 1 .
Revised: Hence, we use Minkowski Engine [21], a PyTorch library for sparse tensors, to design a 3D version of IMPALA-Net with sparse convolutions ( Sparse3D-IMPALA-Net ) 1 .

Original: The Sparse3D-IMPALA network takes as input the point cloud W t , and outputs an embedding vector which is concatenated with the embedding vector of ( q t , a t − 1 ) .
Revised: The Sparse3D-IMPALA network takes as input the point cloud W t , and outputs an embedding vector which is concatenated with the embedding vector of ( q t , a t − 1 ) .

Original: Afterward, a recurrent network is used and outputs the action a t .
Revised: Afterward, a recurrent network is used and outputs the action a t .

Original: The detailed architecture is illustrated in Figure C.5 in the appendix.
Revised: The detailed architecture is illustrated in Figure C.5 in the appendix.

Original: Mitigating the object symmetry issue : π E is trained with the the ground-truth state information s E t including the object orientation α ot and goal orientation α g .
Revised: Mitigating the object symmetry issue : π E is trained with the the ground-truth state information s E t including the object orientation α ot and goal orientation α g .

Original: The vision policy does not take any orientation information as input.
Revised: The vision policy does not take any orientation information as input.

Original: If an object is symmetric, the two different orientations of the object may correspond to the same point cloud observation.
Revised: If an object is symmetric, the two different orientations of the object may correspond to the same point cloud observation.

Original: This makes it problematic to use the difference in orientation angles ( ∆ θ ≤ ¯ θ ) as the stopping and success criterion.
Revised: This makes it problematic to use the difference in orientation angles ( ∆ θ ≤ ¯ θ ) as the stopping and success criterion.

Original: To mitigate this issue, we use
Revised: To mitigate this issue, we use

Original: Chamfer distance [22] to compute the distance between the object point cloud in α ot and the goal point cloud (i.e., the object rotated by α g ) as the evaluation criterion.
Revised: Chamfer distance [22] to compute the distance between the object point cloud in α ot and the goal point cloud (i.e., the object rotated by α g ) as the evaluation criterion.

Original: The Chamfer distance is computed as d C = (cid:80) a ∈ W ot min b ∈ W g
Revised: The Chamfer distance is computed as d [Equation]

Original: (cid:107) a − b (cid:107) 22 + (cid:80) b ∈ W g min 
Revised: 22 + [Equation] min [Equation]

Revised: ot

Original: a ∈ W ot (cid:107) a − b (cid:107) 22 , where W ot is the object point cloud in its current orientation.
Revised: [Equation] − [Equation] 22 , where W ot is the object point cloud in its current orientation.

Original: Both W ot and W g are scaled to ﬁt in a unit sphere for computing d C .
Revised: Both W ot and W g are scaled to ﬁt in a unit sphere for computing d C .

Original: We check Chamfer distance in each rollout step.
Revised: We check Chamfer distance in each rollout step.

Original: If d C ≤ ¯ d C ( ¯ d C is a threshold value for d C ), we consider the episode to be successful.
Revised: If d C ≤ ¯ d C ( ¯ d C is a threshold value for d C ), we consider the episode to be successful.

Original: Hence, the success criterion is (∆ θ ≤ ¯ θ ) ∨ ( d C ≤ ¯ d C ) .
Revised: Hence, the success criterion is (∆ θ ≤ ¯ θ ) ∨ ( d C ≤ ¯ d C ) .

Original: In training, if the success criterion is satisﬁed, the episode is terminated and used for updating π S .
Revised: In training, if the success criterion is satisﬁed, the episode is terminated and used for updating π S .

Original:  Experimental Setup
Revised:  Experimental Setup

Original: We use the simulated Shadow Hand [23] in NVIDIA Isaac Gym [24].
Revised: We use the simulated Shadow Hand [23] in NVIDIA Isaac Gym [24].

Original: Shadow Hand is an anthropomorphic robotic hand with 24 degrees of freedom (DoF).
Revised: Shadow Hand is an anthropomorphic robotic hand with 24 degrees of freedom (DoF).

Original: In our experiments, we assume the base of the hand to be ﬁxed.
Revised: We assume the base of the hand to be ﬁxed.

Original: Twenty joints are actuated by agonist–antagonist tendons and the remaining fourjoints are under-actuated.
Revised: Twenty joints are actuated by agonist–antagonist tendons and the remaining four are under-actuated.

Original: Object datasets : We use the EGAD dataset [25] and YCB dataset [26] that contain objects with diverse shapes (see Figure B.2) for in-hand manipulation experiments.
Revised: Object datasets : We use the EGAD dataset [25] and YCB dataset [26] that contain objects with diverse shapes (see Figure B.2) for in-hand manipulation experiments.

Original: EGAD contains 2282 geometrically diverse textureless object meshes, while the YCB dataset includes textured meshes for objects of daily life with different shapes and textures.
Revised: EGAD contains 2282 geometrically diverse textureless object meshes, while the YCB dataset includes textured meshes for objects of daily life with different shapes and textures.

Original: We use the 78 YCB object models collected with the
Revised: We use the 78 YCB object models collected with the

Original: Google scanner.
Revised: Google scanner.

Original: Since most YCB objects are too big for in-hand manipulation, we proportionally scale down the YCB meshes.
Revised: Since most YCB objects are too big for in-hand manipulation, we proportionally scale down the YCB meshes.

Original: To further increase the diversity of the datasets, we create 5 variants for each object mesh by randomly scaling the mesh.
Revised: To further increase the diversity of the datasets, we create 5 variants for each object mesh by randomly scaling the mesh.

Original: More details of the object datasets are in Section B.2.
Revised: More details of the object datasets are in Section B.2.

Original: Setup for visual observations : For the vision experiments, we trained policies for the scenario of hand facing upwards.
Revised: Setup for visual observations : For the vision experiments, we trained policies for the scenario of hand facing upwards.

Original: We place two RGBD cameras above the hand (Figure B.4).
Revised: We place two RGBD cameras above the hand (Figure B.4).

Original: The data from these cameras is combined to create a point cloud observation of the scene [20].
Revised: The data from these cameras is combined to create a point cloud observation of the scene [20].

Original: For downstream computation, the point cloud is voxelized with a resolution of 0 .
Revised: For downstream computation, the point cloud is voxelized with a resolution of 0 .

Original: 003 m .
Revised: 003 m .

Original: Setup with the upward facing hand : We ﬁrst consider the case where the Shadow Hand faces upward and is required to reorient objects placed in the hand (see Figure 2a).
Revised: Setup with the upward facing hand : We ﬁrst consider the case where the Shadow Hand faces upward and is required to reorient objects placed in the hand (see Figure 2a).

Original: We use the coordinate system where the z -axis points vertically upward and the xy -plane denotes the horizontal plane.
Revised: We use the coordinate system where the z -axis points vertically upward and the xy -plane denotes the horizontal plane.

Original: The object pose is initialized with the following procedure: xy position of the object’s center of mass (COM) p o 0 ,xy is randomly sampled from a square region of size 0 .
Revised: The object pose is initialized with the following procedure: xy position of the object’s center of mass (COM) p o 0 ,xy is randomly sampled from a square region of size 0 .

Original: 09 m × 0 .
Revised: 09 m × 0 .

Original: 09 m .
Revised: 09 m .

Original: The center of this square is approximately located on the intersection of the middle ﬁnger and the palm so that the sampling region covers both the ﬁngers and the palm.
Revised: The center of this square is approximately located on the intersection of the middle ﬁnger and the palm so that the sampling region covers both the ﬁngers and the palm.

Original: The z position of the object is ﬁxed to 0 .
Revised: The z position of the object is ﬁxed to 0 .

Original: 13 m above the base of the hand to ensure that the object does not collide with the hand at initialization.
Revised: 13 m above the base of the hand to ensure that the object does not collide with the hand at initialization.

Original: The initial and goal orientations are randomly sampled from the full SO (3) space.
Revised: The initial and goal orientations are randomly sampled from the full SO (3) space.

Original: Setup with the downward facing hand : Next, we consider the cases where the hand faces downward.
Revised: Setup with the downward facing hand : Next, we consider the cases where the hand faces downward.

Original: We experiment with two scenarios: with and without a table below the hand.
Revised: We experiment with two scenarios: with and without a table below the hand.

Original: In the ﬁrst case, we place a table with the tabletop being 0 .
Revised: In the ﬁrst case, we place a table with the tabletop being 0 .

Original: 12 m below the hand base.
Revised: 12 m below the hand base.

Original: We place objects in a random pose between the hand and the table so that the objects will fall onto the table.
Revised: We place objects in a random pose between the hand and the table so that the objects will fall onto the table.

Original: We will describe the setup for the second case (without a table) in Section 4.2.2.
Revised: We will describe the setup for the second case (without a table) in Section 4.2.2.

Original: Evaluation criterion : For non-vision experiments, a policy rollout is considered a success if θ ≤ ¯ θ .
Revised: Evaluation criterion : For non-vision experiments, a policy rollout is considered a success if θ ≤ ¯ θ .

Original: ¯ θ = 0 .
Revised: ¯ θ = 0 .

Original: 1 rad .
Revised: 1 rad .

Original: For vision experiments, we also check d C ≤ ¯ d C as another criterion and ¯ θ = 0 .
Revised: For vision experiments, we also check d C ≤ ¯ d C as another criterion and ¯ θ = 0 .

Original: 2 rad , ¯ d C = 0 .
Revised: 2 rad , ¯ d C = 0 .

Original: The initial and goal orientation are randomly sampled from SO (3) space in all the experiments.
Revised: The initial and goal orientation are randomly sampled from SO (3) space in all the experiments.

Original: We report performance as the percentage of successful episodes when the agent is tasked to reorient each training object 100 times from arbitrary start to goal orientation.
Revised: We report performance as the percentage of successful episodes when the agent is tasked to reorient each training object 100 times from arbitrary start to goal orientation.

Original: We report the mean and standard deviation of success rate from 3 seeds.
Revised: We report the mean and standard deviation of success rate from 3 seeds.

Original:  Results
Revised:  Results

Original: We evaluate the performance of reorientation policies with the hand facing upward and downward.
Revised: We evaluate the performance of reorientation policies with the hand facing upward and downward.

Original: Further we analyze the generalization of the learned policies to unseen object shapes.
Revised: Further we analyze the generalization of the learned policies to unseen object shapes.

Original: 4.1 Reorient objects with the hand facing upwardTrain a
Revised: 4.1 Reorient objects with the hand facing upward

Original:  teacher policy with full-state information We train our teacher MLP and RNN policies using the full state information using all objects in the EGAD and YCB datasets separately.
Revised: Train a teacher policy with full-state information We train our teacher MLP and RNN policies using the full state information using all objects in the EGAD and YCB datasets separately.

Original: The progression of success rate during training is shown inthe appendix Figure D.6.
Revised: The progression of success rate during training is shown in Figure D.6 in Appendix D.1 .

Original: Figure D.6 also shows that using privileged information substantially speeds up policy learning.
Revised: Figure D.6 also shows that using privileged information substantially speeds up policy learning.

Original: Results reported in
Revised: Results reported in

Original: Table 1 indicate that the RNN policies achieve a success rate greater than 90% on the EGAD dataset (entry B1) and greater than 80% on the YCB dataset (entry G1) without any explicit knowledge of the object shape 2 .
Revised: Table 1 indicate that the RNN policies achieve a success rate greater than 90% on the EGAD dataset (entry B1) and greater than 80% on the YCB dataset (entry G1) without any explicit knowledge of the object shape 2 .

Original: This result is surprising because apriori one might believe that shape information is important for in-hand reorientation of diverse objects.
Revised: This result is surprising because apriori one might believe that shape information is important for in-hand reorientation of diverse objects.

Original: The visualization of policy rollout reveals that the agent employs a clever strategy that is invariant to object geometry for re-orienting objects.
Revised: The visualization of policy rollout reveals that the agent employs a clever strategy that is invariant to object geometry for re-orienting objects.

Original: The agent throws the object in the air with a spin and catches it at the precise time when the object’s orientation matches the goal orientation.
Revised: The agent throws the object in the air with a spin and catches it at the precise time when the object’s orientation matches the goal orientation.

Original: Throwing the object with a spin is a dexterous skill that automatically emerges!
Revised: Throwing the object with a spin is a dexterous skill that automatically emerges!

Original: One possibility for the emergence of this skill is that we used very light objects.
Revised: One possibility for the emergence of this skill is that we used very light objects.

Original: This is not true because we trained with objects in the range of 50-150g which spans many hand-held objects used by humans (e.g., an egg weighs about 50g, a glass cup weighs around 100g, iPhone 12 weighs 162g, etc.).
Revised: This is not true because we trained with objects in the range of 50-150g which spans many hand-held objects used by humans (e.g., an egg weighs about 50g, a glass cup weighs around 100g, iPhone 12 weighs 162g, etc.).

Original: To further probe this concern, we evaluated zero-shot performance on objects weighing up to 500g 3 and found that the learned policy can successfully reorient them.
Revised: To further probe this concern, we evaluated zero-shot performance on objects weighing up to 500g 3 and found that the learned policy can successfully reorient them.

Original: We provide further analysis in the appendix showing that forces applied by the hand for such manipulation are realistic.
Revised: We provide further analysis in the appendix showing that forces applied by the hand for such manipulation are realistic.

Original: While there is still room for the possibility that the learned policy is exploiting the simulator to reorient objects by throwing them in the air, our analysis so far indicates otherwise.
Revised: While there is still room for the possibility that the learned policy is exploiting the simulator to reorient objects by throwing them in the air, our analysis so far indicates otherwise.

Original: Next, to understand the failure modes, we collected one hundred unsuccessful trajectories on YCB dataset and manually documented the reason for failure.
Revised: Next, to understand the failure modes, we collected one hundred unsuccessful trajectories on YCB dataset and manually analyzed them.

Original: The primarysource of failure is  manipulating long, small, or thin objects, which accounts for 60% of the failures.
Revised: The primary failure is in manipulating long, small, or thin objects, which accounts for 60% of all errors.

Original: In such cases,  the objects slip through the ﬁngers and fall, or are hard to be ﬂipped when the objects land on the palm.Results in
Revised: In such cases, either the object slips through the ﬁngers and falls, or is hard to be manipulated when the objects land on the palm.

Original: Figure D.10 conﬁrm that sphere-like objects such as tennis balls and orange are easiest to reorient.

Original: Long or thin objects such as knives and forks are the hardest ones to manipulate.

Original: Another cause[Table] of failures ( 19% ) is that objects are reoriented close to the goal orientation but not close enough to satisfy ∆ θ < ¯ θ .
Revised: Another cause of failures ( 19% ) is that objects are reoriented close to the goal orientation but not close enough to satisfy ∆ θ < ¯ θ .

Original: Finally, the performance on YCB is lower than EGAD because objects in the YCB dataset have greater diversity in their aspect ratios.
Revised: Finally, the performance on YCB is lower than EGAD because objects in the YCB dataset are more diverse in their aspect ratios.

Original: Scaling these objects by constraining l max ∈ [0 . 05 , 0 .
Revised: Scaling these objects by constraining l max ∈ [0 . 05 , 0 .

Original: 12] m (see Section 3) makes some of the objects either too small, too big, or too thin and consequently results in failure (see Figure D.7)
Revised: 12] m (see Section 3) makes some of these objects either too small, too big, or too thin and consequently results in failure (see Figure D.7).

Original: [Table]
Revised: A detailed object-wise quantitative analysis of performance is reported in appendix Figure D.10. Results conﬁrm that sphere-like objects such as tennis balls and orange are easiest to reorient, while long/thin objects such as knives and forks are the hardest to manipulate.

Original: Train a student policy with a reduced state space
Revised: Train a student policy with a reduced state space

Original: The student policy state is s S t ∈ R 31 .
Revised: The student policy state is s S t ∈ R 31 .

Original: In
Revised: In

Original: Table 1 (entries E1 and J1), we can see that π S R can get similarly high success rates as π E R .
Revised: Table 1 (entries E1 and J1), we can see that π S R can get similarly high success rates as π E R .

Original: The  last two columns in Table 1 also show that the policy is more robust to dynamics variations and observation/action noise after being trained with domain randomization.
Revised: The [Table] last two columns in Table 1 also show that the policy is more robust to dynamics variations and observation/action noise after being trained with domain randomization [Table]

Original: 4.2 Reorient objects with the hand facing downward
Revised: 4.2 Reorient objects with the hand facing downward

Original: The results above demonstrate that when the hand faces upwards, RL can be used to train policies for reorienting a diverse set of geometrically different objects.
Revised: The results above demonstrate that when the hand faces upwards, RL can be used to train policies for reorienting a diverse set of geometrically different objects.

Original: A natural question to ask is then does this still hold true when the hand is ﬂipped upside down?
Revised: A natural question to ask is, does this still hold true when the hand is ﬂipped upside down?

Original: Intuitively, this task is much more challenging because the objects willfall immediately  without proper control on the ﬁngers, which means the exploration becomes much harder for the RL agent.
Revised: Intuitively, this task is much more challenging because the objects will immediately fall down without speciﬁc ﬁnger movements that stabilize the object. Because with the hand facing upwards, the object primarily rests on the palm, such speciﬁc ﬁnger movements are not required. Therefore, the hand facing downwards scenario presents a much harder exploration challenge. To verify this hypothesis

Original: To see if this is the case, we ran an experimenting where we ﬂipped the hand upside down, placed the  objects  underneath the hand (see Figure 2b), and used the same reward function (Equation (1)) to train the reorientation policy.
Revised: [Table] we trained a policy with the downward-facing hand, objects placed underneath the hand (see Figure 2b) [Table] and using the same reward function (Equation (1)) as before.

Original: Unsurprisingly, theagent was unable to learn and the success rate was 0% .
Revised: Unsurprisingly, the success rate was 0% .

Original: The poor success can be attributed to policy needing to learn to both stabilize the object under the effect of gravity and simultaneously reorienting it.
Revised: The agent’s failure can be attributed to policy needing to learn to both stabilize the object under the effect of gravity and simultaneously reorient it.

Original: We found that deployment of the learned policy simply leads to object falling down showing that this is a hard-exploration  problem.
Revised: Deploying this policy simply results in an object falling down, conﬁrming the hard-exploration nature of this problem.

Original: 4.2.1 Reorient objects on a table
Revised: 4.2.1 Reorient objects on a table

Original: To tackle the  problem of reorienting objects with the hand facing downward, we start with a simpler version of the task where there is a table below the hand (see Figure 2c).
Revised: To tackle the hard problem of reorienting objects with the hand facing downward, we started with a simpliﬁed task setup that included a table under the hand (see Figure 2c).

Original: The presence of the table mitigates the exploration issue because it prevents the objects from falling.
Revised: Table eases exploration by preventing the objects from falling.

Original: We train π E M using the same reward function Equation (1) on objects sampled from the EGAD and YCB datasets.
Revised: We train π E M using the same reward function Equation (1) on objects sampled from the EGAD and YCB datasets.

Original: The success rate for the EGAD dataset with an MLP policy using full state information  is 95 .
Revised: The success rate using an MLP policy using full state information for EGAD and YCB is 95 .

Original: 31% ± 0 .
Revised: 31% ± 0 .

Original: The success rate for the YCB dataset is 81 .
Revised: 9% and 81 .

Original: 59% ± 0 .
Revised: 59% ± 0 .

Revised: 7% respectively.

Original: Using external support for in-hand manipulation has been a challenging problem in robotics.
Revised: Making use of external support for in-hand manipulation has been a challenging problem in robotics.

Original: Recent works approach this problem by building analytical models and constructing motion cones [27]. However, building the analytical models is especially difﬁcult for objects with complex geometry.
Revised: Prior work approach this problem by building analytical models and constructing motion cones [27], which is challenging for objects with complex geometry.

Original: Our experiments show that model-free RL is able to learn to use the support effectively to reorient many objects with the hand facing downward.
Revised: Our experiments show that model-free RL provides an effective alternative for learning manipulation strategies capable of using external support surfaces.

Original: 4.2.2 Reorient objects in air with hand facing downward
Revised: 4.2.2 Reorient objects in air with hand facing downward

Original: While providing a table can help RL learn a good reorientation policy, we want the skill to be more general. We would like the policy to reorient objects with the hand facing downwards and without any external support.
Revised: To enable the agent to operate in more general scenarios, we tackled the re-orientation problem with the hand facing downwards and without any external support.

Original: In this setup, one might hypothesize that object shape information (e.g., from vision) is critical because ﬁnding the strategy in Section 4.1 is not easy when the hand needs to overcome gravity and stabilize the object while reorienting it.
Revised: In this setup, one might hypothesize that object shape information (e.g., from vision) is critical because ﬁnding the strategy in Section 4. is not easy when the hand needs to overcome gravity and stabilize the object while reorienting it.

Original: We experimentally verify that even in this case, the policies achieve a reasonably high success rate without any knowledge of object shape.
Revised: We experimentally verify that even in this case, the policies achieve a reasonably high success rate without any knowledge of object shape.

Original: A good pose initialization is what you need : The difﬁculty of directly training the RL policies when the hand faces downward is mainly because of the hard-exploration issue in learning to catch the objects that are moving downward.
Revised: A good pose initialization is what you need : The difﬁculty of directly training the RL policies when the hand faces downward is mainly because of the hard-exploration issue in learning to catch the objects that are moving downward.

Original: However, catching is not necessary for the reorientation.
Revised: However, catching is not necessary for the reorientation.

Original: Even for

Original: [Table] [Table] [Table] human, we only reorient the object after we grasp it.
Revised: Even for human, we only reorient the object after we grasp it.

Original: More speciﬁcally, we ﬁrst train an object-lifting policy to lift objects from the table, collect the ending state (joint positions q T , object position p oT and orientation α oT ) in each successful lifting rollout episode, and reset the hand and objects to these states (velocities are all 0 ) for the pose initialization in training the reorientation policy.
Revised: More speciﬁcally, we ﬁrst train an object-lifting policy to lift objects from the table, collect the ending state (joint positions q T , object position p oT and orientation α o T ) in each successful lifting rollout episode, and reset the hand and objects to these states (velocities are all 0 ) for the pose initialization in training the reorientation policy.

Original: The objects have randomly initialized poses and are dropped onto the table.
Revised: The objects have randomly initialized poses and are dropped onto the table.

Original: We trained a separate RNN policy for each dataset using the reward function in Section C.2.
Revised: We trained a separate RNN policy for each dataset using the reward function in Section C.2.

Original: The success rate on the EGAD dataset is 97 .
Revised: The success rate on the EGAD dataset is 97 .

Original: 80% , while the success rate on the YCB dataset is 90 .
Revised: 80% , while the success rate on the YCB dataset is 90 .

Original: Note that objects need to be grasped ﬁrst to be lifted.
Revised: Note that objects need to be grasped ﬁrst to be lifted.

Revised: Our high success rates on object lifting also indicate that using an anthropomorphic hand

Original: Our high success rates on object lifting also indicate that using an anthropomorphic hand makes object grasping an easy task, while many prior works [28, 29] require much more involved training techniques to learn grasping skills with parallel-jaw grippers.
Revised: [Table] [Table] [Table] makes object grasping an easy task, while many prior works [28, 29] require much more involved training techniques to learn grasping skills with parallel-jaw grippers.

Original: After we train the lifting policy, we collect about 250 ending states for each object respectively from the successful lifting episodes.
Revised: After we train the lifting policy, we collect about 250 ending states for each object respectively from the successful lifting episodes.

Original: In every reset during the reorientation policy training, ending states are randomly sampled and used as the initial pose of the ﬁngers and objects.
Revised: In every reset during the reorientation policy training, ending states are randomly sampled and used as the initial pose of the ﬁngers and objects.

Original: With a good pose initialization, policies are able to learn to reorient objects with high success rates.
Revised: With a good pose initialization, policies are able to learn to reorient objects with high success rates.

Original: π E R trained on EGAD dataset gets a success rate more than 80% while π E R trained on YCB dataset gets a success rate greater than 50% .
Revised: π E R trained on EGAD dataset gets a success rate more than 80% while π E R trained on YCB dataset gets a success rate greater than 50% .

Original: More results on the different policies with and without domain randomization are available in Table D.6 in the appendix.
Revised: More results on the different policies with and without domain randomization are available in Table D.6 in the appendix.

Original: This setup is challenging because if at any time step in an episode the ﬁngers take a bad action, the object will fall.
Revised: This setup is challenging because if at any time step in an episode the ﬁngers take a bad action, the object will fall.

Original: Improving performance using gravity curriculum : Since the difﬁculty of training the reorientation policy with the hand facing downward is due to the gravity, we propose to build a gravity curriculum to learn the policy π E .
Revised: Improving performance using gravity curriculum : Since the difﬁculty of training the reorientation policy with the hand facing downward is due to the gravity, we propose to build a gravity curriculum to learn the policy π E .

Original: Since π E already performs very well on EGAD objects, we apply gravity curriculum to train π E on YCB objects.
Revised: Since π E already performs very well on EGAD objects, we apply gravity curriculum to train π E on YCB objects.

Original: Our gravity curriculum is constructed as follows: we start the training with g = 1 m / s 2 , then we gradually decrease g in a step-wise fashion if the evaluation success rate ( w ) is above a threshold value ( ¯ w ) until g = − 9 .
Revised: Our gravity curriculum is constructed as follows: we start the training with g = 1 m / s 2 , then we gradually decrease g in a step-wise fashion if the evaluation success rate ( w ) is above a threshold value ( ¯ w ) until g = − 9 .

Original: 81 m / s 2 .
Revised: 81 m / s 2 .

Original: More details about gravity curriculum are in Section C.4.
Revised: More details about gravity curriculum are in Section C.4.

Original: In Table D.

Revised: In Table D.

Original: (Exp Q and T) in the appendix, we can see that adding gravity curriculum ( g -curr) signiﬁcantly boost the success rates on the YCB dataset.
Revised: (Exp Q and T) in the appendix, we can see that adding gravity curriculum ( g -curr) signiﬁcantly boost the success rates on the YCB dataset.

Original: 4.3 Zero-shot policy transfer across datasets
Revised: 4.3 Zero-shot policy transfer across datasets

Original: We have shown the testing performance on the same training dataset so far.
Revised: We have shown the testing performance on the same training dataset so far.

Original: How would the policies work on a different dataset?
Revised: How would the policies work on a different dataset?

Original: To answer this, we test our policies across datasets: policies trained with
Revised: To answer this, we test our policies across datasets: policies trained with

Original: EGAD objects are now tested with YCB objects and vice versa.
Revised: EGAD objects are now tested with YCB objects and vice versa.

Original: We used the RNN policies trained with full-state information and reduced-state information respectively (without domain randomization) and tested them on the other dataset with the hand facing upward and downward.
Revised: We used the RNN policies trained with full-state information and reduced-state information respectively (without domain randomization) and tested them on the other dataset with the hand facing upward and downward.

Original: In the case of the hand facing downward, we tested the RNN policy trained with gravity curriculum.
Revised: In the case of the hand facing downward, we tested the RNN policy trained with gravity curriculum.

Original: Table 2 shows that policies still perform well on the untrained dataset.
Revised: Table 3 shows that policies still perform well on the untrained dataset.

Original: 4.4 Object Reorientation with RGBD sensors
Revised: 4.4 Object Reorientation with RGBD sensors

Original: In this section, we investigate whether we can train a vision policy to reorient objects with the hand facing upward.
Revised: In this section, we investigate whether we can train a vision policy to reorient objects with the hand facing upward.

Original: As vision-based experiments require much more compute resources, we train one vision policy for each object individually on six objects shown in Table 3.
Revised: As vision-based experiments require much more compute resources, we train one vision policy for each object individually on six objects shown in Table 4.

Original: We leave training a single vision policy for all objects to future work.
Revised: We leave training a single vision policy for all objects to future work.

Original: We use the expert MLP policy trained in Section 4.
Revised: We use the expert MLP policy trained in Section 4.

Original: to supervise the vision policy.
Revised: to supervise the vision policy.

Original: We also performed data augmentation on the point cloud input to the policy network at each time step in both training and testing.
Revised: We also performed data augmentation on the point cloud input to the policy network at each time step in both training and testing.

Original: The data augmentation includes the random translation of the point cloud, random noise on the point positions, random dropout on the points, and random variations on the point color.
Revised: The data augmentation includes the random translation of the point cloud, random noise on the point positions, random dropout on the points, and random variations on the point color.

Original: More details about the data augmentation are in Section D.5.
Revised: More details about the data augmentation are in Section D.5.

Original: We can see from Table 3 that reorienting the non-symmetric objects including the toy and the mug has high success rates (greater than 80% ).
Revised: We can see from Table 4 that reorienting the non-symmetric objects including the toy and the mug has high success rates (greater than 80% ).

Original: While training the policy for symmetric objects is much harder, Table 3 shows that using d C as an episode termination criterion enables the policies to achieve a success rate greater than 50% .
Revised: While training the policy for symmetric objects is much harder, Table 4 shows that using d C as an episode termination criterion enables the policies to achieve a success rate greater than 50% .

Original:  Related Work
Revised:  Related Work

Original: Dexterous manipulation has been studied for decades, dating back to [30, 31].
Revised: Dexterous manipulation has been studied for decades, dating back to [30, 31].

Original: In contrast to paralleljaw grasping, pushing, pivoting [32], or pick-and-place, dexterous manipulation typically involves continuously controlling force to the object through the ﬁngertips of a robotic hand [33].
Revised: In contrast to paralleljaw grasping, pushing, pivoting [32], or pick-and-place, dexterous manipulation typically involves continuously controlling force to the object through the ﬁngertips of a robotic hand [33].

Original: Some prior works used analytical kinematics and dynamics models of the hand and object, and used trajectory optimization to output control policies [1, 2, 34] or employed kinodynamic planning to ﬁnd a feasible motion plan [35].
Revised: Some prior works used analytical kinematics and dynamics models of the hand and object, and used trajectory optimization to output control policies [1, 2, 34] or employed kinodynamic planning to ﬁnd a feasible motion plan [35].

Original: However, due to the large number of active contacts on the hand and the objects, model simpliﬁcations such as simple ﬁnger and object geometries are usually necessary to make the optimization or planning tractable.
Revised: However, due to the large number of active contacts on the hand and the objects, model simpliﬁcations such as simple ﬁnger and object geometries are usually necessary to make the optimization or planning tractable.

Original: Sundaralingam and Hermans [34] moved objects in hand but assumes that there is no contact breaking or making between the ﬁngers and the object.
Revised: Sundaralingam and Hermans [34] moved objects in hand but assumes that there is no contact breaking or making between the ﬁngers and the object.

Original: Furukawa et al.
Revised: Furukawa et al.

Original: [36] achieved a high-speed dynamic regrasping motion on a cylinder using a high-speed robotic hand and a high-speed vision system.
Revised: [36] achieved a high-speed dynamic regrasping motion on a cylinder using a high-speed robotic hand and a high-speed vision system.

Original: Prior works have also explored the use of a vision system for manipulating an object to track a planned path [37], detecting manipulation modes [38], precision manipulation [39] with a limited number of objects with simple shapes using a two-ﬁngered gripper.
Revised: Prior works have also explored the use of a vision system for manipulating an object to track a planned path [37], detecting manipulation modes [38], precision manipulation [39] with a limited number of objects with simple shapes using a two-ﬁngered gripper.

Original: Recent works have explored the application of reinforcement learning to dexterous manipulation.
Revised: Recent works have explored the application of reinforcement learning to dexterous manipulation.

Original: Model-based RL works learned a linear [4, 7] or deep neural network [5] dynamics model from the rollout data, and used online optimal control to rotate a pen or Baoding balls on a Shadow hand.
Revised: Model-based RL works learned a linear [4, 7] or deep neural network [5] dynamics model from the rollout data, and used online optimal control to rotate a pen or Baoding balls on a Shadow hand.

Original: However, when the system is unstable, collecting informative trajectories for training a good dynamics model that generalizes to different objects remains challenging.
Revised: However, when the system is unstable, collecting informative trajectories for training a good dynamics model that generalizes to different objects remains challenging.

Original: Another line of works uses model-free RL algorithms to learn a dexterous manipulation policy.
Revised: Another line of works uses model-free RL algorithms to learn a dexterous manipulation policy.

Original: For example, Andrychowicz et al.
Revised: For example, OpenAI et al.

Original: [11] and OpenAI et al.
Revised: [11] and OpenAI et al.

Original: [12] learned a controller to reorient a block or a Rubik’s cube.
Revised: [12] learned a controller to reorient a block or a Rubik’s cube.

Original: Van Hoof et al.
Revised: Van Hoof et al.

Original: [40] learned the tactile informed policy via RL for a three-ﬁnger manipulator to move an object on the table.
Revised: [40] learned the tactile informed policy via RL for a three-ﬁnger manipulator to move an object on the table.

Original: To reduce the sample complexity of model-free learning, [9, 13, 8, 41, 6] combined imitation learning with RL to learn to reorient a pen, open a door, assemble LEGO blocks, etc.
Revised: To reduce the sample complexity of model-free learning, [9, 13, 8, 41, 6] combined imitation learning with RL to learn to reorient a pen, open a door, assemble LEGO blocks, etc.

Original: However, collecting expert demonstration data from humans is expensive, time-consuming, and even incredibly difﬁcult for contact-rick tasks [8].
Revised: However, collecting expert demonstration data from humans is expensive, time-consuming, and even incredibly difﬁcult for contact-rick tasks [8].

Original: Our method belongs to the category of model-free learning.
Revised: Our method belongs to the category of model-free learning.

Original: We use the teacher-student learning paradigm to speed up the deployment policy learning.
Revised: We use the teacher-student learning paradigm to speed up the deployment policy learning.

Original: Our learned policies also generalize to new shapes and show strong zero-shot transfer performance.
Revised: Our learned policies also generalize to new shapes and show strong zero-shot transfer performance.

Original: To the best of our knowledge, our system is the ﬁrst work that demonstrates the capabilities of reorienting a diverse set of objects that have complex geometries with both the hand facing upward and downward.
Revised: To the best of our knowledge, our system is the ﬁrst work that demonstrates the capabilities of reorienting a diverse set of objects that have complex geometries with both the hand facing upward and downward.

Revised: A recent work [42] (after our CORL submission) learns a shape-conditioned policy to reorient objects around z -axis with an upward-facing hand.

Revised: Our work tackles more general tasks (more diverse objects, any goal orientation in SO (3) , hand faces upward and downward) and shows that even without knowing any object shape information, the policies can get surprisingly high success rates in these tasks.

Original:  Discussion and Conclusion
Revised:  Discussion and Conclusion

Original: Our results show that model-free RL with simple deep learning architectures can be used to train policies to reorient a large set of geometrically diverse objects.
Revised: Our results show that model-free RL with simple deep learning architectures can be used to train policies to reorient a large set of geometrically diverse objects.

Original: Further, for learning with the hand facing downwards, we found that a good pose initialization obtained from a lifting policy was necessary, and the gravity curriculum substantially improved performance.
Revised: Further, for learning with the hand facing downwards, we found that a good pose initialization obtained from a lifting policy was necessary, and the gravity curriculum substantially improved performance.

Original: The agent also learns to use an external surface (i.e., the table).
Revised: The agent also learns to use an external surface (i.e., the table).

Original: The most surprising observation is that information about shape is not required despite the fact that we train a single policy to manipulate multiple objects.
Revised: The most surprising observation is that information about shape is not required despite the fact that we train a single policy to manipulate multiple objects.

Original: Perhaps in hindsight, it is not as surprising – after all, humans can close their eyes and easily manipulate novel objects into a speciﬁc orientation.
Revised: Perhaps in hindsight, it is not as surprising – after all, humans can close their eyes and easily manipulate novel objects into a speciﬁc orientation.

Original: But, even in this scenario, shape can be implicitly inferred using touch signals.

Original: However, our system has no access to touch data.

Original: As such, our work can serve a strong baseline for future in-hand object reorientation works that incorporate object shape in the observation space to improve performance.
Revised: Our work can serve a strong baseline for future in-hand object reorientation works that incorporate object shape in the observation space.

Original: While we only present results in simulation, we also provide evidence that our policies can be transferred to the real world.
Revised: While we only present results in simulation, we also provide evidence that our policies can be transferred to the real world.

Original: The experiments with domain randomization indicate that learned policies can work with noisy inputs.
Revised: The experiments with domain randomization indicate that learned policies can work with noisy inputs.

Original: Analysis of peak torques during manipulation (see Figure D. in the appendix) also reveals that generated torque commands are feasible to actuate on an actual robotic hand.
Revised: Analysis of peak torques during manipulation (see Figure D. in the appendix) also reveals that generated torque commands are feasible to actuate on an actual robotic hand.

Original: Finally, Figure D.10 and Figure D.11 in the appendix show that the success rate varies substantially with object shape.
Revised: Finally, Figure D.10 and Figure D.11 in the appendix show that the success rate varies substantially with object shape.

Original: This suggests that in the future, a training curriculum based on object shapes can improve performance.
Revised: This suggests that in the future, a training curriculum based on object shapes can improve performance.

Original: Another future work is to directly train one vision policy for a diverse set of objects.
Revised: Another future work is to directly train one vision policy for a diverse set of objects.

Original: A major bottleneck in vision-based experiments is the demand for much larger GPU memory.
Revised: A major bottleneck in vision-based experiments is the demand for much larger GPU memory.

Original: Learning visual representations of point cloud data that can ease the computational bottleneck is, therefore, an important avenue for future research.
Revised: Learning visual representations of point cloud data that can ease the computational bottleneck is, therefore, an important avenue for future research.

Original: Acknowledgments
Revised: Acknowledgments

Original: We thank the anonymous reviewers for their helpful comments in revising the paper.
Revised: We thank the anonymous reviewers for their helpful comments in revising the paper.

Original: We thank the members of Improbable AI lab for providing valuable feedback on research idea formulation and manuscript.
Revised: We thank the members of Improbable AI lab for providing valuable feedback on research idea formulation and manuscript.

Original: This research is funded by Toyota Research Institute, Amazon Research Award and
Revised: This research is funded by Toyota Research Institute, Amazon Research Award and

Original: DARPA Machine Common Sense Program.
Revised: DARPA Machine Common Sense Program.

Original: We also acknowledge the MIT SuperCloud and Lincoln
Revised: We also acknowledge the MIT SuperCloud and Lincoln

Original: Laboratory Supercomputing Center for providing HPC resources that have contributed to the research results reported within this paper.
Revised: Laboratory Supercomputing Center for providing HPC resources that have contributed to the research results reported within this paper.