Original: A Simple Method for Complex In-Hand Manipulation
Revised: A Simple Method for Complex In-Hand Manipulation

Original: Anonymous Author(s) AfﬁliationAddress email Abstract: In-hand object reorientation has been a challenging problem in roboticsdue to high dimensional actuation space and the frequent change in contact statebetween the ﬁngers and the objects.
Revised: Abstract: In-hand object reorientation has been a challenging problem in robotics due to high dimensional actuation space and the frequent change in contact state between the ﬁngers and the objects.

Original: We present a simple model-free framework thatcan learn to reorient objects with both the hand facing upwards and downwards.
Revised: We present a simple model-free framework that can learn to reorient objects with both the hand facing upwards and downwards.

Original: Wedemonstrate the capability of reorienting over 2000 geometrically different objectsin both cases.
Revised: We demonstrate the capability of reorienting over 2000 geometrically different objects in both cases.

Original: The learned policies show strong zero-shot transfer performance onnew objects.
Revised: The learned policies show strong zero-shot transfer performance on new objects.

Original: We provide evidence that these policies are amenable to real-worldoperation by distilling them to use observations easily available in the real world.
Revised: We provide evidence that these policies are amenable to realworld operation by distilling them to use observations easily available in the real world.

Original: The videos of the learned policies are available at: https://sites.google.com/view/in-hand-reorientation .
Revised: The videos of the learned policies are available at: https://taochenshh. github io/projects/in-hand-reorientation .

Original:  Introduction
Revised:  Introduction

Original: A common maneuver in many tasks of daily living is to pick an object, reorient it in hand and theneither place it or use it as a tool.
Revised: A common maneuver in many tasks of daily living is to pick an object, reorient it in hand and then either place it or use it as a tool.

Original: Consider three simpliﬁed variants of this maneuver  in Figure 1.
Revised: Consider three simpliﬁed variants of this maneuver shown in Figure 1.

Original: Theﬁrst taskshown in the top row requires an upward-facing multi-ﬁnger hand to manipulate an arbitraryobject in a random orientation to a goal conﬁguration.
Revised: The task in the top row requires an upward-facing multi-ﬁnger hand to manipulate an arbitrary object in a random orientation to a goal conﬁguration shown in the rightmost column.

Original: The next two rows showthe second andthird tasks where the hand is facing downward and is required to reorient the object either using thetable as a support or without the aid of any support surface respectively.
Revised: The next two rows show tasks where the hand is facing downward and is required to reorient the object either using the table as a support or without the aid of any support surface respectively.

Original: The last task is the hardestbecause the object is in an intrinsically unstable conﬁguration owing to the downward gravitationalforce and lack of support from the palm.
Revised: The last task is the hardest because the object is in an intrinsically unstable conﬁguration owing to the downward gravitational force and lack of support from the palm.

Original: Additional challenges in making such maneuvers with amulti-ﬁnger robotic hand stem from the control space being high-dimensional and reasoning aboutmultiple transitions in the contact state between the ﬁnger and the object.
Revised: Additional challenges in performing such manipulation with a multi-ﬁnger robotic hand stem from the control space being high-dimensional and reasoning about the multiple transitions in the contact state between the ﬁnger and the object.

Original: Due to its practical utilityand several unsolved issues, in-hand object reorientation remains an active area of research.
Revised: Due to its practical utility and several unsolved issues, in-hand object reorientation remains an active area of research.

Original: Past work has tackled the in-hand reorientation problem via several approaches: (i)
Revised: Past work has tackled the in-hand reorientation problem via several approaches: (i)

Original: The use ofanalytical models with powerful trajectory optimization methods [1, 2, 3].
Revised: The use of analytical models with powerful trajectory optimization methods [1, 2, 3].

Original: While these methodsdemonstrated remarkable performance, the results were largely in simulation with simple objectgeometries and required detailed knowledge of the object model and physical parameters.
Revised: While these methods demonstrated remarkable performance, the results were largely in simulation with simple object geometries and required detailed knowledge of the object model and physical parameters.

Original: As such, itremains unclear how to scale these methods to real-world and generalize to new objects.
Revised: As such, it remains unclear how to scale these methods to real-world and generalize to new objects.

Original: Another lineof work has employed (ii) model-based reinforcement learning [4, 5]; or (iii) model-free reinforcement learning with [6, 7, 8, 9] and without expert demonstrations [10, 11, 12, 13].
Revised: Another line of work has employed (ii) model-based reinforcement learning [4, 5]; or (iii) model-free reinforcement learning with [6, 7, 8, 9] and without expert demonstrations [10, 11, 12, 13].

Original: While some of theseworks demonstrated learned skills onthe real robots, they required substantial engineering andadditional apparatus  (e.g., motion capture system) to infer the object state, and the learned policiesdid not generalize to diverse objects.
Revised: While some of these works demonstrated learned skills on real robots, it required use of additional sensory apparatus not readily available in the real-world (e.g., motion capture system) to infer the object state, and the learned policies did not generalize to diverse objects.

Original: Furthermore, nearly all of these methods operate in a simpliﬁedsetting where the hand is only allowed to face upward (with pick-and-place being an exception, butpick-and-place doesn’t incorporate any in-hand manipulation).
Revised: Furthermore, most prior methods operate in the simpliﬁed setting of the hand facing upwards. The only exception is pick-and-place, but it does not involve any in-hand re-orientation.

Original: A detailed discussion of prior researchis provided in Section 5.
Revised: A detailed discussion of prior research is provided in Section 5.

Original: In this paper, our goal is to study the object reorientation problem with a multi-ﬁnger hand in itsgeneral form.
Revised: In this paper, our goal is to study the object reorientation problem with a multi-ﬁnger hand in its general form.

Original: We desire (a) manipulation with hand facing upward or downward; (b) the skill of usingexternal surfaces to aid manipulation; (c) the ability to reorient objects of novel shapes to arbitraryorientations; (d) operation from sensory data that can be easily obtained in the real world such as RGBD images and joint positions of the hand.
Revised: We desire (a) manipulation with hand facing upward or downward; (b) the ability of using external surfaces to aid manipulation; (c) the ability to reorient objects of novel shapes to arbitrary orientations; (d) operation from sensory data that can be easily obtained in the real world such as RGBD images and joint positions of the hand.

Original: While some of these aspects have been individuallydemonstrated in prior work, we are unaware of any published method that realizes all four.
Revised: While some of these aspects have been individually demonstrated in prior work, we are unaware of any published method that realizes all four.

Original: Our maincontribution is building a simple framework that can achieve these desiderata.
Revised: Our main contribution is building a simple framework that achieves the desiderata.

Original: We experimentally testthis framework with a simulated Shadow hand.
Revised: Our method is simple in the following aspects. First, we do not make use of explicit models of the object, the manipulator, or the contact dynamics. Our method builds upon a well-established model-free RL algorithm.

Revised: The three key components: teacher-student learning, gravity curriculum, and stable initialization of objects are general ideas that are likely to be useful even outside the context of in-hand re-orientation tasks.

Revised: Second, we do not require any special pre-processing of sensory observations.

Revised: For example, our vision-based policies can work directly from RGBD point clouds captured by cameras.

Revised: We experimentally test our framework using a simulated Shadow hand.

Original: Due to the scope of the problem and the ongoingpandemic, we limit our experiments to be in simulation.
Revised: Due to the scope of the problem and the ongoing pandemic, we limit our experiments to be in simulation.

Original: However, we provide evidence indicatingthat the learned policies can be transferred to the real world in the future.
Revised: However, we provide evidence indicating that the learned policies can be transferred to the real world in the future.

Original: Method
Revised: A Surprising Finding : While seemingly counterintuitive, we found that a single policy that had no access to shape information can manipulate a large number of previously unseen objects in all the three settings mentioned above. At the start of the project, we hypothesized that developing visual processing architecture for inferring shape while the robot manipulates the object would be the primary research challenge. On the contrary, our results show that it is possible to learn control strategies for general in-hand object re-orientation that are shape-agnostic. Our results, therefore, suggest that visual perception may be less important for in-hand manipulation than previously thought.

Revised: Of course, we still believe that the performance of our system can be improved by incorporating shape information.

Revised: However, our ﬁndings suggest a different framework of thinking: a lot can be achieved without vision, and that vision might be the icing on the cake instead of the cake itself.

Revised:  Method

Original: To our surprise, simple extensions to existing techniques in robot learning can be used to constructa system for general object reorientation.
Revised: Against our expectation, we found that simple extensions to existing techniques in robot learning can be used to construct a dexterous system for general object reorientation.

Original: First, to avoid explicit modeling of the highly non-linearand frequent changes in the contact state between the object and the hand, we use model-freereinforcement learning (RL).
Revised: First, to avoid explicit modeling of the highly non-linear and frequent changes in the contact state between the object and the hand, we use model-free reinforcement learning (RL).

Original: An added advantage is that model-free RL is amenable to directlyoperating from rawRGBD point cloud observations.
Revised: An added advantage is that model-free RL is amenable to directly operating from raw point cloud observations.

Original: Second, we make use of privileged stateinformation such as the positions/orientations/velocities of the object/ﬁngertips that is easily availablein the simulator but not in the real world.
Revised: Second, we make use of privileged state information such as the positions/orientations/velocities of the object/ﬁngertips that is easily available in the simulator but not in the real world.

Original: Such state information can simplify policy learning andincrease data efﬁciency.
Revised: Such state information can simplify policy learning and increase data efﬁciency.

Original: We train an RL policy with privileged state information to reorient morethan two thousand objects of diverse shapes (see Section 2.1).
Revised: We train an RL policy with privileged state information to reorient more than two thousand objects of diverse shapes (see Section 2.1).

Original: However such a policy cannot betransferred to the real robot due to the difﬁculty of getting the priviledged information in real world.
Revised: However, such a policy cannot be transferred to the real robot due to the challenges in obtaining such privileged information in real world.

Original: We address this issue via teacher-student learning.
Revised: We address this issue via teacher-student learning.

Original: The idea of ﬁrst learning an expert or teacher policy ( π E ) from privileged information and thenusing it to guide the learning of a student policy ( π S ) that only uses sensory inputs available in thereal world is known as teacher-student training [14, 15].
Revised: The idea of ﬁrst learning an expert or teacher policy ( π E ) from privileged information, and then using it to guide the learning of a student policy ( π S ) that only uses sensory inputs available in the real world, is known as teacher-student training [14, 15].

Original: We leverage this idea to demonstrate thepossibility of transferring the learned policies to the real world in the future.
Revised: We leverage this idea to demonstrate the possibility of transferring the learned policies to the real world in the future.

Original: Let the state spacecorresponding to π E and π S be S E and S S respectively.
Revised: Let the state space corresponding to π E and π S be S E and S S respectively.

Original: In general, S E (cid:54) =
Revised: In general, S E (cid:54) =

Original: S S .
Revised: S S .

Original: We ﬁrst detail themethod for distilling π E to a student policy using a reduced state space consisting of only the jointpositions of the hand, the object position, and the difference in orientation from the goal conﬁguration(see Section 2.2.1).
Revised: We ﬁrst detail the method for distilling π E to a student policy using a reduced state space consisting of only the joint positions of the hand, the object position, and the difference in orientation from the goal conﬁguration (see Section 2.2.1).

Original: However, in the real world, the object position and relative orientation mustbe inferred from sensory observation.
Revised: However, in the real world, the object position and relative orientation must be inferred from sensory observation.

Original: Not only does this process require substantial engineeringeffort (e.g., a motion capture or a pose estimation system), but also inferring the pose of a symmetricobject is prone to errors.
Revised: Not only does this process require substantial engineering effort (e.g., a motion capture or a pose estimation system), but also inferring the pose of a symmetric object is prone to errors.

Original: This is because a symmetric object at multiple orientations looks exactlythe same in sensory space such as RGBD images.
Revised: This is because a symmetric object at multiple orientations looks exactly the same in sensory space such as RGBD images.

Original: To mitigate these issues, we further distill π E tooperate directly from theRGBD point cloud and position of all the hand joints (see Section 2.2.2).
Revised: To mitigate these issues, we further distill π E to operate directly from the point cloud and position of all the hand joints (see Section 2.2.2).

Original: We propose a simple modiﬁcation that generalizes an existing 2D CNN  to make this possible.
Revised: We propose a simple modiﬁcation that generalizes an existing 2D CNN architecture [16] to make this possible.

Original: While the procedure described above works well for manipulation with the hand facing upwards anddownwards with a table  as a support, we ﬁnd that in the case of the hand facing downward without atable, a good initialization of object pose such as the one obtained by an  objectlifting policy is keyfor successful reorientation, and the proposed gravity curriculum substantially improves performance.
Revised: The procedure described above works well for manipulation with the hand facing upwards and downwards when a table is available as a support. However, when the hand faces downward without an underlying support surface, it is important to have the object initialized in a stable conﬁguration. In practical scenarios, an agent would ﬁrst lift the object and then re-orient it. The act of lifting results in a stable initialization for further re-orientation.

Revised: We, therefore, train a lifting policy that precedes in-hand manipulation.

Revised: Finally, training in a curriculum where gravity is slowly introduced (i.e., gravity curriculum ) substantially improves performance.

Original: 2.1 Learning the teacher policy
Revised: 2.1 Learning the teacher policy

Original: We use model-free RL to learn the teacher policy ( π E ) for reorienting an object ( { O i | i = 1 , ..., N } )from an initial orientation α o 0 to a target orientation α g ( α o 0 (cid:54) =
Revised: We use model-free RL to learn the teacher policy ( π E ) for reorienting an object ( { O i | i = 1 , ..., N } ) from an initial orientation α o 0 to a target orientation α g ( α o 0 (cid:54) =

Original: α g ).
Revised: α g ).

Original: At every time step t , the agentobserves the state s t , executes the action a t sampled from the policy π E , and receives a reward r t .
Revised: At every time step t , the agent observes the state s t , executes the action a t sampled from the policy π E , and receives a reward r t .

Original: π Eis optimized to maximize the expected discounted return: π = arg max π E E (cid:2) (cid:80) T − 1 t =

Revised: π E is optimized to maximize the expected discounted return: π = arg max π E E (cid:2) (cid:80) T − 1 t =

Original: γ t r t (cid:3) , where γis the discount factor.
Revised: γ t r t (cid:3) , where γ is the discount factor.

Original: The task is successful if the angle difference ∆ θ between the object’s current( α ot ) and the goal orientation ( α g ) is smaller than the threshold value ¯ θ , i.e. , ∆ θ ≤ ¯ θ .
Revised: The task is successful if the angle difference ∆ θ between the object’s current ( α o t ) and the goal orientation ( α g ) is smaller than the threshold value ¯ θ , i.e. , ∆ θ ≤ ¯ θ .

Original: To encourage the policy to be smooth, the previous action is appended to the inputs to the policy ( i.e. ,a t = π E ( s t , a t − 1 ) ) and large actions are penalized in the reward function.
Revised: To encourage the policy to be smooth, the previous action is appended to the inputs to the policy ( i.e. , a t = π E ( s t , a t − 1 ) ) and large actions are penalized in the reward function.

Original: We experiment with twoarchitectures for π E : (1) an MLP policy π M , (2) an RNN policy π R .
Revised: We experiment with two architectures for π E : (1) an MLP policy π M , (2) an RNN policy π R .

Original: We use PPO [16] to optimizeπ E .
Revised: We use PPO [17] to optimize π E .

Original: All the experiments in the paper were run on at most 2 GPUs with a 32 GB memory.

Original: More detailsabout thenetwork architecture and training are in Section C.1 and Section C.2 in the appendix.
Revised: More details about the training are in Section C.1 and Section C.2 in the appendix.

Original: Observation and action space : We deﬁne S E to include joint, ﬁngertip, object, and goal informationas detailed in Table B.1 in the appendix.
Revised: Observation and action space : We deﬁne S E to include joint, ﬁngertip, object, and goal information as detailed in Table B.1 in the appendix.

Original: Note that S E does not include object shape or informationabout friction, damping, contact states between the ﬁngers and the object, etc.
Revised: Note that S E does not include object shape or information about friction, damping, contact states between the ﬁngers and the object, etc.

Original: We control the jointmovements by commanding the relative change in the target joint angle ( q targett ) on each actuatedjoint (action a t ∈ R 20 ): q targett + = q targett + a t × ∆ t , where ∆ t is the control time step.
Revised: We control the joint movements by commanding the relative change in the target joint angle ( q targett ) on each actuated joint (action a t ∈ R 20 ): q targett + = q targett + a t × ∆ t , where ∆ t is the control time step.

Original: We clamp theaction command if necessary to make sure | ∆ q targett | ≤ 0 .
Revised: We clamp the action command if necessary to make sure | ∆ q targett | ≤ 0 .

Original: 33 rad where ∆ q targett = q targett + − q targett.
Revised: 33 rad .

Original: The control frequency is 60 Hz .
Revised: The control frequency is 60 Hz .

Original: Reward function :

Original: For training π E , the reward function is deﬁned as:

Original: [Equation]

Original: (1) where c θ 1 > 0 , c θ 2 > 0 and c 3 < 0 are the coefﬁcients, ∆ θ t is the difference between the currentobject orientation and the target orientation, (cid:15) θ is a constant, 1 is an indicator function that identiﬁeswhether the object is in the target orientation.

Original: The ﬁrst two reward terms encourage the policy toreorient the object to the desired orientation while the last term is a regularizer to suppress largeaction commands.

Original: In our experiments, we use c θ 1 = 1 , c θ 2 = 800 , c 3 = 0 .

Original: 1 , (cid:15) θ = 0 . 1 , ¯ θ = 0 . 1rad . Dynamics randomization : Even though we do not test our policies on a real robot, we train andevaluate policies with domain randomization [17] to provide evidence that our work has the potentialto be transferred to a real robotics system in the future.
Revised: Dynamics randomization : Even though we do not test our policies on a real robot, we train and evaluate policies with domain randomization [18] to provide evidence that our work has the potential to be transferred to a real robotic system in the future.

Original: We randomize the object mass, frictioncoefﬁcient, joint damping and add noise to the state observation s t and the action a t .
Revised: We randomize the object mass, friction coefﬁcient, joint damping and add noise to the state observation s t and the action a t .

Original: More detailsabout domain randomization are provided in Table C.4 in the appendix.
Revised: More details about domain randomization are provided in Table C.4 in the appendix.

Original: 2.2 Learning the student policy
Revised: 2.2 Learning the student policy

Original: We distill the teacher π E into the student policy π S using Dagger [18], a learning-from-demonstrationmethod that overcomes the covariate shift problem.
Revised: We distill the teacher π E into the student policy π S using Dagger [19], a learning-from-demonstration method that overcomes the covariate shift problem.

Original: We optimize π S by minimizing the KL-divergencebetween π S and π E : π S = arg min π S D KL (cid:0) π E ( s E t , a t − 1 )
Revised: We optimize π S by minimizing the KL-divergence between π S and π E : π S = arg min π S D KL (cid:0) π E ( s E t , a t − 1 )

Original: || π S ( s S t , a t − 1 )) (cid:1) .
Revised: || π S ( s S t , a t − 1 )) (cid:1) .

Original: Based on observationdata available in real-world settings, we investigate two different choices of S S .
Revised: Based on observation data available in real-world settings, we investigate two different choices of S S .

Original: 2.2.1 Training student policy from low-dimensional state
Revised: 2.2.1 Training student policy from low-dimensional state

Original: In the ﬁrst case, we consider a non-vision student policy π S ( s S t , a t − 1 ) .
Revised: In the ﬁrst case, we consider a non-vision student policy π S ( s S t , a t − 1 ) .

Original: s S t ∈ R 31 includes the jointpositions q t ∈ R 24 , object position p ot ∈ R 3 , quaternion difference between the object’s currentand target orientation β t ∈ R 4 .
Revised: s S t ∈ R 31 includes the joint positions q t ∈ R 24 , object position p ot ∈ R 3 , quaternion difference between the object’s current and target orientation β t ∈ R 4 .

Original: In this case, S S ⊂ S E , and we assume the availability of objectpose information, but do not require velocity information.
Revised: In this case, S S ⊂ S E , and we assume the availability of object pose information, but do not require velocity information.

Original: We use the same MLP and RNN networkarchitectures used for π E on π S except the input dimension changes as the state dimension is different.
Revised: We use the same MLP and RNN network architectures used for π E on π S except the input dimension changes as the state dimension is different.

Original: 2.2.2 Training student policy from vision
Revised: 2.2.2 Training student policy from vision

Original: In the second case, S S only uses direct observations from RGBD cameras and the joint position ( q t )of the robotic hand.
Revised: In the second case, S S only uses direct observations from RGBD cameras and the joint position ( q t ) of the robotic hand.

Original: We convert the RGB and Depth data into a colored point cloud using a pinholecamera model [19].
Revised: We convert the RGB and Depth data into a colored point cloud using a pinhole camera model [20].

Original: Our vision policy takes as input the voxelized point cloud of the scene W t , q t ,and previous action command a t − 1 , and outputs the action a t , i.e. , a t = π S
Revised: Our vision policy takes as input the voxelized point cloud of the scene W t , q t , and previous action command a t − 1 , and outputs the action a t , i.e. , a t = π S

Original: ( W t , q t , a t − 1 ) .
Revised: ( W t , q t , a t − 1 ) .

Original:  Goal speciﬁcation : To avoidthe dependency on any manually deﬁned coordinate frame for specifyingthe goal quaternion, weonly provide the object’s point cloud rotated to the desired orientation W gto the policy, i.e. , we only show the policy how the object should look like in the end (see the topleft of Figure C.5).
Revised: [Figure] Goal speciﬁcation : To avoid manually deﬁning per-object coordinate frame for specifying the goal quaternion, we provide the goal to the policy as an object point cloud rotated to the desired orientation W g , i.e. , we only show the policy how the object should look like in the end (see the top left of Figure C.5).

Original: The input to π S is the point cloud W t =
Revised: The input to π S is the point cloud W t =

Original: W s t ∪ W g . We obtain W g by sampling5000 points from the object’s CAD mesh using the Barycentric coordinate, rotating the points by thedesired orientation, and translating them so that these points are next to the hand.
Revised: W st ∪ W g where W st is the actual point cloud of the current scene obtained from the cameras.

Original: Note that one canalso put the object in the desired orientation right next to the hand in the simulator and render theentire scene altogether to remove the need for CAD models. We use CAD models for W g just tosave the computational cost of rendering another object while we still use RGBD cameras to get W st .
Revised: Details of obtaining W g are in Section C.2.

Original: Sparse3D-IMPALA-Net : To convert a voxelized point cloud into a lower-dimensional featurerepresentation, we use a sparse convolutional neural network.
Revised: Sparse3D-IMPALA-Net : To convert a voxelized point cloud into a lower-dimensional feature representation, we use a sparse convolutional neural network.

Original: We extend the IMPALA policyarchitecture [20] for processing RGB images to process colored point cloud data using 3D convolution.
Revised: We extend the IMPALA policy architecture [16] for processing RGB images to process colored point cloud data using 3D convolution.

Original: Since many voxels are unoccupied, the use of regular 3D convolution wastes substantial computationtime.
Revised: Since many voxels are unoccupied, the use of regular 3D convolution substantially increases computation time.

Original: Hence, we use Minkowski Engine [21], a PyTorch library for sparse tensors, to design a 3Dversion of IMPALA-Net with sparse convolutions ( Sparse3D-IMPALA-Net ) 1 .
Revised: Hence, we use Minkowski Engine [21], a PyTorch library for sparse tensors, to design a 3D version of IMPALA-Net with sparse convolutions ( Sparse3D-IMPALA-Net ) 1 .

Original: The point cloud W tis processed by a series of CNN residual modules and projected into an embedding vector.

Original: q t anda t − 1 are concatenated together and projected into an embedding vector via an MLP. Both embeddingvectors from W t and ( q t , a t − 1 ) are concatenated and passed through a recurrent network to outputthe action a t .
Revised: The Sparse3D-IMPALA network takes as input the point cloud W t , and outputs an embedding vector which is concatenated with the embedding vector of ( q t , a t − 1 ) . Afterward, a recurrent network is used and outputs the action a t .

Original: The detailed architecture is illustrated in Figure C.5 in the appendix.
Revised: The detailed architecture is illustrated in Figure C.5 in the appendix.

Original: Mitigating the object symmetry issue : π E is trained with the the ground-truth state informations E t including the object orientation α ot and goal orientation α g .
Revised: Mitigating the object symmetry issue : π E is trained with the the ground-truth state information s E t including the object orientation α ot and goal orientation α g .

Original: The vision policy does not take anyorientation information as input.
Revised: The vision policy does not take any orientation information as input.

Original: If an object is symmetric, the two different orientations of the objectmay correspond to the same point cloud observation.
Revised: If an object is symmetric, the two different orientations of the object may correspond to the same point cloud observation.

Original: This makes it problematic to use the differencein orientation angles ( ∆ θ ≤ ¯ θ ) as the stopping and success criterion.
Revised: This makes it problematic to use the difference in orientation angles ( ∆ θ ≤ ¯ θ ) as the stopping and success criterion.

Original: To mitigate this issue, we use
Revised: To mitigate this issue, we use

Original: Chamfer distance [22] to compute the distance between the object point cloud in α ot and the goal pointcloud (i.e., the object rotated by α g ) as the evaluation criterion.
Revised: Chamfer distance [22] to compute the distance between the object point cloud in α ot and the goal point cloud (i.e., the object rotated by α g ) as the evaluation criterion.

Original: The Chamfer distance is computedas d C = (cid:80) [Equation] ot min 
Revised: The Chamfer distance is computed as d C = (cid:80) a ∈ W ot min b ∈ W g

Original: [Equation] g (cid:107) a − b (cid:107) 22 + (cid:80) b ∈ W g min a ∈ W ot (cid:107) a − b (cid:107) 22 , where W ot is the object pointcloud in its current orientation.
Revised: (cid:107) a − b (cid:107) 22 + (cid:80) b ∈ W g min a ∈ W ot (cid:107) a − b (cid:107) 22 , where W ot is the object point cloud in its current orientation.

Original: Both W o t and W g are scaled to ﬁt in a unit sphere for computing d C .
Revised: Both W ot and W g are scaled to ﬁt in a unit sphere for computing d C .

Original: We check Chamfer distance in each rollout step.
Revised: We check Chamfer distance in each rollout step.

Original: If d C ≤ ¯ d C ( ¯ d C is a threshold value for d C ), weconsider the episode to be successful.
Revised: If d C ≤ ¯ d C ( ¯ d C is a threshold value for d C ), we consider the episode to be successful.

Original: Hence, the success criterion is (∆ θ ≤ ¯ θ ) ∨ ( d C ≤ ¯ d C ) .
Revised: Hence, the success criterion is (∆ θ ≤ ¯ θ ) ∨ ( d C ≤ ¯ d C ) .

Original: Intraining, if the success criterion is satisﬁed, the episode is terminated and used for updating π S .
Revised: In training, if the success criterion is satisﬁed, the episode is terminated and used for updating π S .

Original:  Experimental Setup
Revised:  Experimental Setup

Original: We use the simulated Shadow Hand [23] in NVIDIA Isaac Gym [24].
Revised: We use the simulated Shadow Hand [23] in NVIDIA Isaac Gym [24].

Original: Shadow Hand is an anthropomorphic robotic hand with 24 degrees of freedom (DoF).
Revised: Shadow Hand is an anthropomorphic robotic hand with 24 degrees of freedom (DoF).

Original: In our experiments, we assume the base ofthe hand to be ﬁxed.
Revised: In our experiments, we assume the base of the hand to be ﬁxed.

Original: Twenty joints are actuated by agonist–antagonist tendons and the remainingfour joints are under-actuated.
Revised: Twenty joints are actuated by agonist–antagonist tendons and the remaining four joints are under-actuated.

Original: Object datasets : We use the EGAD dataset [25] and YCB dataset [26] that contain objects withdiverse shapes (see Figure B.2) for in-hand manipulation experiments.
Revised: Object datasets : We use the EGAD dataset [25] and YCB dataset [26] that contain objects with diverse shapes (see Figure B.2) for in-hand manipulation experiments.

Original: EGAD contains 2282 geometrically diverse textureless object meshes, while the YCB dataset includes textured meshes for objectsof daily life with different shapes and textures.
Revised: EGAD contains 2282 geometrically diverse textureless object meshes, while the YCB dataset includes textured meshes for objects of daily life with different shapes and textures.

Original: We use the 78 YCB object models collected with the
Revised: We use the 78 YCB object models collected with the

Original: Google scanner.
Revised: Google scanner.

Original: Since most YCB objects are too big for in-hand manipulation, we proportionallyscale down the YCB meshes.
Revised: Since most YCB objects are too big for in-hand manipulation, we proportionally scale down the YCB meshes.

Original: To further increase the diversity of the datasets, we create 5 variantsfor each object mesh by randomly scaling the mesh.
Revised: To further increase the diversity of the datasets, we create 5 variants for each object mesh by randomly scaling the mesh.

Original: The scaling ratios are randomly sampled suchthat the longest side of the objects’ bounding boxes l max lies in [0 . 05 , 0 . 08] m for EGAD objects,and l max ∈ [0 . 05 , 0 .

Original: 12] m for YCB objects.

Original: Mass is randomly sampled from [0 . 05 , 0 . 15] kg for everyobject.

Original: When we randomly scale YCB objects, some objects become too small or too thin, making thereorientation task even more challenging.

Original: In total, we use 11410 EGAD object meshes and 390 YCBobject meshes for training.

Original: More details on the dataset areavailable in Section B.2 in the appendix.
Revised: More details of the object datasets are in Section B.2.

Original: Setup for visual observations : For the vision experiments, we experiment with the scenario wherethe hand faces upward.
Revised: Setup for visual observations : For the vision experiments, we trained policies for the scenario of hand facing upwards.

Original: We place two RGBD cameras above the hand (Figure B.4).
Revised: We place two RGBD cameras above the hand (Figure B.4).

Original: The data fromthese cameras is combined to create a point cloud observation of the scene [19].
Revised: The data from these cameras is combined to create a point cloud observation of the scene [20].

Original: For downstreamcomputation, the point cloud is voxelized with a resolution of 0 .
Revised: For downstream computation, the point cloud is voxelized with a resolution of 0 .

Original: 003 m .
Revised: 003 m .

Original: Setup with the upward facing hand : We ﬁrst consider the case where the Shadow Hand facesupward and is required to reorient objects placed in the hand (see Figure 2a).
Revised: Setup with the upward facing hand : We ﬁrst consider the case where the Shadow Hand faces upward and is required to reorient objects placed in the hand (see Figure 2a).

Original: We use the coordinatesystem where the z -axis points vertically upward and the xy -plane denotes the horizontal plane.
Revised: We use the coordinate system where the z -axis points vertically upward and the xy -plane denotes the horizontal plane.

Original: Theobject is initialized as following: xy position of the object’s center of mass (COM) p o 0 ,xy is randomlysampled from a square region of size 0 .
Revised: The object pose is initialized with the following procedure: xy position of the object’s center of mass (COM) p o 0 ,xy is randomly sampled from a square region of size 0 .

Original: 09 m × 0 .
Revised: 09 m × 0 .

Original: 09 m .
Revised: 09 m .

Original: The center of this square is approximatelylocated on the intersection of the middle ﬁnger and the palm so that the sampling region covers boththe ﬁngers and the palm.
Revised: The center of this square is approximately located on the intersection of the middle ﬁnger and the palm so that the sampling region covers both the ﬁngers and the palm.

Original: The z position of the object is ﬁxed to 0 .
Revised: The z position of the object is ﬁxed to 0 .

Original: 13 m above the base of the hand toensure that the object does not collide with the hand at initialization.
Revised: 13 m above the base of the hand to ensure that the object does not collide with the hand at initialization.

Original: The initial and goal orientationsare randomly sampled from the full SO (3) space.
Revised: The initial and goal orientations are randomly sampled from the full SO (3) space.

Original: [Figure]

Original: Setup with the downward facing hand : Next, we consider the cases where the hand faces downward.
Revised: Setup with the downward facing hand : Next, we consider the cases where the hand faces downward.

Original: We experiment with two scenarios: with and without a table below the hand.
Revised: We experiment with two scenarios: with and without a table below the hand.

Original: In the ﬁrst case,we place a table with the tabletop being 0 .
Revised: In the ﬁrst case, we place a table with the tabletop being 0 .

Original: 12 m below the hand base.
Revised: 12 m below the hand base.

Original: We place objects in a randompose between the hand and the table so that the objects will fall onto the table.
Revised: We place objects in a random pose between the hand and the table so that the objects will fall onto the table.

Original: We will describe thesetup for the second case (without a table) in Section 4.2.2.
Revised: We will describe the setup for the second case (without a table) in Section 4.2.2.

Original: Evaluation criterion : For non-vision experiments, a policy rollout is considered a success if θ ≤ ¯ θ .¯
Revised: Evaluation criterion : For non-vision experiments, a policy rollout is considered a success if θ ≤ ¯ θ .

Original:  θ = 0 .
Revised: ¯ θ = 0 .

Original: 1 rad .
Revised: 1 rad .

Original: For vision experiments, we also check d C ≤ ¯ d C as another criterion and ¯ θ =0 .
Revised: For vision experiments, we also check d C ≤ ¯ d C as another criterion and ¯ θ = 0 .

Original: 2 rad , ¯ d C = 0 .
Revised: 2 rad , ¯ d C = 0 .

Original: The initial and goal orientation are randomly sampled from SO (3) space in allthe experiments.
Revised: The initial and goal orientation are randomly sampled from SO (3) space in all the experiments.

Original: We report performance as the percentage of successful episodes when the agent istasked to reorient all the training objects 100 times from arbitrary start to goal orientation.
Revised: We report performance as the percentage of successful episodes when the agent is tasked to reorient each training object 100 times from arbitrary start to goal orientation.

Original: We reportthe mean and standard deviation of success rate from 3 seeds.
Revised: We report the mean and standard deviation of success rate from 3 seeds.

Original:  Results
Revised:  Results

Revised: We evaluate the performance of reorientation policies with the hand facing upward and downward.

Revised: Further we analyze the generalization of the learned policies to unseen object shapes.

Original: 4.1 Reorient objects with the hand facing upward 
Revised: 4.1 Reorient objects with the hand facing upward Train a

Original: Train a teacher policy with full-state information We train our teacher MLP and RNN policiesusing the full state information on the EGAD and YCB datasets separately.
Revised: teacher policy with full-state information We train our teacher MLP and RNN policies using the full state information using all objects in the EGAD and YCB datasets separately.

Original: The  success rateas afunction of environment steps during training is shown inFigure D.6 in the appendix.
Revised: The progression of success rate during training is shown in the appendix Figure D.6.

Original: Figure D.6also shows that using the full-state information can speed up policy learning signiﬁcantly.
Revised: Figure D.6 also shows that using privileged information substantially speeds up policy learning.

Original: The testingresults in Table 1 show thatboth theMLP and RNN policiesare able to achieve a success rate greaterthan 90% on the EGAD dataset (entries A1, B1) and greater than 70% on the YCB dataset (entries F1, G1) without any explicit knowledge of the object shape.
Revised: Results reported in Table 1 indicate that the RNN policies achieve a success rate greater than 90% on the EGAD dataset (entry B1) and greater than 80% on the YCB dataset (entry G1) without any explicit knowledge of the object shape 2 .

Original: This result is surprising because intuitively,one would assume thatinformation about the object shape  is important for in-hand reorientation.
Revised: This result is surprising because apriori one might believe that shape information is important for in-hand reorientation of diverse objects.

Original: The visualization ofthe policy rollout reveals that the agent is able to reorient different objects withoutknowing their shapes by ﬁnding a clever strategy that is invariant to object geometry.
Revised: The visualization of policy rollout reveals that the agent employs a clever strategy that is invariant to object geometry for re-orienting objects.

Original: The agentthrows the object in the air with a spin and catches it at the precise time when the object’s orientationmatches the goal orientation.
Revised: The agent throws the object in the air with a spin and catches it at the precise time when the object’s orientation matches the goal orientation.

Original: Throwing the object with a spin is a dexterous spin that automaticallyemerges!
Revised: Throwing the object with a spin is a dexterous skill that automatically emerges!

Original: One possibility for the emergence of this skill is that we used very light objects.
Revised: One possibility for the emergence of this skill is that we used very light objects.

Original: Thisis not true because we trained with objects in the range of 50-150g which spans many hand-heldobjects used by humans (e.g., an egg weighs about 50g, a glass cup weighs around 100g, iPhone 12weighs 162g, etc.).
Revised: This is not true because we trained with objects in the range of 50-150g which spans many hand-held objects used by humans (e.g., an egg weighs about 50g, a glass cup weighs around 100g, iPhone 12 weighs 162g, etc.).

Original: To further probe this concern, we evaluated zero-shot performance on objectsweighing up to 500g 2 and found that the learned policy can successfully reorient them.
Revised: To further probe this concern, we evaluated zero-shot performance on objects weighing up to 500g 3 and found that the learned policy can successfully reorient them.

Original: We providefurther analysis in the appendix showing that forces applied by the hand for such manipulation arerealistic.
Revised: We provide further analysis in the appendix showing that forces applied by the hand for such manipulation are realistic.

Original: While there is still room for the possibility that the learned policy is exploiting the simulatorto reorient objects by throwing them in the air, our analysis so far indicates otherwise.
Revised: While there is still room for the possibility that the learned policy is exploiting the simulator to reorient objects by throwing them in the air, our analysis so far indicates otherwise.

Original: Next, to understand the failure modes, we collected one hundred failed trajectories on YCB datasetand manually documented the reason for failure.
Revised: Next, to understand the failure modes, we collected one hundred unsuccessful trajectories on YCB dataset and manually documented the reason for failure.

Original: It turns out that the primary  failure is with long,small, or thin objects, which accounts for 60% of the failures.
Revised: The primary source of failure is manipulating long, small, or thin objects, which accounts for 60% of the failures.

Original: In such cases, the objects slip throughthe ﬁngers and fall down, or get stuck on the palm.
Revised: In such cases, the objects slip through the ﬁngers and fall, or are hard to be ﬂipped when the objects land on the palm.

Original: Results in Figure D.9 conﬁrm that sphere-likeobjects such as tennis balls and orange are easiest to reorient.
Revised: Results in Figure D.10 conﬁrm that sphere-like objects such as tennis balls and orange are easiest to reorient.

Original: Long or thin objects such as knives andforks are the hardest ones to manipulate.
Revised: Long or thin objects such as knives and forks are the hardest ones to manipulate.

Original: Another cause  of failures ( 19% ) is that objects are reorientedclose to the goal orientation but not close enough to satisfy ∆ θ < ¯ θ .
Revised: Another cause [Table] of failures ( 19% ) is that objects are reoriented close to the goal orientation but not close enough to satisfy ∆ θ < ¯ θ .

Original: Finally, we note that theperformance on YCB is lower than EGAD because objects in the YCB dataset have greater diversityin the aspect ratios.
Revised: Finally, the performance on YCB is lower than EGAD because objects in the YCB dataset have greater diversity in their aspect ratios.

Original: Scaling these objects by constraining l max ∈ [0 . 05 , 0 .
Revised: Scaling these objects by constraining l max ∈ [0 . 05 , 0 .

Original: 12] m (see Section 3) makessome of the objects either too small, too big, or too thin and consequently results in failure (see Figure D.11).
Revised: 12] m (see Section 3) makes some of the objects either too small, too big, or too thin and consequently results in failure (see Figure D.7)

Original: [Table]
Revised: [Table]

Original: Train a student policy with a reduced state spaceWe experimented with the following threecombinations: (1) distill π E M into π S M , (2) distill π E M into π S R , (3) distill π E R into π S R .
Revised: Train a student policy with a reduced state space

Original: The studentpolicy state is s S t ∈ R 31 .
Revised: The student policy state is s S t ∈ R 31 .

Original: In Table 1 (entries C1-E1, H1-J1), we can see that πE R → π S R gives thehighest success rate on π S , while π E M → π S M leads to much worse performance ( 36% drop of successrate in EGAD, and 47% drop in YCB).
Revised: In Table 1 (entries E1 and J1), we can see that π S R can get similarly high success rates as π E R .

Original: This shows that π S requires temporal information due toreduced state space.

Original: The last two columns in Table 1 also show that the policy is more robust todynamics variations and observation/action noise after being trained with domain randomization.
Revised: The last two columns in Table 1 also show that the policy is more robust to dynamics variations and observation/action noise after being trained with domain randomization.

Original: 4.2 Reorient objects with the hand facing downward
Revised: 4.2 Reorient objects with the hand facing downward

Original: The results above demonstrate that when the hand faces upwards, RL can be used to train policies forreorienting a diverse set of geometrically different objects.
Revised: The results above demonstrate that when the hand faces upwards, RL can be used to train policies for reorienting a diverse set of geometrically different objects.

Original: A natural question to ask is then does thisstill hold true when the hand is ﬂipped upside down?
Revised: A natural question to ask is then does this still hold true when the hand is ﬂipped upside down?

Original: Intuitively, this task is much more challengingbecause the objects will fall immediately without proper control on the ﬁngers, which means theexploration becomes much harder for the RL agent.
Revised: Intuitively, this task is much more challenging because the objects will fall immediately without proper control on the ﬁngers, which means the exploration becomes much harder for the RL agent.

Original: To see if this is the case, we experimented withthe following settings: We ﬁrst ﬂip the hand upside down, placed the objects underneath the hand(see Figure 2b), and use the same reward function (Equation (1)) to train the reorientation policy.
Revised: To see if this is the case, we ran an experimenting where we ﬂipped the hand upside down, placed the objects underneath the hand (see Figure 2b), and used the same reward function (Equation (1)) to train the reorientation policy.

Original: Unsurprisingly, the agent is unable to learn and results in 0% success.
Revised: Unsurprisingly, the agent was unable to learn and the success rate was 0% .

Original: This is because the policyneeds to learn to both stabilize the objectto prevent it from falling under the effect of gravity andsimultaneously reorient it.
Revised: The poor success can be attributed to policy needing to learn to both stabilize the object under the effect of gravity and simultaneously reorienting it.

Original: We found that the application of the method developed until now leads toobject falling down, showing that this is a hard-exploration problem.
Revised: We found that deployment of the learned policy simply leads to object falling down showing that this is a hard-exploration problem.

Original: 4.2.1 Reorient objects on a table
Revised: 4.2.1 Reorient objects on a table

Original: To tackle the problem of reorienting objects with the hand facing downward, we start with a simplerversion of the task where there is a table below the hand (see Figure 2c).
Revised: To tackle the problem of reorienting objects with the hand facing downward, we start with a simpler version of the task where there is a table below the hand (see Figure 2c).

Original: The presence of the tablemitigates the exploration issue because it prevents the objects from falling.
Revised: The presence of the table mitigates the exploration issue because it prevents the objects from falling.

Original: We train π E M using thesame reward function Equation (1) on objects sampled from the EGAD and YCB datasets.
Revised: We train π E M using the same reward function Equation (1) on objects sampled from the EGAD and YCB datasets.

Original: The successrate for the EGAD dataset with an MLP policy using full state information is 95 .
Revised: The success rate for the EGAD dataset with an MLP policy using full state information is 95 .

Original: 31% ± 0 .
Revised: 31% ± 0 .

Original: Thesuccess rate for the YCB dataset is 81 .
Revised: The success rate for the YCB dataset is 81 .

Original: 59% ± 0 .
Revised: 59% ± 0 .

Original: Using external support for in-hand manipulationhas been a challenging problem in robotics.
Revised: Using external support for in-hand manipulation has been a challenging problem in robotics.

Original: Recent works approach this problem by buildinganalytical models and constructing motion cones [27].
Revised: Recent works approach this problem by building analytical models and constructing motion cones [27].

Original: However, building the analytical models isespecially difﬁcult for objects with complex geometry.
Revised: However, building the analytical models is especially difﬁcult for objects with complex geometry.

Original: Our experiments show that model-free RL isable to learn to use the support effectively to reorient many objects with the hand facing downward.
Revised: Our experiments show that model-free RL is able to learn to use the support effectively to reorient many objects with the hand facing downward.

Original: 4.2.2 Reorient objects in air with hand facing downward
Revised: 4.2.2 Reorient objects in air with hand facing downward

Original: While providing a table can help RL learn a good reorientation policy, we want the skill to be moregeneral.
Revised: While providing a table can help RL learn a good reorientation policy, we want the skill to be more general.

Original: We would like the policy to reorient objects with the hand facing downwards and withoutany external support.
Revised: We would like the policy to reorient objects with the hand facing downwards and without any external support.

Original: In this setup, one might hypothesize that object shape information (e.g., fromvision) is critical because ﬁnding the strategy in Section 4.1 is not easy when the hand needs toovercome gravity and stabilize the object while reorienting it.
Revised: In this setup, one might hypothesize that object shape information (e.g., from vision) is critical because ﬁnding the strategy in Section 4.1 is not easy when the hand needs to overcome gravity and stabilize the object while reorienting it.

Original: We experimentally verify that even inthis case, the policies achieve a reasonably high success rate without any knowledge of object shape.
Revised: We experimentally verify that even in this case, the policies achieve a reasonably high success rate without any knowledge of object shape.

Original: A good pose initialization is what you need : The difﬁculty of directly training the RL policies whenthe hand faces downward is mainly because of the hard-exploration issue in learning to catch the 
Revised: A good pose initialization is what you need : The difﬁculty of directly training the RL policies when the hand faces downward is mainly because of the hard-exploration issue in learning to catch the objects that are moving downward. However, catching is not necessary for the reorientation. Even for

Original: [Table]
Revised: [Table]

Original: [Table][Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: objects that are moving downward.

Original: However, catching is not necessary for the reorientation.

Original: Even forhuman, we only reorient the object after we grasp it.
Revised: human, we only reorient the object after we grasp it.

Original: More speciﬁcally, we ﬁrst train an object-liftingpolicy to lift objects from the table, collect the ending state (joint positions q T , object position p oT andorientation α oT ) in each successful lifting rollout episode, and reset the hand and objects to these states(velocities are all 0 ) for the pose initialization in training the reorientation policy.To train the liftingpolicy, we use the following reward function: r ( s t , a t ) = c h
Revised: More speciﬁcally, we ﬁrst train an object-lifting policy to lift objects from the table, collect the ending state (joint positions q T , object position p oT and orientation α oT ) in each successful lifting rollout episode, and reset the hand and objects to these states (velocities are all 0 ) for the pose initialization in training the reorientation policy.

Original: [Table] [Equation] [Table] [Equation]
Revised: The objects have randomly initialized poses and are dropped onto the table.

Original: 22where ∆ h t = max( p b,zt − p o,zt , 0) and p b,zt is the height ( z coordinate) of the Shadow Hand baseframe, p o,zt is the height of the object, ¯ h is the threshold of the height difference. In our experiments,c h 1 = 0 .
Revised: We trained a separate RNN policy for each dataset using the reward function in Section C.2.

Original: 05 , (cid:15) h
Revised: The success rate on the EGAD dataset is 97 .

Original: = 0 . 02 , ¯ h = 0 . 04 , c h 2 = 800 , c 3 = 0 .
Revised: 80% , while the success rate on the YCB dataset is 90 .

Original: The objects have randomly initialized posesand are dropped onto the table.
Revised: Note that objects need to be grasped ﬁrst to be lifted.

Revised: Our high success rates on object lifting also indicate that using an anthropomorphic hand makes object grasping an easy task, while many prior works [28, 29] require much more involved training techniques to learn grasping skills with parallel-jaw grippers.

Original: After we train the lifting policy, we collect about 250 ending states foreach object respectively from the successful lifting episodes.
Revised: After we train the lifting policy, we collect about 250 ending states for each object respectively from the successful lifting episodes.

Original: In every reset during the reorientationpolicy training, ending states are randomly sampled and used as the initial pose of the ﬁngers andobjects.
Revised: In every reset during the reorientation policy training, ending states are randomly sampled and used as the initial pose of the ﬁngers and objects.

Original: With a good pose initialization, policies are able to learn to reorient objects with high successrates.
Revised: With a good pose initialization, policies are able to learn to reorient objects with high success rates.

Original: π E R trained on EGAD dataset gets a success rate more than 80% while π E R trained on YCBdataset gets a success rate greater than 50% .
Revised: π E R trained on EGAD dataset gets a success rate more than 80% while π E R trained on YCB dataset gets a success rate greater than 50% .

Original: More results on the different policies with and withoutdomain randomization are available in Table D.5 in the appendix.
Revised: More results on the different policies with and without domain randomization are available in Table D.6 in the appendix.

Original: This setup is challenging becauseif at any time step in an episode the ﬁngers take a bad action, the object will fall.
Revised: This setup is challenging because if at any time step in an episode the ﬁngers take a bad action, the object will fall.

Original: Improving performance using gravity curriculum : Since the difﬁculty of training the reorientation policy with the hand facing downward is due to the gravity, we propose to build agravity curriculum to learn the policy π E .
Revised: Improving performance using gravity curriculum : Since the difﬁculty of training the reorientation policy with the hand facing downward is due to the gravity, we propose to build a gravity curriculum to learn the policy π E .

Original: Since π E already performs very well on EGAD objects,we apply gravity curriculum to train π E on YCB objects.
Revised: Since π E already performs very well on EGAD objects, we apply gravity curriculum to train π E on YCB objects.

Original: Our gravity curriculum is constructed asfollows: we start the training with g = 1 m / s 2 , then we gradually decrease g in a step-wise fashion ifthe evaluation success rate ( w ) is above a threshold value ( ¯ w ) until g = − 9 .
Revised: Our gravity curriculum is constructed as follows: we start the training with g = 1 m / s 2 , then we gradually decrease g in a step-wise fashion if the evaluation success rate ( w ) is above a threshold value ( ¯ w ) until g = − 9 .

Original: 81 m / s 2 .
Revised: 81 m / s 2 .

Original: More detailsabout gravity curriculum are in Section C.4.
Revised: More details about gravity curriculum are in Section C.4.

Original: In Table D.5{+D.
+} (Exp Q and T) in the appendix, we can seethat adding gravity curriculum ( g -curr) signiﬁcantly boost the success rates on the YCB dataset.
Revised: In Table {+D.
+} (Exp Q and T) in the appendix, we can see that adding gravity curriculum ( g -curr) signiﬁcantly boost the success rates on the YCB dataset.

Original: 4.3 Zero-shot policy transfer across datasets
Revised: 4.3 Zero-shot policy transfer across datasets

Original: We have shown the testing performance on the same training dataset so far.
Revised: We have shown the testing performance on the same training dataset so far.

Original: How would the policieswork on a different dataset?
Revised: How would the policies work on a different dataset?

Original: To answer this, we test our policies across datasets: policies trained with
Revised: To answer this, we test our policies across datasets: policies trained with

Original: EGAD objects are now tested with YCB objects and vice versa.
Revised: EGAD objects are now tested with YCB objects and vice versa.

Original: We used the RNN policies trainedwith full-state information and reduced-state information respectively (without domain randomization)and tested them on the other dataset with the hand facing upward and downward.
Revised: We used the RNN policies trained with full-state information and reduced-state information respectively (without domain randomization) and tested them on the other dataset with the hand facing upward and downward.

Original: In the case of thehand facing downward, we tested the RNN policy trained with gravity curriculum.
Revised: In the case of the hand facing downward, we tested the RNN policy trained with gravity curriculum.

Original: Table 2 showsthat policies still perform well on the untrained dataset.
Revised: Table 2 shows that policies still perform well on the untrained dataset.

Original: 4.4 Reorient objects with RGBD sensors
Revised: 4.4 Object Reorientation with RGBD sensors

Original: In this section, we investigate whether we can train a vision policy to reorient objects with the handfacing upward.
Revised: In this section, we investigate whether we can train a vision policy to reorient objects with the hand facing upward.

Original: As vision-based experiments require much more compute resources, we train onevision policy for each object individually on six objects shown in Table 3.
Revised: As vision-based experiments require much more compute resources, we train one vision policy for each object individually on six objects shown in Table 3.

Original: We leave training a singlevision policy for all objects to future work.
Revised: We leave training a single vision policy for all objects to future work.

Original: We use the expert MLP policy trained in Section 4.1 tosupervise the vision policy.
Revised: We use the expert MLP policy trained in Section 4. to supervise the vision policy.

Revised: We also performed data augmentation on the point cloud input to the policy network at each time step in both training and testing.

Original:  We can see from Table 3 that reorienting the  toy has a high success ratewhile reorienting the mug has a much lower success rate. For the mug, the policy reorients the mugto a pose where the body of the mug match how the mug should look like in the goal orientation, butthe cup handle does not match.
Revised: The data augmentation includes the random translation of the point cloud, random noise on the point positions, random dropout on the points, and random variations on the point color. More details about the data augmentation are in Section D.5. We can see from Table 3 that reorienting the non-symmetric objects including the toy and the mug has high success rates (greater than 80% ).

Original: This is likely due to the difﬁculty of the network capturing the smallmug handle structure from the entire point cloud.

Original: Using d C as an episode termination criterion, wecan train a vision policy for the symmetric objects (cup and legos) individually with high successrates as shown in Table 3.
Revised: While training the policy for symmetric objects is much harder, Table 3 shows that using d C as an episode termination criterion enables the policies to achieve a success rate greater than 50% .

Original:  Related Work
Revised:  Related Work

Original: Dexterous manipulation has been studied for decades, dating back to [28, 29].
Revised: Dexterous manipulation has been studied for decades, dating back to [30, 31].

Original: In contrast to paralleljaw grasping, pushing, pivoting [30], or pick-and-place, dexterous manipulation typically involvescontinuously controlling force to the object through the ﬁngertips of a robotic hand [31].
Revised: In contrast to paralleljaw grasping, pushing, pivoting [32], or pick-and-place, dexterous manipulation typically involves continuously controlling force to the object through the ﬁngertips of a robotic hand [33].

Original: Someprior works used analytical kinematics and dynamics models of the hand and object, and usedtrajectory optimization to output control policies [1, 2, 32] or employed kinodynamic planning toﬁnd a feasible motion plan [33].
Revised: Some prior works used analytical kinematics and dynamics models of the hand and object, and used trajectory optimization to output control policies [1, 2, 34] or employed kinodynamic planning to ﬁnd a feasible motion plan [35].

Original: However, due to the large number of active contacts on the hand andthe objects, model simpliﬁcations such as simple ﬁnger and object geometries are usually necessaryto make the optimization or planning tractable.
Revised: However, due to the large number of active contacts on the hand and the objects, model simpliﬁcations such as simple ﬁnger and object geometries are usually necessary to make the optimization or planning tractable.

Original: Sundaralingam and Hermans [32] moved objects inhand without contact breaking or making between the ﬁngers and the object.
Revised: Sundaralingam and Hermans [34] moved objects in hand but assumes that there is no contact breaking or making between the ﬁngers and the object.

Original: Furukawa et al.
Revised: Furukawa et al.

Original: [34]achieved a high-speed dynamic regrasping motion on a cylinder using a high-speed robotic handand a high-speed vision system.
Revised: [36] achieved a high-speed dynamic regrasping motion on a cylinder using a high-speed robotic hand and a high-speed vision system.

Original: Prior works have also explored the using vision  for manipulatingan object to track a  path [35], detecting manipulation modes [36], precision manipulation [37] witha limited number of objects with simple shapes using a two-ﬁngered gripper.
Revised: Prior works have also explored the use of a vision system for manipulating an object to track a planned path [37], detecting manipulation modes [38], precision manipulation [39] with a limited number of objects with simple shapes using a two-ﬁngered gripper.

Original: Recent works haveexplored the application of reinforcement learning to dexterous manipulation.
Revised: Recent works have explored the application of reinforcement learning to dexterous manipulation.

Original: Model-based RL workslearned a linear [4, 7] or deep neural network [5] dynamics model from the rollout data, and usedonline optimal control to rotate a pen or Baoding balls on a Shadow hand.
Revised: Model-based RL works learned a linear [4, 7] or deep neural network [5] dynamics model from the rollout data, and used online optimal control to rotate a pen or Baoding balls on a Shadow hand.

Original: However, when the systemis unstable, collecting informative trajectories for training a good dynamics model that generalizesto different objects remains challenging.
Revised: However, when the system is unstable, collecting informative trajectories for training a good dynamics model that generalizes to different objects remains challenging.

Original: Another line of works uses model-free RL algorithms tolearn a dexterous manipulation policy.
Revised: Another line of works uses model-free RL algorithms to learn a dexterous manipulation policy.

Original: For example, Andrychowicz et al.
Revised: For example, Andrychowicz et al.

Original: [11] and OpenAI et al.[12] learned a controller to reorient a block or a Rubik’s cube.
Revised: [11] and OpenAI et al. [12] learned a controller to reorient a block or a Rubik’s cube.

Original: Van Hoof et al.
Revised: Van Hoof et al.

Original: [38] learned thetactile informed policy via RL for a three-ﬁnger manipulator to move an object on the table.
Revised: [40] learned the tactile informed policy via RL for a three-ﬁnger manipulator to move an object on the table.

Original: Toreduce the sample complexity of model-free learning, [9, 13, 8, 39, 6] combined imitation learningwith RL to learn to reorient a pen, open a door, assemble LEGO blocks, etc.
Revised: To reduce the sample complexity of model-free learning, [9, 13, 8, 41, 6] combined imitation learning with RL to learn to reorient a pen, open a door, assemble LEGO blocks, etc.

Original: However, collectingexpert demonstration data from humans is expensive, time-consuming, and even incredibly difﬁcultfor contact-rick tasks [8].
Revised: However, collecting expert demonstration data from humans is expensive, time-consuming, and even incredibly difﬁcult for contact-rick tasks [8].

Original: Our method belongs to the category of model-free learning.
Revised: Our method belongs to the category of model-free learning.

Original: We use theteacher-student learning paradigm to speed up the deployment policy learning.
Revised: We use the teacher-student learning paradigm to speed up the deployment policy learning.

Original: Our learned policiesalso generalize to new shapes and show strong zero-shot transfer performance.
Revised: Our learned policies also generalize to new shapes and show strong zero-shot transfer performance.

Original: To the best of ourknowledge, our system is the ﬁrst work that demonstrates the capabilities of reorienting a diverse setof objects that have complex geometries with both the hand facing upward and downward.
Revised: To the best of our knowledge, our system is the ﬁrst work that demonstrates the capabilities of reorienting a diverse set of objects that have complex geometries with both the hand facing upward and downward.

Original:  Discussion 
Revised:  Discussion and Conclusion

Original: Our results show that model-free RL with simple deep learning architectures can be used to trainpolicies to reorient a large set of geometrically diverse objects.
Revised: Our results show that model-free RL with simple deep learning architectures can be used to train policies to reorient a large set of geometrically diverse objects.

Original: Further for learning with  hand facingdownwards, we found that a good pose initialization obtained from a lifting policy was necessary,and the gravity curriculum substantially improved performance.
Revised: Further, for learning with the hand facing downwards, we found that a good pose initialization obtained from a lifting policy was necessary, and the gravity curriculum substantially improved performance.

Original: The agent also learns to use anexternal surface (i.e., the table).
Revised: The agent also learns to use an external surface (i.e., the table).

Original: The most surprising observation is that information about shape isnot required despite the fact that we train a single policy to manipulate multiple objects.
Revised: The most surprising observation is that information about shape is not required despite the fact that we train a single policy to manipulate multiple objects.

Original: Perhapsin hindsight, it is not as surprising – after all, humans can close their eyes and easily manipulatenovel objects into a speciﬁc orientation.
Revised: Perhaps in hindsight, it is not as surprising – after all, humans can close their eyes and easily manipulate novel objects into a speciﬁc orientation.

Original: However, even in this scenario, shape can be implicitlyinferred using touch signals.
Revised: But, even in this scenario, shape can be implicitly inferred using touch signals.

Revised: However, our system has no access to touch data.

Original: As such, our work can serve a strong baseline for future in-hand objectreorientation works that incorporate object shape information as part of the observation space toimprove performance.
Revised: As such, our work can serve a strong baseline for future in-hand object reorientation works that incorporate object shape in the observation space to improve performance.

Original: It should be noted that successful training of the teacher policy critically relied on shape informationbeing expendable but required privileged information – a major obstacle in the real-world deployment.

Original: While we were not able to train reorientation policies from scratch using visual inputs, we showthat we can distill this policy to operate from RGBD point clouds and joint positions that are readilyavailable in the real world.

Original: While we only present results in simulation, we also provide evidence thatour policies can be transferred to the real world.
Revised: While we only present results in simulation, we also provide evidence that our policies can be transferred to the real world.

Original: The experiments with domain randomization indicatethat learned policies can work with noisy inputs.
Revised: The experiments with domain randomization indicate that learned policies can work with noisy inputs.

Original: Analysis of peak torques during manipulation (see Figure D.12 in the appendix) also reveals that generated torque commands are feasible to actuate onan actual robotic hand.
Revised: Analysis of peak torques during manipulation (see Figure D. in the appendix) also reveals that generated torque commands are feasible to actuate on an actual robotic hand.

Original: Finally, Figure D.9 and Figure D.10 in the appendix show that the success rate varies substantiallywith object shape.
Revised: Finally, Figure D.10 and Figure D.11 in the appendix show that the success rate varies substantially with object shape.

Original: This suggests that in the future, a training curriculum based on object shapes canimprove performance.
Revised: This suggests that in the future, a training curriculum based on object shapes can improve performance.

Original: Another future work is to directly train one vision policy for a diverse set ofobjects.
Revised: Another future work is to directly train one vision policy for a diverse set of objects.

Original: A major bottleneck in vision-based experiments is the demand for much larger GPU memory.
Revised: A major bottleneck in vision-based experiments is the demand for much larger GPU memory.

Original: Learning visual representations of point cloud data that can ease the computational bottleneck is,therefore, an important avenue for future research.
Revised: Learning visual representations of point cloud data that can ease the computational bottleneck is, therefore, an important avenue for future research.

Revised: Acknowledgments

Revised: We thank the anonymous reviewers for their helpful comments in revising the paper.

Revised: We thank the members of Improbable AI lab for providing valuable feedback on research idea formulation and manuscript.

Revised: This research is funded by Toyota Research Institute, Amazon Research Award and

Revised: DARPA Machine Common Sense Program.

Revised: We also acknowledge the MIT SuperCloud and Lincoln

Revised: Laboratory Supercomputing Center for providing HPC resources that have contributed to the research results reported within this paper.