Original: A Simple Method for Complex In-Hand Manipulation
Revised: A Simple Method for Complex In-Hand Manipulation

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract: In-hand object reorientation has been a challenging problem in roboticsdue to high dimensional actuation space and the frequent change in contact statebetween the ﬁngers and the objects.
Revised: Abstract: In-hand object reorientation has been a challenging problem in roboticsdue to high dimensional actuation space and the frequent change in contact statebetween the ﬁngers and the objects.

Original: We present a simple model-free framework thatcan learn to reorient objects with both the hand facing upwards and downwards.
Revised: We present a simple model-free framework thatcan learn to reorient objects with both the hand facing upwards and downwards.

Original: Wedemonstrate the capability of reorienting over 2000 geometrically different objectsin both cases.
Revised: Wedemonstrate the capability of reorienting over 2000 geometrically different objectsin both cases.

Original: The learned policies show strong zero-shot transfer performance onnew objects.
Revised: The learned policies show strong zero-shot transfer performance onnew objects.

Original: We provide evidence that these policies are amenable to real-worldoperation by distilling them to use observations easily available in the real world.
Revised: We provide evidence that these policies are amenable to real-worldoperation by distilling them to use observations easily available in the real world.

Original: The videos of the learned policies are available at: https://sites.google.com/view/in-hand-reorientation .
Revised: The videos of the learned policies are available at: https://sites.google.com/view/in-hand-reorientation .

Original:  Introduction
Revised:  Introduction

Original: A common maneuver in many tasks of daily living is to pick an object, reorient it in hand and theneither place it or use it as a tool.
Revised: A common maneuver in many tasks of daily living is to pick an object, reorient it in hand and theneither place it or use it as a tool.

Original: Consider three simpliﬁed variants of this maneuver in Figure 1.
Revised: Consider three simpliﬁed variants of this maneuver in Figure 1.

Original: Theﬁrst task shown in the top row requires an upward-facing multi-ﬁnger hand to manipulate an arbitraryobject in a random orientation to a goal conﬁguration.
Revised: Theﬁrst task shown in the top row requires an upward-facing multi-ﬁnger hand to manipulate an arbitraryobject in a random orientation to a goal conﬁguration.

Original: The next two rows show the second andthird tasks where the hand is facing downward and is required to reorient the object either using thetable as a support or without the aid of any support surface respectively.
Revised: The next two rows show the second andthird tasks where the hand is facing downward and is required to reorient the object either using thetable as a support or without the aid of any support surface respectively.

Original: These two tasks are harderbecause the object is in an intrinsically unstable conﬁguration owing to the downward gravitationalforce and lack of support from the palm.
Revised: The last task is the hardestbecause the object is in an intrinsically unstable conﬁguration owing to the downward gravitationalforce and lack of support from the palm.

Original: Additional challenges in making such maneuvers with amulti-ﬁnger robotic hand stem from the control space being high-dimensional and reasoning aboutmultiple transitions in the contact state between the ﬁnger and the object.
Revised: Additional challenges in making such maneuvers with amulti-ﬁnger robotic hand stem from the control space being high-dimensional and reasoning aboutmultiple transitions in the contact state between the ﬁnger and the object.

Original: Due to its practical utilityand several unsolved issues, in-hand object reorientation remains an active area of research.
Revised: Due to its practical utilityand several unsolved issues, in-hand object reorientation remains an active area of research.

Original: Past work has tackled the in-hand reorientation problem via several approaches: (i)
Revised: Past work has tackled the in-hand reorientation problem via several approaches: (i)

Original: The use ofanalytical models with powerful trajectory optimization methods [1, 2, 3].
Revised: The use ofanalytical models with powerful trajectory optimization methods [1, 2, 3].

Original: While these methodsdemonstrated remarkable performance, the results were largely in simulation with simple objectgeometries and required detailed knowledge of the object model and physical parameters.
Revised: While these methodsdemonstrated remarkable performance, the results were largely in simulation with simple objectgeometries and required detailed knowledge of the object model and physical parameters.

Original: As such, itremains unclear how to scale these methods to real-world and generalize to new objects.
Revised: As such, itremains unclear how to scale these methods to real-world and generalize to new objects.

Original: Another lineof work has employed (ii) model-based reinforcement learning [4, 5]; or (iii) model-free reinforcement learning with [6, 7, 8, 9] and without expert demonstrations [10, 11, 12, 13].
Revised: Another lineof work has employed (ii) model-based reinforcement learning [4, 5]; or (iii) model-free reinforcement learning with [6, 7, 8, 9] and without expert demonstrations [10, 11, 12, 13].

Original: While some of theseworks demonstrated learned skills on the real robots, they required substantial engineering andadditional apparatus (e.g., motion capture system) to infer the object state, and the learned policiesdid not generalize to diverse objects.
Revised: While some of theseworks demonstrated learned skills on the real robots, they required substantial engineering andadditional apparatus (e.g., motion capture system) to infer the object state, and the learned policiesdid not generalize to diverse objects.

Original: Furthermore, nearly all of these methods operate in a simpliﬁedsetting where the hand is only allowed to face upward (with pick-and-place being an exception, butpick-and-place doesn’t incorporate any in-hand manipulation).
Revised: Furthermore, nearly all of these methods operate in a simpliﬁedsetting where the hand is only allowed to face upward (with pick-and-place being an exception, butpick-and-place doesn’t incorporate any in-hand manipulation).

Original: A detailed discussion of prior researchis provided in Section 5.
Revised: A detailed discussion of prior researchis provided in Section 5.

Original: In this paper, our goal is to study the object reorientation problem with a multi-ﬁnger hand in itsgeneral form.
Revised: In this paper, our goal is to study the object reorientation problem with a multi-ﬁnger hand in itsgeneral form.

Original: We desire (a) manipulation with hand facing upward or downward; (b) the skill of usingexternal surfaces to aid manipulation; (c) the ability to reorient objects of novel shapes to arbitraryorientations; (d) operation from sensory data that can be easily obtained in the real world such as
Revised: We desire (a) manipulation with hand facing upward or downward; (b) the skill of usingexternal surfaces to aid manipulation; (c) the ability to reorient objects of novel shapes to arbitraryorientations; (d) operation from sensory data that can be easily obtained in the real world such as

Original: RGBD images and joint positions of the hand.
Revised: RGBD images and joint positions of the hand.

Original: While some of these aspects have been individuallydemonstrated in prior work, we are unaware of any published method that realizes all four.
Revised: While some of these aspects have been individuallydemonstrated in prior work, we are unaware of any published method that realizes all four.

Original: Our maincontribution is building a simple framework that can achieve these desiderata.
Revised: Our maincontribution is building a simple framework that can achieve these desiderata.

Original: We experimentally testthis framework with a simulated Shadow hand.
Revised: We experimentally testthis framework with a simulated Shadow hand.

Original: Due to the scope of the problem and the ongoingpandemic, we limit our experiments to be in simulation.
Revised: Due to the scope of the problem and the ongoingpandemic, we limit our experiments to be in simulation.

Original: However, we provide evidence indicatingthat the learned policies can be transferred to the real world in the future.
Revised: However, we provide evidence indicatingthat the learned policies can be transferred to the real world in the future.

Original:  Method
Revised:  Method

Original: To our surprise, simple extensions to existing techniques in robot learning can be used to constructa system for general object reorientation.
Revised: To our surprise, simple extensions to existing techniques in robot learning can be used to constructa system for general object reorientation.

Original: First, to avoid explicit modeling of the highly non-linearand frequent changes in the contact state between the object and the hand, we use model-freereinforcement learning (RL).
Revised: First, to avoid explicit modeling of the highly non-linearand frequent changes in the contact state between the object and the hand, we use model-freereinforcement learning (RL).

Original: An added advantage is that model-free RL is amenable to directlyoperating from raw RGBD point cloud observations.
Revised: An added advantage is that model-free RL is amenable to directlyoperating from raw RGBD point cloud observations.

Original: Second, we make use of privileged stateinformation such as the positions/orientations/velocities of the object/ﬁngertips that is easily availablein the simulator but not in the real world.
Revised: Second, we make use of privileged stateinformation such as the positions/orientations/velocities of the object/ﬁngertips that is easily availablein the simulator but not in the real world.

Original: Such state information can simplify policy learning andincrease data efﬁciency.
Revised: Such state information can simplify policy learning andincrease data efﬁciency.

Original: We train an RL policy with privileged state information to reorient morethan two thousand objects of diverse shapes (see Section 2.1).
Revised: We train an RL policy with privileged state information to reorient morethan two thousand objects of diverse shapes (see Section 2.1).

Original: However such a policy cannot betransferred to the real robot due to the difﬁculty of getting the priviledged information in real world.
Revised: However such a policy cannot betransferred to the real robot due to the difﬁculty of getting the priviledged information in real world.

Original: We address this issue via teacher-student learning.
Revised: We address this issue via teacher-student learning.

Original: The idea of ﬁrst learning an expert or teacher policy ( π E ) from privileged information and thenusing it to guide the learning of a student policy ( π S ) that only uses sensory inputs available in thereal world is known as teacher-student training [14, 15].
Revised: The idea of ﬁrst learning an expert or teacher policy ( π E ) from privileged information and thenusing it to guide the learning of a student policy ( π S ) that only uses sensory inputs available in thereal world is known as teacher-student training [14, 15].

Original: We leverage this idea to demonstrate thepossibility of transferring the learned policies to the real world in the future.
Revised: We leverage this idea to demonstrate thepossibility of transferring the learned policies to the real world in the future.

Original: Let the state spacecorresponding to π E and π S be S E and S S respectively.
Revised: Let the state spacecorresponding to π E and π S be S E and S S respectively.

Original: In general, S E (cid:54) =
Revised: In general, S E (cid:54) =

Original: S S .
Revised: S S .

Original: We ﬁrst detail themethod for distilling π E to a student policy using a reduced state space consisting of only the jointpositions of the hand, the object position, and the difference in orientation from the goal conﬁguration(see Section 2.2.1).
Revised: We ﬁrst detail themethod for distilling π E to a student policy using a reduced state space consisting of only the jointpositions of the hand, the object position, and the difference in orientation from the goal conﬁguration(see Section 2.2.1).

Original: However, in the real world, the object position and relative orientation mustbe inferred from sensory observation.
Revised: However, in the real world, the object position and relative orientation mustbe inferred from sensory observation.

Original: Not only does this process require substantial engineeringeffort (e.g., a motion capture or a pose estimation system), but also inferring the pose of a symmetricobject is prone to errors.
Revised: Not only does this process require substantial engineeringeffort (e.g., a motion capture or a pose estimation system), but also inferring the pose of a symmetricobject is prone to errors.

Original: This is because a symmetric object at multiple orientations looks exactlythe same in sensory space such as RGBD images.
Revised: This is because a symmetric object at multiple orientations looks exactlythe same in sensory space such as RGBD images.

Original: To mitigate these issues, we further distill π E tooperate directly from the RGBD point cloud and position of all the hand joints (see Section 2.2.2).
Revised: To mitigate these issues, we further distill π E tooperate directly from the RGBD point cloud and position of all the hand joints (see Section 2.2.2).

Original: We propose a simple modiﬁcation that generalizes an existing 2D CNN to make this possible.
Revised: We propose a simple modiﬁcation that generalizes an existing 2D CNN to make this possible.

Original: While the procedure described above works well for manipulation with the hand facing upwards anddownwards with a table as a support, we ﬁnd that in the case of the hand facing downward without atable, a good initialization of object pose such as the one obtained by an object lifting policy is keyfor successful reorientation, and the proposed gravity curriculum substantially improves performance.
Revised: While the procedure described above works well for manipulation with the hand facing upwards anddownwards with a table as a support, we ﬁnd that in the case of the hand facing downward without atable, a good initialization of object pose such as the one obtained by an object lifting policy is keyfor successful reorientation, and the proposed gravity curriculum substantially improves performance.

Original: 2.1 Learning the teacher policy
Revised: 2.1 Learning the teacher policy

Original: We use model-free RL to learn the teacher policy ( π E ) for reorienting an object ( { O i | i = 1 , ..., N } )from an initial orientation α o 0 to a target orientation α g ( α o 0 (cid:54) =
Revised: We use model-free RL to learn the teacher policy ( π E ) for reorienting an object ( { O i | i = 1 , ..., N } )from an initial orientation α o 0 to a target orientation α g ( α o 0 (cid:54) =

Original: α g ).
Revised: α g ).

Original: At every time step t , the agentobserves the state s t , executes the action a t sampled from the policy π E , and receives a reward r t .
Revised: At every time step t , the agentobserves the state s t , executes the action a t sampled from the policy π E , and receives a reward r t .

Original: π Eis optimized to maximize the expected discounted return: π = arg max π E E (cid:2) (cid:80) T − 1 t =

Revised: π Eis optimized to maximize the expected discounted return: π = arg max π E E (cid:2) (cid:80) T − 1 t =

Original: γ t r t (cid:3) , where γis the discount factor.
Revised: γ t r t (cid:3) , where γis the discount factor.

Original: The task is successful if the angle difference ∆ θ between the object’s current( α ot ) and the goal orientation ( α g ) is smaller than the threshold value ¯ θ , i.e. , ∆ θ ≤ ¯ θ .
Revised: The task is successful if the angle difference ∆ θ between the object’s current( α ot ) and the goal orientation ( α g ) is smaller than the threshold value ¯ θ , i.e. , ∆ θ ≤ ¯ θ .

Original: To encourage the policy to be smooth, the previous action is appended to the inputs to the policy ( i.e. ,a t = π E ( s t , a t − 1 ) ) and large actions are penalized in the reward function.
Revised: To encourage the policy to be smooth, the previous action is appended to the inputs to the policy ( i.e. ,a t = π E ( s t , a t − 1 ) ) and large actions are penalized in the reward function.

Original: We experiment with twoarchitectures for π E : (1) an MLP policy π M , (2) an RNN policy π R .
Revised: We experiment with twoarchitectures for π E : (1) an MLP policy π M , (2) an RNN policy π R .

Original: We use PPO [16] to optimizeπ E .
Revised: We use PPO [16] to optimizeπ E .

Original: All the experiments in the paper were run on at most 2 GPUs with a 32 GB memory.
Revised: All the experiments in the paper were run on at most 2 GPUs with a 32 GB memory.

Original: More detailsabout the network architecture and training are in Section C.1 and Section C.2 in the appendix.
Revised: More detailsabout the network architecture and training are in Section C.1 and Section C.2 in the appendix.

Original: Observation and action space : We deﬁne S E to include joint, ﬁngertip, object, and goal informationas detailed in Table B.1 in the appendix.
Revised: Observation and action space : We deﬁne S E to include joint, ﬁngertip, object, and goal informationas detailed in Table B.1 in the appendix.

Original: Note that S E does not include object shape or informationabout friction, damping, contact states between the ﬁngers and the object, etc.
Revised: Note that S E does not include object shape or informationabout friction, damping, contact states between the ﬁngers and the object, etc.

Original: We control the jointmovements by commanding the relative change in the target joint angle ( q targett ) on each actuatedjoint (action a t ∈ R 20 ): q targett + = q targett + a t × ∆ t , where ∆ t is the control time step.
Revised: We control the jointmovements by commanding the relative change in the target joint angle ( q targett ) on each actuatedjoint (action a t ∈ R 20 ): q targett + = q targett + a t × ∆ t , where ∆ t is the control time step.

Original: We clamp theaction command if necessary to make sure | ∆ q targett | ≤ 0 .
Revised: We clamp theaction command if necessary to make sure | ∆ q targett | ≤ 0 .

Original: 33 rad .
Revised: 33 rad where ∆ q targett = q targett + − q targett.

Original: The control frequency is 60 Hz .
Revised: The control frequency is 60 Hz .

Original: Reward function :
Revised: Reward function :

Original: For training π E , the reward function is deﬁned as:
Revised: For training π E , the reward function is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: ) where c θ 1 > 0 , c θ 2 > 0 and c 3 < 0 are the coefﬁcients, ∆ θ t is the difference between the currentobject orientation and the target orientation, (cid:15) θ is a constant, 1 is an indicator function that identiﬁeswhether the object is in the target orientation.
Revised: (1) where c θ 1 > 0 , c θ 2 > 0 and c 3 < 0 are the coefﬁcients, ∆ θ t is the difference between the currentobject orientation and the target orientation, (cid:15) θ is a constant, 1 is an indicator function that identiﬁeswhether the object is in the target orientation.

Original: The ﬁrst two reward terms encourage the policy toreorient the object to the desired orientation while the last term is a regularizer to suppress largeaction commands.
Revised: The ﬁrst two reward terms encourage the policy toreorient the object to the desired orientation while the last term is a regularizer to suppress largeaction commands.

Original: In our experiments, we use c θ 1 = 1 , c θ 2 = 800 , c 3 = 0 .
Revised: In our experiments, we use c θ 1 = 1 , c θ 2 = 800 , c 3 = 0 .

Original: 1 , (cid:15) θ = 0 .
Revised: 1 , (cid:15) θ = 0 .

Original: 1 , ¯ θ = 0 .
Revised: 1 , ¯ θ = 0 .

Original: 1rad .
Revised: 1rad .

Original: Dynamics randomization : Even though we do not test our policies on a real robot, we train andevaluate policies with domain randomization [17] to provide evidence that our work has the potentialto be transferred to a real robotics system in the future.
Revised: Dynamics randomization : Even though we do not test our policies on a real robot, we train andevaluate policies with domain randomization [17] to provide evidence that our work has the potentialto be transferred to a real robotics system in the future.

Original: We randomize the object mass, frictioncoefﬁcient, joint damping and add noise to the state observation s t and the action a t .
Revised: We randomize the object mass, frictioncoefﬁcient, joint damping and add noise to the state observation s t and the action a t .

Original: More detailsabout domain randomization are provided in Table C.2 in the appendix.
Revised: More detailsabout domain randomization are provided in Table C.4 in the appendix.

Original: 2.2 Learning the student policy
Revised: 2.2 Learning the student policy

Original: We distill the teacher π E into the student policy π S using Dagger [18], a learning-from-demonstrationmethod that overcomes the covariate shift problem.
Revised: We distill the teacher π E into the student policy π S using Dagger [18], a learning-from-demonstrationmethod that overcomes the covariate shift problem.

Original: We optimize π S by minimizing the KL-divergencebetween π S and π E :
Revised: We optimize π S by minimizing the KL-divergencebetween π S and π E :

Original: π S = arg
Revised: π S = arg

Original: min π S D KL (cid:0) π E
Revised: min π S D KL (cid:0) π E

Original: ( s E t , a t − 1 )
Revised: ( s E t , a t − 1 )

Original: || π S ( s S t , a t − 1 )) (cid:1) .
Revised: || π S ( s S t , a t − 1 )) (cid:1) .

Original: Based on observationdata available in real-world settings, we investigate two different choices of S S .
Revised: Based on observationdata available in real-world settings, we investigate two different choices of S S .

Original: 2.2.1 Training student policy from low-dimensional state
Revised: 2.2.1 Training student policy from low-dimensional state

Original: In the ﬁrst case, we consider a non-vision student policy π S ( s S t , a t − 1 ) .
Revised: In the ﬁrst case, we consider a non-vision student policy π S ( s S t , a t − 1 ) .

Original: s S t ∈ R 31 includes the jointpositions q t ∈ R 24 , object position p ot ∈ R 3 , quaternion difference between the object’s currentand target orientation β t ∈ R 4 .
Revised: s S t ∈ R 31 includes the jointpositions q t ∈ R 24 , object position p ot ∈ R 3 , quaternion difference between the object’s currentand target orientation β t ∈ R 4 .

Original: In this case, S S ⊂ S E , and we assume the availability of objectpose information, but do not require velocity information.
Revised: In this case, S S ⊂ S E , and we assume the availability of objectpose information, but do not require velocity information.

Original: We use the same MLP and RNN networkarchitectures used for π E on π S except the input dimension changes as the state dimension is different.
Revised: We use the same MLP and RNN networkarchitectures used for π E on π S except the input dimension changes as the state dimension is different.

Original: 2.2.2 Training student policy from vision
Revised: 2.2.2 Training student policy from vision

Original: In the second case, S S only uses direct observations from RGBD cameras and the joint position ( q t )of the robotic hand.
Revised: In the second case, S S only uses direct observations from RGBD cameras and the joint position ( q t )of the robotic hand.

Original: We convert the RGB and Depth data into a colored point cloud using a pinholecamera model [19].
Revised: We convert the RGB and Depth data into a colored point cloud using a pinholecamera model [19].

Original: Our vision policy takes as input the voxelized point cloud of the scene W t , q t ,and previous action command a t − 1 , and outputs the action a t , i.e. , a t = π S
Revised: Our vision policy takes as input the voxelized point cloud of the scene W t , q t ,and previous action command a t − 1 , and outputs the action a t , i.e. , a t = π S

Original: ( W t , q t , a t − 1 ) .
Revised: ( W t , q t , a t − 1 ) .

Original: Goal speciﬁcation : To avoid the dependency on any manually deﬁned coordinate frame for specifyingthe goal quaternion, we only provide the object’s point cloud rotated to the desired orientation W gto the policy, i.e. , we only show the policy how the object should look like in the end (see the topleft of Figure C.5).
Revised: Goal speciﬁcation : To avoid the dependency on any manually deﬁned coordinate frame for specifyingthe goal quaternion, we only provide the object’s point cloud rotated to the desired orientation W gto the policy, i.e. , we only show the policy how the object should look like in the end (see the topleft of Figure C.5).

Original: The input to π S is the point cloud W t
Revised: The input to π S is the point cloud W t

Original: =
Revised: =

Original: W s t ∪ W g .
Revised: W s t ∪ W g .

Original: We obtain W g by sampling5000 points from the object’s CAD mesh using the Barycentric coordinate, rotating the points by thedesired orientation, and translating them so that these points are next to the hand.
Revised: We obtain W g by sampling5000 points from the object’s CAD mesh using the Barycentric coordinate, rotating the points by thedesired orientation, and translating them so that these points are next to the hand.

Original: Note that one canalso put the object in the desired orientation right next to the hand in the simulator and render theentire scene altogether to remove the need for CAD models.
Revised: Note that one canalso put the object in the desired orientation right next to the hand in the simulator and render theentire scene altogether to remove the need for CAD models.

Original: We use CAD models for W g just tosave the computational cost of rendering another object while we still use RGBD cameras to get W st .
Revised: We use CAD models for W g just tosave the computational cost of rendering another object while we still use RGBD cameras to get W st .

Original: Sparse3D-IMPALA-Net : To convert a voxelized point cloud into a lower-dimensional featurerepresentation, we use a sparse convolutional neural network.
Revised: Sparse3D-IMPALA-Net : To convert a voxelized point cloud into a lower-dimensional featurerepresentation, we use a sparse convolutional neural network.

Original: We extend the IMPALA policyarchitecture [20] for processing RGB images to process colored point cloud data using 3D convolution.
Revised: We extend the IMPALA policyarchitecture [20] for processing RGB images to process colored point cloud data using 3D convolution.

Original: Since many voxels are unoccupied, the use of regular 3D convolution wastes substantial computationtime.
Revised: Since many voxels are unoccupied, the use of regular 3D convolution wastes substantial computationtime.

Original: Hence, we use Minkowski Engine [21], a PyTorch library for sparse tensors, to design a 3Dversion of IMPALA-Net with sparse convolutions ( Sparse3D-IMPALA-Net ) 1 .
Revised: Hence, we use Minkowski Engine [21], a PyTorch library for sparse tensors, to design a 3Dversion of IMPALA-Net with sparse convolutions ( Sparse3D-IMPALA-Net ) 1 .

Original: The point cloud W tis processed by a series of CNN residual modules and projected into an embedding vector.
Revised: The point cloud W tis processed by a series of CNN residual modules and projected into an embedding vector.

Original: q t anda t − 1 are concatenated together and projected into an embedding vector via an MLP.
Revised: q t anda t − 1 are concatenated together and projected into an embedding vector via an MLP.

Original: Both embeddingvectors from W t and ( q t , a t − 1 ) are concatenated and passed through a recurrent network to outputthe action a t .
Revised: Both embeddingvectors from W t and ( q t , a t − 1 ) are concatenated and passed through a recurrent network to outputthe action a t .

Original: The detailed architecture is illustrated in Figure C.5 in the appendix.
Revised: The detailed architecture is illustrated in Figure C.5 in the appendix.

Original: Mitigating the object symmetry issue : π E is trained with the the ground-truth state informations E t including the object orientation α ot and goal orientation α g .
Revised: Mitigating the object symmetry issue : π E is trained with the the ground-truth state informations E t including the object orientation α ot and goal orientation α g .

Original: The vision policy does not take anyorientation information as input.
Revised: The vision policy does not take anyorientation information as input.

Original: If an object is symmetric, the two different orientations of the objectmay correspond to the same point cloud observation.
Revised: If an object is symmetric, the two different orientations of the objectmay correspond to the same point cloud observation.

Original: This makes it problematic to use the differencein orientation angles ( ∆ θ ≤ ¯ θ ) as the stopping and success criterion.
Revised: This makes it problematic to use the differencein orientation angles ( ∆ θ ≤ ¯ θ ) as the stopping and success criterion.

Original: To mitigate this issue, we use
Revised: To mitigate this issue, we use

Original: Chamfer distance [22] to compute the distance between the object point cloud in α ot and the goal pointcloud (i.e., the object rotated by α g ) as the evaluation criterion.
Revised: Chamfer distance [22] to compute the distance between the object point cloud in α ot and the goal pointcloud (i.e., the object rotated by α g ) as the evaluation criterion.

Original: The Chamfer distance is computedas [Equation] = 
Revised: The Chamfer distance is computedas d C = (cid:80)

Original: [Equation]
Revised: [Equation]

Original:  min
Revised: ot min

Original: [Equation]
Revised: [Equation]

Original:  (cid:107) a − b (cid:107) 22 + (cid:80) [Equation] ∈ W g min a ∈ W ot (cid:107) a − b (cid:107) 22 , where W ot is the object pointcloud in its current orientation.
Revised: g (cid:107) a − b (cid:107) 22 + (cid:80) b ∈ W g min a ∈ W ot (cid:107) a − b (cid:107) 22 , where W ot is the object pointcloud in its current orientation.

Original: Both W o t and W g are scaled to ﬁt in a unit sphere for computing d C .
Revised: Both W o t and W g are scaled to ﬁt in a unit sphere for computing d C .

Original: We check Chamfer distance in each rollout step.
Revised: We check Chamfer distance in each rollout step.

Original: If d C ≤ ¯ d C ( ¯ d C is a threshold value for d C ), weconsider the episode to be successful.
Revised: If d C ≤ ¯ d C ( ¯ d C is a threshold value for d C ), weconsider the episode to be successful.

Original: Hence, the success criterion is (∆ θ ≤ ¯ θ ) ∨ ( d C ≤ ¯ d C ) .
Revised: Hence, the success criterion is (∆ θ ≤ ¯ θ ) ∨ ( d C ≤ ¯ d C ) .

Original: Intraining, if the success criterion is satisﬁed, the episode is terminated and used for updating π S .
Revised: Intraining, if the success criterion is satisﬁed, the episode is terminated and used for updating π S .

Original:  Experimental Setup
Revised:  Experimental Setup

Original: We use the simulated Shadow Hand [23] in NVIDIA Isaac Gym [24].
Revised: We use the simulated Shadow Hand [23] in NVIDIA Isaac Gym [24].

Original: Shadow Hand is an anthropomorphic robotic hand with 24 degrees of freedom (DoF).
Revised: Shadow Hand is an anthropomorphic robotic hand with 24 degrees of freedom (DoF).

Original: In our experiments, we assume the base ofthe hand to be ﬁxed.
Revised: In our experiments, we assume the base ofthe hand to be ﬁxed.

Original: Twenty joints are actuated by agonist–antagonist tendons and the remainingfour joints are under-actuated.
Revised: Twenty joints are actuated by agonist–antagonist tendons and the remainingfour joints are under-actuated.

Original: Object datasets : We use the EGAD dataset [25] and YCB dataset [26] that contain objects withdiverse shapes (see Figure B.2) for in-hand manipulation experiments.
Revised: Object datasets : We use the EGAD dataset [25] and YCB dataset [26] that contain objects withdiverse shapes (see Figure B.2) for in-hand manipulation experiments.

Original: EGAD contains 2282 geometrically diverse textureless object meshes, while the YCB dataset includes textured meshes for objectsof daily life with different shapes and textures.
Revised: EGAD contains 2282 geometrically diverse textureless object meshes, while the YCB dataset includes textured meshes for objectsof daily life with different shapes and textures.

Original: We use the 78 YCB object models collected with the
Revised: We use the 78 YCB object models collected with the

Original: Google scanner.
Revised: Google scanner.

Original: Since most YCB objects are too big for in-hand manipulation, we proportionallyscale down the YCB meshes.
Revised: Since most YCB objects are too big for in-hand manipulation, we proportionallyscale down the YCB meshes.

Original: To further increase the diversity of the datasets, we create 5 variantsfor each object mesh by randomly scaling the mesh.
Revised: To further increase the diversity of the datasets, we create 5 variantsfor each object mesh by randomly scaling the mesh.

Original: The scaling ratios are randomly sampled suchthat the longest side of the objects’ bounding boxes l max lies in [0 . 05 , 0 . 08] m for EGAD objects,and l max ∈ [0 . 05 , 0 .
Revised: The scaling ratios are randomly sampled suchthat the longest side of the objects’ bounding boxes l max lies in [0 . 05 , 0 . 08] m for EGAD objects,and l max ∈ [0 . 05 , 0 .

Original: 12] m for YCB objects.
Revised: 12] m for YCB objects.

Original: Mass is randomly sampled from [0 . 05 , 0 . 15] kg for everyobject.
Revised: Mass is randomly sampled from [0 . 05 , 0 . 15] kg for everyobject.

Original: When we randomly scale YCB objects, some objects become too small or too thin, making thereorientation task even more challenging.
Revised: When we randomly scale YCB objects, some objects become too small or too thin, making thereorientation task even more challenging.

Original: In total, we use 11410 EGAD object meshes and 390 YCBobject meshes for training.
Revised: In total, we use 11410 EGAD object meshes and 390 YCBobject meshes for training.

Original: More details on the dataset are available in Section B.2 in the appendix.
Revised: More details on the dataset are available in Section B.2 in the appendix.

Original: Setup for visual observations : For the vision experiments, we experiment with the scenario wherethe hand faces upward.
Revised: Setup for visual observations : For the vision experiments, we experiment with the scenario wherethe hand faces upward.

Original: We place two RGBD cameras above the hand (Figure B.4).
Revised: We place two RGBD cameras above the hand (Figure B.4).

Original: The data fromthese cameras is combined to create a point cloud observation of the scene [19].
Revised: The data fromthese cameras is combined to create a point cloud observation of the scene [19].

Original: For downstreamcomputation, the point cloud is voxelized with a resolution of 0 .
Revised: For downstreamcomputation, the point cloud is voxelized with a resolution of 0 .

Original: 003 m .
Revised: 003 m .

Original: Setup with the upward facing hand : We ﬁrst consider the case where the Shadow Hand facesupward and is required to reorient objects placed in the hand (see Figure 2a).
Revised: Setup with the upward facing hand : We ﬁrst consider the case where the Shadow Hand facesupward and is required to reorient objects placed in the hand (see Figure 2a).

Original: We use the coordinatesystem where the z -axis points vertically upward and the xy -plane denotes the horizontal plane.
Revised: We use the coordinatesystem where the z -axis points vertically upward and the xy -plane denotes the horizontal plane.

Original: Theobject is initialized as following: xy position of the object’s center of mass (COM) p o 0 ,xy is randomlysampled from a square region of size 0 .
Revised: Theobject is initialized as following: xy position of the object’s center of mass (COM) p o 0 ,xy is randomlysampled from a square region of size 0 .

Original: 09 m × 0 .
Revised: 09 m × 0 .

Original: 09 m .
Revised: 09 m .

Original: The center of this square is approximatelylocated on the intersection of the middle ﬁnger and the palm so that the sampling region covers boththe ﬁngers and the palm.
Revised: The center of this square is approximatelylocated on the intersection of the middle ﬁnger and the palm so that the sampling region covers boththe ﬁngers and the palm.

Original: The z position of the object is ﬁxed to 0 .
Revised: The z position of the object is ﬁxed to 0 .

Original: 13 m above the base of the hand toensure
Revised: 13 m above the base of the hand toensure

Original: that the object does not collide with the hand at initialization.
Revised: that the object does not collide with the hand at initialization.

Original: The initial and goal orientationsare randomly sampled from the full SO (3) space.
Revised: The initial and goal orientationsare randomly sampled from the full SO (3) space.

Original: [Figure]
Revised: [Figure]

Original: Setup with the downward facing hand : Next, we consider the cases where the hand faces downward.
Revised: Setup with the downward facing hand : Next, we consider the cases where the hand faces downward.

Original: We experiment with two scenarios: with and without a table below the hand.
Revised: We experiment with two scenarios: with and without a table below the hand.

Original: In the ﬁrst case,we place a table with the tabletop being 0 .
Revised: In the ﬁrst case,we place a table with the tabletop being 0 .

Original: 12 m below the hand base.
Revised: 12 m below the hand base.

Original: We place objects in a randompose between the hand and the table so that the objects will fall onto the table.
Revised: We place objects in a randompose between the hand and the table so that the objects will fall onto the table.

Original: We will describe thesetup for the second case (without a table) in Section 4.2.2.
Revised: We will describe thesetup for the second case (without a table) in Section 4.2.2.

Original: Evaluation criterion : For non-vision experiments, a policy rollout is considered a success if θ ≤ ¯ θ .¯
Revised: Evaluation criterion : For non-vision experiments, a policy rollout is considered a success if θ ≤ ¯ θ .¯

Original: θ = 0 .
Revised: θ = 0 .

Original: 1 rad .
Revised: 1 rad .

Original: For vision experiments, we also check d C ≤ ¯ d C as another criterion and ¯ θ =0 .
Revised: For vision experiments, we also check d C ≤ ¯ d C as another criterion and ¯ θ =0 .

Original: 2 rad , ¯ d C = 0 .
Revised: 2 rad , ¯ d C = 0 .

Original: The initial and goal orientation are randomly sampled from SO (3) space in allthe experiments.
Revised: The initial and goal orientation are randomly sampled from SO (3) space in allthe experiments.

Original: We report performance as the percentage of successful episodes when the agent istasked to reorients all the training objects 100 times from arbitrary start to goal orientation.
Revised: We report performance as the percentage of successful episodes when the agent istasked to reorient all the training objects 100 times from arbitrary start to goal orientation.

Original: We reportthe mean and standard deviation of success rate from 3 seeds.
Revised: We reportthe mean and standard deviation of success rate from 3 seeds.

Original:  Results
Revised:  Results

Original: 4.1 Reorient objects with the hand facing upward
Revised: 4.1 Reorient objects with the hand facing upward

Original: Train a teacher policy with full-state information We train our teacher MLP and RNN policiesusing the full state information on the EGAD and YCB datasets separately.
Revised: Train a teacher policy with full-state information We train our teacher MLP and RNN policiesusing the full state information on the EGAD and YCB datasets separately.

Original: The success rate as afunction of environment steps during training is shown in Figure D.6 in the appendix.
Revised: The success rate as afunction of environment steps during training is shown in Figure D.6 in the appendix.

Original: Figure D.6also shows that using the full-state information can speed up policy learning signiﬁcantly.
Revised: Figure D.6also shows that using the full-state information can speed up policy learning signiﬁcantly.

Original: The testingresults in Table 1 show that both the MLP and RNN policies are able to achieve a success rate greaterthan 90% on the EGAD dataset (entries A1, B1) and greater than 70% on the YCB dataset (entries F1,
Revised: The testingresults in Table 1 show that both the MLP and RNN policies are able to achieve a success rate greaterthan 90% on the EGAD dataset (entries A1, B1) and greater than 70% on the YCB dataset (entries F1,

Original: G1)
Revised: G1)

Original: without any explicit knowledge of the object shape.
Revised: without any explicit knowledge of the object shape.

Original: This result is surprising because intuitively,one would assume that information about the object shape is important for in-hand reorientation.
Revised: This result is surprising because intuitively,one would assume that information about the object shape is important for in-hand reorientation.

Original: The visualization of the policy rollout reveals that the agent is able to reorient different objects withoutknowing their shapes by ﬁnding a clever strategy that is invariant to object geometry.
Revised: The visualization of the policy rollout reveals that the agent is able to reorient different objects withoutknowing their shapes by ﬁnding a clever strategy that is invariant to object geometry.

Original: The agentthrows the object in the air with a spin and catches it at the precise time when the object’s orientationmatches the goal orientation.
Revised: The agentthrows the object in the air with a spin and catches it at the precise time when the object’s orientationmatches the goal orientation.

Original: Throwing the object with a spin is a dexterous spin that automaticallyemerges!
Revised: Throwing the object with a spin is a dexterous spin that automaticallyemerges!

Original: One possibility for the emergence of this skill is that we used very light objects.
Revised: One possibility for the emergence of this skill is that we used very light objects.

Original: Thisis not true because we trained with objects in the range of 50-150g which spans many hand-heldobjects used by humans (e.g., an egg weighs about 50g, a glass cup weighs around 100g, iPhone 12weighs 162g, etc.).
Revised: Thisis not true because we trained with objects in the range of 50-150g which spans many hand-heldobjects used by humans (e.g., an egg weighs about 50g, a glass cup weighs around 100g, iPhone 12weighs 162g, etc.).

Original: To further probe this concern, we evaluated zero-shot performance on objectsweighing up to 500g 2 and found that the learned policy can successfully reorient them.
Revised: To further probe this concern, we evaluated zero-shot performance on objectsweighing up to 500g 2 and found that the learned policy can successfully reorient them.

Original: We providefurther analysis in the appendix showing that forces applied by the hand for such manipulation arerealistic.
Revised: We providefurther analysis in the appendix showing that forces applied by the hand for such manipulation arerealistic.

Original: While there is still room for the possibility that the learned policy is exploiting the simulatorto reorient objects by throwing them in the air, our analysis so far indicates otherwise.
Revised: While there is still room for the possibility that the learned policy is exploiting the simulatorto reorient objects by throwing them in the air, our analysis so far indicates otherwise.

Original: Next, to understand the failure modes, we collected one hundred failed trajectories on YCB datasetand manually documented the reason for failure.
Revised: Next, to understand the failure modes, we collected one hundred failed trajectories on YCB datasetand manually documented the reason for failure.

Original: It turns out that the primary failure is with long,small, or thin objects, which accounts for 60% of the failures.
Revised: It turns out that the primary failure is with long,small, or thin objects, which accounts for 60% of the failures.

Original: In such cases, the objects slip throughthe ﬁngers and fall down, or get stuck on the palm.
Revised: In such cases, the objects slip throughthe ﬁngers and fall down, or get stuck on the palm.

Original: Results in Figure D.9 conﬁrm that sphere-likeobjects such as tennis balls and orange are easiest to reorient.
Revised: Results in Figure D.9 conﬁrm that sphere-likeobjects such as tennis balls and orange are easiest to reorient.

Original: Long or thin objects such as knives andforks are the hardest ones to manipulate.
Revised: Long or thin objects such as knives andforks are the hardest ones to manipulate.

Original: Another cause of failures ( 19% ) is that objects are reorientedclose to the goal orientation but not close enough to satisfy ∆ θ < ¯ θ .
Revised: Another cause of failures ( 19% ) is that objects are reorientedclose to the goal orientation but not close enough to satisfy ∆ θ < ¯ θ .

Original: Finally, we note that theperformance on YCB is lower than EGAD because objects in the YCB dataset have greater diversityin the aspect ratios.
Revised: Finally, we note that theperformance on YCB is lower than EGAD because objects in the YCB dataset have greater diversityin the aspect ratios.

Original: Scaling these objects by constraining l max ∈ [0 . 05 , 0 .
Revised: Scaling these objects by constraining l max ∈ [0 . 05 , 0 .

Original: 12] m (see Section 3) makessome of the objects either too small, too big, or too thin and consequently results in failure (see
Revised: 12] m (see Section 3) makessome of the objects either too small, too big, or too thin and consequently results in failure (see

Original: Figure D.11).
Revised: Figure D.11).

Original: [Table]
Revised: [Table]

Original: Train a student policy with a reduced state space We experimented with the following threecombinations: (1) distill π E M into π S M , (2) distill π E M into π S R , (3) distill π E R into π S R .
Revised: Train a student policy with a reduced state space We experimented with the following threecombinations: (1) distill π E M into π S M , (2) distill π E M into π S R , (3) distill π E R into π S R .

Original: The studentpolicy state is s S t ∈ R 31 .
Revised: The studentpolicy state is s S t ∈ R 31 .

Original: In Table 1 (entries C1-E1, H1-J1), we can see that π E R → π S R gives thehighest success rate on π S , while π E M → π S M leads to much worse performance ( 36% drop of successrate in EGAD, and 47% drop in YCB).
Revised: In Table 1 (entries C1-E1, H1-J1), we can see that π E R → π S R gives thehighest success rate on π S , while π E M → π S M leads to much worse performance ( 36% drop of successrate in EGAD, and 47% drop in YCB).

Original: This shows that π S requires temporal information due toreduced state space.
Revised: This shows that π S requires temporal information due toreduced state space.

Original: The last two columns in Table 1 also show that the policy is more robust todynamics variations and observation/action noise after being trained with domain randomization.
Revised: The last two columns in Table 1 also show that the policy is more robust todynamics variations and observation/action noise after being trained with domain randomization.

Original: 4.2 Reorient objects with the hand facing downward
Revised: 4.2 Reorient objects with the hand facing downward

Original: The results above demonstrate that when the hand faces upwards, RL can be used to train policies forreorienting a diverse set of geometrically different objects.
Revised: The results above demonstrate that when the hand faces upwards, RL can be used to train policies forreorienting a diverse set of geometrically different objects.

Original: A natural question to ask is then does thisstill hold true when the hand is ﬂipped upside down?
Revised: A natural question to ask is then does thisstill hold true when the hand is ﬂipped upside down?

Original: Intuitively, this task is much more challengingbecause the objects will fall immediately without proper control on the ﬁngers, which means theexploration becomes much harder for the RL agent.
Revised: Intuitively, this task is much more challengingbecause the objects will fall immediately without proper control on the ﬁngers, which means theexploration becomes much harder for the RL agent.

Original: To see if this is the case, we experimented withthe following settings: We ﬁrst ﬂip the hand upside down, placed the objects underneath the hand(see Figure 2b), and use the same reward function (Equation (1)) to train the reorientation policy.
Revised: To see if this is the case, we experimented withthe following settings: We ﬁrst ﬂip the hand upside down, placed the objects underneath the hand(see Figure 2b), and use the same reward function (Equation (1)) to train the reorientation policy.

Original: Unsurprisingly, the agent is unable to learn and results in 0% success.
Revised: Unsurprisingly, the agent is unable to learn and results in 0% success.

Original: This is because the policyneeds to learn to both stabilize the object to prevent it from falling under the effect of gravity andsimultaneously reorient it.
Revised: This is because the policyneeds to learn to both stabilize the object to prevent it from falling under the effect of gravity andsimultaneously reorient it.

Original: We found that the application of the method developed until now leads toobject falling down, showing that this is a hard-exploration problem.
Revised: We found that the application of the method developed until now leads toobject falling down, showing that this is a hard-exploration problem.

Original: 4.2.1 Reorient objects on a table
Revised: 4.2.1 Reorient objects on a table

Original: To tackle the problem of reorienting objects with the hand facing downward, we start with a simplerversion of the task where there is a table below the hand (see Figure 2c).
Revised: To tackle the problem of reorienting objects with the hand facing downward, we start with a simplerversion of the task where there is a table below the hand (see Figure 2c).

Original: The presence of the tablemitigates the exploration issue because it prevents the objects from falling.
Revised: The presence of the tablemitigates the exploration issue because it prevents the objects from falling.

Original: We train π E M using thesame reward function Equation (1) on objects sampled from the EGAD and YCB datasets.
Revised: We train π E M using thesame reward function Equation (1) on objects sampled from the EGAD and YCB datasets.

Original: The successrate for the EGAD dataset with an MLP policy using full state information is 95 .
Revised: The successrate for the EGAD dataset with an MLP policy using full state information is 95 .

Original: 3% ± 0 .
Revised: 31% ± 0 .

Original: Thesuccess rate for the YCB dataset is 81 .
Revised: Thesuccess rate for the YCB dataset is 81 .

Original: 59% ± 0 .
Revised: 59% ± 0 .

Original: Using external support for in-hand manipulationhas been a challenging problem in robotics.
Revised: Using external support for in-hand manipulationhas been a challenging problem in robotics.

Original: Recent works approach this problem by buildinganalytical models and constructing motion cones [27].
Revised: Recent works approach this problem by buildinganalytical models and constructing motion cones [27].

Original: However, building the analytical models isespecially difﬁcult for objects with complex geometry.
Revised: However, building the analytical models isespecially difﬁcult for objects with complex geometry.

Original: Our experiments show that model-free RL isable to learn to use the support effectively to reorient many objects with the hand facing downward.
Revised: Our experiments show that model-free RL isable to learn to use the support effectively to reorient many objects with the hand facing downward.

Original: 4.2.2 Reorient objects in air with hand facing downwards
Revised: 4.2.2 Reorient objects in air with hand facing downward

Original: While providing a table can help RL learn a good reorientation policy, we want the skill to be moregeneral.
Revised: While providing a table can help RL learn a good reorientation policy, we want the skill to be moregeneral.

Original: We would like the policy to reorient objects with the hand facing downwards and withoutany external support.
Revised: We would like the policy to reorient objects with the hand facing downwards and withoutany external support.

Original: In this setup, one might hypothesize that object shape information (e.g., fromvision) is critical because ﬁnding the strategy in Section 4.1 is not easy when the hand needs toovercome gravity and stabilize the object while reorienting it.
Revised: In this setup, one might hypothesize that object shape information (e.g., fromvision) is critical because ﬁnding the strategy in Section 4.1 is not easy when the hand needs toovercome gravity and stabilize the object while reorienting it.

Original: We experimentally verify that even inthis case, the policies achieve a reasonably high success rate without any knowledge of object shape.
Revised: We experimentally verify that even inthis case, the policies achieve a reasonably high success rate without any knowledge of object shape.

Original: A good pose initialization is what you need : The difﬁculty of directly training the RL policies whenthe hand faces downward is mainly because of the hard-exploration issue in learning to catch the
Revised: A good pose initialization is what you need : The difﬁculty of directly training the RL policies whenthe hand faces downward is mainly because of the hard-exploration issue in learning to catch the

Original: [Table]
Revised: [Table]

Original: [Table] 
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: objects that are moving downward.
Revised: objects that are moving downward.

Original: However, catching is not necessary for the reorientation.
Revised: However, catching is not necessary for the reorientation.

Original: Even forhuman, we only reorient the object after we grasp it.
Revised: Even forhuman, we only reorient the object after we grasp it.

Original: More speciﬁcally, we ﬁrst train an object-liftingpolicy to lift objects from the table, collect the ending state (joint positions q T , object position p
Revised: More speciﬁcally, we ﬁrst train an object-liftingpolicy to lift objects from the table, collect the ending state (joint positions q T , object position p

Original: oT andorientation α oT ) in each successful lifting rollout episode, and reset the hand and objects to these states(velocities are all 0 ) for the pose initialization in training the reorientation policy.
Revised: oT andorientation α oT ) in each successful lifting rollout episode, and reset the hand and objects to these states(velocities are all 0 ) for the pose initialization in training the reorientation policy.

Original: To train the liftingpolicy, we use the following reward function: r ( s t , a t ) =
Revised: To train the liftingpolicy, we use the following reward function: r ( s t , a t ) =

Original: c h
Revised: c h

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where ∆ h t = max( p b,zt − p o,zt , 0) and p b,zt is the height ( z coordinate) of the Shadow Hand baseframe, p o,zt is the height of the object, ¯ h is the threshold of the height difference.
Revised: 22where ∆ h t = max( p b,zt − p o,zt , 0) and p b,zt is the height ( z coordinate) of the Shadow Hand baseframe, p o,zt is the height of the object, ¯ h is the threshold of the height difference.

Original: In our experiments,c h 1 = 0 .
Revised: In our experiments,c h 1 = 0 .

Original: 05 , (cid:15) h
Revised: 05 , (cid:15) h

Original: = 0 .
Revised: = 0 .

Original: 02 , ¯ h = 0 .
Revised: 02 , ¯ h = 0 .

Original: 04 , c h 2 = 800 , c 3 = 0 .
Revised: 04 , c h 2 = 800 , c 3 = 0 .

Original: The objects have randomly initialized posesand are dropped onto the table.
Revised: The objects have randomly initialized posesand are dropped onto the table.

Original: After we train the lifting policy, we collect about 250 ending states foreach object respectively from the successful lifting episodes.
Revised: After we train the lifting policy, we collect about 250 ending states foreach object respectively from the successful lifting episodes.

Original: In every reset during the reorientationpolicy training, ending states are randomly sampled and used as the initial pose of the ﬁngers andobjects.
Revised: In every reset during the reorientationpolicy training, ending states are randomly sampled and used as the initial pose of the ﬁngers andobjects.

Original: With a good pose initialization, policies are able to learn to reorient objects with high successrates.
Revised: With a good pose initialization, policies are able to learn to reorient objects with high successrates.

Original: π E R trained on EGAD dataset gets a success rate more than 80% while π E R trained on YCBdataset gets a success rate greater than 50% .
Revised: π E R trained on EGAD dataset gets a success rate more than 80% while π E R trained on YCBdataset gets a success rate greater than 50% .

Original: More results on the different policies with and withoutdomain randomization are available in Table [-D.
-] in the appendix.
Revised: More results on the different policies with and withoutdomain randomization are available in Table [-D.
-]D.5 in the appendix.

Original: This setup is challenging becauseif at any time step in an episode the ﬁngers take a bad action, the object will fall.
Revised: This setup is challenging becauseif at any time step in an episode the ﬁngers take a bad action, the object will fall.

Original: Improving performance using gravity curriculum : Since the difﬁculty of training the reorientation policy with the hand facing downward is due to the gravity, we propose to build agravity curriculum to learn the policy π E .
Revised: Improving performance using gravity curriculum : Since the difﬁculty of training the reorientation policy with the hand facing downward is due to the gravity, we propose to build agravity curriculum to learn the policy π E .

Original: Since π E already performs very well on EGAD objects,we apply gravity curriculum to train π E on YCB objects.
Revised: Since π E already performs very well on EGAD objects,we apply gravity curriculum to train π E on YCB objects.

Original: Our gravity curriculum is constructed asfollows: we start the training with g = 1 m / s 2 , then we gradually decrease g in a step-wise fashion ifthe evaluation success rate ( w ) is above a threshold value ( ¯ w ) until g = − 9 .
Revised: Our gravity curriculum is constructed asfollows: we start the training with g = 1 m / s 2 , then we gradually decrease g in a step-wise fashion ifthe evaluation success rate ( w ) is above a threshold value ( ¯ w ) until g = − 9 .

Original: 81 m / s 2 .
Revised: 81 m / s 2 .

Original: More detailsabout gravity curriculum are in Section C.4.
Revised: More detailsabout gravity curriculum are in Section C.4.

Original: In Table D.3 (Exp Q and T) in the appendix, we can seethat adding gravity curriculum ( g -curr) signiﬁcantly boost the success rates on the YCB dataset.
Revised: In Table D.5 (Exp Q and T) in the appendix, we can seethat adding gravity curriculum ( g -curr) signiﬁcantly boost the success rates on the YCB dataset.

Original: 4.3 Zero-shot policy transfer across datasets
Revised: 4.3 Zero-shot policy transfer across datasets

Original: We have shown the testing performance on the same training dataset so far.
Revised: We have shown the testing performance on the same training dataset so far.

Original: How would the policieswork on a different dataset?
Revised: How would the policieswork on a different dataset?

Original: To answer this, we test our policies across datasets: policies trained with
Revised: To answer this, we test our policies across datasets: policies trained with

Original: EGAD objects are now tested with YCB objects and vice versa.
Revised: EGAD objects are now tested with YCB objects and vice versa.

Original: We used the RNN policies trainedwith full-state information and reduced-state information respectively (without domain randomization)and tested them on the other dataset with the hand facing upward and downward.
Revised: We used the RNN policies trainedwith full-state information and reduced-state information respectively (without domain randomization)and tested them on the other dataset with the hand facing upward and downward.

Original: In the case of thehand facing downward, we tested the RNN policy trained with gravity curriculum.
Revised: In the case of thehand facing downward, we tested the RNN policy trained with gravity curriculum.

Original: Table 2 showsthat policies still perform well on the untrained dataset.
Revised: Table 2 showsthat policies still perform well on the untrained dataset.

Original: 4.4 Reorient objects with RGBD sensors
Revised: 4.4 Reorient objects with RGBD sensors

Original: In this section, we investigate whether we can train a vision policy to reorient objects with the handfacing upward.
Revised: In this section, we investigate whether we can train a vision policy to reorient objects with the handfacing upward.

Original: As vision-based experiments require much more compute resources, we train onevision policy for each object individually on six objects shown in Table 3.
Revised: As vision-based experiments require much more compute resources, we train onevision policy for each object individually on six objects shown in Table 3.

Original: We leave training a singlevision policy for all objects to future work.
Revised: We leave training a singlevision policy for all objects to future work.

Original: We use the expert MLP policy trained in Section 4.1 tosupervise the vision policy.
Revised: We use the expert MLP policy trained in Section 4.1 tosupervise the vision policy.

Original: We can see from Table 3 that reorienting the toy has a high success ratewhile reorienting the mug has a much lower success rate.
Revised: We can see from Table 3 that reorienting the toy has a high success ratewhile reorienting the mug has a much lower success rate.

Original: For the mug, the policy reorients the mugto a pose where the body of the mug match how the mug should look like in the goal orientation, butthe cup handle does not match.
Revised: For the mug, the policy reorients the mugto a pose where the body of the mug match how the mug should look like in the goal orientation, butthe cup handle does not match.

Original: This is likely due to the difﬁculty of the network capturing the smallmug handle structure from the entire point cloud.
Revised: This is likely due to the difﬁculty of the network capturing the smallmug handle structure from the entire point cloud.

Original: Using d C as an episode termination criterion, we can train a vision policy for the symmetric objects (cup and legos) individually with high successrates as shown in Table 3.
Revised: Using d C as an episode termination criterion, wecan train a vision policy for the symmetric objects (cup and legos) individually with high successrates as shown in Table 3.

Original:  Related Work
Revised:  Related Work

Original: Dexterous manipulation has been studied for decades, dating back to [28, 29].
Revised: Dexterous manipulation has been studied for decades, dating back to [28, 29].

Original: In contrast to paralleljaw grasping, pushing,  or pick-and-place, dexterous manipulation typically involves continuouslycontrolling force to the object through the ﬁngertips of a robotic hand [30].
Revised: In contrast to paralleljaw grasping, pushing, pivoting [30], or pick-and-place, dexterous manipulation typically involvescontinuously controlling force to the object through the ﬁngertips of a robotic hand [31].

Original: Some prior works usedanalytical kinematics and dynamics models of the hand and object, and used trajectory optimizationto output control policies [1, 2] or employed kinodynamic planning to ﬁnd a feasible motion plan [31].
Revised: Someprior works used analytical kinematics and dynamics models of the hand and object, and usedtrajectory optimization to output control policies [1, 2, 32] or employed kinodynamic planning toﬁnd a feasible motion plan [33].

Original: However, due to the large number of active contacts on the hand and the objects, model simpliﬁcationssuch as simple ﬁnger and object geometries are usually necessary to make the optimization or planningtractable.
Revised: However, due to the large number of active contacts on the hand andthe objects, model simpliﬁcations such as simple ﬁnger and object geometries are usually necessaryto make the optimization or planning tractable.

Revised: Sundaralingam and Hermans [32] moved objects inhand without contact breaking or making between the ﬁngers and the object.

Original: Furukawa et al.
Revised: Furukawa et al.

Original: [32] achieved a high-speed dynamic regrasping motion on a cylinder using ahigh-speed robotic hand and a high-speed vision system.
Revised: [34]achieved a high-speed dynamic regrasping motion on a cylinder using a high-speed robotic handand a high-speed vision system.

Revised: Prior works have also explored the using vision for manipulatingan object to track a path [35], detecting manipulation modes [36], precision manipulation [37] witha limited number of objects with simple shapes using a two-ﬁngered gripper.

Original: Recent works have explored the applicationof reinforcement learning to dexterous manipulation.
Revised: Recent works haveexplored the application of reinforcement learning to dexterous manipulation.

Original: Model-based RL works learned a linear [4, 7]or deep neural network [5] dynamics model from the rollout data, and used online optimal control torotate a pen or Baoding balls on a Shadow hand.
Revised: Model-based RL workslearned a linear [4, 7] or deep neural network [5] dynamics model from the rollout data, and usedonline optimal control to rotate a pen or Baoding balls on a Shadow hand.

Original: However, when the system is unstable, collectinginformative trajectories for training a good dynamics model that generalizes to different objectsremains challenging.
Revised: However, when the systemis unstable, collecting informative trajectories for training a good dynamics model that generalizesto different objects remains challenging.

Original: Another line of works uses model-free RL algorithms to learn a dexterousmanipulation policy.
Revised: Another line of works uses model-free RL algorithms tolearn a dexterous manipulation policy.

Original: For example, Andrychowicz et al.
Revised: For example, Andrychowicz et al.

Original: [11] and Akkaya et al. [12] learned acontroller to reorient a block or a Rubik’s cube.
Revised: [11] and OpenAI et al.[12] learned a controller to reorient a block or a Rubik’s cube.

Revised: Van Hoof et al.

Revised: [38] learned thetactile informed policy via RL for a three-ﬁnger manipulator to move an object on the table.

Original: To reduce the sample complexity of model-freelearning, [9, 13, 8, 33, 6] combined imitation learning with RL to learn to reorient a pen, open adoor, assemble LEGO blocks, etc.
Revised: Toreduce the sample complexity of model-free learning, [9, 13, 8, 39, 6] combined imitation learningwith RL to learn to reorient a pen, open a door, assemble LEGO blocks, etc.

Original: However, collecting expert demonstration data from humansis expensive, time-consuming, and even incredibly difﬁcult for contact-rick tasks [8].
Revised: However, collectingexpert demonstration data from humans is expensive, time-consuming, and even incredibly difﬁcultfor contact-rick tasks [8].

Original: Our methodbelongs to the category of model-free learning.
Revised: Our method belongs to the category of model-free learning.

Original: We use the teacher-student learning paradigm tospeed up the deployment policy learning.
Revised: We use theteacher-student learning paradigm to speed up the deployment policy learning.

Original: Our learned policies also generalize to new shapes andshow strong zero-shot transfer performance.
Revised: Our learned policiesalso generalize to new shapes and show strong zero-shot transfer performance.

Original: To the best of our knowledge, our system is the ﬁrstwork that demonstrates the capabilities of reorienting a diverse set of objects that have complexgeometries with both the hand facing upward and downward.
Revised: To the best of ourknowledge, our system is the ﬁrst work that demonstrates the capabilities of reorienting a diverse setof objects that have complex geometries with both the hand facing upward and downward.

Original:  Discussion
Revised:  Discussion

Original: Our results show that model-free RL with simple deep learning architectures can be used to trainpolicies to reorient a large set of geometrically diverse objects.
Revised: Our results show that model-free RL with simple deep learning architectures can be used to trainpolicies to reorient a large set of geometrically diverse objects.

Original: Further for learning with hand facingdownwards, we found that a good pose initialization obtained from a lifting policy was necessary,and the gravity curriculum substantially improved performance.
Revised: Further for learning with hand facingdownwards, we found that a good pose initialization obtained from a lifting policy was necessary,and the gravity curriculum substantially improved performance.

Original: The agent also learns to use anexternal surface (i.e., the table).
Revised: The agent also learns to use anexternal surface (i.e., the table).

Original: The most surprising observation is that information about shape isnot required despite the fact that we train a single policy to manipulate multiple objects.
Revised: The most surprising observation is that information about shape isnot required despite the fact that we train a single policy to manipulate multiple objects.

Original: Perhapsin hindsight, it is not as surprising – after all, humans can close their eyes and easily manipulatenovel objects into a speciﬁc orientation.
Revised: Perhapsin hindsight, it is not as surprising – after all, humans can close their eyes and easily manipulatenovel objects into a speciﬁc orientation.

Original: However, even in this scenario, shape can be implicitlyinferred using touch signals.
Revised: However, even in this scenario, shape can be implicitlyinferred using touch signals.

Original: As such, our work can serve a strong baseline for future in-hand objectreorientation works that incorporate object shape information as part of the observation space toimprove performance.
Revised: As such, our work can serve a strong baseline for future in-hand objectreorientation works that incorporate object shape information as part of the observation space toimprove performance.

Original: It should be noted that successful training of the teacher policy critically relied on shape informationbeing expendable but required privileged information – a major obstacle in the real-world deployment.
Revised: It should be noted that successful training of the teacher policy critically relied on shape informationbeing expendable but required privileged information – a major obstacle in the real-world deployment.

Original: While we were not able to train reorientation policies from scratch using visual inputs, we showthat we can distill this policy to operate from RGBD point clouds and joint positions that are readilyavailable in the real world.
Revised: While we were not able to train reorientation policies from scratch using visual inputs, we showthat we can distill this policy to operate from RGBD point clouds and joint positions that are readilyavailable in the real world.

Original: While we only present results in simulation, we also provide evidence thatour policies can be transferred to the real world.
Revised: While we only present results in simulation, we also provide evidence thatour policies can be transferred to the real world.

Original: The experiments with domain randomization indicatethat learned policies can work with noisy inputs.
Revised: The experiments with domain randomization indicatethat learned policies can work with noisy inputs.

Original: Analysis of peak torques during manipulation (see
Revised: Analysis of peak torques during manipulation (see

Original: Figure D.12 in the appendix) also reveals that generated torque commands are feasible to actuate onan actual robotic hand.
Revised: Figure D.12 in the appendix) also reveals that generated torque commands are feasible to actuate onan actual robotic hand.

Original: Finally, Figure D.9 and Figure D.10 in the appendix show that the success rate varies substantiallywith object shape.
Revised: Finally, Figure D.9 and Figure D.10 in the appendix show that the success rate varies substantiallywith object shape.

Original: This suggests that in the future, a training curriculum based on object shapes canimprove performance.
Revised: This suggests that in the future, a training curriculum based on object shapes canimprove performance.

Original: Another future work is to directly train one vision policy for a diverse set ofobjects.
Revised: Another future work is to directly train one vision policy for a diverse set ofobjects.

Original: A major bottleneck in vision-based experiments is the demand for much larger GPU memory.
Revised: A major bottleneck in vision-based experiments is the demand for much larger GPU memory.

Original: Learning visual representations of point cloud data that can ease the computational bottleneck is,therefore, an important avenue for future research.
Revised: Learning visual representations of point cloud data that can ease the computational bottleneck is,therefore, an important avenue for future research.