Learning Generalizable Dexterous Manipulation from Human Grasp Affordance
Learned Policy Visualization on Unseen Test Objects
C a m e r a
B o tt l e
C an
M ug
Affordance Demonstrations for Training
Object HumanGrasp
RobotGrasp
Object HumanGrasp
RobotGrasp
Object
Policy Visualization
Abstract: Dexterous manipulation with a multi-finger hand is one of the most challenging problems in robotics.
While recent progress in imitation learning has largely improved the sample efficiency compared to Reinforcement Learning, the learned policy can hardly generalize to manipulate novel objects, given limited expert demonstrations.
In this paper, we propose to learn dexterous manipulation using large-scale demonstrations with diverse 3D objects in a category, which are generated from a human grasp affordance model.
This generalizes the policy to novel object instances within the same category.
To train the policy, we propose a novel imitation learning objective jointly with a geometric representation learning objective using our demonstrations.
By experimenting with relocating diverse objects in simulation, we show that our approach outperforms baselines with a large margin when manipulating novel objects.
We also ablate the importance of 3D object representation learning for manipulation.
We include additional videos and information on the project website - https://kristery.github.io/ILAD/ .
Introduction
Human hands provide the primary means for our daily life interactions with the physical world.
Our hands exhibit tremendous flexibility in operating objects around us.
To enable the robot the same flexibility in assisting humans in daily life, dexterous manipulation with multi-finger robot hands has been one of the core problems in robotics.
At the same time, it is one of the most challenging problems in robotics given its high Degree-of-Freedom joints (e.g., 24 to 30 DoF).
While recent progress in Reinforcement Learning (RL) has shown encouraging results on complex dexterous manipulation [1, 2, 3], it is still limited by the requirement of a large number of samples in training, and the trained policy can hardly generalize to novel objects during deployment.
To improve the sample efficiency in training, one promising direction is to perform imitation learning from human demonstrations [4, 5, 6, 7, 8, 9].
The expert demonstrations for dexterous manipulation can be collected by a human from teleoperation in a Virtual Reality (VR) [5], using Mocap [10] and videos [8].
Guided by human demonstrations, it not only reduces sample complexity in learning but also helps robot hands perform human-like and safe behaviors.
However, the current setup on data collection largely limits the scale and diversity of the demonstrations.
For instance, data collection with VR in [5] only leads to 25 demonstrations per task with one single object instance.
With limited data, the learned policy can hardly generalize and transfer to unseen objects in test time.
Instead of focusing on small scale of data, can we obtain larger scale demonstrations from reasoning the key interactions between hand and objects?
In this paper, we propose to leverage the human grasp affordance model for generalizing dexterous manipulation to novel object instances in the same category.
We will first generate large-scale demonstrations on human hands interacting with diverse objects within the same category from affordance reasoning (left columns in Fig. 1).
Specifically, we can generate a hand grasp pose and a way to contact by leveraging the human grasp affordance model [11].
We utilize motion planning to generate a trajectory that moves the robot hand from a start state to the target grasp.
This trajectory provides a partial demonstration of how the robot hand can reach and stably grasp the object like humans do, preparing for the downstream tasks.
We then use imitation learning to train a policy by augmenting RL with these demonstrations and test on unseen objects (right columns in Fig. 1).
Our policy takes the object point cloud and the robot hand state as inputs for decision making.
We tackle generalization by jointly learning: (i) skill generalization with a new imitation learning objective; and (ii) geometric representation generalization with a behavior cloning objective.
We illustrate these two components of our learning approach as following.
Imitation Learning Objective.
To learn the policy, we augment RL with our generated demonstrations for imitation learning.
Previous approaches weighted all demonstrations equally during learning [5, 7].
In order to take advantage of diverse and large-scale demonstrations, we propose a novel ranking function to encourage the policy to learn from trajectories that it is less likely to reproduce.
In addition, we estimate advantage values for state-action pairs from demonstrations with a growing weight so that the policy can still benefit from the given demonstrations at late training phases.
Geometric Representation Learning.
The policy needs to understand the object shape given the point cloud inputs to manipulate it accordingly.
We utilize PointNet [12] to encode the input object and pre-train the representation with a behavior cloning task using our large-scale demonstrations.
As the policy interacts with the environment during learning, we collect new data to continue fine-tuning the PointNet with behavior cloning.
Our pipeline jointly optimizes the imitation learning objective for skill generalization and the behavior cloning objective for representation generalization.
We perform experiments in simulation with five different object categories.
We train one policy for each object category on the relocate task, which requires the multi-finger robot hand to relocate an object instance from an initial position to a goal location.
We focus on evaluating the metric on generalization to relocate novel objects that are not seen in training.
We not only observe significant improvement over RL and imitation learning baselines but also ablate the effectiveness of our novel imitation learning objective and geometric representation learning using our demonstrations.
We highlight our main contributions as following: (i) A novel approach on generating large-scale dexterous manipulation demonstrations on diverse objects; (ii) A novel imitation learning objective and 3D geometric representation learning approach for generalizing dexterous manipulation; (iii)
Significant improvement over baselines for dexterous manipulation on novel objects.
Related Work
Dexterous Manipulation.
Dexterous manipulation with a multi-finger hand has been one of the core robotics problems [13, 14, 15, 16, 17, 18, 19, 4, 20, 21, 22, 23, 24].
Recent success has been shown in using Reinforcement Learning (RL) for solving complex dexterous manipulation tasks [1, 2].
Extending from this line of research, recent efforts [25, 26] have been made on using model-free RL to generalize diverse object in-hand reorientation.
While these works have shown encouraging results, it is unclear how well it works for grasping and relocating objects, which is the main focus for our work.
We emphasize that both types of manipulation tasks are important problems that have wide applications, but with different focuses .
Sample complexity and unrealistic actions are still challenges for these approaches.
We will compare to this line of approaches in our experiments.
Our work is more related to recent efforts on using affordance and contact reasoning to design an auxiliary loss to guide human-like dexterous grasping [27].
However, the small scale of contact examples limits generalization ability of the policy, and the policy is not required to handle different environment configurations with different goals.
In this paper, we propose a novel method to generate large-scale demonstrations from grasp affordance, and novel imitation learning algorithms for better generalization on a more challenging task setting.
Imitation learning.
Imitation learning aims at recovering the expert policy that generates the given demonstrations.
It includes using behavior cloning [28, 29, 30],
Inverse Reinforcement Learning (IRL) [31, 32, 33, 34, 35, 36, 37, 38], and augmenting expert demonstrations to the online collected data for Reinforcement Learning [39, 40, 41, 5, 7].
Our work falls into the last line of research.
For example, Rajeswaran et al.
[5] propose to take maximum likelihood with demonstrations as an auxiliary term during RL training.
However, they utilize perfect demonstrations collected via
VR at a small scale.
On the other hand, we propose to utilize imperfect partial demonstrations at a large scale for better generalization.
Our work is also related to imitation learning and RL with videos [6, 42, 43, 44, 45, 46].
However, most of these works focus on relatively simple manipulators (i.e., a 2-jaw parallel gripper).
Recent concurrent works have also proposed to collect dexterous manipulation demonstrations from human videos or visual teleoperation [8, 9, 47, 48], however, the scale of collected demonstrations is still much less than what we achieved in this paper.
Visual representation learning for decision making.
Visual inputs provides not only the appearance but also the geometric information of the scene and object.
A lot of recent efforts have been made on learning decision making directly from visual inputs [49, 50, 51, 52, 53, 54, 55, 56].
For example, Mu et al.
[56] propose to train policy directly taking point clouds as inputs for manipulation tasks using a robot arm.
Inspired by these works, our policy takes point clouds as inputs and uses a PointNet [12] to capture the object shape for manipulation.
To improve generalization, we leverage the affordance model on reasoning how humans grasp an object [57, 58, 59, 60, 11, 61, 62] to generate large-scale demonstrations for training the PointNet representation.
Method
We propose to learn a policy using imitation learning with demonstrations generated from the human grasp affordance model.
Our policy takes the object point clouds together with the hand joint states as inputs for decision making.
We introduce our approach I mitation L earning from A ffordance
D emonstration (ILAD) as a 2-stage pipeline:
(i) Affordance Demonstration Generation .
We leverage a state-of-the-art affordance model
GraspCVAE [11] to generate diverse grasps on diverse objects within the same category.
With the generated grasps, we utilize motion planning to obtain trajectories to reach these grasps.
While these trajectories do not show how to perform a particular task, they can serve as partial demonstrations for guiding our policy to achieve the right contacts in grasping
(ii) Imitation Learning with Representation Learning .
We propose a novel imitation learning objective to learn the policy with the affordance demonstrations.
As we utilize a
PointNet [12] en-
coder to
extract
the
3D object shape information
, we propose
a
3D geometric representation
learning approach
jointly with imitation
learning.
[Figure]
3.1 Affordance Demonstration Generation
We propose to generate demonstrations from human grasp affordance in 2 steps as shown in Fig.
Grasp generation.
Given diverse 3D objects, we adopt the pre-trained GraspCVAE model [11] to generate diverse human grasps for each object.
Specifically, the GraspCVAE will take the object point cloud observation as inputs, and outputs the grasp pose represented by the MANO [63] model parameterized by shape parameter β and pose parameter α .
With these parameters, we can compute the human hand joints using the forward kinematics function j h =
J h ( β, α ) .
Motion planning for grasp trajectory.
Given the target grasp hand joints j h , our goal is to find an robot hand action sequence a 1 , ..., a K which generates a robot hand joint sequence j r 0 , ..., j rK so that the last robot hand joint positions j rK reaches j h .
Note the initial robot hand joints j r 0 are given.
The objective for motion planning is min a 1 ,...,a K ∥
j rK − j h ∥ 2 + λ ∥ p K − p 1 ∥ 2 , where pand p K are object poses at time step 1 and K , and constant λ = 10 .
The first term of the objective encourages the robot hand to reach the human grasp, and the second term prevents the object from moving during the process.
We use cross-entropy method (CEM) [64] for motion planning given this objective.
We first sample actions from the Gaussian distribution G ( µ, diag ( σ 2 )) and pick a fixed number of candidates with lower costs.
We update the µ and diag( σ 2 ) with these candidates and repeat iteratively and use model predictive control (MPC) to execute the planned trajectories until the objective below a threshold δ .
In the following subsections, we will first introduce our novel imitation learning objective using these demonstrations, and then the policy training pipeline using both the imitation objective and a geometric representation learning objective to enhance the generalizability of the learned policy.
3.2 Imitation Learning Objective
We perform imitation learning using demonstrations generated from our planning algorithm, under a setting where a reward function for Reinforcement Learning and demonstrations are both given.
Preliminaries.
We first consider a standard Markov Decision Process (MDP).
It is represented by a tuple ⟨ S, A, P, R, γ ⟩ , where S and A are state and action space, P ( s t +1 | s t , a t ) is the transition density of state s t +1 at step t + 1 given action a t made under state s t , R
( s, a ) is the reward, and γ is the discount factor.
The goal of RL is to maximize the expected reward with a policy π ( a | s ) .
We build our approach upon an imitation learning baseline algorithm called Demo Augmented Policy Gradient (DAPG)
The learning objective function at epoch k can be represented as,
[Equation]
where A π θ is the advantage function [65] that is used to estimate the difference of the discounted reward sum starting from ( s, a ) and s according to policy π θ , D π E are state-action pairs from the expert demonstrations, D π πθ are state-action pairs collected with policy π θ , and λ 0 and λ 1 hyperparameters.
In the implementation of [5], max ( s,a ) ∈ D
πθ A π θ ( s, a ) is set to 1 for stability.
Learning from partial demonstrations with multiple objects.
For generalizing dexterous manipulation to multiple objects, where there are easier and more challenging shapes, the demonstrations
[Figure]
Cloning
[Figure]
Pre
[Figure]
DemonstrationsTraining:
[Figure]
should not be taken equally during training.
We propose to adaptively rank the demonstrations based on the difficulties of objects and the learning progress of the policy with the objective, g
[Equation]
where w k ( s, a ) in the second term is computed as the negative of a normalized log likelihood to encourage the policy to learn from trajectories that it is hard to reproduce by the current policy.
A π θ ϕ ( s, a ) in the third term is an advantage function estimated with a model parameterized by ϕ for state-action pairs of the demonstrations.
Formally, w k ( s, a ) can be represented as a normalized value (scaled between 0 and 1) of the negative of the log likelihood w k ( s, a ) =
[Figure]
[Equation]
where τ s,a is a trajectory from the demonstrations that contains state-action pair ( s, a ) , and the negative of the log likelihood for a trajectory l k ( τ ) = − 1 | τ | (cid:80) ( s,a ) ∈ τ log Pr( s, a | π θ ) .
We explain our key innovations on the second term and the third term in Eq.
1 as following.
Normalized likelihood weights in the second term in Eq.
In our experiments, we find that the policy could easily learn to manipulate a certain kind of object while neglecting the others.
To encourage the policy to generalize on diverse objects, we dynamically assign larger weights w k ( s, a ) for demonstrations that have a smaller likelihood in the current epoch.
Advantage approximation for demonstrations in the third term in Eq.
1 is designed to further elevate the utilization of demonstrations.
While in the previous approach [5] the advantage function for demonstrations is taken as 1 as mentioned, we approximate the true advantage for a more accurate estimation of the gradients.
We train a neural network
A π θ ϕ ( s, a ) to predict the advantage function.
This new advantage function is trained with data collected by RL and applied to the partial demonstrations.
We set λ ′
= 0 .
1 λ 0 across all experiments to prevent excessive parameter tuning.
3.3 Policy Training with Geometric Representation
Our policy takes both the point cloud of the object, object 6D pose, robot hand joint states as inputs, and predicts the actions.
Specifically, to represent the object shape, we utilize the PointNet [25] encoder θ pc for the point cloud inputs.
Given the point cloud embedding, we concatenate it with the object 6D pose parameters and hand joint states together and forward them together to a 3-layer
MLP θ p network for decision making.
Thus the policy network is parameterized by θ = { θ pc , θ p } .
To perform training, besides optimizing towards the imitation learning objective, we design a geometric representation learning objective for training the PointNet jointly.
Our overall model architecture and training pipeline are visualized in Fig.
Behavior cloning for geometric representation learning.
We utilize behavior cloning to provide an objective to train our PointNet encoder.
We obtain the training data directly from the examples D π θ collected during the interaction with the environment in policy learning.
Specifically, the behavior cloning objective can be represented as,
[Equation]
where we still utilize the whole network θ = { θ pc , θ p } including the decision making MLP θ p to compute the loss, we only optimize the PointNet parameters θ pc through backpropagation.
This part of training corresponds to the upper part of Fig.
Pre-training.
The same objective Eq.
2 can also be used to pre-train the PointNet encoder and policy network before policy learning with demonstrations D π E .
1: Input: partial demonstrations D π E , PointNet θ pc , policy network θ p 2: Pre-train θ pc and θ p , according to Eq.3: for t = 0 , 1 , 2 , ... do 4:
Sample trajectories
D π θ = { ( s i , a i ) } ni
= 5: if t ≡ 0 (mod T ) then 6:
Update θ pc , according to Eq.7: end if 8:
Update θ p , according to Eq.9: end for
Joint learning with both objectives.
We train our policy jointly with both the imitation learning objective and the behavior cloning objective as shown in
Fig.
Empirically, we find training the PointNet with policy gradient in RL (Eq. 1) makes the representation unstable for decision making.
The variance of gradients is much larger in RL and very challenging to learn an encoder with high-dimensional inputs directly [51, 52].
Thus, we propose to share the network parameters θ = { θ pc , θ p } for both objectives, but use policy gradient to optimize the decision making MLP θ p and behavior cloning to optimize the
PointNet encoder θ pc .
That is, the gradients of Eq.becomes with respect to θ p , instead of θ .
For better stability, we perform slower updates on the PointNet encoder so that the decision-making learns from similar representations over time.
Specifically, while we update the MLP θ p for every epoch using policy gradients, we perform an update on the PointNet encoder θ pc every T epochs using behavior cloning.
In Fig.
3, arrows with different colors represent different update strategies.
We summarize our learning procedure in Algorithm 1.
Experiments 4.1 Experiment and Comparison Settings.
We conduct experiments on Relocate task with five categories: bottle, remote, mug, can, and camera.
In this task, an object is placed on a table with random orientation and position and the robot is required to grasp the object and move it to a random target position.
For each category, we use 40 objects for training and about 30 unseen objects (differ by category) for testing to evaluate the generalizability.
The unseen objects did not appear during training but are within the same category as training objects.
Both training and testing objects are from ShapeNet [66].
There are two settings of the demonstration size: 100 and 1000 for each category.
Baselines and Compared Methods.
We adopt TRPO [67] as our RL baseline.
We also perform comparison to DAPG [5].
For fair comparison, we use the same PointNet pre-training method as
ILAD and use it to process point clouds inputs for DAPG, namely DAPG+PC .
Since the code of [26] is not released and it is using a different simulator, we construct a baseline using RL with pointcloud inputs as an approximation.
We add the joint learning of PointNet for this baseline, namely RL+PC where PC represents “PointCloud”.
Note the demonstrations are not adopted in this baseline.
We use
[Table] [Table]
the same RL algorithm (TRPO) with the same hyper-parameters for all approaches.
We use the same
PointNet encoder architecture as our method for both DAPG+PC and RL+PC.
The performance is evaluated with five individual random seeds and the seeds are the same across all comparisons.
4.2 Main Comparisons
Success rate.
We perform comparisons using both small and large number of demonstrations.
We use “(large)” to denote training with large scale number of demonstrations, and small number of demonstrations otherwise.
The results are presented in terms of success rate on unseen objects that are not shown during training in Tab. 1.
The performance is evaluated via 100 trials for five random seeds.
A trial is counted as a success when the final object position is within 0 .
1 unit length to the target.
Both the initial object position and target position are randomized.
Tab. 1 shows that ILAD outperforms baselines with a large margin.
The RL baseline only achieves an average success rate of 16%.
While ILAD achieves an average success rate of 96%, which is about 43% improvement over DAPG+PC.
At the same time, a larger number of demonstrations always performs better for both DAPG+PC and ILAD.
This indicates the importance of our demonstration generation method which enables automatic large-scale demonstrations.
Tab. 1 also shows that the RL+PC baseline (approximation of [26]) fails to benefit from using point cloud inputs, as it achieves similar or worse results compared to pure RL.
We attribute it to the fact that the relocate task for diverse objects is hard to learn without demonstrations, and the low learning efficiency of RL is insufficient to collect informative data for training the 3D representation.
It validates that the performance of representation learning relies on the quality of datasets [68, 69].
Joint-learning updating interval T .
We use different learning intervals T = 20 and T = 50 with large number of demonstrations (Tab. 1).
Our method is robust to the choice of T as we observe there are only small differences between them.
Learning with larger T achieves slightly better results as the representations change more smoothly for more stable RL training.
4.3 Ablation Study
Generalization.
We summarize the success rate on unseen objects under the same category in
Tab. 2.
We show ILAD achieves much better results than DAPG+PC and using our new imitation objectives with λ ′ 0 is essential.
The joint learning significantly enhances the success rate for unseen objects from 67% to 89% on average.
This suggests that updating the encoded point cloud from behavior cloning objective in Eq. 2 greatly alleviates the overfitting on the training objects .
We show the average return of all the methods trained with five different random seeds using a small number ( 100 per category) of demonstrations in Fig.
The x-axis is the update iterations during training and the average return of the y-axis is normalized to the same range for all five categories.
1500 Remote 0.0 0.
A v e r age
R e t u r n   1500 Bottle 0.0 0.
1500 Can 0.0 0.
1500 Mug 0.0 0.
1500 Camera 0.0 0.
ILAD
ILAD (w/o J L )
ILAD (w/o J L and λ 0 0 = 0 )
DAPG+PC
RL
RL+PC
[Table]
[Table] [Table]
ing demonstrations.
We emphasize Fig.
4 reports results on training objects , and both Tab. 1 and Tab. 2 report results on unseen objects .
While all methods except for RL perform competitively in terms of average return from training environments in Fig.
4, fitting the same on training environments do not necessarily mean they can all transfer well in unseen objects.
Among these methods, Tab. 1 and Tab. 2 both show that ILAD is able to achieve much better results on unseen objects compared to other approaches.
This comparison indicates the superior generalization power of our method on unseen objects .
Ablation on demonstration quality.
We ablate the performance of the policy on relocating bottles depending on different demonstration generation approaches in Tab. 3.
We ablate threshold δ and approach with and without hand grasp information.
For the approach without hand grasp pose, we minimize the distance between the robot palm hand and the object during planning.
With the same threshold δ , the demonstrations without hand grasp pose fail to provide sufficient information for imitation learning, indicating the importance of modeling hand-object interactions .
A smaller threshold δ indicates that the final states of the generated demonstration are closer to the desired grasp poses.
Tab.
3 shows that the performance will drop without precise planning for grasps.
This provides evidence that the generated demonstrations do not only facilitate the learning of the reaching phase but also give the agent a better instruction to grasp objects of different shapes.
We also compare the proposed sampling method with Rapidly-exploring random trees (RRT) and we leave the results to supplementary materials due to the limited space.
User study on naturalness.
We compare demonstrations generated with and without hand grasp from the affordance model and perform a user study on Amazon Mechanical Turk.
We ask users to choose which demonstrations look more natural and the results are shown in Tab. 4.
The survey suggests that 71% users would choose our demonstrations as more natural grasping demonstrations.
Along with the ablation study in Tab. 3, it shows that naturalness is indeed a key factor to allow the policy to acquire a higher success rate on unseen objects.
Conclusion and limitation
Conclusion.
We propose ILAD to generalize dexterous manipulation with point cloud inputs.
To achieve generalization on diverse unseen objects, we introduce a novel approach to generate largescale demonstrations and new imitation learning methods.
We believe such ability of generalization is also essential for Sim2Real transfer for real robot dexterous hand manipulation.
We are committed to releasing our code, environments, and dataset .
Limitation.
While this paper has not include real robot experiments, we can perform randomization on object point clouds to simulate real-world occlusions, which can potentially lead to direct
Sim2Real deployment of our method.
The main focus of this work is unseen object generalization, and we will conduct the real-robot extension as our future work.
Acknowledgments
This work was supported, in part, by grants from NSF CCF-2112665 (TILOS), NSF 1730158 CINew: Cognitive Hardware and Software Ecosystem Community Infrastructure (CHASE-CI), NSF
ACI-1541349 CC*DNI Pacific Research Platform, the Industrial Technology Innovation Program (20018112, Development of autonomous manipulation and gripping technology using imitation learning based on visualtactile sensing) funded by the Ministry of Trade, Industry and Energy of the Republic of Korea and gifts from Meta, Google, Qualcomm.