Goal-Auxiliary Actor-Critic for 6D Robotic Grasping with Point Clouds
Abstract: 6D robotic grasping beyond top-down bin-picking scenarios is a challenging task.
Previous solutions based on 6D grasp synthesis with robot motion planning usually operate in an open-loop setting, which are sensitive to grasp synthesis errors.
In this work, we propose a new method for learning closed-loop control policies for 6D grasping.
Our policy takes a segmented point cloud of an object from an egocentric camera as input, and outputs continuous 6D control actions of the robot gripper for grasping the object.
We combine imitation learning and reinforcement learning and introduce a goal-auxiliary actor-critic algorithm for policy learning.
We demonstrate that our learned policy can be integrated into a tabletop 6D grasping system and a human-robot handover system to improve the grasping performance of unseen objects.
Videos and code are available at https://sites.google.com/view/gaddpg .
Introduction
Robotic grasping of arbitrary objects is a challenging task.
A robot needs to deal with objects it has never seen before, and generates a motion trajectory to grasp an object.
Due to the complexity of the problem, majority works in the literature focus on bin-picking tasks, where top-down grasping is sufﬁcient to pick up an object.
Both grasp detection approaches [1, 2, 3] and reinforcement learning-based methods [4, 5] are introduced to tackle the top-down grasping problem.
However, it is difﬁcult for these methods to grasp objects in environments where 6D grasping is necessary, i e., 3D translation and 3D rotation of the robot gripper, such as a cereal box on a tabletop or in a cabinet.
While 6D grasp synthesis has been studied using 3D models of objects [6, 7] and partial observations [8, 9, 10], these methods only generate 6D grasp poses of the robot gripper for an object, instead of generating a trajectory to reach and grasp the object.
As a result, a motion planner is needed to plan the grasping motion according to the grasp poses.
Usually, the planned trajectory is executed in an open-loop fashion since re-planning is expensive, and perception feedback during grasping as well as dynamics and contacts of the object are often ignored, which makes the grasping sensitive to grasp synthesis errors.
In this work, to overcome the limitations of open-loop 6D grasping based on grasp synthesis, we introduce a new method for learning closed-loop 6D grasping polices from partially-observed point clouds of objects (Fig. 1).
Our policy takes a segmented point cloud of an object as input and directly outputs the control action of the robot gripper, which is the relative 6D pose transformation of the gripper.
Our policy can be combined with different perception methods for object segmentation.
In order to learn the policy, we combine Imitation Learning (IL) and Reinforcement Learning (RL).
Because the chance of lifting an object in 6D grasping is very rare if only RL is used for exploration.
We obtain demonstrations using a joint motion and grasp planner [11] in simulation.
Consequently, we can efﬁciently obtain a large number of 6D grasping trajectories using ShapeNet objects [12] with the planner.
Then we learn the grasping policy based on the Deep Deterministic Policy Gradient (DDPG) algorithm [13], which is an actor-critic algorithm in RL that can utilize off-policy data from demonstrations.
More importantly, we introduce a goal prediction auxiliary task to improve the policy learning, where the actor and the critic in DDPG are trained to predict the ﬁnal 6D grasping 5th Conference on Robot Learning (CoRL 2021), London, UK.
[Figure]
pose of the robot gripper as well.
The supervision on goal prediction comes from the expert planner for objects with known 3D shape and pose.
For unknown objects without 3D models available, we can still obtain the grasping goals from successful grasping rollouts of the policy, i.e., hindsight goals.
This property enables our learned policy to be ﬁne-tuned on unknown objects, which is critical for continual learning in the real world.
We see that adding the auxiliary losses stabilizes training and improves the learned policy.
Overall, our contributions are: 1) We introduce a Goal-Auxiliary DDPG (GA-DDPG) algorithm for joint imitation and reinforcement learning of 6D robotic grasping.
We demonstrate that our learned policy can be integrated into a tabletop 6D grasping system to improve grasping performance.
We demonstrate that our closed-loop policy can improve human-to-robot handover.
Related Work
Vision-based Robotic Grasping.
Grasp synthesis can be used in a planning and control pipeline for robotic grasping [2, 3, 14, 15].
Alternatively, end-to-end policy learning methods [16, 4, 17] make use of large-scale data to learn closed-loop vision-based grasping.
Although RGB images are widely used as the state representation, it requires the policy to infer 3D information from 2D images.
Recently, depth and segmentation masks [3], shape completion [9], deep visual features [18], keypoints [19], point clouds [20], multiple cameras [21], and egocentric views [22, 23] have been considered to improve the state representation.
Motivated by these methods, we utilize point clouds of objects as our state representation, which transfers well from simulation to the real world.
Combining Imitation Learning and Reinforcement Learning.
Model-free RL [4, 5] requires a large number of interactions even with full-state information.
Therefore, many works have proposed to combine imitation learning [24] in RL.
[25] augments policy gradient update with demonstration data to circumvent reward shaping and the exploration challenge.
[26] uses inverse RL to learn dexterous manipulation tasks with a few human demonstrations in simulation.
The closest related works to ours are [27, 28] that utilize demonstration data with off-policy RL.
Despite the focus on different tasks, the main difference is that our demonstrations come from an expert planner instead of human demonstrators.
Thus, we can obtain a large number of consistent demonstrations and query the expert planner during training to provide supervision on the on-policy data.
Goals and Auxiliary Tasks in Policy Learning.
Goals are often used as extra signals to guide policy learning [29].
For goal-conditioned policies, [30, 31, 32] make the observation that every trajectory is a successful demonstration of the goal state that it reaches, thereby these methods relabel goals in rollout trajectories for effective learning.
However, goals still need to be provided in testing.
In 6D grasping, estimating the grasping goals is a challenging problem.
On the other hand, auxiliary tasks have been used to improve RL as well [33, 34, 35, 36].
We utilize the grasping goal prediction as an auxiliary task that requires the policy to predict how to grasp the target object.
Learning 6D Grasping Policies
Our goal is to learn a closed-loop policy for 6D robotic grasping.
We ﬁrst introduce related background knowledge and then present our method for learning the policy.
3.1 Background
Markov Decision Process.
A Markov Decision Process (MDP) is deﬁned using the tuple: M = {S , R , A , O , P , ρ 0 , γ } .
S , A , and O represent the state, action, and observation space.
R : S×A → R is the reward function.
P : S×A → S is the transition dynamics.
ρ 0 is the probability distribution over initial states and γ =
[0 , 1) is a discount factor.
Let π : S
→ A be a policy which maps states to actions.
In the partially observable case, at each time t , the policy maps a partial observation o t of the environment to an action a t =
π ( o t ) .
Our goal is to learn a policy that maximizes the expected cumulative rewards
E π [ (cid:80) ∞ t =0 γ t r t ] , where r t is the reward at time t .
The Q-function of the policy for a state-action pair is Q ( s, a ) =
R ( s, a ) + γ E s (cid:48) ,π [ (cid:80) ∞ t =0 γ t r t | s
= s (cid:48) ] , where s (cid:48) represents the next state of taking action a in state s according to the transition dynamics.
Deep Deterministic Policy Gradient.
DDPG [13] is an actor-critic algorithm that uses off-policy data, deterministic policy gradient, and temporal difference learning.
It has successful applications in continuous control [37, 27].
Speciﬁcally, the actor in DDPG learns the policy π θ ( s ) , while the critic approximates the Q-function Q φ ( s, a ) , where θ and φ denote the parameters of the actor and the critic, respectively.
A replay buffer of transitions D
= { ( s, a, r, s (cid:48) ) } is maintained during training, and examples sampled from it are used to optimize the actor and critic alternately.
DDPG minimizes the following Bellman error with respect to φ :
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Then the deterministic policy π θ is trained to maximize the learned Q-function with max θ E s ∼D ( Q φ ( s, π θ ( s ))) , which resembles a policy evaluation and improvement schedule.
3.2 6D Grasping Policy From Point Clouds
We consider the task of grasping an arbitrary object in a closed-loop manner.
Our goal is to learn a 6D grasping policy π : s t → a t that maps the state s t at time t to an action a t .
To simplify the setting, we still use s to represent the input of the policy even though observations are used.
At time t , the action a t is parametrized by the relative 3D translation and the 3D rotation of the robot endeffector.
Therefore, our learned visuomotor policy represents an operational space controller for 6D grasping.
We utilize 3D point clouds of objects to represent the states, which can be computed using depth images and foreground masks of the objects.
These points are transformed from the camera frame to the robot end-effector frame to represent the state.
Different segmentation methods can be used to obtain a foreground mask of a target object.
Our learned policy is robust to noises in the segmentation.
For grasping static objects, we aggregate the point clouds from previous time steps as described in Appendix Section 6.2.
For dynamic objects, no aggregation is used.
3.3 Demonstrations from a Joint Motion and Grasp Planner
To obtain demonstrations for 6D grasping, we utilize the Optimization-based Motion and Grasp
Planner (OMG Planner)
[11] as our expert.
Given a planning scene and a set of pre-deﬁned grasps on a target object from a grasp planner [6, 7], the OMG Planner plans a trajectory of the robot to grasp the object.
Depending on the initial conﬁguration of the robot, it selects different grasps as the goal.
This property enables us to learn a policy that grasps objects in different ways according to the initial pose of the robot gripper.
Let ξ =
( T 0 , T 1 , . .
. , T T ) be a trajectory of the robot gripper pose generated from the OMG planner to grasp an object, where T t ∈ SE (3) is the gripper pose in the robot base frame at time t .
Then the expert action at time t can be computed as a ∗ t =
T t +1 T − 1 t , which is the relative transformation of the gripper between t and t + 1 .
Furthermore, T T from the planner is a successful grasp pose of the object.
We deﬁne the expert goal at time t as g ∗ t =
T T T − 1 t , which is the relative transformation between the current gripper pose and the ﬁnal grasp.
Finally, the state s t can be obtained by computing the point cloud as in Section 3.2.
In this way, we construct an ofﬂine dataset of demonstrations from the planner as D expert = { ( s t , a ∗ t , g ∗ t , s (cid:48) t ) } , where s (cid:48) t is the next state.
3.4 Behavior Cloning and DAGGER
Using the demonstration dataset D expert , we can already train a policy network π θ by applying Behavior Cloning (BC) to predict an action from a state.
To minimize the distance with expert actions,
[Figure]
different loss functions on SE (3) can be used.
In particular, we adopt the point matching loss function as deﬁned in [38] to jointly optimize translation and rotation:
[Equation]
where T 1 , T 2 ∈ SE (3) , X g denotes a set of pre-deﬁned 3D points on the robot gripper, and L1 norm is used.
Let a θ
= π θ ( s ) be an action predicted from the policy and a ∗ =
π ∗ ( s ) be an action from the expert, the behavior cloning loss is L BC ( a ∗ , a θ ) =
L POSE ( a ∗ , a θ ) .
We apply DAGGER [39] to augment supervisions for rollouts from the learner.
Given a sampled initial state s 0 , the current policy π θ can roll out a trajectory of states s 0 , ..., s T and actions a 0 , ..., a T with end-effector poses ˆ T 0 , ..., ˆ T T .
The expert then treats each state as the initial state and generates a trajectory to grasp the object ξ ( s t ) =
( ˆ T t , T t +1 , ..., T T ) .
The ﬁrst action in the plan a ∗ t = π ∗ ( s t ) =
T t +1 ˆ T − 1 t is used as supervision to correct deviations of the learner from ξ ( s t ) .
A corresponding expert goal g ∗ t =
T T ˆ T − 1 t can also be obtained for state s t .
These action and goal labels are used to construct a dataset D dagger =
{ ( s t , a t , a ∗ t , g ∗ t , s (cid:48) t ) } .
3.5 Goal-Auxiliary DDPG
To handle contact-rich scenarios that are rare in the expert trajectories, we further improve the policy learning by combining RL with expert demonstrations.
In our RL setting, each episode ends when the agent attempts a grasp or reaches a maximum horizon T .
The sparse reward r is an indicator function given at the end of the episode denoting if the target object is lifted or not.
Similar to DAGGER, we collect on-policy data for DDPG training.
The only difference is that we do not have expert demonstration for the action a ∗ t and goal g ∗ t for a state s t from the learner policy.
Instead, we ﬁrst ﬁnd the nearest goal from a goal set G by (cid:101) g t =
arg min g ∈G L POSE ( g, T t ) , where
T t is the pose of the robot gripper at time t .
Then the heuristic goal for s t is g t =
(cid:101) g t T − 1 t .
Note that the goal set G is only available if we have the 3D shape and pose of the target object.
Finally, we construct a dataset for DDPG D ddpg = { ( s t , a t , g t , r t , s (cid:48) t ) } , where r t is the reward at state s t and s (cid:48) t is the next state of s t from the on-policy rollout.
The replay buffer of DDPG training is
D = D expert ∪ D dagger ∪ D ddpg , where we augment D expert and D dagger with the sparse reward.
Our network architecture for training the actor-critic is shown in Fig.
We introduce two auxiliary tasks for the actor-critic to predict goals, which help to learn the policy and the Q-function by adding goal supervision.
Given a data sample ( s, a, g, r, s (cid:48) ) from the replay buffer, the critic loss is
[Equation]
where y =
r + γQ φ (cid:48) (cid:0) s (cid:48) , π θ (cid:48)
( s (cid:48) )
+ (cid:15) (cid:1) is the Bellman target and g φ is the predicted goal.
Q φ (cid:48) and π θ (cid:48) are the target networks and (cid:15) is a pre-deﬁned clipped noise as in TD3 [41].
The auxiliary loss is
L AUX ( g, g φ ) =
L POSE ( g, g φ ) for measuring pose errors in goal prediction.
Given a data sample ( s, a ∗ , g ) from the replay buffer, the loss function for the actor is deﬁned as
[Equation]
where a θ and g θ are the action and the goal predicted from the actor, respectively, and λ is a weighting hyper-parameter to balance the losses from the expert and from the learned value function.
The BC loss L BC ( a ∗ , a θ ) =
L POSE ( a ∗ , a θ ) as deﬁned before, which prevents the learner from moving too far away from the expert policy.
The deterministic policy loss is deﬁned as
L DDPG ( s, a θ ) = − Q φ ( s, a θ ) in order to maximize Q values.
Note that the expert action a ∗ is only available if the data sample is from D expert or D dagger .
Otherwise, we do not include the BC loss in Eq.
Even though our method can be used with any off-policy actor-critic algorithms, in practice, we choose TD3 for its improved performance over vanilla DDPG.
3.6 Hindsight Goals For Fine-tuning on Unknown Objects
For unknown objects without 3D models, we cannot obtain neither expert demonstrations nor grasping goals from a pre-deﬁned goal set.
We can use hindsight goals to ﬁne-tune our policy on unknown objects.
Based on the current policy π θ , each learner episode rolls out a trajectory of state-actions with end-effector poses ˆ T 0 , ..., ˆ T T .
If the grasp in the episode is successful, we know that ˆ T T is a success grasp for the target object, which is also known as the hindsight goal.
Then we can compute the grasp goal for state s t in the episode as ˆ g t
= ˆ T T
ˆ T − 1 t , which is used to supervise the actor and the critic via the goal auxiliary tasks.
By using hindsight goals, we construct a dataset using on-policy rollouts on unknown objects D hindsight =
{ ( s t , a t , ˆ g t , r t , s (cid:48) t ) } , where ˆ g t is the hindsight goal.
We ﬁnetune the pretrained actor-critic network on the dataset D = D expert ∪ D hindsight .
The algorithm for training with hindsight goals can be found in Appendix Section 6.1.
Experiments
Simulation Environment.
We experiment with the Franka Emika Panda arm, a 7 -DOF arm with a parallel gripper.
We use ShapeNet [12] and YCB objects [42] as our object repository.
A task scene is generated by dropping a sampled target object with a random pose on a table in the PyBullet
Simulator [43].
The maximum horizon for the policy is T = 30 .
An episode is terminated once a grasp is completed.
The observation image size is 112 × 112 in PyBullet using a hand camera.
More environment details can be found in Appendix Section 6.2.
Training and Testing.
We use approximately 1 , 500 ShapeNet objects from 169 different classes in the ACRONYM dataset [44] for training, where each object has 100 pre-computed grasps from [7] for planning.
For testing, we use 9 selected YCB objects with 10 scenes per object andholdout
ShapeNet objects within 157 scenes.
We train each model three times with different random seeds.
We run each YCB scene 5 times and each ShapeNet scene 3 times and then compute the mean grasping success rate.
More details on network architecture and training can be found in Appendix
Section 6.3 and Appendix Section 6.4, respectively.
4.1 Ablation Studies in Simulation
Ablations on State Representations.
We ﬁrst conduct experiments with BC to investigate the effect of using different state representations.
Table 1 presents the grasping success rates.
When using images, there are three variations: “RGB”, “RGBD” and “RGBDM” indicating whether depth (D) or foreground mask (M) of the object are used or not.
Depth and/or mask are concatenated with the
RGB image, and ResNet18 [45] is used for feature extraction.
“Point” in Table 1 indicates the point cloud state representation.
“Ofﬂine” means a ﬁxed-size ofﬂine dataset D expert is used for training which contains 50,000 data points from expert demonstrations.
“Online” means the expert planner is running in parallel with training, which keeps adding new data to the dataset.
“DAGGER” uses on-policy rollout data from the learner policy with expert supervision D dagger , and “Goal-Auxiliary” indicates whether we add the grasping goal prediction task or not in BC training.
From Table 1, we can see that: i) using the point clouds achieves better performance compared to using images of the current view.
This indicates that the 3D features in object point clouds generalizes better for 6D grasping.
ii)
Adding depth or foreground mask to the image-based representation improves the performance.
iii) “Online” is overall better than “Ofﬂine” by utilizing more data for training.
iv) Both DAGGER and adding the auxiliary task improve success rates.
Ablations on Goal-auxiliary vs. Goal-conditioned.
We evaluate our goal-auxiliary DDPG algorithm for 6D grasping, where the dataset D = D expert ∪ D dagger ∪ D ddpg is constructed for training.
We also consider another common strategy of using goals for comparison: goal-conditioned policies such as in [30, 28, 31], where a goal is concatenated with the state as the input for the network.
In this case, in order to use goals in testing, we need to train a separate network to predict goals from states.
Table 2 displays the evaluation results, where we test different combinations of adding the goal-auxiliary task or using the goal-conditioned input to the actor and the critic.
[Table] [Table]
From Table 2, we can see that: i) using the goal-auxiliary loss for the critic signiﬁcantly improves the performance since it regularizes the Q-learning.
ii)
The goal-conditioned policies perform worse than the goal-auxiliary counterparts.
When the predicted goal is not accurate, it affects the grasping success.
We also observe some instability in goal-conditioned policy training.
iii)
Adding the goalauxiliary task to both the actor and the critic achieves the best performance.
iv) Comparing to the supervised BC training in Table 1, our GA-DDPG algorithm further improves the policy, especially in the contact-rich scenarios.
Imitation learning alone cannot handle these contacts with the target object before closing the ﬁngers since these are rarely seen in the expert demonstrations.
Ablations on Fine-tuning with Hindsight Goals.
Our policy trained on ShapetNet objects achieves 91.3% and 88.2% success rate for grasping unseen ShapeNet and unseen YCB objects, respectively, as shown in Table 2.
We can further improve the policy using RL with unseen objects.
To resemble real-world RL, we assume that the 3D shape and pose of the YCB objects are not available.
Therefore, there is no expert supervision on these objects.
We ﬁne-tune the policy using the dataset
D = D expert ∪D hindsight with hindsight goals.
We see that the success rate is improved from 88.2% to 93.5% after ﬁne-tuning.
We also experiment with the simulation dynamics by decreasing the lateral frictions of the object, which makes object easier to slide during contacts.
In this case, the success rate of the pre-trained policy reduces to 81.6%.
After ﬁne-tuning with hindsight goals, the policy adapts well and achieves 88.5% success rate, which demonstrates the robustness of the training procedure to contact model errors.
Ablations on Other Design Choices.
We conduct ablation studies on several design choices in
GA-DDPG as shown in Table 3.
We can see that: i) RL without BC failed to learn a useful policy due to the high-dimensional state-action space and sparse reward.
ii) Both online interaction for adding data and DAGGER help.
iii)
Our aggregated point cloud representation is better than using “RGBDM” images or single-frame point clouds.
iv)
Early fusion of the state-action in the critic is better than late fusion, i.e., concatenating features of state and action.
v) “Expert Init.”
: we also use expert-aided exploration by rolling out expert plans for a few steps to initiate the start states during learner rollouts, which is beneﬁcial.
vi)
The point matching loss deﬁned in Eq.
(2) is better than a direct L2 loss on translation and rotation.
Overall, we observe that these design choices both stabilize training and improve the performance.
Ablation experiments on grasp prediction performance can be found in Appendix Section 6.5.
[Table]
Fixed initial joint
[Figure]
35/ 38/ 37/ 31/ 26/ 34/45 Unseen 8/ 5/ 8/
7/ 7/ 7/10 Success Rate 78.
[Figure]
4.2 Tabletop 6D Grasping in the Real World
We conducted tabletop grasping experiments on a Franka robot with a realsense D415 camera on the robot gripper.
We evaluated two different settings: ﬁxed initial setting and varying initial setting.
In the ﬁrst setting, the robot always starts from the same initial position but the object positions change to evaluate the robustness of GA-DDPG with respect to different object poses.
In the second setting, we evaluate the robustness of the system against different intial positions of the robot.
For each setting we consider 3 baselines: a) Policy: we use GA-DDPG during the whole trajectory and grasping the object.
b) Planning: we use GraspNet [10] to generate grasps for the object and OMG planner [11] for planning.
GraspNet trains a variational autoencoder to generate 6D grasps from an input point cloud of an object, while the OMG planner is a trajectory optimization method for joint motion planning and grasp selection.
c)
Combined:
The planner plans trajectories with length ofsteps.
We execute the ﬁrst 20 steps of the plan and switch to GA-DDPG for the ﬁnal part.
Objects are segmented using unseen object instance segmentation method [46] for all the settings.
We evaluated different variations on 9 YCB objects and 10 unseen objects with varying shapes and materials.
GA-DDPG is ﬁnetuned on YCB objects.
For each YCB object, we tested 5 different conﬁgurations according to the ﬁxed/varying initial settings.
Table 4 presents the numbers of successful grasps among trials.
Fig.
9 in the Appendix shows all the objects in our experiments.
We can see that the learned policy performs on par with the open-loop planning with ﬁxed initial robot poses and outperforms it in the varying initial setting.
Combining planning with the policy achieves the best performance in both settings.
For varying initial poses, open-loop planning suffers more due to perception errors and control errors, since the robot started further away from the target object.
Using the policy for the last steps of grasping can ﬁx some open-loop failures in the combined version.
However, since the policy is trained in simulation, it suffers from the sim-to-real gap, especially in contact modeling with objects.
Improving the ﬁdelity of physics simulation and also ﬁne-tuning with real-world data would help bridge the gap.
A discussion on design limitations can be found in Appendix Section 6.8.
Fig.
3 shows some successful and failed grasping examples.
Most failures in the real world are due to the sim-to-real gap in contacting modeling.
4.3 Human-to-Robot Handovers in the Real World
Our closed-loop policy can be applied to human-to-robot (H2R) handovers.
Based on the perception system introduced in [47], we integrate GA-DDPG to compute actions for robot control.
Speciﬁcally, the point cloud of an object in hand was segmented using an external Azure Kinect RGB-D camera according to [47].
Then the object point cloud is fed into the policy network to compute the control action.
As shown in Fig.
4 (a), we used a subset of 10 household objects proposed in [47], and measured the time to successful handovers , success rate and the number of grasp attempts .
We ﬁrst conducted a systematic evaluation to measure the performance of our approach with respect to different objects and varied handover locations.
Then we conducted a user study with 6 participants recruited from the lab.
We asked the participants to answer a questionnaire with Likert-scale questions and open-ended questions after the experiments.
[Figure] [Figure]
Systematic Evaluation.
We handover each object from three handover locations, i.e., left, middle , and right , with respect to the robot base location.
Then we tried to move the object from left to right, and from right to left, after the robot starts to move.
We conducted experiments on all the 10 household objects, and report the success rate in Fig.
4 (b) with an example of Medicine Box on these 5 settings.
Please refer to Table 7 in Appendix for detailed results.
Overall, reactive handover with moving objects is more challenging than static handover.
The robot sometimes cannot adjust its grasping after the object moves.
In our setting, the camera is on the right side.
When the object is on the left side, i.e., further from the camera, we see performance drop due to point cloud sensing noises, especially for the right to left reactive handover.
Additionally, the Azure Kinect only runs at 15fps, which also causes delays in reactive handover.
User Study Results.
As reported in Fig.
5 Top, our method achieves comparable performance against [47] which is speciﬁcally designed and tuned for the task of H2R handovers.
Interestingly, our approach has higher success rate for some difﬁcult objects, such as mug and pen , which demonstrates that our approach can not only generalize to moving objects, but also adapts well to unseen environments (non table-top settings) and views (external camera).
Subjective Evaluation.
Fig.
5 Bottom reports participants’ agreement with statements from [47].
Overall, the participants agreed with the ﬂuency and commented “[the robot] ﬂuently moved toward the object” and “could adjust it’s position when I moved”.
They felt safe, saying “[I] never felt it would overshoot or become aggressive”.
They also sensed the robot was aware of their actions since “[it] adapted fast when I moved my arm”, and was aware of the difference of objects “because it used different grasps for different objects”.
One was “surprised it worked on very thin objects”.
Several participants thought the robot motion was slow when it got close to the object.
This is because the system evaluates the grasping score using the grasp evaluator in GraspNet [10].
Only when the score is higher than a pre-deﬁned threshold, the robot closes its gripper.
Conclusion
We introduce the goal-auxiliary DDPG algorithm for efﬁciently learning of 6D grasping control polices from point clouds.
Our method uses demonstrations from an expert motion and grasp planner and utilizes grasping goal prediction as an auxiliary task to improve the performance of the actor and the critic.
We demonstrate that our policy trained in simulation can be integrated in a tabletop 6D grasping system and a human-to-robot handover system to improve grasping performance of unseen objects.
For future work, we plan to investigate the sim-to-real gap in policy learning due to contact modeling in simulation and extend the method to 6D grasping in cluttered scenes.