Goal-Auxiliary Actor-Critic for 6D Robotic Grasping with Point Clouds
Abstract: 6D robotic grasping beyond top-down bin-picking scenarios is a challenging task.
Previous solutions based on 6D grasp synthesis with robot motionplanning usually operate in an open-loop setting, which are sensitive to grasp synthesis errors.
In this work, we propose a new method for learning closed-loopcontrol policies for 6D grasping.
Our policy takes a segmented point cloud of anobject from an egocentric camera as input, and outputs continuous 6D control actions of the robot gripper for grasping the object.
We combine imitation learningand reinforcement learning and introduce a goal-auxiliary actor-critic algorithmfor policy learning.
We demonstrate that our learned policy can be integrated intoa tabletop 6D grasping system and a human-robot handover system to improve thegrasping performance of unseen objects.
Introduction
Robotic grasping of arbitrary objects is a challenging task.
A robot needs to deal with objects it hasnever seen before, and generates a motion trajectory to grasp an object.
Due to the complexity ofthe problem, majority works in the literature focus on bin-picking tasks, where top-down graspingis sufﬁcient to pick up an object.
Both grasp detection approaches [1, 2, 3] and reinforcementlearning-based methods [4, 5] are introduced to tackle the top-down grasping problem.
However, itis difﬁcult for these methods to grasp objects in environments where 6D grasping is necessary, i.e.,3D translation and 3D rotation of the robot gripper, such as a cereal box on a tabletop or in a cabinet.
While 6D grasp synthesis has been studied using 3D models of objects [6, 7] and partial observations[8, 9, 10], these methods only generate 6D grasp poses of the robot gripper for an object, insteadof generating a trajectory to reach and grasp the object.
As a result, a motion planner is needed toplan the grasping motion according to the grasp poses.
Usually, the planned trajectory is executedin an open-loop fashion since re-planning is expensive, and perception feedback during grasping aswell as dynamics and contacts of the object are often ignored, which makes the grasping sensitiveto grasp synthesis errors.
In this work, to overcome the limitations of open-loop 6D grasping based on grasp synthesis, weintroduce a new method for learning closed-loop 6D grasping polices from partially-observed pointclouds of objects (Fig. 1).
Our policy takes a segmented point cloud of an object as input and directlyoutputs the control action of the robot gripper, which is the relative 6D pose transformation of thegripper.
Our policy can be combined with different perception methods for object segmentation.
In order to learn the policy, we combine Imitation Learning (IL) and Reinforcement Learning (RL).
Because the chance of lifting an object in 6D grasping is very rare if only RL is used for exploration.
We obtain demonstrations using a joint motion and grasp planner [11] in simulation.
Consequently,we can efﬁciently obtain a large number of 6D grasping trajectories using ShapeNet objects [12]with the planner.
Then we learn the grasping policy based on the Deep Deterministic Policy Gradient(DDPG) algorithm [13], which is an actor-critic algorithm in RL that can utilize off-policy data from
[Figure]
demonstrations.
More importantly, we introduce a goal prediction auxiliary task to improve thepolicy learning, where the actor and the critic in DDPG are trained to predict the ﬁnal 6D graspingpose of the robot gripper as well.
The supervision on goal prediction comes from the expert plannerfor objects with known 3D shape and pose.
For unknown objects without 3D models available, wecan still obtain the grasping goals from successful grasping rollouts of the policy, i.e., hindsightgoals.
This property enables our learned policy to be ﬁne-tuned on unknown objects, which iscritical for continual learning in the real world.
We see that adding the auxiliary losses stabilizestraining and improves the learned policy.
Overall, our contributions are: 1) We introduce a Goal-Auxiliary DDPG (GA-DDPG) algorithmfor joint imitation and reinforcement learning of 6D robotic grasping. 2)
We demonstrate that ourlearned policy can be integrated into a tabletop 6D grasping system to improve grasping performance.
We demonstrate that our closed-loop policy can improve human-to-robot handover.
Related Work
Vision-based Robotic Grasping.
Grasp synthesis can be used in a planning and control pipelinefor robotic grasping [2, 3, 14, 15].
Alternatively, end-to-end policy learning methods [16, 4, 17]make use of large-scale data to learn closed-loop vision-based grasping.
Although RGB imagesare widely used as the state representation, it requires the policy to infer 3D information from 2Dimages.
Recently, depth and segmentation masks [3], shape completion [9], deep visual features[18], keypoints [19], point clouds [20], multiple cameras [21], and egocentric views [22, 23] havebeen considered to improve the state representation.
Motivated by these methods, we utilize pointclouds of objects as our state representation, which transfers well from simulation to the real world.
Combining Imitation Learning and Reinforcement Learning.
Model-free RL [4, 5] requires alarge number of interactions even with full-state information.
Therefore, many works have proposedto combine imitation learning [24] in RL.
[25] augments policy gradient update with demonstrationdata to circumvent reward shaping and the exploration challenge.
[26] uses inverse RL to learndexterous manipulation tasks with a few human demonstrations in simulation.
The closest relatedworks to ours are [27, 28] that utilize demonstration data with off-policy RL.
Despite the focus ondifferent tasks, the main difference is that our demonstrations come from an expert planner insteadof human demonstrators.
Thus, we can obtain a large number of consistent demonstrations andquery the expert planner during training to provide supervision on the on-policy data.
Goals and Auxiliary Tasks in Policy Learning.
Goals are often used as extra signals to guidepolicy learning [29].
For goal-conditioned policies, [30, 31, 32] make the observation that everytrajectory is a successful demonstration of the goal state that it reaches, thereby these methods relabel goals in rollout trajectories for effective learning.
However, goals still need to be provided intesting.
In 6D grasping, estimating the grasping goals is a challenging problem.
On the other hand,auxiliary tasks have been used to improve RL as well [33, 34, 35, 36].
We utilize the grasping goalprediction as an auxiliary task that requires the policy to predict how to grasp the target object.
Learning 6D Grasping Policies
Our goal is to learn a closed-loop policy for 6D robotic grasping.
We ﬁrst introduce related background knowledge and then present our method for learning the policy.
3.1 Background
Markov Decision Process.
A Markov Decision Process (MDP) is deﬁned using the tuple: M ={S , R , A , O , P , ρ 0 , γ } .
S , A , and O represent the state, action, and observation space.
R : S×A → 81 R is the reward function.
P : S×A → S is the transition dynamics.
ρ 0 is the probability distributionover initial states and γ =
[0 , 1) is a discount factor.
Let π : S
→ A be a policy which maps statesto actions.
In the partially observable case, at each time t , the policy maps a partial observation o t ofthe environment to an action a t =
π ( o t ) .
Our goal is to learn a policy that maximizes the expectedcumulative rewards
E π [ (cid:80) ∞ t =0 γ t r t ] , where r t is the reward at time t .
The Q-function of the policyfor a state-action pair is Q ( s, a ) =
R ( s, a ) + γ E s (cid:48) ,π [ (cid:80) ∞ t =0 γ t r t | s
= s (cid:48) ] , where s (cid:48) represents thenext state of taking action a in state s according to the transition dynamics.
Deep Deterministic Policy Gradient.
DDPG [13] is an actor-critic algorithm that uses off-policydata, deterministic policy gradient, and temporal difference learning.
It has successful applicationsin continuous control [37, 27].
Speciﬁcally, the actor in DDPG learns the policy π θ ( s ) , whilethe critic approximates the Q-function Q φ ( s, a ) , where θ and φ denote the parameters of the actorand the critic, respectively.
A replay buffer of transitions D
= { ( s, a, r, s (cid:48) ) } is maintained duringtraining, and examples sampled from it are used to optimize the actor and critic alternately.
DDPGminimizes the following Bellman error with respect to φ :
[Equation]
Then the deterministic policy π θ is trained to maximize the learned Q-function with 96 max θ E s ∼D ( Q φ ( s, π θ ( s ))) , which resembles a policy evaluation and improvement schedule.
3.2 6D Grasping Policy From Point Clouds
We consider the task of grasping an arbitrary object in a closed-loop manner.
Our goal is to learna 6D grasping policy π : s t → a t that maps the state s t at time t to an action a t .
To simplify thesetting, we still use s to represent the input of the policy even though observations are used.
At timet , the action a t is parametrized by the relative 3D translation and the 3D rotation of the robot endeffector.
Therefore, our learned visuomotor policy represents an operational space controller for 6Dgrasping.
We utilize 3D point clouds of objects to represent the states, which can be computed usingdepth images and foreground masks of the objects.
These points are transformed from the cameraframe to the robot end-effector frame to represent the state.
Different segmentation methods can beused to obtain a foreground mask of a target object. Our learned policy is robust to noises in thesegmentation. For grasping static objects, we aggregate the point clouds from previous time stepsas described in Appendix Section 6.2. For dynamic objects, no aggregation is used.
3.3 Demonstrations from a Joint Motion and Grasp Planner
To obtain demonstrations for 6D grasping, we utilize the Optimization-based Motion and Grasp
Planner (OMG Planner)
[11] as our expert.
Given a planning scene and a set of pre-deﬁned graspson a target object from a grasp planner [6, 7], the OMG Planner plans a trajectory of the robot tograsp the object.
Depending on the initial conﬁguration of the robot, it selects different grasps asthe goal.
This property enables us to learn a policy that grasps objects in different ways accordingto the initial pose of the robot gripper.
Let ξ =
( T 0 , T 1 , . .
. , T T ) be a trajectory of the robot gripper pose generated from the OMG plannerto grasp an object, where T t ∈ SE (3) is the gripper pose in the robot base frame at time t .
Then theexpert action at time t can be computed as a ∗ t
= T t +1 T − 1 t , which is the relative transformation ofthe gripper between t and t + 1 .
Furthermore, T T from the planner is a successful grasp pose of theobject.
We deﬁne the expert goal at time t as g ∗ t =
T T T − 1 t , which is the relative transformation between the current gripper pose and the ﬁnal grasp.
Finally, the state s t can be obtained by computingthe point cloud as in Section 3.2.
In this way, we construct an ofﬂine dataset of demonstrations fromthe planner as D expert =
{ ( s t , a ∗ t , g ∗ t , s (cid:48) t ) } , where s (cid:48) t is the next state.
3.4 Behavior Cloning and DAGGER
Using the demonstration dataset D expert , we can already train a policy network π θ by applying Behavior Cloning (BC) to predict an action from a state.
To minimize the distance with expert actions,
[Figure]
different loss functions on SE (3) can be used.
In particular, we adopt the point matching loss function as deﬁned in [38] to jointly optimize translation and rotation:
[Equation]
where T 1 , T 2 ∈ SE (3) , X g denotes a set of pre-deﬁned 3D points on the robot gripper, and L1 normis used.
Let a θ
= π θ ( s ) be an action predicted from the policy and a ∗ =
π ∗ ( s ) be an action fromthe expert, the behavior cloning loss is L BC ( a ∗ , a θ ) =
L POSE ( a ∗ , a θ ) .
We apply DAGGER [39] to augment supervisions for rollouts from the learner.
Given a sampledinitial state s 0 , the current policy π θ can roll out a trajectory of states s 0 , ..., s T and actions a 0 , ..., a Twith end-effector poses ˆ T 0 , ..., ˆ T T .
The expert then treats each state as the initial state and generatesa trajectory to grasp the object ξ ( s t ) =
( ˆ T t , T t +1 , ..., T T ) .
The ﬁrst action in the plan a ∗ t = π ∗ ( s t ) =
T t +1 ˆ T − 1 t is used as supervision to correct deviations of the learner from ξ ( s t ) .
A correspondingexpert goal g ∗ t = T T ˆ T − 1 t can also be obtained for state s t .
These action and goal labels are used toconstruct a dataset D dagger =
{ ( s t , a t , a ∗ t , g ∗ t , s (cid:48) t ) } .
3.5 Goal-Auxiliary DDPG
To handle contact-rich scenarios that are rare in the expert trajectories, we further improve the policylearning by combining RL with expert demonstrations.
In our RL setting, each episode ends whenthe agent attempts a grasp or reaches a maximum horizon T .
The sparse reward r is an indicatorfunction given at the end of the episode denoting if the target object is lifted or not.
Similar to DAGGER, we collect on-policy data for DDPG training.
The only difference is that wedo not have expert demonstration for the action a ∗ t and goal g ∗ t for a state s t from the learner policy.
Instead, we ﬁrst ﬁnd the nearest goal from a goal set G by (cid:101) g t =
arg min g ∈G L POSE ( g, T t ) , where
T t is the pose of the robot gripper at time t .
Then the heuristic goal for s t is g t =
(cid:101) g t T − 1 t .
Notethat the goal set G is only available if we have the 3D shape and pose of the target object.
Finally,we construct a dataset for DDPG D ddpg = { ( s t , a t , g t , r t , s (cid:48) t ) } , where r t is the reward at state s tand s (cid:48) t is the next state of s t from the on-policy rollout.
The replay buffer of DDPG training is
D = D expert ∪ D dagger ∪ D ddpg , where we augment D expert and D dagger with the sparse reward.
Our network architecture for training the actor-critic is shown in Fig.
We introduce two auxiliarytasks for the actor-critic to predict goals, which help to learn the policy and the Q-function by addinggoal supervision.
Given a data sample ( s, a, g, r, s (cid:48) ) from the replay buffer, the critic loss is L φ =
[Equation]
where y =
r + γQ φ (cid:48) (cid:0) s (cid:48) , π θ (cid:48)
( s (cid:48) )
+ (cid:15) (cid:1) is the Bellman target and g φ is the predicted goal.
Q φ (cid:48) andπ θ (cid:48) are the target networks and (cid:15) is a pre-deﬁned clipped noise as in TD3 [41].
The auxiliary loss is
L AUX ( g, g φ ) =
L POSE ( g, g φ ) for measuring pose errors in goal prediction.
Given a data sample ( s, a ∗ , g ) from the replay buffer, the loss function for the actor is deﬁned as
[Equation]
where a θ and g θ are the action and the goal predicted from the actor, respectively, and λ is aweighting hyper-parameter to balance the losses from the expert and from the learned value function.
The BC loss L BC ( a ∗ , a θ ) =
L POSE ( a ∗ , a θ ) as deﬁned before, which prevents the learner from moving too far away from the expert policy.
The deterministic policy loss is deﬁned as
L DDPG ( s, a θ ) = − Q φ ( s, a θ ) in order to maximize Q values.
Note that the expert action a ∗ isonly available if the data sample is from D expert or D dagger .
Otherwise, we do not include the BCloss in Eq.
Even though our method can be used with any off-policy actor-critic algorithms, inpractice, we choose TD3 for its improved performance over vanilla DDPG.
3.6 Hindsight Goals For Fine-tuning on Unknown Objects
For unknown objects without 3D models, we cannot obtain neither expert demonstrations nor grasping goals from a pre-deﬁned goal set.
We can use hindsight goals to ﬁne-tune our policy on unknownobjects.
Based on the current policy π θ , each learner episode rolls out a trajectory of state-actionswith end-effector poses ˆ T 0 , ..., ˆ T T .
If the grasp in the episode is successful, we know that ˆ T T is asuccess grasp for the target object, which is also known as the hindsight goal.
Then we can computethe grasp goal for state s t in the episode as ˆ g t
= ˆ T T
ˆ T − 1 t , which is used to supervise the actor and thecritic via the goal auxiliary tasks.
By using hindsight goals, we construct a dataset using on-policyrollouts on unknown objects D hindsight =
{ ( s t , a t , ˆ g t , r t , s (cid:48) t ) } , where ˆ g t is the hindsight goal.
Weﬁnetune the pretrained actor-critic network on the dataset D = D expert ∪ D hindsight .
The algorithm fortraining with hindsight goals can be found in Appendix Section 6.1.
Experiments
Simulation Environment.
We experiment with the Franka Emika Panda arm, a 7 -DOF arm with aparallel gripper.
We use ShapeNet [12] and YCB objects [42] as our object repository.
A task sceneis generated by dropping a sampled target object with a random pose on a table in the PyBullet
Simulator [43].
The maximum horizon for the policy is T = 30 .
An episode is terminated once agrasp is completed.
The observation image size is 112 × 112 in PyBullet using a hand camera.
Moreenvironment details can be found in Appendix Section 6.2.
Training and Testing.
We use approximately 1 , 500 ShapeNet objects from 169 different classesin the ACRONYM dataset [44] for training, where each object has 100 pre-computed grasps from[7] for planning.
For testing, we use 9 selected YCB objects with 10 scenes per object and 138holdout ShapeNet objects within 157 scenes.
We train each model three times with different randomseeds.
We run each YCB scene 5 times and each ShapeNet scene 3 times and then compute the meangrasping success rate.
More details on network architecture and training can be found in Appendix
Section 6.3 and Appendix Section 6.4, respectively.
4.1 Ablation Studies in Simulation
Ablations on State Representations.
We ﬁrst conduct experiments with BC to investigate the effectof using different state representations.
Table 1 presents the grasping success rates.
When usingimages, there are three variations: “RGB”, “RGBD” and “RGBDM” indicating whether depth (D)or foreground mask (M) of the object are used or not.
Depth and/or mask are concatenated with the
RGB image, and ResNet18 [45] is used for feature extraction.
“Point” in Table 1 indicates the pointcloud state representation.
“Ofﬂine” means a ﬁxed-size ofﬂine dataset D expert is used for trainingwhich contains 50,000 data points from expert demonstrations.
“Online” means the expert planneris running in parallel with training, which keeps adding new data to the dataset.
“DAGGER” useson-policy rollout data from the learner policy with expert supervision D dagger , and “Goal-Auxiliary”indicates whether we add the grasping goal prediction task or not in BC training.
From Table 1, we can see that: i) using the point clouds achieves better performance compared tousing images of the current view.
This indicates that the 3D features in object point clouds generalizes better for 6D grasping.
ii)
Adding depth or foreground mask to the image-based representationimproves the performance.
iii) “Online” is overall better than “Ofﬂine” by utilizing more data fortraining.
iv) Both DAGGER and adding the auxiliary task improve success rates.
Ablations on Goal-auxiliary vs. Goal-conditioned.
We evaluate our goal-auxiliary DDPG algorithm for 6D grasping, where the dataset D = D expert ∪ D dagger ∪ D ddpg is constructed for training.
We also consider another common strategy of using goals for comparison: goal-conditioned policiessuch as in [30, 28, 31], where a goal is concatenated with the state as the input for the network.
Inthis case, in order to use goals in testing, we need to train a separate network to predict goals fromstates.
Table 2 displays the evaluation results, where we test different combinations of adding thegoal-auxiliary task or using the goal-conditioned input to the actor and the critic.
[Table] [Table]
From Table 2, we can see that: i) using the goal-auxiliary loss for the critic signiﬁcantly improvesthe performance since it regularizes the Q-learning.
ii)
The goal-conditioned policies perform worsethan the goal-auxiliary counterparts.
When the predicted goal is not accurate, it affects the graspingsuccess.
We also observe some instability in goal-conditioned policy training.
iii)
Adding the goalauxiliary task to both the actor and the critic achieves the best performance.
iv) Comparing to thesupervised BC training in Table 1, our GA-DDPG algorithm further improves the policy, especiallyin the contact-rich scenarios.
Imitation learning alone cannot handle these contacts with the targetobject before closing the ﬁngers since these are rarely seen in the expert demonstrations.
Ablations on Fine-tuning with Hindsight Goals.
Our policy trained on ShapetNet objects achieves91.3% and 88.2% success rate for grasping unseen ShapeNet and unseen YCB objects, respectively,as shown in Table 2.
We can further improve the policy using RL with unseen objects.
To resemble real-world RL, we assume that the 3D shape and pose of the YCB objects are not available.
Therefore, there is no expert supervision on these objects.
We ﬁne-tune the policy using the dataset
D = D expert ∪D hindsight with hindsight goals.
We see that the success rate is improved from 88.2% to93.5% after ﬁne-tuning.
We also experiment with the simulation dynamics by decreasing the lateralfrictions of the object, which makes object easier to slide during contacts.
In this case, the successrate of the pre-trained policy reduces to 81.6%.
After ﬁne-tuning with hindsight goals, the policy adapts well and achieves 88.5% success rate, which demonstrates the robustness of the trainingprocedure to contact model errors.
Ablations on Other Design Choices.
We conduct ablation studies on several design choices in
GA-DDPG as shown in Table 3.
We can see that: i) RL without BC failed to learn a useful policydue to the high-dimensional state-action space and sparse reward.
ii)
Both online interaction foradding data and DAGGER help.
iii)
Our aggregated point cloud representation is better than using“RGBDM” images or single-frame point clouds.
iv)
Early fusion of the state-action in the critic isbetter than late fusion, i.e., concatenating features of state and action.
v) “Expert Init.”
: we also useexpert-aided exploration by rolling out expert plans for a few steps to initiate the start states duringlearner rollouts, which is beneﬁcial.
vi)
The point matching loss deﬁned in Eq.
(2) is better than adirect L2 loss on translation and rotation.
Overall, we observe that these design choices both stabilizetraining and improve the performance.
Ablation experiments on grasp prediction performance canbe found in Appendix Section 6.5.
[Table]
Fixed initial joint
[Figure]
35/ 38/ 37/ 31/ 26/ 34/45 Unseen 8/ 5/ 8/
7/ 7/ 7/10 Success Rate 78.
[Figure]
4.2 Tabletop 6D Grasping in the Real World
We conducted tabletop grasping experiments on a Franka robot with a realsense D415 camera onthe robot gripper.
We evaluated two different settings: ﬁxed initial setting and varying initial setting.
In the ﬁrst setting, the robot always starts from the same initial position but the object positionschange to evaluate the robustness of GA-DDPG with respect to different object poses.
In the secondsetting, we evaluate the robustness of the system against different intial positions of the robot.
Foreach setting we consider 3 baselines: a) Policy: we use GA-DDPG during the whole trajectory andgrasping the object.
b) Planning: we use GraspNet [10] to generate grasps for the object and OMGplanner [11] for planning.
GraspNet trains a variational autoencoder to generate 6D grasps from aninput point cloud of an object, while the OMG planner is a trajectory optimization method for jointmotion planning and grasp selection.
c)
Combined:
The planner plans trajectories with length of 25steps.
We execute the ﬁrst 20 steps of the plan and switch to GA-DDPG for the ﬁnal part.
Objects aresegmented using unseen object instance segmentation method [46] for all the settings.
We evaluateddifferent variations on 9 YCB objects and 10 unseen objects with varying shapes and materials.
GA-DDPG is ﬁnetuned on YCB objects.
For each YCB object, we tested 5 different conﬁgurationsaccording to the ﬁxed/varying initial settings.
Table 4 presents the numbers of successful graspsamong trials.
Fig.
9 in the Appendix shows all the objects in our experiments.
We can see that the learned policy performs on par with the open-loop planning with ﬁxed initialrobot poses and outperforms it in the varying initial setting.
Combining planning with the policyachieves the best performance in both settings.
For varying initial poses, open-loop planning suffersmore due to perception errors and control errors, since the robot started further away from thetarget object.
Using the policy for the last steps of grasping can ﬁx some open-loop failures in thecombined version.
However, since the policy is trained in simulation, it suffers from the sim-to-realgap, especially in contact modeling with objects.
Improving the ﬁdelity of physics simulation andalso ﬁne-tuning with real-world data would help bridge the gap.
A discussion on design limitationscan be found in Appendix Section 6.8.
Fig.
3 shows some successful and failed grasping examples.
Most failures in the real world are due to the sim-to-real gap in contacting modeling.
4.3 Human-to-Robot Handovers in the Real World
Our closed-loop policy can be applied to human-to-robot (H2R) handovers.
Based on the perceptionsystem introduced in [47], we integrate GA-DDPG to compute actions for robot control.
Speciﬁcally, the point cloud of an object in hand was segmented using an external Azure Kinect RGB-Dcamera according to [47].
Then the object point cloud is fed into the policy network to compute thecontrol action.
As shown in Fig.
4 (a), we used a subset of 10 household objects proposed in [47],and measured the time to successful handovers , success rate and the number of grasp attempts .
Weﬁrst conducted a systematic evaluation to measure the performance of our approach with respectto different objects and varied handover locations.
Then we conducted a user study with 6 participants recruited from the lab.
We asked the participants to answer a questionnaire with Likert-scalequestions and open-ended questions after the experiments. [Figure] [Figure]
Systematic Evaluation.
We handover each object from three handover locations, i.e., left, middle ,and right , with respect to the robot base location.
Then we tried to move the object from left toright, and from right to left, after the robot starts to move.
We conducted experiments on all the10 household objects, and report the success rate in Fig.
4 (b) with an example of Medicine Box onthese 5 settings.
Please refer to Table 7 in Appendix for detailed results.
Overall, reactive handoverwith moving objects is more challenging than static handover.
The robot sometimes cannot adjustits grasping after the object moves.
In our setting, the camera is on the right side.
When the objectis on the left side, i.e., further from the camera, we see performance drop due to point cloud sensingnoises, especially for the right to left reactive handover.
Additionally, the Azure Kinect only runs at15fps, which also causes delays in reactive handover.
User Study Results
As reported in Fig.
5 Top, our method achieves comparable performanceagainst [47] which is speciﬁcally designed and tuned for the task of H2R handovers.
Interestingly,our approach has higher success rate for some difﬁcult objects, such as mug and pen , which demonstrates that our approach can not only generalize to moving objects, but also adapts well to unseenenvironments (non table-top settings) and views (external camera).
Subjective Evaluation.
Fig.
5 Bottom reports participants’ agreement with statements from [47].
Overall, the participants agreed with the ﬂuency and commented “[the robot] ﬂuently moved towardthe object” and “could adjust it’s position when I moved”.
They felt safe, saying “[I] never felt itwould overshoot or become aggressive”.
They also sensed the robot was aware of their actions since“[it] adapted fast when I moved my arm”, and was aware of the difference of objects “because it useddifferent grasps for different objects”.
One was “surprised it worked on very thin objects”.
Severalparticipants thought the robot motion was slow when it got close to the object.
This is because thesystem evaluates the grasping score using the grasp evaluator in GraspNet [10].
Only when the scoreis higher than a pre-deﬁned threshold, the robot closes its gripper.
Conclusion
We introduce the goal-auxiliary DDPG algorithm for efﬁciently learning of 6D grasping controlpolices from point clouds.
Our method uses demonstrations from an expert motion and grasp plannerand utilizes grasping goal prediction as an auxiliary task to improve the performance of the actor andthe critic.
We demonstrate that our policy trained in simulation can be integrated in a tabletop 6Dgrasping system and a human-to-robot handover system to improve grasping performance of unseenobjects.
For future work, we plan to investigate the sim-to-real gap in policy learning due to contactmodeling in simulation and extend the method to 6D grasping in cluttered scenes.