L EARNING TO C OORDINATE M ANIPULATION S KILLS VIA S KILL B EHAVIOR D IVERSIFICATION
Anonymous authors
Paper under double-blind review
A BSTRACT
When mastering a complex manipulation task, humans often decompose the task into sub-skills of their body parts, practice the sub-skills independently, and then execute the sub-skills together.
Similarly, a robot with multiple end-effectors can perform a complex task by coordinating sub-skills of each end-effector.
To realize temporal and behavioral coordination of skills, we propose a hierarchical framework that ﬁrst individually trains sub-skills of each end-effector with skill behavior diversiﬁcation, and learns to coordinate end-effectors using diverse behaviors of the skills.
We demonstrate that our proposed framework is able to efﬁciently learn sub-skills with diverse behaviors and coordinate them to solve challenging collaborative control tasks such as picking up a long bar, placing a block inside a container while pushing the container with two robot arms, and pushing a box with two ant agents.
I NTRODUCTION
Imagine you wish to play Chopin’s Fantaisie Impromptu on the piano.
With little prior knowledge about the piece, you would ﬁrst practice playing the piece with each hand separately.
After independently mastering the left hand part and the right hand part, you would move on to practicing with both hands simultaneously, trying to coordinate between the movement of the two hands to together create a complete piece of music.
Coordinating movements of two hands requires adjusting one-handed playing of each hand to the synchronized and coordinated movements of two hands, which requires you to learn variations in playing the same melody.
Through the decomposition of skills into sub-skills of two hands and learning variations of sub-skills, humans make the learning process of manipulation skills much faster than learning everything at once.
Can autonomous agents efﬁciently learn complicated tasks with coordination of different skills from multiple end-effectors like humans?
Learning to perform collaborative and composite tasks from scratch requires a huge amount of environment interaction and extensive reward engineering, which often results in undesired behaviors (Riedmiller et al., 2018).
Hence, instead of learning a task at once, modular approaches (Andreas et al., 2017; Oh et al., 2017; Frans et al., 2018; Peng et al., 2019; Goyal et al., 2019) suggest to learn reusable primitive skills and solve more complex tasks by recombining the skills.
However, all these approaches are focused on working with a single end-effector or agent with learned primitive skills, and learning to coordinate has not been addressed.
To this end, we propose a hierarchical framework that learns to coordinate multiple end-effectors with their primitive skills for various robotics tasks, such as bi-manual manipulation.
The main challenge is that naive simultaneous execution of primitive skills from multiple end-effectors can often cause unintended behaviors (e.g. collisions between end-effectors).
Thus, our model needs to learn to “correctly” coordinate each end-effector; and thus needs a way to obtain, represent, and control detailed behaviors of each primitive skill.
To address these intuitions, our method consists of two parts: (1) acquiring primitive skills with diverse behaviors by mutual information maximization, and (2) learning a meta policy that selects a skill for each end-effector and coordinates the chosen skills by controlling the behavior of each skill.
The main contributions of this paper is a hierarchical approach that tackles cooperative manipulation tasks with multiple end-effectors by (1) learning primitive skills of each end-effector independently with skill behavior diversiﬁcation and (2) coordinating end-effectors using diverse behaviors of the skills.
Our empirical results indicate that our proposed method is able to efﬁciently learn primitive
[Figure]
skills with diverse behaviors and coordinate these skills to solve challenging collaborative control tasks such as picking up a long bar, placing a block inside the container on the right side, and pushing a box with two ant agents.
We will make the training code and environments publicly available.
R ELATED WORK
Hierarchical reinforcement learning:
Deep reinforcement learning for continuous control is an active research area.
However, learning a complex task either from a sparse reward or a heavily engineered reward becomes computationally impractical as the target task becomes more complicated.
Instead of learning from scratch, complex tasks can be tackled by decomposing the tasks into easier and reusable sub-tasks.
Hierarchical reinforcement learning temporally splits a task into a sequence of temporally extended meta actions.
It often consists of one high-level policy and a set of lowlevel policies, such as option framework (Sutton et al., 1999).
The high-level policy decides which low-level controller to activate and the chosen low-level controller generates an action sequence until the high-level policy switches it to another low-level controller.
The option can be discovered unsupervisedly (Schmidhuber, 1990; Levy et al., 2017; Bacon et al., 2017; Nachum et al., 2018), meta-learned (Frans et al., 2018), pre-deﬁned (Kulkarni et al., 2016;
Oh et al., 2017; Lee et al., 2019), or attained from additional supervision signals (Andreas et al., 2017).
However, these option frameworks are not ﬂexible to solve a task that requires simultaneous activation or interpolation of multiple skills since only one skill can be activated at each time step.
Composition of multiple policies
To solve compositional tasks multiple policies can be simultaneously activated by adding Q-functions (Haarnoja et al., 2018a), additive composition (Qureshi et al., 2019; Goyal et al., 2019), or multiplicative composition (Peng et al., 2019).
As each of policies takes the whole observation as input and controls the whole agent, it is not robust to changes in unrelated part of observation.
Hence, these skill composition approaches can fail when an agent encounters a new combination of skills or a new skill is introduced since a policy cannot achieve its own goal under unseen circumstances.
Multi-agent reinforcement learning:
Instead of having a policy with the full observation and action space, multi-agent reinforcement learning (MARL) proposes to explicitly split the observation and action space according to agents (e.g. robots or end-effectors), which allows efﬁcient lowlevel policy training as well as ﬂexible skill composition.
For cooperative tasks, sharing policy parameters (Gupta et al., 2017) and decentralized actor with centralized critic (Lowe et al., 2017;
Foerster et al., 2018) have been actively used.
However, these approaches suffer from the credit assignment problem (Sutton, 1984) among agents and the lazy agent problem (Sunehag et al., 2018).
As agents have more complicated morphology and larger observation space, learning a policy for a multi-agent system from scratch requires extremely long training time.
To resolve these issues, we propose to ﬁrst train reusable skills for each agent separately, instead of learning primitive skills of multiple agents together.
Then, we recombine these skills to complete more complicated tasks with proper coordination between the skills.
[Figure]
Learning diverse skills:
Coordination of multiple skills from multiple agents requires the skills to be ﬂexible; hence, a skill can be adjusted to collaborate with other agents’ skills.
Maximum entropy policies (Haarnoja et al., 2017; 2018a;b) can learn diverse ways to achieve a goal by maximizing not only reward but also entropy of the policy.
Eysenbach et al.
(2019) proposes to discover diverse skills without reward by maximizing entropy as well as mutual information between resulting states and skill embeddings (Hausman et al., 2018).
To this end, our method leverages a maximum entropy policy with a discriminability objective (Eysenbach et al., 2019) to learn a primitive skill with diverse behaviors conditioned on a controllable skill embedding, which will be later used as a behavior embedding for a high-level policy to adapt low-level policies for coordination.
M ETHOD
In this work, we address the problem of solving cooperative manipulation tasks that require collaboration between multiple end-effectors or agents (we use the terms ”multiple end-effectors” and ”multiple agents” interchangeably in this paper).
Instead of learning a multi-agent task from scratch (Lowe et al., 2017; Gupta et al., 2017; Sunehag et al., 2018;
Foerster et al., 2018), modular approaches (Andreas et al., 2017; Frans et al., 2018; Peng et al., 2019) suggest to learn reusable primitive skills and solve more complex tasks by recombining these skills.
However, concurrent execution of primitive skills of multiple agents fails when agents never experienced a combination of skills during the pre-training stage or skills require temporal or behavioral coordination.
Therefore, we propose a hierarchical framework that learns to coordinate multiple agents with primitive skills to compose a complex skill.
Moreover, during primitive skill training, we propose to learn a latent behavior embedding, which provides controllability of the primitive skill to the meta policy for skill coordination.
In Section 3.2, we describe our hierarchical framework in details.
Next, in Section 3.3, we elaborate how controllable primitive skills can be achieved.
Lastly, we describe how the meta policy learns to coordinate primitive skills in Section 3.4.
3. P RELIMINARIES
We formulate our problem as a Markov decision process deﬁned by a tuple {S , A , T , R, ρ, γ } of states, actions, transition probability, reward, initial state distribution, and discount factor.
In our formulation, we assume the environment includes N agents.
To promote consistency in our terminology, we use superscripts to denote the index of agent and subscripts to denote time or primitive skill index.
Hence, the state space and action space for an agent i can be represented as S i and A i where each element of S i is a subset of the corresponding element in S and A
=
A 1 × A 2 × · · · × A N , respectively.
For each agent i , we provide a set of m i skills, Π i = { π i 1 , . . .
, π im i } . A policy of an agent i is represented as π ic it ( a it | s it )
∈
Π i , where c it is a skill index, s it ∈ S
i is a state, a it ∈
A i is an agent action at time t .
An initial state s 0 is sampled from ρ , and then, N agents take actions a 1 t , a 2 t , . . . , a Nt sampled a t + [Equation] = ( a 1 t +
[Equation]
. , a Nt + [Equation] ) ∼ ( π [Equation]
[Equation]
( s t + [Equation] , z 1 t ) , . . . , π Nc Nt ( s t +
[Equation]
, z Nt )
) s 12: t ← τ 13: end while from a composite policy π ( a 1 t , a 2 t , . . . , a N t | s t , c 1 t , c 2 t , . . . , c N t ) and receives a single reward r t , where π ( a 1 t , a 2 t , . .
. , a Nt | s t , c 1 t , c 2 t , . .
c Nt ) =
( π 1 c 1 t ( a 1 t | s t ) , π 2 c 2 t ( a 2 t | s t ) , . . .
, π Nc Nt ( a Nt | s t )) .
The performance is evaluated based on a discounted return R = (cid:80) T − 1 t =
γ t r t , where T is the episode horizon.
3. H IERARCHICAL FRAMEWORK
As illustrated in Figure 2, our hierarchical model is composed of two components: a meta policy π meta and a set of primitive skills of multiple agents Π 1 , . .
, Π N .
The meta policy chooses one primitive skill for each agent ( π 1 c 1 , π 2 c 2 , . .
, π Nc N ) ∈ Π 1 × Π 2 × · · · × Π N .
To control the chosen primitive skills, the meta policy also outputs a set of latent behavior embeddings ( z 1 , z 2 , . .
z N ) ∈ R N z × R N z × · · · × R N z which parameterize diverse behaviors of the corresponding skills.
Then, each agent simultaneously executes the selected skill
π ic i parameterized by the behavior embedding z i .
The behavior embedding enables the meta policy to control primitive skills and it is necessary for coordinating multiple agents’ skills.
For example, in the task illustrated by Figure 1, naively placing a block in the left hand to the container being moved by the right hand can cause collision between two robot arms.
The arms can avoid collision while performing their skills by properly adjusting their end-effector skill behaviors (e.g. placing the block from the left side and pushing the container while leaning the hand toward the right side).
A set of skills are selected every T low time steps.
Algorithmillustrates the overall rollout process.
We denote the meta policy as π meta ( c 1 , . . .
, c N , z 1 , . . .
, z N | s t ) , where c i ∈ [1 , m i ] represents a skill index of an agent i ∈ [1 , N ] and z i represents a behavior embedding of the skill.
The meta policy π meta is modeled as a mixture of N categorical distributions for skill indexes and N Gaussian distributions for behavior embeddings.
The observation of the meta policy contains all available information while each agent has limited access to a partial observation.
Our meta policy selects a skill to execute for each agent, rather than selecting one primitive skill for the entire multi-agent system to execute.
Also, we give the meta policy the capability to select which variant of a skill to execute (see Section 3.4).
Once a set of primitive skills { π 1 c 1 , . . .
, π Nc N } are chosen to be executed, each primitive skill generates an action a i ∼ π
ic i ( a i | s i , z i ) based on the current state s i and the latent vector
z i .
Note that we will not differentiate state spaces for primitive polices and omit time step t in the rest of the paper due to the simplicity of notations.
We want primitive skills to be agent-speciﬁc, controlling motion of only one agent.
This will not only make the learning of primitive skills efﬁcient but also enable generalization to different collaboration scenarios (see Section 3.3).
3. T RAINING PER - AGENT PRIMITIVE POLICIES WITH DIVERSE BEHAVIORS
For a primitive skill to adjust to and collaborate with skills of other agents in a new environment, the skill needs to support variations of skill behaviors when executed at a given state.
Moreover, a behavioral variation of a skill should be controllable by a meta policy for skill coordination.
In order to make our primitive policies generate diverse behaviors controlled by a latent vector z , we leverage the entropy and mutual information maximization objective introduced in Eysenbach et al.
More speciﬁcally, a primitive policy of an agent i takes as input the current state s ∈ S and a latent behavior embedding z ∼ p ( z ) , and output action a ∈ A , where the prior distribution p ( z ) is Gaussian (we omit agent i in this section for the simplicity of notations).
Diverse behaviors conditioned on a random sample z can be achieved by maximizing the mutual information between skills and states, MI ( s | z ) , while minimizing the mutual information between skills and actions given the state, MI ( a, z | s ) , together with maximizing the entropy of the policy for the diverse behaviors, H ( a | s ) .
The objective can be written as follows (we refer the readers to Eysenbach et al.
for derivation):
[Equation]
where the learned discriminator q φ ( z | s ) approximates the posterior p ( z | s ) .
In this paper, we train a primitive policy by maximizing a weighted sum of the reward and Equation (3):
[Equation]
Maximizing Equation (3) encourages multi-modal exploration strategies by learning policies represented by expressive energy-based models while achieving their own goal.
Moreover, by maximizing identiﬁability of behaviors, the latent vector z , named behavior embedding , can represent a variation of the learned policy and thus can be used to control the behavior of the policy.
For example, when training a robot to move an object, a policy learns to move the object quickly as well as slowly, and these diverse behaviors map to different latent vectors z .
We empirically show that the policies with diverse behaviors achieve better compositionality with other agents in our experiments.
3. C OMPOSING PRIMITIVE SKILLS WITH META POLICY
We design the meta policy π meta : S → Π Ni =1 (cid:0) [1 , m i ] × R N z (cid:1) to take as input the current state containing information about the environment as well as the states of all agents and output (1) the primitive skill, denoted as c i ∈ [1 , m i ] , to execute for each agent; and (2) the behavior embedding vector z i ∈ R N z to be used as input for the primitive skill c i .
Since there are a ﬁnite number of skills for each agent to execute, the meta action space for each agent [1 , m i ] is discrete, while the behavior embedding space for each agent R N z is continuous.
Thus, the meta policy can be implemented using a ( m i + N z ) -head neural network which represents a mixture of m i -way categorical distributions for skill selection and N z Gaussian distributions for behavior control of the chosen skill.
3. I MPLEMENTATION
We model the primitive policies and posterior distributions q φ as neural networks.
We train the primitive policies using soft actor-critic (Haarnoja et al., 2018b).
When we train a primitive policy, we use unit Gaussian distribution as the prior distribution of latent variables p ( z ) . We use 5 as the size of latent behavior embedding z .
Each primitive policy outputs the mean and standard deviation of a Gaussian distribution over an action space.
For a primitive policy, we apply tanh activation to normalize the action between [ − 1 , 1] .
We model the meta policy as neural network with multiple heads that outputs the skill c
i and behavior embedding z
i for each agent.
The meta policy is trained using PPO (Schulman et al., 2017; 2016; Dhariwal et al., 2017).
All policy networks in this paper consist of 3 fully connected layers of 64 hidden units with ReLU nonlinearities.
The discriminator network for primitive training with DIAYN (Eysenbach et al., 2019) is a 2-fully connected layers network with 64 hidden units.
E XPERIMENTS
To demonstrate the effectiveness of our framework, we compare our method to prior methods in the ﬁeld of multi-agent RL and ablate the components of our framework to understand their importance.
We conducted experiments on a set of challenging robot control environments that require coordination of different agents to complete collaborative robotic manipulation and locomotion tasks.
Through our experiments, we aim to answer the following questions: (1) can our framework efﬁciently learn to combine primitive skills to execute a complicated task; (2) can our learned agent exhibit
[Figure]
collaborative behaviors during task execution; (3) can our framework leverage the controllable behavior variations of the primitive skills to achieve better coordination?
For details about environments and training, please refer to the supplementary material.
As the performance of training algorithms varies between runs, we train each method on each Jaco task with 6 different random seeds and Ant-push task with 3 different random seeds, and report mean and standard deviation of each method’s success rate.
4. B ASELINES
We compare the performance of our method with various single- and multi-agent RL methods:
Centralized: a single policy takes as input the full observation and outputs all agents’ actions. Decentralized: a set of N policies takes the observation of each agent as input and outputs an action for that agent.
All policies share the global critic from a single task reward (Lowe et al., 2017).
Centralized with Skill Behavior Diversiﬁcation (Centralized w/ SBD) : a hierarchical framework composed of a meta policy and a single low-level policy. The meta policy outputs a behavior embedding and the low-level policy outputs all agents’ actions conditioned on the behavior embedding.
Decentralized with Skill Behavior Diversiﬁcation (Decentralized w/ SBD) : a hierarchical framework composed of a meta policy and a set of N low-level policies for N agents. The meta policy selects N behavior embeddings and each low-level policy outputs each agent’s action conditioned on its observation and behavior embedding.
Modular Network (Ours w/o SBD): a modular framework composed of a meta policy and N sets of primitive skills.
The meta policy takes the full observation as input and selects a primitive skill for each agent policy.
Each agent policy outputs action for that agent.
Modular Network with Skill Behavior Diversiﬁcation (Ours w/ SBD): our proposed method consists of a meta policy and N sets of primitive skills, where each primitive skill is conditioned on a behavior embedding z .
The meta policy takes the full observation as input and selects both a primitive skill and a behavior embedding for each agent.
Then, each primitive skill outputs action for that agent.
4. J ACO P ICK -P USH -P LACE
We developed J ACO -P ICK -P USH -P LACE and J ACO -B AR -M OVING environments using two Kinova Jaco arms, where each Jaco arm is a 9 DoF robotic arm with 3 ﬁngers. The task starts with a block on the left and a container on the right.
To complete the task, the robotic arms need to pick up the block, push the container to the center, and place the block inside the container. [Figure] [Figure] [Figure]
[Figure]
S u cc e ss r a t e
CentralizedDecentralizedCentralized
[Figure]
[Figure]
Decentralized
[Figure] / SBD [Figure] [Figure]
SBD [Figure]
/ SBD
[Figure]
Primitives skills.
There are three primitive skills available to each arm: Picking up , Pushing , and
Placing (see Figure 3a).
Picking up requires a robotic arm to pick up a small block, which is randomly placed on the table.
If the block is not picked up after a certain
Composite task.
Our method can successfully perform J ACO P ICK -P USH -P LACE task while all baselines fail to compose primitive skills as shown in Figure 4a.
The centralized and decentralized baselines cannot learn the composite task mainly because the agent requires to learn the combinatorial number of skill composition and to solve the credit assignment problem between multiple agents.
Since the composite task requires multiple primitive skills to learn for multiple agents, a reward signal about a failure case cannot be assigned to the correct agent or skill.
By using pre-trained primitive skills, the credit assignment problem is relaxed and all agents can perform their skills concurrently.
4. J ACO B AR -M OVING
Two Jaco arms need to pick up a long bar together, move the bar towards a target location while maintaining its rotation, and place it on the table (see Figure 3b).
The initial position of the bar will be randomly initialized every episode and an agent needs to ﬁnd appropriate coordination between two arms for each initialization.
Primitives skills.
There are three pre-trained primitive skills available to each arm: Picking up ,
Moving , and Placing .
Picking up and Placing are same as described in Section 4.2.
Moving learns to move a block in the hand to the speciﬁed location.
The agent fails if it drops the block or does not reach the target position within a certain amount of time.
Composite task.
The J ACO B AR -M
OVING task requires the two end-effectors to work very closely together.
For example, picking up skill of both arms should be synchronized when they start to lift the bar and two arms require to lift the bar while maintaining the relative position between them since they are connected by holding the bar.
The modular framework without explicit coordination of skills can synchronize the execution of picking, moving, and placing.
But the inability to micro-adjust to the movement of the other arm causes instability of bar picking and moving, which results in degraded success rates compared to the modular framework with explicit coordination. On the other hand, all baselines without pre-deﬁned primitive skills fail to learn the J ACO B AR -M OVING task. [Table]
4. A NT P USH
We developed a multi-ant environment, A NT -P USH , inspired from Nachum et al.
(2019), simulated in the Mujoco (Todorov et al., 2012) physics engine and models from OpenAI Gym (Brockman et al., 2016).
In this environment, two ants need to push a large object toward a green target place, collaborating with each other to keep the angle of the object as stable as possible (see Figure 3c)
[Table]
Primitives skills.
We train walking skills of an ant agent in 4 directions: up, down, left, and right.
During primitive skill training, a block is randomly placed and pushing the block gives an additional reward to the agent.
The learned policies have different speed and trajectories conditioned on the latent behavior embedding.
Composite task.
Our method achieves 32.3% success rate on A NT P USH task while all baselines fail to compose primitive skills as shown in Figure 4c and Table 1.
The poor performance of centralized and decentralized baselines shows the difﬁculty of credit assignment between agents, which leads one of the ants moves toward a block and pushes it but another ant does not move.
This result illustrates the importance of coordination of agents, which helps synchronizing and controlling the velocities of both ant agents to push the block toward the goal position while maintaining its rotation.
4. E FFECT OF DIVERSITY OF PRIMITIVE SKILLS
[Figure]
To analyze the effect of the diversity of primitive skills, we compare our model with primitive skills trained with different diversity coefﬁcients λ 2 = { 0 . 0 , 0 . 05 , 0 . 1 , 0 . 5 , 1 . 0 } in Equation (4).
The reward curves in Figure 5 show that with small diversity coefﬁcients λ 2 = { 0 . 05 , 0 . 1 } , the agent can control detailed behaviors of primitive skills while primitive skills without diversity ( λ 2 = 0 ) cannot be coordinated.
The meta policy tries to synchronize two ant agents’ positions and velocities by switching primitive skills, but it cannot achieve proper coordination without diversiﬁed skills.
On the other hand, large diversity coefﬁcients λ 2 = { 0 . 5 , 1 . 0 } make the primitive skills often focus on demonstrating diverse behaviors and fail to achieve the goals of the skills.
Hence, these primitives do not have enough functionality to solve the target task.
C ONCLUSIONS
In this paper, we propose a hierarchical framework with skill coordination to tackle challenges of compositions of sub-skills.
Speciﬁcally, we use entropy maximization policies with the mutual information maximization to train controllable primitive skills with diverse behaviors.
To coordinate learned primitive skills, the meta policy predicts not only the skill to execute for each end-effector but also the behavior embedding that controls the chosen primitive skill’s behavior.
Our experimental results on robotic manipulation and locomotion tasks demonstrate that our proposed framework is able to efﬁciently learn primitive skills with diverse behaviors and coordinate multiple end-effectors to solve challenging cooperative control tasks. A A DDITIONAL ANALYSIS A. E FFECT OF DIVERSITY OF PRIMITIVE SKILLS
For the sake of consistency in reported results, we provide the success rate curves of our model with primitive skills trained with different diversity coefﬁcients in Figure 6 along with episode reward curves in Figure 5.
We put the episode reward curves not the success rate curves since all experiments except λ 2 = { 0 . 1 , 0 .
05 } fail to succeed, which makes hard to compare the effects of different diversity coefﬁcients.
[Figure] A. E FFECT OF SKILL SELECTION INTERVAL T low To analyze the effect of the skill selection interval hyperparameter T low , we compare our model trained with T low = { 1 , 2 , 3 , 5 , 10 } in Algorithm 1. The success rate curves in Figure 7 demonstrate that smaller T low values in range [1 , 3] lead to better performance.
This can be because the agent can realize more ﬂexible skill coordination by adjusting the behavior embedding more frequently.
In addition to the ﬁxed T low values, we also consider the variation of our method in which the skill behavior embedding is only sampled when the meta policy updates its skill selection.
Concretely, we set the value of T low to 1 but update ( z 1 t , . . .
, z Nt ) only if ( c 1 t , . . .
, c Nt ) (cid:54) =
( c 1 t − 1 , . . .
c
Nt − 1 ) .
We observe that in this setting, the meta policy at times switch back and forth between two skills in two consecutive time steps, leading to worse performance compared to experiments using low T low values.
This indicates that the meta policy needs to adjust the behavior embedding in order to optimally coordinate skills of the different agents.
[Figure] B E NVIRONMENT DETAILS The details of observation spaces, action spaces, number of agents, and episode lengths are described in Table 2.
All units in this section are in meters unless otherwise speciﬁed.
[Table]
B. E NVIRONMENT D ESCRIPTIONS
In both Jaco environments, the robot works on a table with size (1 . 6 , 1 . 6) and top center position (0 , 0 , 0 . 82) .
The two Jaco arms are initialized at positions ( − 0 .
16 , − 0 .
16 , 1 . 2) and ( − 0 .
Left arm and right arm objects are initialized around (0 . 3 , 0 . 2 , 0 . 86) and (0 . 3 , − 0 . 2 , 0 . 86) respectively in all primitive training and composite task training environments, with small random position and rotation perturbation.
In the Jaco Pick-Push-Place task, the right jaco arm needs to pick up the object and place it into the container initialized at the other side of the table.
Success is deﬁned by contact between the object and the inner top side of the container.
In the Jaco Bar-Moving task, the two Jaco arms need to together pick the long bar up by height of 0 . 7 , move it towards the arms by distance of 0 . 15 , and place it back on the table.
Success is deﬁned by (1) the bar being placed within 0 .
04 away from the desired destination both in height and in xy-position and (2) the bar having been picked 0 .
7 above the table.
In the Ant Push task, the two ant agents need to push a big box together to the goal position.
The box has a size of 8 .
× 1 . 6 × 1 . 6 .
The distance between ants and the box is 20 cm and the distance between the box and the goal is 30 cm.
Initial positions have 1 cm of randomness and the agent has a randomness of 0 . 01 .
The task is considered as success when both the distances between left and right end of the box and the goal are within 5 cm.
B. R EWARD D ESIGN
For every task, we add a control penalty, − 0 .
001 ∗ (cid:107) a (cid:107) 2 , to regularize the magnitude of actions where a is a torque action performed by an agent.
Jaco Pick: to help the agent learn to reach, pick, and hold the picked object, we provide dense reward to the agent deﬁned by the weighted sum of pick reward, gripper-to-cube distance reward, cube position and quaternion stability reward, hold duration reward, success reward, and robot control reward.
More concretely,
[Equation]
where λ pick = 500 , λ dist = 100 , λ pos = 1000 , λ quat = 1000 , λ hold = 10 , λ success = 100 , λ ctrl = 1 × 10 − 4 .
Jaco Place: reward for place primitive is deﬁned by the weighted sum of xy-distance reward, height reward (larger when cube close to ﬂoor), success reward, and robot control reward. [Equation]
, where λ xy = 500 , λ z = 500 , λ success = 500 , λ ctrl = 1 × 10 − 4 .
Jaco Push: reward for push primitive is deﬁned by the weighted sum of gripper reaching reward, box-to-destination distance reward, quaternion stability reward, hold duration reward, success reward, and robot control reward. [Equation]
[Table] where λ reaching = 100 , λ pos = 500 , λ quat = 30 , λ hold = 10 , λ success = 1000 , λ ctrl = 1 × 10 − 4 . Jaco Pick-Push-Place: reward for Pick-Push-Place is deﬁned by the weighted sum of gripper contact reward, per-stage reach/pick/push/place rewards, success reward, and control reward.
We tune the reward carefully for all baselines. [Equation]
where λ reach = 10 , λ contact = 10 , λ pick = λ place = λ place = 10 ,
λ success = 50 , λ ctrl = 0 , and 1 reaching , 1 pick , and 1 place are indicator functions specifying whether the agent is in reaching, pick or place stage.
Agent stages are determined by how many multiples of 25 steps the agent has stepped through in the environment.
Jaco Bar-Moving: reward for Bar-Moving is deﬁned by the weighted sum of per-stage reach/pick/move/place rewards, success reward, and control reward. [Equation] [Table] [Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where λ reach = 10 , λ pick = 30 , λ move = 100 , λ place = 100 , λ success = 100 , λ ctrl = 1 × 10 − 4 , and 1 pick and 1 place are indicator functions specifying whether the agent is in pick or place stage.
Agent stages are determined by whether the pick objective is fulﬁlled or not.
Ant Moving: reward for A NT P USH is deﬁned by upright, velocity towards the desired direction.
We provide a dense reward to encourage the desired locomotion behavior using velocity, stability, and posture, as following:
[Equation]
where λ vel = 50 , λ boxvel = 20 , λ upright = 1 , λ height = 0 . 5 .
For A NT P USH , we provide an additional reward based on distance between the box and the goal position with λ goal = 200 .
B. T RAINING DETAILS
For Jaco tasks, we train a total of 4 primitive skills – right arm pick, right arm place-to-center, right arm place-towards-arm, and left-arm-push – to be composed by meta-policy.
For Jaco Pick-PushPlace, we provide the meta-policy with right arm pick and right arm place-to-center as right arm primitives and left arm push as left arm primitives; for Jaco Bar-Moving, we provide the meta-policy with right arm pick and right arm place-towards-arm as right arm primitivies and left arm pick and right arm place-towards-arm as left arm primitives.
We obtain left arm primitives for bar-moving task by using the learned right arm primitives directly.
To obtain the 4 primitives skills described above, we ﬁrst train right arm pick with DIAYN reward weight 0.01 and left arm push with DIAYN reward weight 0.1.
The destination of right arm push is set to (0 . 3 , − 0 . 03 , 0 . 86) , which is slightly left of the center of the table.
After pick primitive is trained, we train the two right arm place primitives where episodes are initialized by intermediate states of successful right arm pick episodes where the height of the box is larger than 0.94 (0.
higher than the target pick height).
The place destinations for towards-arm and to-center primitives are (0 . 15 , − 0 . 2 , 0 . 86) and (0 . 3 , − 0 .
02 , 0 . 86) , respectively.
We use 0.1 as the DIAYN reward weights for the place primitive trainings.
C E XPERIMENT DETAILS
We use PyTorch (Paszke et al., 2017) for our implementation and all experiments are conducted on a workstation with Intel Xeon Gold 6154 CPU and 4 NVIDIA GeForce RTX 2080 Ti GPUs. [Table] C. N ETWORK A RCHITECTURES Actor Networks:
In all experiments, we model our actor network for each primitive skill as a 3-layer
MLP with hidden layer size 64. The last layer of the MLP is two-headed – one for the mean of the action distribution and the other for the standard deviation of it. We use ReLU as activation function in hidden layers. We do not apply any activation function for the ﬁnal output layer.
The action distribution output represents per-dimension normal distribution, from which single actions can be sampled and executed in the environment.
Critic Networks: The critic network for each primitive skill and meta policy is modeled as a 2-layer MLP with hidden layer size 128. It uses ReLU as activation function in the hidden layer and doesn’t use any activation function in the ﬁnal output layer.
The critic network output is used to assess the value of a given state-action pair, and is trained by ﬁtting its outputs to the target Q-value clamped by ± 100 .
Meta Policy: The meta policy is modeled as a 3-layer MLP with hidden layer size of 64.
Since the actions of meta policy are sampled from N categorical distributions for each end-effector/agent and N normal distributions for behavior embeddings, the output dimension of the meta policy is (cid:80)
Ni =1 ( m i + N z ) .
The meta policy uses ReLU as activation function for all layers except for the ﬁnal output layer.