L EARNING TO C OORDINATE M ANIPULATION S KILLS VIA S KILL B EHAVIOR D IVERSIFICATION
A BSTRACT
When mastering a complex manipulation task, humans often decompose the task into sub-skills of their body parts, practice the sub-skills independently, and then execute the sub-skills together.
Similarly, a robot with multiple end-effectors can perform complex tasks by coordinating sub-skills of each end-effector.
To realize temporal and behavioral coordination of skills, we propose a modular framework that ﬁrst individually trains sub-skills of each end-effector with skill behavior diversiﬁcation, and then learns to coordinate end-effectors using diverse behaviors of the skills.
We demonstrate that our proposed framework is able to efﬁciently coordinate skills to solve challenging collaborative control tasks such as picking up a long bar, placing a block inside a container while pushing the container with two robot arms, and pushing a box with two ant agents. Videos and code are available at https://clvrai.com/coordination .
I NTRODUCTION
Imagine you wish to play Chopin’s Fantaisie Impromptu on the piano.
With little prior knowledge about the piece, you would ﬁrst practice playing the piece with each hand separately.
After independently mastering the left and right hand parts, you would move on to practicing with both hands simultaneously. To ﬁnd the synchronized and non-interfering movements of two hands, you would try variable ways of playing the same melody with each hand, and eventually create a complete piece of music.
Through the decomposition of skills into sub-skills of two hands and learning variations of sub-skills, humans make the learning process of manipulation skills much faster than learning everything at once.
Can autonomous agents efﬁciently learn complicated tasks with coordination of different skills from multiple end-effectors like humans?
Learning to perform collaborative and composite tasks from scratch requires a huge amount of environment interaction and extensive reward engineering, which often results in undesired behaviors (Riedmiller et al., 2018).
Hence, instead of learning a task at once, modular approaches (Andreas et al., 2017; Oh et al., 2017; Frans et al., 2018; Lee et al., 2019; Peng et al., 2019; Goyal et al., 2020) suggest to learn reusable primitive skills and solve more complex tasks by recombining the skills.
However, all these approaches either focus on working with single end-effector manipulation or single agent locomotion, and these do not scale to multi-agent problems.
To this end, we propose a modular framework that learns to coordinate multiple end-effectors with their primitive skills for various robotics tasks, such as bimanual manipulation.
The main challenge is that naive simultaneous execution of primitive skills from multiple end-effectors can often cause unintended behaviors (e.g. collisions between end-effectors).
Thus, as illustrated in Figure 1, our model needs to learn to appropriately coordinate end-effectors; and hence needs a way to obtain, represent, and control detailed behaviors of each primitive skill.
Inspired by these intuitions, our method consists of two parts: (1) acquiring primitive skills with diverse behaviors by mutual information maximization, and (2) learning a meta policy that selects a skill for each end-effector and coordinates the chosen skills by controlling the behavior of each skill.
The main contribution of this paper is a modular and hierarchical approach that tackles cooperative manipulation tasks with multiple end-effectors by (1) learning primitive skills of each end-effector independently with skill behavior diversiﬁcation and (2) coordinating end-effectors using diverse
[Figure]
behaviors of the skills. Our empirical results indicate that our proposed method is able to efﬁciently learn primitive skills with diverse behaviors and coordinate these skills to solve challenging collaborative control tasks such as picking up a long bar, placing a block inside the container on the right side, and pushing a box with two ant agents.
We provide additional qualitative results and code at https://clvrai.com/coordination .
R ELATED WORK
Deep reinforcement learning (RL) for continuous control is an active research area.
However, learning a complex task either from a sparse reward or a heavily engineered reward becomes computationally impractical as the target task becomes complicated.
Instead of learning from scratch, complex tasks can be tackled by decomposing the tasks into easier and reusable sub-tasks.
Hierarchical reinforcement learning temporally splits a task into a sequence of temporally extended meta actions.
It often consists of one meta policy (high-level policy) and a set of low-level policies, such as options framework (Sutton et al., 1999).
The meta policy decides which low-level policy to activate and the chosen low-level policy generates an action sequence until the meta policy switches it to another lowlevel policy.
Options can be discovered without supervision (Schmidhuber, 1990; Bacon et al., 2017; Nachum et al., 2018; Levy et al., 2019), meta-learned (Frans et al., 2018), pre-deﬁned (Kulkarni et al., 2016;
Oh et al., 2017; Merel et al., 2019; Lee et al., 2019), or attained from additional supervision signals (Andreas et al., 2017; Ghosh et al., 2018).
However, option frameworks are not ﬂexible to solve a task that requires simultaneous activation or interpolation of multiple skills since only one skill can be activated at each time step.
To solve composite tasks multiple policies can be simultaneously activated by adding Qfunctions (Haarnoja et al., 2018a), additive composition (Qureshi et al., 2020; Goyal et al., 2020), or multiplicative composition (Peng et al., 2019).
As each policy takes the whole observation as input and controls the whole agent, it is not robust to changes in unrelated parts of the observation.
For example, a left arm skill can be affected by the pose change in the right arm, which is not relevant to the left arm skill.
Hence, these skill composition approaches can fail when an agent encounters a new combination of skills or a new skill is introduced since the agent will experience unseen observations.
Instead of having a policy with the full observation and action space, multi-agent reinforcement learning (MARL) suggests to explicitly split the observation and action space according to agents (e.g. robots or end-effectors), which allows efﬁcient low-level policy training as well as ﬂexible skill composition.
For cooperative tasks, communication mechanisms (Sukhbaatar et al., 2016; Peng et al., 2017; Jiang & Lu, 2018), sharing policy parameters (Gupta et al., 2017), and decentralized actor with centralized critic (Lowe et al., 2017; Foerster et al., 2018) have been actively used.
However, these approaches suffer from the credit assignment problem (Sutton, 1984) among agents and the lazy agent problem (Sunehag et al., 2018).
As agents have more complicated morphology and larger observation space, learning a policy for a multi-agent system from scratch requires extremely long training time.
Moreover, the credit assignment problem becomes more challenging when the complexity of cooperative tasks increases and all agents need to learn completely from scratch. To resolve these issues, we propose to ﬁrst train reusable skills for each agent in isolation, instead of
[Figure]
learning primitive skills of multiple agents together.
Then, we recombine these skills (Maes & Brooks, 1990) to complete more complicated tasks with learned coordination of the skills.
To coordinate skills from multiple agents, the skills have to be ﬂexible; hence, a skill can be adjusted to collaborate with other agents’ skills.
Maximum entropy policies (Haarnoja et al., 2017; 2018a;b) can learn diverse ways to achieve a goal by maximizing not only reward but also entropy of the policy.
In addition, Eysenbach et al.
(2019) proposes to discover diverse skills without reward by maximizing entropy as well as mutual information between resulting states and latent representations of skills (i.e. skill embeddings).
Our method leverages the maximum entropy policy (Haarnoja et al., 2018b) with the discriminability objective (Eysenbach et al., 2019) to learn a primitive skill with diverse behaviors conditioned on a controllable skill embedding. This controllable skill embedding will be later used as a behavior embedding for the meta policy to adjust a primitive skill’s behavior for coordination.
M ETHOD
In this paper, we address the problem of solving cooperative manipulation tasks that require collaboration between multiple end-effectors or agents. Note that we use the terms “end-effector” and “agent” interchangeably in this paper.
Instead of learning a multi-agent task from scratch (Lowe et al., 2017; Gupta et al., 2017; Sunehag et al., 2018;
Foerster et al., 2018), modular approaches (Andreas et al., 2017; Frans et al., 2018; Peng et al., 2019) suggest to learn reusable primitive skills and solve more complex tasks by recombining these skills.
However, concurrent execution of primitive skills of multiple agents fails when agents never experienced a combination of skills during the pre-training stage, or skills require temporal or behavioral coordination.
Therefore, we propose a modular and hierarchical framework that learns to coordinate multiple agents with primitive skills to perform a complex task.
Moreover, during primitive skill training, we propose to learn a latent behavior embedding , which provides controllability of each primitive skill to the meta policy while coordinating skills.
In Section 3.2, we describe our modular framework in detail.
Next, in Section 3.3, we elaborate how controllable primitive skills can be acquired.
Lastly, we describe how the meta policy learns to coordinate primitive skills in Section 3.4.
3. P RELIMINARIES
We formulate our problem as a Markov decision process deﬁned by a tuple {S , A , T , R, ρ, γ } of states, actions, transition probability, reward, initial state distribution, and discount factor.
In our formulation, we assume the environment includes N agents.
To promote consistency in our terminology, we use superscripts to denote the index of agent and subscripts to denote time or primitive skill index.
Hence, the state space and action space for an agent i can be represented as S i and A i where each element of S i is a subset of the corresponding element in S and A =
A 1 ×A 2 ×· · ·×A N , respectively.
For each agent i , we provide a set of m i skills, Π i = {
π i 1 , . . .
, π i m i } .
A policy of an agent i is represented as [Figure] π i c it ( a it | s i t ) ∈
Π i , where c i t is a skill index, s i t ∈ S
i is a state, and a i t ∈
A i is an agent action at time t .
An initial state s 0 is sampled from ρ , and then, N agents take actions a 1 t , a 2 t , . . .
, a Nt sampled from a composite policy π ( a 1 t , a 2 t , . .
. , a Nt | s t , c 1 t , c 2 t , . .
c Nt ) = ( π
[Equation]
t (
[Equation]
1 t | s t ) , π
[Equation]
[Equation]
2 t | s t ) , . .
, π Nc Nt ( a Nt | s t )) and receive a single reward r t .
The performance is evaluated based on a discounted return R = (cid:80) T −t =
γ t r t , where T is the episode horizon.
3. M ODULAR FRAMEWORK
As illustrated in Figure 2, our model is composed of two components: a meta policy π meta and a set of primitive skills of N agents Π 1 , . .
, Π N .
Note that each primitive skill π ic i ∈
Π i contains variants of behaviors parameterized by an N z -dimensional latent behavior embedding z
i
(see Section 3.3).
The meta policy selects a skill to execute for each agent, rather than selecting one primitive skill for the entire multi-agent system to execute.
Also, we give the meta policy the capability to select which variant of the skill to execute (see Section 3.4).
Then, the chosen primitive skills are simultaneously executed for T low time steps. The concurrent execution of multiple skills often leads to undesired results and therefore requires coordination between the skills.
For example, naively placing a block in the left hand to a container being moved by the right hand can cause collision between the two robot arms.
The arms can avoid collision while performing the skills by properly adjusting their skill behaviors (e.g. the left arm leaning to the left side while placing the block and the right arm leaning to the right side while pushing the container) as shown in Figure 1. In our method, the meta policy learns to coordinate multiple agents’ skills by manipulating the behavior embeddings (i.e. selecting a proper behavior from diverse behaviors of each skill).
3. T RAINING AGENT - SPECIFIC PRIMITIVE SKILLS WITH DIVERSE BEHAVIORS
To adjust a primitive skill to collaborate with other agents’ skills in a new environment, the skill needs to support variations of skill behaviors when executed at a given state.
Moreover, a behavioral variation of a skill should be controllable by the meta policy for skill coordination.
In order to make our primitive skill policies generate diverse behaviors controlled by a latent vector z , we leverage the entropy and mutual information maximization objective introduced in Eysenbach et al.
More speciﬁcally, a primitive policy of an agent i outputs an action a ∈ A conditioned on the current state s ∈ S and a latent behavior embedding z ∼ p ( z ) , where the prior distribution p ( z ) is Gaussian (we omit agent i in this section for the simplicity of notations).
Diverse behaviors conditioned on a random sample z can be achieved by maximizing the mutual information between behaviors and states MI ( s, z ) , while minimizing the mutual information between behaviors and actions given the state MI ( a, z | s ) , together with maximizing the entropy of the policy H ( a | s ) to encourage diverse behaviors.
The objective can be written as follows (we refer the readers to Eysenbach et al.
1: Input: Meta policy π meta , sets of primitive policies Π 1 , ..., Π N , and meta horizon T low 2: Initialize an episode t ← 0 and receive initial state s3: while episode is not terminated do 4: Sample skill indexes and behavior embeddings ( c 1 t , . . c Nt ) , ( z 1 t , . . z Nt ) ∼ π meta ( s t ) 5: τ
←6: while τ < T low and episode is not terminated do 7:
[Equation]
8: s t + τ +1 , r t + τ ← ENV ( s t + τ , a t + τ ) 9: τ ← τ +10: end while 11:
Add a transition s t , ( c 1 t , . .
c Nt ) , ( z 1 t , . .
z Nt ) , s t
+ τ , r t : t + τ − 1 to the rollout buffer B 12: t ← t + τ 13: end while for derivation):
[Equation] where the learned discriminator q φ ( z | s ) approximates the posterior p ( z | s ) .
To achieve a primitive skill with diverse behaviors, we augment Equation (3) to the environment reward:
[Equation]
where λ 1 is the entropy coefﬁcient and λ 2 is the diversity coefﬁcient which corresponds identiﬁability of behaviors.
Maximizing Equation (3) encourages multi-modal exploration strategies while maximizing the reward r t forces to achieve its own goal.
Moreover, by maximizing identiﬁability of behaviors, the latent vector z , named behavior embedding , can represent a variation of the learned policy and thus can be used to control the behavior of the policy.
For example, when training a robot to move an object, a policy learns to move the object quickly as well as slowly, and these diverse behaviors map to different latent vectors z .
We empirically show that the policies with diverse behaviors achieve better compositionality with other agents in our experiments.
3. C OMPOSING PRIMITIVE SKILLS WITH META POLICY
We denote the meta policy as π meta ( c 1 , . . . , c N , z 1 , . . .
, z N | s t ) , where c i ∈ [1 , m i ] represents a skill index of an agent i ∈ [1 , N ] and z i ∈ R N z represents a behavior embedding of the skill. Every T low time steps, the meta policy chooses one primitive skill π ic i ∈
Π i for each agent i . Also, the meta policy outputs a set of latent behavior embeddings ( z 1 , z 2 , . . . , z N ) and feeds them to the corresponding skills (i.e. π ic i ( a i | s i , z i ) for agent i ). Once a set of primitive skills { π 1 c 1 , . . .
, π Nc N } are chosen to be executed, each primitive skill generates an action a i ∼ π
ic i ( a i | s i , z i ) based on the current state s i and the latent vector z i .
Algorithm 1 illustrates the overall rollout process.
Since there are a ﬁnite number of skills for each agent to execute, the meta action space for each agent [1 , m i ] is discrete, while the behavior embedding space for each agent R N z is continuous.
Thus, the meta policy is modeled as a ( 2 × N )-head neural network where the ﬁrst N heads represent m i -way categorical distributions for skill selection and the last N heads represent N z -dimensional Gaussian distributions for behavior control of the chosen skill.
3. I MPLEMENTATION
We model the primitive policies and posterior distributions q φ as neural networks.
We train the primitive policies using soft actor-critic (Haarnoja et al., 2018b).
When we train a primitive policy, we use a unit Gaussian distribution as the prior distribution of latent variables p ( z ) .
We use 5 as the size of latent behavior embedding N z .
Each primitive policy outputs the mean and standard [Figure] deviation of a Gaussian distribution over an action space.
For a primitive policy, we apply tanh activation to normalize the action between [ − 1 , 1] .
We model the meta policy as neural network with multiple heads that output the skill index
c i and behavior embedding z
i for each agent.
The meta policy is trained using PPO (Schulman et al., 2017; 2016; Dhariwal et al., 2017).
All policy networks in this paper consist of 3 fully connected layers of 64 hidden units with ReLU nonlinearities.
The discriminator q φ in Equation (4) is a 2-layer fully connected network with 64 hidden units.
E XPERIMENTS
To demonstrate the effectiveness of our framework, we compare our method to prior methods in the ﬁeld of multi-agent RL and ablate the components of our framework to understand their importance.
We conducted experiments on a set of challenging robot control environments that require coordination of different agents to complete collaborative robotic manipulation and locomotion tasks.
Through our experiments, we aim to answer the following questions: (1) can our framework efﬁciently learn to combine primitive skills to execute a complicated task; (2) can our learned agent exhibit collaborative behaviors during task execution; and (3) can our framework leverage the controllable behavior variations of the primitive skills to achieve better coordination?
For details about environments and training, please refer to the supplementary material.
As the performance of training algorithms varies between runs, we train each method on each task withdifferent random seeds and report mean and standard deviation of each method’s success rate.
4. B ASELINES
We compare the performance of our method with various single- and multi-agent RL methods illustrated in Figure 3:
Single-agent RL (RL): A vanilla RL method where a single policy takes as input the full observation and outputs all agents’ actions.
Multi-agent RL (MARL): A multi-agent RL method where each of N policies takes as input the observation of the corresponding agent and outputs an action for that agent.
All policies share the global critic learned from a single task reward (Lowe et al., 2017).
Modular
Framework (Modular): A modular framework composed of a meta policy and N sets of
primitive skills (i
e. one
or more primitive skills per agent).
Every T low time steps, the meta policy selects a primitive skill for each agent based on the full observation. Then the chosen skills are
executed for T low time steps. [Figure] Single-agent RL with Skill Behavior Diversiﬁcation (RL-SBD): An RL method augmented with the behavior diversiﬁcation objective.
A meta policy is employed to generate a behavior embedding for a low-level policy, and the low-level policy outputs all agents’ actions conditioned on the behavior embedding and the full observation for T low time steps.
The meta policy and the low-level policy are jointly trained with the behavior diversiﬁcation objective described in Equation (4).
Multi-agent RL with Skill Behavior Diversiﬁcation (MARL-SBD): A MARL method augmented with the behavior diversiﬁcation objective.
A meta policy generates N behavior embeddings. Then, each low-level policy outputs each agent’s action conditioned on its observation and behavior embedding for T low time steps.
All policies are jointly trained to maximize Equation (4).
Modular Framework with Skill Behavior Diversiﬁcation (Modular-SBD, Ours): Our method which coordinates primitive skills of multiple agents. The modular framework consists of a meta policy and N sets of primitive skills, where each primitive skill is conditioned on a behavior embedding z .
The meta policy takes as input the full observation and selects both a primitive skill and a behavior embedding for each agent.
Then, each primitive skill outputs action for each agent.
4. J ACO P ICK -P USH -P LACE
We developed J ACO P ICK -P USH -P LACE and J ACO B AR -M
OVING environments using two Kinova
Jaco arms, where each Jaco arm is a 9 DoF robotic arm with 3 ﬁngers.
J ACO P ICK -P USH -P LACE starts with a block on the left and a container on the right.
The robotic arms need to pick up the block, push the container to the center, and place the block inside the container.
For successful completion of the task, the two Jaco arms have to concurrently execute their distinct sets of skills and dynamically adjust their picking, pushing, and placing directions to avoid collision between arms.
Primitives skills.
There are three primitive skills available to each arm: Picking up , Pushing , and
Placing to center (see Figure 4a).
Picking up requires a robotic arm to pick up a small block, which is randomly placed on the table.
If the block is not picked up after a certain amount of time or the arm drops the block, the agent fails. Pushing learns to push a big container to its opposite side (e.g. from left to the center or from right to center). The agent fails if it cannot place the container to the center. Placing to center requires placing an object in the gripper to the table.
The agent only succeeds when it stably places the object at the desired location on the container.
Composite task.
Our method (Modular-SBD) can successfully perform J ACO P ICK -P USH -P LACE task while all baselines fail to compose primitive skills as shown in Figure 5a.
The RL and MARL baselines cannot learn the composite task mainly because the agent requires to learn the combinatorial number of skill compositions and to solve the credit assignment problem across multiple agents.
Since the composite task requires multiple primitive skills of multiple agents to be performed properly at the same time, a reward signal about a failure case cannot be assigned to the correct agent or skill.
By using pre-trained primitive skills, the credit assignment problem is relaxed and all agents can [Table] perform their skills concurrently.
Therefore, the Modular baseline learns to achieve success but shows signiﬁcantly lower performance than our method (Modular-SBD).
This is because the lack of skill behavior diversiﬁcation makes it impossible to adjust pushing and placing trajectories during skill composition time, which resulting in frequent end-effector collisions.
4. J ACO B AR -M OVING
In J ACO B AR -M OVING
[Table] two Jaco arms need to pick up a long bar together, move the bar towards a target location while maintaining its rotation, and place it on the table (see Figure 4b).
The initial position of the bar is randomly initialized every episode and an agent needs to ﬁnd appropriate coordination between two arms for each initialization.
Compared to J ACO P ICK -P USH -P LACE , this task requires that the two arms synchronize their movements and perform more micro-level adjustments to their behaviors.
Primitives skills.
There are two pre-trained primitive skills available to each arm: Picking up and
Placing towards arm .
Picking up is same as described in Section 4.2.
Placing towards arm learns to move a small block (half size of the block used in the composite task) in the hand towards the robotic arm and then place it on the table.
The agent fails if it cannot place the block to the target location.
Composite task.
The J ACO B AR -M
OVING task requires the two arms to work very closely together.
For example, the Picking up skill of both arms should be synchronized when they start to lift the bar and two arms require to lift the bar while maintaining the relative position between them since they are connected by holding the bar.
The modular framework without explicit coordination of skills (Modular) can synchronize the execution of picking, moving, and placing.
But the inability to micro-adjust the movement of the other arm causes instability of bar picking and moving. This results in degraded success rates compared to the modular framework with explicit coordination.
Meanwhile, all baselines without pre-deﬁned primitive skills fail to learn J ACO B AR -M OVING .
4. A NT P USH
We developed a multi-ant environment, A NT -P USH , inspired from Nachum et al.
(2019), simulated in the MuJoCo (Todorov et al., 2012) physics engine. We use the ant model in OpenAI Gym (Brockman et al., 2016).
In this environment, two ants need to push a large object toward a green target place, collaborating with each other to keep the angle of the object as stable as possible (see Figure 4c).
Primitives skills.
We train walking skills of an ant agent in 4 directions: up, down, left, and right.
During primitive skill training, a block (half size of the block used in the composite task) and an ant agent are randomly placed. Pushing the block gives an additional reward to the agent, which prevents an ant to avoid the block.
The learned primitive skills have different speed and trajectories conditioned on the latent behavior embedding.
Composite task.
Our method achieves 32.3% success rate on A NT P USH task while all baselines fail to compose primitive skills as shown in Figure 5c and Table 1.
The poor performance of RL, MARL, RL-SBD, and MARL-SBD baselines shows the difﬁculty of credit assignment between agents, which leads one of the ants moves toward a block and pushes it but another ant does not move.
Moreover, the Modular baseline with primitive skills also fails to learn the pushing task.
This result illustrates the importance of coordination of agents, which helps synchronizing and controlling the velocities of both ant agents to push the block toward the goal position while maintaining its rotation. [Figure] [Figure]
4. E FFECT OF DIVERSITY OF PRIMITIVE SKILLS
To analyze the effect of the diversity of primitive skills, we compare our model with primitive skills trained with different diversity coefﬁcients λ 2 = { 0 . 0 , 0 . 05 , 0 . 1 , 0 . 5 , 1 . 0 } in Equation (4) on
A NT P USH .
Figure 6 shows that with small diversity coefﬁcients λ 2 = { 0 . 05 , 0 . 1 } , the agent can control detailed behaviors of primitive skills while primitive skills without diversity ( λ 2 = 0 ) cannot be coordinated.
The meta policy tries to synchronize two ant agents’ positions and velocities by switching primitive skills, but it cannot achieve proper coordination without diversiﬁed skills.
On the other hand, large diversity coefﬁcients λ 2 = { 0 . 5 , 1 . 0 } make the primitive skills often focus on demonstrating diverse behaviors and fail to achieve the goals of the skills.
Hence, these primitive skills do not have enough functionality to solve the target task.
The diversity coefﬁcient needs to be carefully chosen to acquire primitive skills with good performance as well as diverse behaviors.
4. E FFECT OF SKILL SELECTION INTERVAL T low
To analyze the effect of the skill selection interval hyperparameter T low , we compare our method trained with T low = { 1 , 2 , 3 , 5 , 10 } on Jaco environments.
The success rate curves in Figuredemonstrate that smaller T low values in range [1 , 3] lead to better performance.
This can be because the agent can realize more ﬂexible skill coordination by adjusting the behavior embedding frequently.
In addition to the ﬁxed T low values, we also consider the variation of our method in which the skill behavior embedding is only sampled when the meta policy updates its skill selection.
Concretely, we set the value of T low to 1 but update ( z 1 t , . . .
, z Nt ) only if ( c 1 t , . . .
, c Nt ) (cid:54) =
( c 1 t − 1 , . . .
c
Nt − 1 ) .
We observe that in this setting, the meta policy at times switch back and forth between two skills in two consecutive time steps, leading to slightly worse performance compared to our method with small T low values.
This indicates that the meta policy needs to adjust the behavior embedding in order to optimally coordinate skills of the different agents.
C ONCLUSION
In this paper, we propose a modular framework with skill coordination to tackle challenges of composition of sub-skills with multiple agents.
Speciﬁcally, we use entropy maximization with mutual information maximization to train controllable primitive skills with diverse behaviors.
To coordinate learned primitive skills, the meta policy predicts not only the skill to execute for each agent (end-effector) but also the behavior embedding that controls the chosen primitive skill’s behavior.
The experimental results on robotic manipulation and locomotion tasks demonstrate that the proposed framework is able to efﬁciently learn primitive skills with diverse behaviors and coordinate multiple agents (end-effectors) to solve challenging cooperative control tasks. Acquiring skills without supervision and extending our method to a visual domain are exciting directions for future work. A CKNOWLEDGMENTS This project was funded by SKT.
The authors would like to thank Karl Pertsch and many members of the USC CLVR lab for helpful discussion.
A E NVIRONMENT DETAILS
The details of observation spaces, action spaces, number of agents, and episode lengths are described in Table 2.
All units in this section are in meters unless otherwise speciﬁed.
[Table]
A. E NVIRONMENT DESCRIPTIONS
In both Jaco environments, the robot works on a table with size (1 . 6 , 1 . 6) and top center position (0 , 0 , 0 . 82) .
The two Jaco arms are initialized at positions ( − 0 .
16 , − 0 .
16 , 1 . 2) and ( − 0 .
Left arm and right arm objects are initialized around (0 . 3 , 0 . 2 , 0 . 86) and (0 . 3 , − 0 . 2 , 0 . 86) respectively in all primitive training and composite task training environments, with small random position and rotation perturbation.
In the Jaco Pick-Push-Place task, the right jaco arm needs to pick up the object and place it into the container initialized at the other side of the table.
Success is deﬁned by contact between the object and the inner top side of the container.
In the Jaco Bar-Moving task, the two Jaco arms need to together pick the long bar up by height of 0 . 7 , move it towards the arms by distance of 0 . 15 , and place it back on the table.
Success is deﬁned by (1) the bar being placed within 0 .
04 away from the desired destination both in height and in xy-position and (2) the bar having been picked 0 .
7 above the table.
In the Ant Push task, the two ant agents need to push a big box together to the goal position.
The box has a size of 8 .
× 1 . 6 × 1 . 6 .
The distance between ants and the box is 20 cm and the distance between the box and the goal is 30 cm.
Initial positions have 1 cm of randomness and the agent has a randomness of 0 . 01 in each joint.
The task is considered as success when both the distances between left and right end of the box and the goal are within 5 cm.
A. R EWARD DESIGN
For every task, we add a control penalty, − 0 .
001 ∗ (cid:107) a (cid:107) 2 , to regularize the magnitude of actions where a is a torque action performed by an agent.
Jaco Pick: To help the agent learn to reach, pick, and hold the picked object, we provide dense reward to the agent deﬁned by the weighted sum of pick reward, gripper-to-cube distance reward, cube position and quaternion stability reward, hold duration reward, success reward, and robot control reward.
More concretely,
[Equation]
a [Equation] where λ pick = 500 , λ dist = 100 , λ pos = 1000 , λ quat = 1000 , λ hold = 10 , λ success = 100 , λ ctrl = 1 × 10 − 4 .
Jaco Place: Reward for place primitive is deﬁned by the weighted sum of xy-distance reward, height reward (larger when cube close to ﬂoor), success reward, and robot control reward.
[Equation]
, where λ xy = 500 , λ z = 500 , λ success = 500 , λ ctrl = 1 × 10 − 4 .
Jaco Push: Reward for push primitive is deﬁned by the weighted sum of gripper reaching reward, box-to-destination distance reward, quaternion stability reward, hold duration reward, success reward, and robot control reward.
[Equation]
where λ reaching = 100 , λ pos = 500 , λ quat = 30 , λ hold = 10 , λ success = 1000 , λ ctrl = 1 × 10 − 4 .
Jaco Pick-Push-Place: Reward for Pick-Push-Place is deﬁned by the weighted sum of gripper contact reward, per-stage reach/pick/push/place rewards, success reward, and control reward.
We tune the reward carefully for all baselines.
[Equation]
where λ reach = 10 , λ contact = 10 , λ pick = λ place = λ place = 10 ,
λ success = 50 , λ ctrl = 0 , and 1 reaching , 1 pick , and 1 place are indicator functions specifying whether the agent is in reaching, pick or place stage.
Agent stages are determined by how many multiples of 25 steps the agent has stepped through in the environment.
Jaco Bar-Moving: Reward for Bar-Moving is deﬁned by the weighted sum of per-stage reach/pick/move/place rewards, success reward, and control reward.
where λ reach = 10 , λ pick = 30 , λ move = 100 , λ place = 100 , λ success = 100 , λ ctrl = 1 × 10 − 4 , and 1 pick and 1 place are indicator functions specifying whether the agent is in pick or place stage.
Agent stages are determined by whether the pick objective is fulﬁlled or not.
Ant Push & Ant Moving: Reward for A NT P USH is deﬁned by upright, velocity towards the desired direction.
We provide a dense reward to encourage the desired locomotion behavior using velocity, stability, and posture, as following:
[Equation]
where λ vel = 50 , λ boxvel = 20 , λ upright = 1 , λ height = 0 . 5 .
For A NT P USH , we provide an additional reward based on distance between the box and the goal position with λ goal = 200 .
B E XPERIMENT DETAILS
We use PyTorch (Paszke et al., 2017) for our implementation and all experiments are conducted on a workstation with Intel Xeon Gold 6154 CPU and 4 NVIDIA GeForce RTX 2080 Ti GPUs. [Table]
[Table]
N ETWORK ARCHITECTURES
Actor Networks:
In all experiments, we model our actor network for each primitive skill as a 3-layer
MLP with hidden layer size 64.
The last layer of the MLP is two-headed – one for the mean of the action distribution and the other for the standard deviation of it.
We use ReLU as activation function in hidden layers.
We do not apply any activation function for the ﬁnal output layer.
The action distribution output represents per-dimension normal distribution, from which single actions can be sampled and executed in the environment.
Critic Networks:
The critic network for each primitive skill and meta policy is modeled as a 2-layer
MLP with hidden layer size 128.
ReLU is used as an activation function in the hidden layers.
The critic network output is used to assess the value of a given state-action pair, and is trained by ﬁtting its outputs to the target Q-value clamped by ± 100 .
Meta Policy: The meta policy is modeled as a 3-layer MLP with hidden layer size of 64.
Since the actions of meta policy are sampled from N categorical distributions for each end-effector/agent and N normal distributions for behavior embeddings, the output dimension of the meta policy is (cid:80)
Ni =1 ( m i + N z ) .
The meta policy uses ReLU as an activation function for all layers except for the ﬁnal output layer.
B. T RAINING DETAILS
For all baselines, we train the meta policies using PPO and the low-level policies using SAC.
We use the same environment conﬁgurations, composite task reward deﬁnitions, and value of T low across all baselines.
For Jaco tasks, we train a total of 4 primitive skills – right arm pick, right arm place-to arm to by meta-policy.
Jaco Pick-PushPlace, we provide the meta-policy with right arm pick and right arm place- center right arm and left arm push as left arm
; for Jaco Bar-Moving, we provide the meta-policy with right arm pick and right arm place-towards-arm as both right and left arm and and arm
We obtain left arm primitives for bar-moving by using the learned right arm directly.
To obtain the 4 primitives skills described above, we train right arm pick with diversity coefﬁcient λ 2 = 0 .
and the other three primitives with λ 2 = 0 .
The destination of right arm push is set to (0 . 3 , − 0 . 03 , 0 . 86) , which is slightly left of the center of the table.
After pick primitive is trained, we train the two right arm place primitives where episodes are initialized by intermediate states of successful right arm pick episodes where the height of the box is larger than 0.94 (0.01 higher than the target pick height).
The place destinations for towards-arm and to-center primitives are (0 . 15 , − 0 . 2 , 0 . 86) and (0 . 3 , − 0 .
02 , 0 . 86) , respectively.
For non-modular baselines that incorporates skill behavior diversiﬁcation, we use λ 2 = 0 .
01 for both
Jaco Pick-Push-Place and Jaco Bar-Moving because both tasks require picking skills, which can only be trained with a small value of λ 2 .