M ULTI - TASK R EINFORCEMENT L EARNING WITH T ASK R EPRESENTATION M ETHOD
A BSTRACT
Multi-task reinforcement learning (RL) algorithms can train agents to acquire generalized skills across various tasks.
However, jointly learning with multiple tasks can induce negative transfer between different tasks, resulting in unstable training.
In this paper, we newly propose a task representation method that prevents negative transfer in policy learning.
The proposed method for multi-task RL adopts a task embedding network in addition to a policy network, where the policy network takes the output of the task embedding network and states as inputs.
Furthermore, we propose a measure of negative transfer and design an overall update method that can minimize the suggested measure.
In addition, we raise an issue of the negative effect on soft Q-function learning resulting in unstable Q learning and introduce the clipping method to reduce this issue.
The proposed multi-task algorithm is evaluated on various robotics manipulation tasks.
Numerical results show that the proposed multi-task RL algorithm effectively minimizes negative transfer and achieves better performance than previous state-of-the-art multi-task
RL algorithms
I NTRODUCTION
Reinforcement learning (RL) with deep neural networks has been applied successfully to various fields, e.g., playing Atari games from raw pixel images (Mnih et al., 2015), the game of Go (Silver et al., 2016), control of locomotion skills (Schulman et al., 2015; Lillicrap et al., 2015; Schulman et al., 2017; Haarnoja et al., 2018b; Fujimoto et al., 2018), etc.
Despite such success of deep RL, most deep RL algorithms solve each task independently so that they suffer from the lack of sample efficiency on complex tasks.
Learning multiple tasks with individual policies requires a large amount of memory and samples, whereas training a single policy network that generalizes across a given set of tasks is challenging in RL.
In particular, generalized skills across diverse tasks are necessary to apply RL algorithms in real robot control.
Multi-task learning (Caruana, 1997) is one approach to this problem.
In multi-task RL, the agent needs to train a policy that can be generalized across diverse sets of tasks.
The agent can train a policy parameterized by a neural network with multiple tasks, improving sample efficiency by sharing and reusing the parameters across different tasks.
[Figure]
However, multi-task RL has a challenging problem called negative transfer in which the training of some tasks can negatively affect other tasks, resulting in instability in training (Sun et al., 2019).
For example, Fig. 1 shows that the gradients of task 1 and tasks 2 and do not align, so the gradient direction of task 1 harms the learning of tasks 2 of 3.
Note that as the number of tasks increases, the negative transfer occurs more often.
To tackle this issue, the policy with multiple modules (Haarnoja et al., 2018a; Andreas et al., 2017; Yang et al., 2020) was introduced to prevent the interference between tasks using the different module for each task.
In particular, the gradient surgery method proposed in (Yu et al., 2020a) keeps off negative transfer by aligning the gradient direction of each task using the projection method. In
Generalizable practice, however, the gradient of each task contains large noisy component because it is computed by sampling a mini-batch from replay buffer, resulting in an inaccurate projection of gradient.
In this paper, we focus on preventing negative transfer on both policy π and its action value function
Q π .
We propose a task representation method which effectively restrains the negative transfer in policy learning.
This approach consists of a task embedding network and a policy network and a learning algorithm to minimize interference between tasks.
The task embedding network receives its input from the task ID designed as a one-hot encoding vector and the current state, and outputs the representation of task information at the current time.
The policy network takes the current state and the representation from the task embedding network as input, and outputs an action at the current time.
In the learning process, we measure the amount of negative transfer according to the update of policy for each task, and train the task embedding network to minimize the measurement through a gradient-based meta-learning approach (Finn et al., 2017; Ji et al., 2020).
Then, we analyze the interference on Q function between different tasks, and introduce a simple clipping method on Q function, allowing stable Q learning.
The detail will be explained in the following sections.
We evaluate the proposed method in Meta-World (Yu et al., 2020b), which contains 50 manipulation tasks using the robotic arm, and achieve better performance than previous state-of-the-art multi-task algorithms.
B ACKGROUND 2. S ETUP
We consider RL composed of an environment E and an agent, and assume that the system is modeled as a discrete-time finite-horizon Markov decision process (MDP) denoted by M = ( S , A , P , r, ρ, γ, H ) , where S is the state space,
A is the action space, P : S ×
A × S → R + is the transition probability, r : S ×
A → R + is the reward function, ρ : S → R
+ is the initial state distribution, γ ∈ [0 , 1) is the discount factor, and H is the time horizon.
We assume that the environment is fully-observable so that the environment state s t is available to the agent.
At each time step t , the agent observes a state s t ∈ S from the environment E and executes an action a t ∈ A based on its policy π ( a t
| s t ) .
Then, the environment gives the agent a reward r t according to the reward function r t = r
( s t , a t ) and makes a transition to a next state s t +1 according to the transition probability P ( s t +1 | s t , a t ) .
The goal of basic RL is to maximize the expected discounted accumulated rewards J ( π ) =
E π (cid:104) (cid:80) Tt =
γ t r ( s t , a t ) (cid:105) .
We use parameterized policy π θ ( a t | s t ) which is trained to maximize J ( π θ ) .
2. S OFT A CTOR -C RITIC
Soft Actor-Critic (SAC) (Haarnoja et al., 2018b) is an off-policy RL algorithm, where the action from the actor is encouraged to be sampled uniformly by increasing the entropy of action.
The policy objective function of SAC is given by J SAC ( π ) =
E π [ (cid:80) ∞ t =
γ t ( r t + β H
( π ( ·| s t )))]
where
H is the entropy function and β > 0 is the weighting for the entropy term.
To optimize J SAC ( π ) ,
SAC optimizes alternately the parameterized Soft Q-function Q ψ and the policy π θ .
The policy loss function and soft Q-function loss are given by
[Equation]
[Equation] [Equation]
where ¯ ψ is a parameter for target Q-value network.
The coefficient β is trained to maintain the entropy H ( π ( ·| s t )) to a target entropy ¯ H (Haarnoja et al., 2018c) with
[Equation]
2. M ULTI - TASK R EINFORCEMENT L EARNING
Under the setup of multi-task RL, we consider a task set C
= {T i } Ni =1 and assume a uniform distribution p ( T ) over the task set C .
Here, each task T i has a different MDP M i =
( S , A , P i , r i , ρ i , γ, H ) , and captures a situation where the reward function r
i and the transition probability P i are different Generalizable across the tasks.
Each task T drawn from the distribution p ( T ) has a maximization objective given by the expected discounted accumulated rewards, J ( π, T ) .
The goal of Multi-task RL is to learn a policy that maximizes the overall expected returns over the tasks E τ ∼ p
( T )
[ J ( π, T )]
In this paper, we train a generalized policy π ( a | s, z ) across the tasks with the SAC algorithm as the background algorithm, where z represents an encoding of the task ID, i.e., one-hot task identification encoding.
The policy objective of multi-task RL is then given by E T ∼ p ( T )
[ J SAC ( π, T )] .
We assume that the agent has an individual soft Q-function Q πi ( s, a ) for each task T i , where each soft Q-function is trained by the loss (2).
The coefficient of entropy β i exists for each task T i in the multi-task RL setting, and each β i is optimized by J ( β i ) =
E a ∼ π, T i [ − β i log π ( a | s, z ) − β
i ¯ H ] .
M ULTI -T ASK R EINFORCEMENT L EARNING WITH T ASK E MBEDDING N ETWORK
In multi-task learning, we wish that the knowledge that is acquired from learning each task can be exploited and helps learn other tasks better.
During multi-task learning, however, the training of one task can negatively affect the training of others, resulting in negative transfer which induces instability in learning.
In this section, we investigate how the negative transfer affects other tasks’ soft Q-function
Q πi learning, and introduce a simple clipping technique to prevent from harming Q πi .
Then, we propose a task embedding network for multi-task RL, which provides task representation to policy and prevents the negative transfer effect on policy training.
From the proposed task embedding network, we introduce a measure of negative transfer for each task, which is used to reduce interference with other tasks in policy training.
Finally, we propose an additional experience replay per task, which stores the top K trajectories on return for the task embedding network training.
3. P OLICY N ETWORK WITH T ASK E MBEDDING N ETWORK
In the proposed method, we have a generalized policy π θ ( a | s, z ) parameterized by θ , and a task embedding network E ϕ ( z | s, z T ) parameterized by ϕ for each task T .
Here, the proposed networks are shared across all tasks.
Note that E ϕ ( ·| s, z T ) is a distribution over the representation space Z .
At each time, the task embedding network takes the input of the current state s t and the embedding of the task ID z T , and outputs the representation z ∼ E ϕ ( z | s, z T ) , where z T is one-hot task identification encoding.
The policy π θ ( a | s, z ) is learned to achieve the goal of multi-task RL by taking the representation z as an additional input in addition to the current state s t .
3. M EASURE OF N EGATIVE T RANSFER
For each task T i , we have the policy objective as J SAC ( π θ , E ϕ , T i ) , and the policy parameter θ should be trained to maximize E T
∼ p ( T )
[ J SAC ( π θ , E ϕ , T )] by gradient ascent.
However, there can exist some task T k ∈ C whose gradient direction for its objective harms the objectives of other tasks, and this makes it challenging to achieve the goal of the multi-task RL.
Therefore, to tackle this problem, we investigate when negative transfer can occur.
Note that for notational simplicity, denote
J SAC ( π θ , E ϕ , T ) as J SAC ( θ, ϕ, T ) .
Negative Transfer: Consider a task objective set J set =
{ J SAC ( θ, ϕ, T i ) } Ni =1 .
From the policy objective J SAC ( θ, ϕ, T i ) of each task T i , we update the policy parameter θ to maximize the policy objective by gradient-ascent.
Let this updated policy parameter be θ i , i.e.,
[Equation]
where α > 0 is the gradient step size.
From the updated policy parameter θ i , we renew the task objective set J set to J iset :
[Equation]
Assuming that a task T k causes many negative transfers to other tasks, the task T k increments the number of tasks whose policy objective is lower than before the parameter was updated to θ k .
Let us denote the number of such tasks as µ ( T k ) , i.e.,
[Equation]
[Figure]
𝒛 ∼ 𝑬 𝝓 (⋅|𝒔,𝒛 𝓣 ) 𝒂 ∼ 𝝅 𝜽 (⋅|𝒔,𝒛) [Figure] Embedding [Figure]
Network
[Figure]
𝒔 Task ID 𝒛 𝓣 𝒛 𝒂 π 𝜽 𝟏 (⋅|𝒔,𝒛) π 𝛉 𝟐 (⋅|𝐬,𝐳) π 𝛉 𝐍 (⋅|𝐬,𝐳) [Figure] 𝑺𝑨𝑪 (𝜽,𝝓,𝓣) 𝑱 [Figure]
(𝝓) ො𝒔 [Figure]
𝝋 (𝒛,𝒛 𝓣 )
[Figure]
Measurement of Negative Transfer: µ ( T k ) in Equation 6 indicates how many tasks are negatively affected by updating the parameter θ to θ k .
Hence, µ ( T k ) can be considered as a measure of negative transfer on task T k .
So, we define µ ( T k ) as the measure of negative transfer caused by task T k .
Note that when µ ( T k ) is large, task T k induces large negative transfer.
3. T RAINING FOR P OLICY N ETWORK
The goal of training policy π θ is to maximize the average expected return over the task set C , i.e., E
T ∼ p ( T )
[ J SAC ( θ, ϕ, T )] .
Based on the measure of negative transfer defined above, we determine the set of tasks benign to other tasks in the middle of learning, and propose an update rule to train the policy with learning stability.
Task Selection:
We choose a benign set C s of tasks from the task set C .
For this, we order the negative transfer measure values µ ( T i ) , i = 1 , · · · , N and then select the tasks with the smallest M values.
That is, C s :=
{T j | µ ( T j ) ≤ µ ( T κ ( M ) ) } , where κ ( i ) is defined as
[Equation]
Then, instead of using the policy loss equally weighted over the task set C , we give the tasks in
C s more weight to reduce negative transfer on the policy learning.
The weight w i for task T i is proportional to the exponential of N − µ ( T i ) as
[Equation]
where N is the number of tasks.
Thus, we consider the following objective function for the policy:
[Equation]
3. T RAINING FOR T ASK E MBEDDING N ETWORK
In order to prevent negative transfer, the task embedding network E ϕ should be trained to decrease µ ( T k ) for all tasks T k ∈ C .
However, it is difficult to decrease µ ( T k ) directly because µ ( T k ) is not differentiable with respect to the parameter ϕ .
Thus, we consider an alternative cost function for ϕ as maximizing the average over the task objective set in Equation 5 for each task T k :
[Equation]
Since the update rule in Equation 4 also depends on the parameter ϕ , we rewrite the Equation 10 as
[Equation]
Thus, optimizing the objective in Equation 11 can be seen as training the task embedding network
E ϕ not to perform well for a given policy π θ but to perform well after updating θ to θ k , similar to gradient-based meta-learning approaches (Finn et al., 2017; Ji et al., 2020).
For training of the embedding network, rather than considering all tasks in C , we focus on the tasks in the set C \ C s , which is the set of harmful tasks in training policy π θ , and mitigate the negative effect by each task
T ∈ C \ C s .
Thus, the final objective function for parameter ϕ to suppress negative transfer on policy learning is written as
[Equation]
Variational Approach to Task Embedding Network: The representation z ∼ E ϕ ( ·| s, z T ) should capture the sufficient feature of each task and the current state so that the policy can generalize over the task set C by taking the representation z as an additional input.
In order to capture relevant features of state and task, we consider an additional objective through a variational approach (Kingma & Welling, 2013).
Let the decoder network D φ ( z, z T ) be parametrized by φ , which takes the representation z from the embedding network E ϕ and the embedding of the task ID z T as input.
The goal of the decoder network is to reconstruct the state s from z ∼ E ϕ ( ·| s, z T ) and z T so that the representation z becomes a sufficient feature for current state s and task information T .
Thus, we consider an additional objective through variational approach in addition to the negative transfer loss:
[Equation]
) where p ( z ) is the standard normal distribution.
Then, we obtain the final objective for the task embedding network and the decoder network as follows:
[Equation]
[Equation]
where c vae > 0 is the weighting coefficient between the two terms.
3. N EGATIVE E FFECTS ON S OFT Q- FUNCTION
In practice, we consider an individual soft Q-function Q ψ i ( s, a ) parameterized by ψ i for each task T
i
so that the parameters { ψ i } Ni =1 are optimized individually.
For a given state, action, reward, and next state pair d = ( s, a, r, s ′ ) , we can write the soft Bellman error of Q-function as
[Equation] [Equation]
where ¯ ψ i is the parameter of the target Q-function network for each task T i .
If the current policy parameter θ is significantly affected from other tasks, the Q-value Q ( s ′ , a ′ ) in
Equation 15 has a poor estimate because the distribution of the replay buffer may not correspond with the distribution under the current policy, which results in unstable learning of soft Q-function (Fujimoto et al., 2019).
[Equation]
e
[Equation]
[Equation]
(a) Reach-v
[Equation]
steps (1e6) 0.
e v a l - s u cc e ss - r a t i o 0 250750 1000 m e a n - s q - b e ll m a n - e rr o r 1 ( 1 e 6 ) (b) Pick-Place-v
To see the negative effect on soft Q-value, we experiment a multi-task SAC algorithm
(Yu et al., 2020b) with a shared policy and individual soft Q-function on MT benchmarks in the Meta-World (Yu et al., 2020b), and investigate the soft Bellman error for each task.
Fig.shows the success ratio and soft Bellman error of two specific tasks (Reach-v2 and Pick-place-v2) in MT10, where the orange curve shows the average of the Bellman error J Q ( d, ψ i ) , and the blue curve shows the average of success ratio of the given task.
It is seen that the values of Bellman error become very large after a certain time step, significantly reducing the success ratio of each task.
Note that the scale of Bellman error is 1e6.
Clipping Q Loss:
In order to solve the instability of individual soft Q-function learning, we propose a simple technique that clips the soft Bellman error (15) for a given pair ( s, a, s ′ ) not to be so large.
Thus, the individual Q-function loss with clipping is given by
[Equation]
where D is mini-batch and V clip is a maximum value of that does not clip.
3. T OP - K E XPERIENCE R EPLAY
The policy objective J SAC is obtained by sampling a mini-batch from the experience replay so that the selected task set C s may change largely depending on the sampled mini-batch.
Thus, instead of selecting the task set C s for every time step, we select the task set
C s using the task selection method for every H steps, where H is the horizon length of environments.
In the original experience replay buffer, there are many trajectories whose return may be high or low, and these diverse samples can make the uncertainty on the task selection method.
Since the chosen task set C s plays an important role in preventing negative transfer in our method, the proposed task selection method should be more reliable.
In order to reduce the uncertainty in the task selection method, we build another buffer that stores K trajectories with high returns from the original buffer, and train the task embedding network E ϕ by sampling mini-batchs from this replay buffer storing high-return trajectories.
The overall architecture of the proposed method is illustrated in Fig.
and is summarized in Algorithm
E XPERIMENT
In this section, we describe the environment setting, baselines, and implementation details.
4. E NVIRONMENT
Each environment used in our experiment has a distinct task set C
= {T 1 , . . .
, T N } , where N is the number of tasks.
Each task’s reward function and transition probability are different, but the remaining setup such as the state and action spaces is identical.
In order to test how well the proposed method can perform on multiple complex tasks, we tested our approach with the environment proposed in
Meta-World (Yu et al., 2020b), which has 50 distinct robotic control tasks with a sawyer arm in the
MuJoCo environment (Todorov et al., 2012).
Our experiments use two multi-task benchmarks, MT (Yu et al., 2020b) and MT20 with 10 and 20 manipulation tasks, respectively, from the Meta-World environment.
The task sets of MT10 and MT20 are given in Appendix B.
4. B ASELINES
We compared the proposed method with the baseline methods: 1) SAC (Haarnoja et al., 2018b) with a single-task (SAC-Ind): An individual policy for each task with the SAC algorithm.
2) SAC with multi-task (SAC-MT): A shared policy with a one-hot task identification encoding and current state as input.
3) SAC with Multi-task Multi-head (SAC-MT-MH) (Yu et al., 2020b):
It is similar to SAC with multi-task but has an independent final layer in the policy network for each task (multi-head).
SAC with soft modularization (SAC-soft-modular) (Yang et al., 2020): Policy with multiple modules with soft modularization technique that gives routing strategy for each task.
5) SAC with clipping in multi-task setting (SAC-MT-with-clipping): SAC-MT with our proposed clipping method on soft
Q-function learning.
R ESULTS
In this section, we evaluate the proposed multi-task SAC with a task embedding network in various robotics manipulation tasks.
We first provide a performance comparison between the proposed method and previous multi-task RL methods on several benchmarks and then give an ablation study on components constituting the proposed algorithm.
5. R ESULTS ON MULTIPLE TASKS
[Figure]
Algorithm
MT
[Table]
Ours
[Table]
We first examined the performance of our approach compared with baseline algorithms on MT10 and MT benchmarks.
Fig.
4 shows the training curve of the average success ratio, and Table 1 shows the final average success ratio for each algorithm.
The success ratio is obtained by the average of all tasks’ success ratios for given task set, where the success ratio of each task is well defined (Yu et al., 2020b).
We ran the experimenttimes for each algorithm with different seeds and plotted the mean success ratio with standard deviation per every 200 episodes.
We trained each algorithm with 5 million samples per task on the MT10 and MT20 task sets.
Results on MT10.
As seen in Table 1, our approach’s final success ratio is not only close to single task
SAC (upper bound) but also 6% better than the best multi-task baseline algorithm SAC-soft-modular (Yang et al., 2020).
Furthermore, SAC-MT combined with our Q clipping method also outperforms other baseline algorithms but has lower performance than our algorithm.
Results on MT20.
As the number of tasks increases, the multi-task RL problem becomes more challenging since more negative transfer can occur.
Nevertheless, as seen in Table 1, our approach achieves the best final success ratio compared with other multi-task baselines, where the proposed method has 4.8% better success ratio than the best multi-task baseline algorithm SAC-MT-MH (Yu et al., 2020b).
Furthermore, the SAC-MT combined with our Q clipping method is also similar to other baseline algorithms but has lower performance than our algorithm.
The training curves for all tasks in the benchmarks are shown in Appendix C.
5. A BLATION S TUDIES Effects on Method for Preventing Negative Transfer
0 1 2 3 4 5 Time steps (1e6) 0.
e v a l - s u cc e ss - r a t i o 0.0 0.
t r a i n - i n d e x (a)
Button-press- topdown-v 0 1 2 3 4 5 Time steps (1e6) 0.
e v a l - s u cc e ss - r a t i o 0.0 0.
t r a i n - i n d e x
[Figure]
open-v
In our method, the measure of negative transfer for a given task is important.
In order to see how the measure changes during the learning, we investigated the relationship between the success ratio and the train frequency for a given task.
Here, the train frequency is the rate at which the policy is trained on a given task over a period of 20 episodes, inversely proportional to the measure of negative transfer from the definition of the task set C s in Equation 6.
The results for two tasks, Button-presstopdown-v2 and Window-open-v2, are shown in Fig.
5, where the blue curve shows the average success ratio and the orange curve shows the average train frequency.
[Figure]
e v a l
[Figure]
s u cc e ss
[Figure]
r a t i o
[Figure]
e a n [Figure] s q [Figure]
b e ll
[Figure]
a n
[Figure]
e rr o r
[Figure]
[Figure]
It is seen that the train frequency decreases as the two tasks are quickly learned at the beginning of learning, indicating that rapid policy learning in a given task can make frequent negative transfer to other tasks.
In addition, Fig. 5 shows our approach not only effectively prevents negative transfer effect on other tasks over time but also maintains the success ratio.
Hence, we conclude that the proposed method can efficiently prevent the interference between different tasks.
Effects on Clipping Method
We propose the simple clipping method to mitigate the adverse effect on soft Q-function for joint learning for multi-task shown in Fig 3.
In order to check the clipping effect, we ran an experiment for SAC-MT combined with the clipping Q-loss.
The result is shown in Fig.
6, where the orange curve shows the average of the Bellman error J Q ( d, ψ i ) , and the blue curve shows the average of the success ratio of the given task.
It is seen that the clipping method effectively reduces the soft Bellman error so that the performance of each task improves.
Effect of Components of Our Method
We analyzed the importance of three techniques in our method with MT10: (1) Variational approach to extract sufficient representation.
(2) Clipping method to mitigate the adverse effects on soft
Q-function.
(3) Top-
k experience replay.
We compared performance by excluding each component in our method.
This is denoted as (1) Ours-VAE, (2) Ours-clip, and (3) Ours-bestbatch.
The result is shown in Fig.
It is seen that the average success ratio is reduced when one of the components is excluded.
Thus, each component has an important role in our approach, and the VAE approach affects performance the most among our components.
C ONCLUSION AND F UTURE W ORK
In this paper, we have considered jointly learning of multi-task RL and have proposed a new method for preventing negative transfer on both policy and soft Q-function.
We have proposed a task embedding network and a Q clipping method to mitigate the negative transfer problem and have proposed top- K experience replay for stable task selection.
Numerical results show that the proposed method effectively suppress the negative transfer between different tasks so that the approach improves the performance compared with other baselines.  ACKNOWLEDGMENTS
This research was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government.
(MSIT) (2021R1A2C2009143)
A R ELATED W ORK
Multi-task learning: Multi-task learning (Caruana, 1997) is one of the active research areas in machine learning.
In computer vision, researchers have shown that training with multiple objectives helps extract features for generalization (Zhang et al., 2014; Dai et al., 2016; Liu et al., 2019).
Multi-task RL also gains increasing attention from the research community (Wilson et al., 2007;
Pinto & Gupta, 2017; Zeng et al., 2018;
Hausman et al., 2018;
Yang et al., 2020).
Whereas sharing parameters with different tasks can effectively transfer the knowledge from each task, it can induce negative transfer between tasks.
One can prevent negative transfer through distillation methods (Rusu et al., 2015; Parisotto et al., 2015; Teh et al., 2017), but these require a distinct network per task.
Another approach to avoiding interference is to utilize different modules in multiple tasks using a modular network (Rusu et al., 2016; Devin et al., 2017;
Andreas et al., 2017; Haarnoja et al., 2018a;
Yang et al., 2020).
Researchers also propose finding a relationship between tasks by gradient from each task and using it to mitigate negative transfer (Zhang & Yeung, 2014;
Chen et al., 2018;
Hu et al., 2019; Yu et al., 2020a).
However, the gradient of a task has significant noise, so the task relationship by noisy gradient can destabilize the training.
Instead of using the gradient similarity, our approach measures the number of negative effects and proposes a task embedding network with its learning algorithm to prevent the negative transfer, allowing more stable training.
Representation learning: Representation learning (Bengio et al., 2013) has widely been studied in machine learning (Kingma & Welling, 2013;
Butepage et al., 2017; Chen et al., 2016; Hjelm et al., 2018; Chen et al., 2020).
Researchers have shown that multi-task learning extracts better features, containing helpful knowledge to learn other tasks (Maurer et al., 2016).
In multi-task RL, the researchers also propose representation methods for multi-task RL (Goyal et al., 2019; Hausman et al., 2018), where they focus on exploration for RL and generalization skills.
While most representation methods focus on finding features that can be helpful to training, our approach further concentrates on preventing negative transfer between different tasks.
B D ESCRIPTION OF B ENCHMARKS AND E XPERIMENT
In our experiment, we make two benchmarks MT10 and MT20 that contain 10 and 20 distinct environments from the 50 environments of Meta-World (Yu et al., 2020b), respectively.
The environments included in each benchmark are as follows:
• MT10 benchmark reach, push, pick and place, open door, open drawer, close drawer, press button topdown, insert peg side, open window, and open box.
• MT20 benchmark assemble nut, basketball, pick bin, close box, press button top-down, press button top wall, press button, press button wall, get coffee, push mug, pull mug, turn dial, disassemble nut, close door, lock door, open door, unlock door, insert hand, open drawer, and close drawer.
In all experiments, the policy, task embedding network, decoder network, and soft Q function are implemented as neural networks with two hidden layers of size 400 and ReLU activation, where the output size of the task embedding network is 8 with the same dimension of the representation space
Z .
In addition, we set the M value in the task selection method as 5 and 10 for MT10 and MT20, respectively.
The maximum clipping value V clip and the number of best trajectories K are set toand 30, respectively.
C T RAINING C URVES FOR EACH T ASK MT10 benchmark
[Equation]
[Table] [Table]
[Table]
[Table]
[Figure]
[Table]
[Figure]
R a t i o
[Figure]
D
A LGORITHM P SEUDO C ODE
Require: N, M, L, K
Initialize parameter θ, ϕ, φ, ψ i , i = 1 , 2 , . .
N
D i ← ∅ for each task T i for each iteration do for each gradient step do
Sample random a mini-batch of size L from top- K trajectories in each replay D
i
Compute J rep ( ϕ, φ ) , C s from the mini-batch ϕ ← ϕ
+ δ ∇ ϕ J rep ( ϕ, φ )
φ ← φ + δ ∇ φ J rep ( ϕ, φ ) end for for each environment step do for each environment T i do
Take action a it using π θ , E ϕ given state s it
Receive r it , s i t +1 from environment
D i ← D
i ∪ { ( s t , a t , r t , s t +1 ) } end for
Sample a random mini-batch of size L from each replay D
i
Compute J policy ( θ ) , { J QCLIP ( ψ i ) }
Ni =1 using the mini-batch and C s θ ← θ + δ ∇ θ J policy ( θ ) ψ i ← ψ
i − δ ∇ ψ
i J QCLIP ( ψ i ) for i = 1 , 2 , . . .
, N end for end for