S HARING K NOWLEDGE IN M ULTI -T ASK D EEP R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
We study the beneﬁt of sharing representations among tasks to enable the effective use of deep neural networks in Multi-Task Reinforcement Learning.
We leverage the assumption that learning from different tasks, sharing common properties, is helpful to generalize the knowledge of them resulting in a more effective feature extraction compared to learning a single task.
Intuitively, the resulting set of features offers performance beneﬁts when used by Reinforcement Learning algorithms.
We prove this by providing theoretical guarantees that highlight the conditions for which is convenient to share representations among tasks, extending the wellknown ﬁnite-time bounds of Approximate Value-Iteration to the multi-task setting.
In addition, we complement our analysis by proposing multi-task extensions of three Reinforcement Learning algorithms that we empirically evaluate on widely used Reinforcement Learning benchmarks showing signiﬁcant improvements over the single-task counterparts in terms of sample efﬁciency and performance.
I NTRODUCTION
Multi-Task Learning (MTL) ambitiously aims to learn multiple tasks jointly instead of learning them separately, leveraging the assumption that the considered tasks have common properties which can be exploited by Machine Learning (ML) models to generalize the learning of each of them.
For instance, the features extracted in the hidden layers of a neural network trained on multiple tasks have the advantage of being a general representation of structures common to each other.
This translates into an effective way of learning multiple tasks at the same time, but it can also improve the learning of each individual task compared to learning them separately (Caruana, 1997).
Furthermore, the learned representation can be used to perform Transfer Learning (TL), i.e. using it as a preliminary knowledge to learn a new similar task resulting in a more effective and faster learning than learning the new task from scratch (Baxter, 2000; Thrun & Pratt, 2012).
The same beneﬁts of extraction and exploitation of common features among the tasks achieved in MTL, can be obtained in Multi-Task Reinforcement Learning (MTRL) when training a single agent on multiple Reinforcement Learning (RL) problems with common structures (Taylor & Stone, 2009; Lazaric, 2012).
In particular, in MTRL an agent can be trained on multiple tasks in the same domain, e.g. riding a bicycle or cycling while going towards a goal, or on different but similar domains, e.g. balancing a pendulum or balancing a double pendulum 1 .
Considering recent advances in Deep Reinforcement Learning (DRL) and the resulting increase in the complexity of experimental benchmarks, the use of Deep Learning (DL) models, e.g. deep neural networks, has become a popular and effective way to extract common features among tasks in MTRL algorithms (Rusu et al., 2015;
Liu et al., 2016; Higgins et al., 2017).
However, despite the high representational capacity of DL models, the extraction of good features remains challenging.
For instance, the performance of the learning process can degrade when unrelated tasks are used together (Caruana, 1997; Baxter, 2000); another detrimental issue may occur when the training of a single model is not balanced properly among multiple tasks (Hessel et al., 2018).
Recent developments in MTRL achieve signiﬁcant results in feature extraction by means of algorithms speciﬁcally developed to address these issues.
While some of these works rely on a single deep neural network to model the multi-task agent (Liu et al., 2016; Yang et al., 2017; Hessel et al., 2018; Wulfmeier et al., 2019), others use multiple deep neural networks, e.g. one for each task and another for the multi-task agent (Rusu et al., 2015;
Parisotto et al., 2015; Higgins et al., 2017; Teh et al., 2017).
Intuitively, achieving good results in MTRL with a single deep neural network is more desirable than using many of them, since the training time is likely much less and the whole architecture is easier to implement.
In this paper we study the beneﬁts of shared representations among tasks.
We theoretically motivate the intuitive effectiveness of our method, deriving theoretical guarantees that exploit the theoretical framework provided by Maurer et al.
(2016), in which the authors present upper bounds on the quality of learning in MTL when extracting features for multiple tasks in a single shared representation.
The signiﬁcancy of this result is that the cost of learning the shared representation decreases with a factor O ( 1 / √ T ) , where T is the number of tasks for many function approximator hypothesis classes.
The main contribution of this work is twofold.
We derive upper conﬁdence bounds for Approximate Value-Iteration (AVI) and Approximate Policy-Iteration (API) 2 (Farahmand, 2011) in the MTRL setting, and we extend the approximation error bounds in Maurer et al.
(2016) to the case of multiple tasks with different dimensionalities.
Then, we show how to combine these results resulting in, to the best of our knowledge, the ﬁrst proposed extension of the ﬁnite-time bounds of AVI/API to MTRL.
Despite being an extension of previous works, we derive these results to justify our approach showing how the error propagation in AVI/API can theoretically beneﬁt from learning multiple tasks jointly.
We leverage these results proposing a neural network architecture, for which these bounds hold with minor assumptions, that allow us to learn multiple tasks with a single regressor extracting a common representation.
We show an empirical evidence of the consequence of our bounds by means of a variant of Fitted Q -Iteration (FQI)
(Ernst et al., 2005), based on our shared network and for which our bounds apply , that we call Multi Fitted Q -Iteration (MFQI).
Then, we perform an empirical evaluation in challenging RL problems proposing multitask variants of the Deep Q -Network (DQN) (Mnih et al., 2015) and Deep Deterministic
Policy Gradient (DDPG) (Lillicrap et al., 2015) algorithms.
These algorithms are practical implementations of the more general AVI/API framework, designed to solve complex problems.
In this case, the bounds apply to these algorithms only with some assumptions, e.g. stationary sampling distribution.
The outcome of the empirical analysis joins the theoretical results, showing signiﬁcant performance improvements compared to the singletask version of the algorithms in various RL problems, including several MuJoCo (Todorov et al., 2012) domains.
P RELIMINARIES
Let B ( X ) be the space of bounded measurable functions w.r.t.
the σ -algebra σ X , and similarly
B ( X , L ) be the same bounded by L < ∞ .
A Markov Decision Process (MDP) is deﬁned as a 5 -tuple M = < S , A , P , R , γ > , where S is the state space,
A is the action space, P : S ×
A → S is the transition distribution where P ( s (cid:48) | s, a ) is the probability of reaching state s (cid:48)
when performing action a in state s , R : S × A × S
→ R is the reward function, and γ ∈ (0 , 1] is the discount factor.
A deterministic policy π maps, for each state, the action to perform: π : S → A .
Given a policy π , the value of an action a in a state s represents the expected discounted cumulative reward obtained by performing a in s and following π thereafter: Q π ( s, a ) (cid:44) E [ (cid:80) ∞
k =0 γ k r i
+ k +
| s i =
s, a i = a, π ] , where r i + is the reward obtained after the i -th transition.
The expected discounted cumulative reward is maximized by following the optimal policy π ∗ which is the one that determines the optimal action values, i.e., the ones that satisfy the Bellman optimality equation (Bellman, 1954): Q ∗ ( s, a ) (cid:44) (cid:82)
S P ( s (cid:48) | s, a )
[ R ( s, a, s (cid:48) ) + γ max a (cid:48)
Q ∗ ( s (cid:48) , a (cid:48) )] ds (cid:48) .
The solution of the Bellman optimality equation is the ﬁxed point of the optimal Bellman operator T ∗ : B ( S × A ) →
B ( S × A ) deﬁned as ( T ∗ Q )( s, a ) (cid:44) (cid:82) S P ( s (cid:48) | s, a )
[ R ( s, a, s (cid:48) ) + γ max a (cid:48) Q ( s (cid:48) , a (cid:48) )] ds (cid:48) .
In the MTRL setting, there are multiple MDPs M ( t )
= <
S ( t ) , A ( t ) , P ( t ) , R ( t ) , γ ( t ) > where t ∈ { 1 , . . .
, T } and T is the number of MDPs.
For each MDP M ( t ) , a deterministic policy π t : S ( t ) →
A ( t ) induces an action-value function Q π t t ( s ( t ) , a ( t ) )
=
E [ (cid:80) ∞ k =0 γ k r ( t ) i + k +
| s i = s ( t ) , a i = a ( t ) , π t ] .
In this setting, the goal is to maximize the sum of the expected cumulative discounted reward of each task.
In our theoretical analysis of the MTRL problem, the complexity of representation plays a central role.
As done in Maurer et al. (2016), we consider the Gaussian complexity, a variant of the well-known
Rademacher complexity, to measure the complexity of the representation.
Given a set ¯X ∈ X Tn of n input samples for each task t ∈ { 1 , . . . , T } , and a class H composed of k ∈ { 1 , . . . , K } functions, the Gaussian complexity of a random set H ( ¯X ) = { [Equation]
h k ( X ti )) : h ∈ H} ⊆
R KTn is deﬁned as follows:
[Equation]
where γ tki are independent standard normal variables.
We also need to deﬁne the following quantity, taken from Maurer (2016): let γ be a vector of m random standard normal variables, and f ∈ F :
Y → R m , with Y ⊆ R n , we deﬁne
[Equation]
Equation 2 can be viewed as a Gaussian average of Lipschitz quotients, and appears in the bounds provided in this work.
Finally, we deﬁne L ( F ) as the upper bound of the Lipschitz constant of all the functions f in the function class F .
T HEORETICAL ANALYSIS
The following theoretical study starts from the derivation of theoretical guarantees for MTRL in the
AVI framework, extending the results of Farahmand (2011) in the MTRL scenario.
Then, to bound the approximation error term in the AVI bound, we extend the result described in Maurer (2006) to MTRL.
As we discuss, the resulting bounds described in this section clearly show the beneﬁt of sharing representation in MTRL.
To the best of our knowledge, this is the ﬁrst general result for
MTRL; previous works have focused on ﬁnite MDPs (Brunskill & Li, 2013) or linear models (Lazaric & Restelli, 2011).
3. M ULTI - TASK REPRESENTATION LEARNING
The multi-task representation learning problem consists in learning simultaneously a set of T tasks µ t , modeled as probability measures over the space of the possible input-output pairs ( x, y ) , with x ∈ X and y ∈ R , being X the input space.
Let w ∈ W : X → R J , h ∈ H :
R J
→ R K and f ∈ F : R K
→ R be functions chosen from their respective hypothesis classes.
The functions in the hypothesis classes must be Lipschitz continuous functions.
Let ¯Z =
( Z 1 , . . .
, Z T ) be the multi-sample over the set of tasks µ = ( µ 1 , . . .
, µ T ) , where Z t =
( Z t 1 , . . .
, Z tn ) ∼ µ nt and
Z ti =
( X ti , Y ti ) ∼ µ t .
We can formalize our regression problem as the following minimization problem:
[Equation]
where we use f = ( f 1 , .
. . , f T ) , w = ( w 1 , . .
. , w T ) , and deﬁne the minimizers of Equation (3) as ˆw , ˆ h , and ˆf .
We assume that the loss function (cid:96) : R × R → [0 , 1] is 1-Lipschitz in the ﬁrst argument for every value of the second argument.
While this assumption may seem restrictive, the result obtained can be easily scaled to the general case.
To use the principal result of this section, for a generic loss function (cid:96) (cid:48) , it is possible to use (cid:96) ( · ) =
(cid:96) (cid:48) ( · ) / (cid:15) max , where (cid:15) max is the maximum value of (cid:96) (cid:48) .
The expected loss over the tasks, given w , h and f is the task-averaged risk:
[Equation]
The minimum task-averaged risk, given the set of tasks µ and the hypothesis classes W , H and F is ε ∗ avg , and the corresponding minimizers are w ∗ , h ∗ and f ∗ .
3. M ULTI - TASK A PPROXIMATE V ALUE I TERATION BOUND
We start by considering the bound for the AVI framework which applies for the single-task scenario.
Theorem 1.
(Theorem 3.4 of Farahmand (2011))
Let K be a positive integer, and Q max ≤ R max 1 − γ .
Then for any sequence ( Q k ) Kk =
⊂ B ( S × A , Q max ) and the corresponding sequence ( ε k ) K − 1 k =0 , where ε k = (cid:107) Q k +1 − T ∗ Q k (cid:107) 2 ν , we have:
[Equation]
where
[Equation]
with E ( ε 0 , . . .
, ε K − 1 ; r ) =
(cid:80) K − 1 k =
α 2 rk ε k , the two coefﬁcients c VI 1 ,ρ,ν , c VI 2 ,ρ,ν , the distributions ρ and ν , and the series α k are deﬁned as in Farahmand (2011).
In the multi-task scenario, let the average approximation error across tasks be:
[Equation]
where Q t,k +
=
ˆ f t,k ◦ ˆ h k ◦ ˆ w t,k , and T ∗ t is the optimal Bellman operator of task t .
In the following, we extend the AVI bound of Theorem 1 to the multi-task scenario, by computing the average loss across tasks and pushing inside the average using Jensen’s inequality.
Theorem 2.
Let K be a positive integer, and Q max ≤ R max 1 − γ .
Then for any sequence ( Q k ) Kk =
⊂ B ( S ×
A , Q max ) and the corresponding sequence ( ε avg ,k ) K − 1 k =0 , where ε avg ,k = 1 T (cid:80) Tt =1 (cid:107) Q t,k +1 −T ∗ t Q t,k (cid:107) 2 ν , we have:
[Equation]
with E avg = (cid:80) K − {+1 k =
+}
α 2 rk ε avg ,k , γ =
[Equation]
T (cid:80) Tt =1 R max ,t and α k =
[Equation]
Remarks Theorem 2 retains most of the properties of Theorem 3.4 of Farahmand (2011), except that the regression error in the bound is now task-averaged.
Interestingly, the second term of the sum in Equation (8) depends on the average maximum reward for each task.
In order to obtain this result we use an overly pessimistic bound on γ and the concentrability coefﬁcients, however this approximation is not too loose if the MDPs are sufﬁciently similar.
3. M ULTI - TASK APPROXIMATION ERROR BOUND
We bound the task-averaged approximation error ε avg at each AVI iteration k involved in (8) following a derivation similar to the one proposed by Maurer et al.
(2016), obtaining:
Theorem 3.
Let µ , W , H and F be deﬁned as above and assume 0 ∈ H and f (0)
= 0 , ∀ f ∈ F .
Then for δ > 0 with probability at least 1 − δ in the draw of ¯ Z ∼ (cid:81)
Tt =1 µ
nt we have that
[Equation]
Remarks The assumptions 0 ∈ H and f (0)
=
for all f ∈ F are not essential for the proof and are only needed to simplify the result.
For reasonable function classes, the Gaussian complexity
G ( W ( X l )) is O ( √ n ) .
If sup w (cid:107) w ( ¯ X ) (cid:107) and sup h, w (cid:107) h ( w ( ¯ X )) (cid:107) can be uniformly bounded, then they are O ( √ nT ) .
For some function classes, the Gaussian average of Lipschitz quotients O ( · ) can be bounded independently from the number of samples.
Given these assumptions, the ﬁrst and the fourth term of the right hand side of Equation (9), which represent respectively the cost of learning the meta-state space w and the task-speciﬁc f mappings, are both O ( 1 / √ n ) .
The second term represents the cost of learning the multi-task representation h and is O ( 1 / √ nT ) , thus vanishing in the multi-task limit T → ∞ .
The third term can be removed if ∀ h ∈ H , ∃ p 0 ∈ P : h ( p ) = 0 ; even when this assumption does not hold, this term can be ignored for many classes of interest, e.g. neural networks, as it can be arbitrarily small.
The last term to be bounded in (9) is the minimum average approximation error ε ∗ avg at each AVI iteration k .
Recalling that the task-averaged approximation error is deﬁned as in (7), applying
Theorem 5.3 by Farahmand (2011) we obtain:
Lemma 4.
Let Q ∗ t,k , ∀ t ∈ { 1 , . .
, T } be the minimizers of ε ∗ avg ,k , ˇ t k = arg max
[Equation]
{ 1 ,...,T }
[Equation]
Q
[Equation]
t,k
[Equation]
− T ∗ t Q t,k (cid:107) 2 ν , and b k,i = (cid:107) Q ˇ t k ,i +1 − T ∗ ˇ t Q ˇ t k ,i (cid:107) ν , then:
[Equation]
with C AE deﬁned as in Farahmand (2011).
Final remarks The bound for MTRL is derived by composing the results in Theorems 2 and 3, and
Lemma 4.
The results above highlight the advantage of learning a shared representation.
The bound in Theorem 2 shows that a small approximation error is critical to improve the convergence towards the optimal action-value function, and the bound in Theorem 3 shows that the cost of learning the shared representation at each AVI iteration is mitigated by using multiple tasks.
This is particularly beneﬁcial when the feature representation is complex, e.g. deep neural networks.
3. D ISCUSSION
As stated in the remarks of Equation (9), the beneﬁt of MTRL is evinced by the second component of the bound, i.e. the cost of learning h , which vanishes with the increase of the number of tasks.
Obviously, adding more tasks require the shared representation to be large enough to include all of them, undesirably causing the term sup h, w (cid:107) h ( w ( ¯X )) (cid:107) in the fourth component of the bound to increase.
This introduces a tradeoff between the number of features and number of tasks; however, for a reasonable number of tasks the number of features used in the single-task case is enough to handle them, as we show in some experiments in Section 5.
Notably, since the AVI/API framework provided by Farahmand (2011) provides an easy way to include the approximation error of a generic function approximator, it is easy to show the beneﬁt in MTRL of the bound in Equation (9).
Despite being just multi-task extensions of previous works, our results are the ﬁrst one to theoretically show the beneﬁt of sharing representation in MTRL.
Moreover, they serve as a signiﬁcant theoretical motivation, besides to the intuitive ones, of the practical algorithms that we describe in the following sections.
S HARING REPRESENTATIONS
We want to empirically evaluate the beneﬁt of our theoretical study in the problem of jointly learning
T different tasks µ t , introducing a neural network architecture for which our bounds hold.
Following
[Figure]
our theoretical framework, the network we propose extracts representations w t from inputs x t for each task µ t , mapping them to common features in a set of shared layers h , specializing the learning of each task in respective separated layers f t , and ﬁnally computing the output y t =
( f t ◦ h ◦ w t )( x t ) =
f t ( h ( w t ( x t )))
(Figure 1(a)).
The idea behind this architecture is not new in the literature.
For instance, similar ideas have already been used in DQN variants to improve exploration on the same task via bootstrapping (Osband et al., 2016) and to perform MTRL (Liu et al., 2016).
The intuitive and desirable property of this architecture is the exploitation of the regularization effect introduced by the shared representation of the jointly learned tasks.
Indeed, unlike learning a single task that may end up in overﬁtting, forcing the model to compute a shared representation of the tasks helps the regression process to extract more general features, with a consequent reduction in the variance of the learned function.
This intuitive justiﬁcation for our approach, joins the theoretical beneﬁt proven in Section 3.
Note that our architecture can be used in any MTRL problem involving a regression process; indeed, it can be easily used in value-based methods as a Q -function regressor, or in policy search as a policy regressor.
In both cases, the targets are learned for each task µ t in its respective output block f t .
Remarkably, as we show in the experimental Section 5, it is straightforward to extend RL algorithms to their multi-task variants only through the use of the proposed network architecture, without major changes to the algorithms themselves.
E XPERIMENTAL RESULTS
To empirically evince the effect described by our bounds, we propose an extension of FQI (Ernst et al., 2005; Riedmiller, 2005), that we call MFQI, for which our AVI bounds apply.
Then, to empirically evaluate our approach in challenging RL problems, we introduce multi-task variants of two well-known DRL algorithms: DQN (Mnih et al., 2015) and DDPG (Lillicrap et al., 2015), which we call Multi Deep Q -Network (MDQN) and Multi Deep Deterministic Policy Gradient (MDDPG) respectively.
Note that for these methodologies, our AVI and API bounds hold only with the simplifying assumption that the samples are i.i.d.
; nevertheless they are useful to show the beneﬁt of our method also in complex scenarios, e.g. MuJoCo
(Todorov et al., 2012).
We remark that in these experiments we are only interested in showing the beneﬁt of learning multiple tasks with a shared representation w.r.t.
learning a single task; therefore, we only compare our methods with the single task counterparts, ignoring other works on MTRL in literature.
Refer to Appendix C for all the details and our motivations about the experimental settings.
5. M ULTI F ITTED Q -I TERATION
As a ﬁrst empirical evaluation, we consider FQI, as an example of an AVI algorithm, to show the effect described by our theoretical AVI bounds in experiments.
We consider the Car-On-Hill problem as described in Ernst et al.
(2005), and select four different tasks from it changing the mass of the car and the value of the actions (details in Appendix C).
Then, we run separate instances of FQI
[Figure]
with a single task network for each task respectively, and one of MFQI considering all the tasks simultaneously.
Figure 1(b) shows the L 1 -norm of the difference between Q ∗ and Q π K averaged over all the tasks.
It is clear how MFQI is able to get much closer to the optimal Q -function, thus giving an empirical evidence of the AVI bounds in Theorem 2.
For completeness, we also show the advantage of MFQI w.r.t.
FQI in performance.
Then, in Figure 1(c) we provide an empirical evidence of the beneﬁt of increasing the number of tasks in MFQI in terms of both quality and stability.
5. M ULTI D EEP Q -N ETWORK
As in Liu et al.
(2016), our MDQN uses separate replay memories for each task and the batch used in each training step is built picking the same number of samples from each replay memory.
Furthermore, a step of the algorithm consists of exactly one step in each task.
These are the only minor changes to the vanilla DQN algorithm
we introduce, while all other aspects, such as the use of the target network, are not modiﬁed.
Thus, the time complexity of MDQN is considerably lower than vanilla DQN thanks to the learning of T tasks with a single model, but at the cost of a higher memory complexity for the collection of samples for each task.
We consider ﬁve problems with similar state spaces, sparse rewards and discrete actions: Cart-Pole , Acrobot , Mountain-Car , Car-On-Hill , and Inverted-Pendulum .
The implementation of the ﬁrst three problems is the one provided by the
OpenAI Gym library Brockman et al.
(2016), while Car-On-Hill is described in Ernst et al.
and Inverted-Pendulum in Lagoudakis & Parr (2003).
Figure 2(a) shows the performance of MDQN w.r.t.
to vanilla DQN that uses a single-task network structured as the multi-task one in the case with T = 1 .
The ﬁrst three plots from the left show good performance of MDQN, which is both higher and more stable than DQN.
In Car-On-Hill, MDQN is slightly slower than DQN to reach the best performance, but eventually manages to be more stable.
Finally, the Inverted-Pendulum experiment is clearly too easy to solve for both approaches, but it is still useful for the shared feature extraction in MDQN.
The described results provide important hints about the better quality of the features extracted by MDQN w.r.t.
DQN.
To further demonstrate this, we evaluate the performance of DQN on Acrobot, arguably the hardest of the ﬁve problems, using a single-task network with the shared parameters in h initialized with the weights of a multi-task network trained with MDQN on the other four problems.
Arbitrarily, the pre-trained weights can be adjusted during the learning of the new task or can be kept ﬁxed and only the remaining randomly initialized parameters in w and f are trained.
From Figure 2(b), the advantages of initializing the weights are clear.
In particular, we compare the performance of DQN without initialization w.r.t.
DQN with initialization in three settings: in Unfreeze-0 the initialized weights are adjusted, in NoUnfreeze they are kept ﬁxed, and in Unfreeze-10 they are kept ﬁxed until epoch 10 after which they start to be optimized.
Interestingly, keeping the shared weights ﬁxed shows a signiﬁcant performance improvement in the earliest epochs, but ceases to improve soon.
On the other hand, the adjustment of weights from the earliest epochs shows improvements only compared to the uninitialized network in the intermediate stages of learning.
The best results are achieved by starting to adjust the shared weights after epoch 10 , which is approximately the point at which the improvement given by the ﬁxed initialization starts to lessen.
[Figure]
5. M ULTI D EEP D ETERMINISTIC P OLICY G RADIENT
In order to show how the ﬂexibility of our approach easily allows to perform MTRL in policy search algorithms
[Figure]
we propose MDDPG as a multi-task variant of DDPG.
As an actor-critic method, DDPG requires an actor network and a critic network.
Intuitively, to obtain MDDPG both the actor and critic networks should be built following our proposed structure.
We perform separate experiments on two sets of MuJoCo Todorov et al.
(2012) problems with similar continuous state and action spaces: the ﬁrst set includes Inverted-Pendulum , Inverted-Double-Pendulum , and Inverted-Pendulum-Swingup as implemented in the pybullet library, whereas the second set includes Hopper-Stand , Walker-Walk , and Half-Cheetah-Run as implemented in the DeepMind Control SuiteTassa et al.
Figure 3(a) shows a relevant improvement of MDDPG w.r.t.
DDPG in the pendulum tasks.
Indeed, while in
Inverted-Pendulum, which is the easiest problem among the three, the performance of MDDPG is only slightly better than DDPG, the difference in the other two problems is signiﬁcant.
The advantage of MDDPG is conﬁrmed in Figure 3(c) where it performs better than DDPG in Hopper and equally good in the other two tasks.
Again, we perform a TL evaluation of DDPG in the problems where it suffers the most, by initializing the shared weights of a single-task network with the ones of a multi-task network trained with MDDPG on the other problems.
Figures 3(b) and 3(d) show evident advantages of pre-training the shared weights and a signiﬁcant difference between keeping them ﬁxed or not.
C ONCLUSION
We have theoretically proved the advantage in RL of using a shared representation to learn multiple tasks w.r.t.
learning a single task.
We have derived our results extending the AVI/API bounds (Farahmand, 2011) to MTRL, leveraging the upper bounds on the approximation error in MTL provided in Maurer et al.
The results of this analysis show that the error propagation during the
AVI/API iterations is reduced according to the number of tasks.
Then, we proposed a practical way of exploiting this theoretical beneﬁt which consists in an effective way of extracting shared representations of multiple tasks by means of deep neural networks.
To empirically show the advantages of our method, we carried out experiments on challenging RL problems with the introduction of multi-task extensions of FQI, DQN, and DDPG based on the neural network structure we proposed.
As desired, the favorable empirical results conﬁrm the theoretical beneﬁt we described.
A R ELATED WORKS
Our work is inspired from both theoretical and empirical studies in MTL and MTRL literature.
In particular, the theoretical analysis we provide follows previous results about the theoretical properties of multi-task algorithms.
For instance, Cavallanti et al.
and Maurer (2006) prove the theoretical advantages of MTL based on linear approximation.
More in detail, Maurer (2006) derives bounds on
MTL when a linear approximator is used to extract a shared representation among tasks.
Then, Maurer et al.
(2016), which we considered in this work, describes similar results that extend to the use of non-linear approximators.
Similar studies have been conducted in the context of MTRL.
Among the others, Lazaric & Restelli (2011) and Brunskill & Li (2013) give theoretical proofs of the advantage of learning from multiple MDPs and introduces new algorithms to empirically support their claims, as done in this work.
Generally, contributions in MTRL assume that properties of different tasks, e.g. dynamics and reward function, are generated from a common generative model.
About this, interesting analyses consider
Bayesian approaches; for instance Wilson et al.
(2007) assumes that the tasks are generated from a hierarchical Bayesian model, and likewise Lazaric & Ghavamzadeh (2010) considers the case when the value functions are generated from a common prior distribution.
Similar considerations, which however does not use a Bayesian approach, are implicitly made in Taylor et al.
(2007), Lazaric et al.
(2008), and also in this work.
In recent years, the advantages of MTRL have been empirically evinced also in DRL, especially exploiting the powerful representational capacity of deep neural networks.
For instance, Parisotto et al.
(2015) and Rusu et al.
(2015) propose to derive a multi-task policy from the policies learned by
DQN experts trained separately on different tasks.
Rusu et al.
(2015) compares to a therein introduced variant of DQN, which is very similar to our MDQN and the one in Liu et al.
(2016), showing how their method overcomes it in the Atari benchmark Bellemare et al.
Further developments, extend the analysis to policy search (Yang et al., 2017; Teh et al., 2017), and to multi-goal RL (Schaul et al., 2015; Andrychowicz et al., 2017).
Finally, Hessel et al.
(2018) addresses the problem of balancing the learning of multiple tasks with a single deep neural network proposing a method that uniformly adapts the impact of each task on the training updates of the agent.
B P ROOFS B. A PPROXIMATED V ALUE -I TERATION BOUNDS
Proof of Theorem 2.
We compute the average expected loss across tasks:
[Equation]
with γ = max t ∈
[Equation]
[Equation] [Equation]
max
[Equation]
[Equation]
[Equation]
K ; t, r ) , and R max , avg =
[Equation] [Equation]
(cid:80)
[Equation]
R
[Equation]
t . Considering the term 1 / T (cid:80) Tt =1 (cid:104) E 12 ( ε t, 0 , . .
. , ε t,K − 1 ; t, r ) (cid:105)
= 1 / T (cid:80)
Tt =1 (cid:16)(cid:80)
K − 1 k =
α 2 rt,k ε t,k (cid:17) 12 let
[Equation]
then we bound
T
[Equation]
(cid:88) t =
[Equation]
T
[Equation]
Using Jensen’s inequality:
[Equation]
So, now we can write (11) as
T
[Equation]
, with ε avg ,k = 1 / T (cid:80) Tt =1 ε t ,k and E avg ( ε avg , 0 , . .
. , ε avg ,K − 1 ; r ) =
(cid:80) K − 1 k =
α 2 rk ε avg ,k .
Proof of Lemma 4.
Let us start from the deﬁnition of optimal task-averaged risk:
[Equation]
where Q ∗ t,k , with t ∈ [1 , T ] , are the minimizers of ε avg ,k .
Consider the task ˇ t such that
[Equation]
we can write the following inequality:
[Equation]
By the application of Theorem 5.3 by Farahmand (2011) to the right hand side, and deﬁning b k,i = (cid:107) Q ˇ t k ,i +1 − T ∗ ˇ t Q ˇ t k ,i (cid:107) ν , we obtain:
[Equation]
− i .
Squaring both sides yields the result:
[Equation]
B. A PPROXIMATED P OLICY -I TERATION BOUNDS
We start by considering the bound for the API framework:
Theorem 5.
(Theorem 3.2 of Farahmand (2011))
Let K be a positive integer, and Q max ≤ R max 1 − γ .
Then for any sequence ( Q k ) K − 1 k =0 ⊂ B ( S × A , Q max ) and the corresponding sequence ( ε k ) K − 1 k =0 , where ε k = (cid:107) Q k − Q π k (cid:107) 2 ν , we have:
[Equation]
where
C PI ,ρ,ν
[Equation]
K ; r
[Equation]
=
[Equation] [Equation]
with E ( ε 0 , . . .
, ε K − 1 ; r ) =
(cid:80) K − 1 k =
α 2 rk ε k , the three coefﬁcients c PI 1 ,ρ,ν , c PI 2 ,ρ,ν , c PI 3 ,ρ,ν , the distributions ρ and ν , and the series α k are deﬁned as in Farahmand (2011).
From Theorem 5, by computing the average loss across tasks and pushing inside the average using
Jensen’s inequality, we derive the API bounds averaged on multiple tasks.
Theorem 6.
Let K be a positive integer, and Q max ≤ R max 1 − γ .
Then for any sequence ( Q k ) K − 1 k =0 ⊂ B
( S×
A , Q max ) and the corresponding sequence ( ε avg ,k ) K − 1 k =0 , where ε avg ,k = 1 T (cid:80) Tt =1 (cid:107) Q t,k − Q π k t (cid:107)
ν
, we have:
[Equation]
with E avg = (cid:80) K − 1 k =
α 2 rk ε avg ,k , γ =
[Equation]
T (cid:80) Tt =1 R max ,t and α k =
[Equation]
Proof of Theorem 6.
The proof is very similar to the one for AVI.
We compute the average expected loss across tasks:
[Equation]
Using Jensen’s inequality as in the AVI scenario, we can write (15) as:
[Equation]
with ε avg ,k = 1 / T (cid:80) Tt =
ε t ,k and E avg ( ε avg , 0 , . . .
, ε avg ,K − 1 ; r ) =
(cid:80) K
[Equation]
k
[Equation] [Equation]
rk ε avg
[Equation]
k .
B. A PPROXIMATION BOUNDS
Proof of Theorem 3.
Let w ∗ 1 , . .
. , w ∗ T , h ∗ and f ∗ 1 , . . .
, f ∗ T be the minimizers of ε ∗ avg , then: ε
[Equation]
We proceed to bound the three components individually:
• C can be bounded using Hoeffding’s inequality, with probability 1 − δ / 2 by (cid:112) ln( 2 / δ ) / (2 nT ) , as it contains only nT random variables bounded in the interval [0 , 1] ; • B can be bounded by 0 , by deﬁnition of ˆw , ˆ h and ˆf , as they are the minimizers of Equation (3); • the bounding of A is less straightforward and is described in the following.
We deﬁne the following auxiliary function spaces:
[Equation] [Equation]
• F (cid:48)
= (cid:8) y ∈ R KTn → ( f t ( y ti )) :
( f 1 , . . .
, f T ) ∈ F T (cid:9) ,
and the following auxiliary sets:
[Equation]
=
[Equation]
(cid:96) ( f t (
[Equation] [Equation]
[Equation] [Equation]
• S (cid:48)
[Equation] [Equation]
[Equation] [Equation]
[Equation]
)))) : f
[Equation]
T
[Equation]
T (cid:9) ⊆
R Tn
[Equation]
• S (cid:48)(cid:48)
=
H (
[Equation]
(cid:48) ( ¯ X )) =
(cid:8) ( h ( w t ( X ti ))) :
h ∈ H , w ∈ W T (cid:9) ⊆
R KTn ,
which will be useful in our proof.
Using Theorem 9 by Maurer et al.
(2016), we can write:
[Equation]
then by Lipschitz property of the loss function (cid:96) and the contraction lemma Corollary 11 Maurer et al.
(2016): G ( S ) ≤ G ( S (cid:48) ) .
By Theorem 12 by Maurer et al.
(2016), for universal constants c (cid:48)
1 and c (cid:48) 2 :
[Equation]
) where L ( F (cid:48) ) is the largest value for the Lipschitz constants in the function space F (cid:48) , and D ( S (cid:48)(cid:48) ) is the Euclidean diameter of the set S (cid:48)(cid:48) .
Using Theorem 12 by Maurer et al.
(2016) again, for universal constants
c (cid:48)(cid:48) 1 and c (cid:48)(cid:48) 2 :
[Equation]
Putting (19) and (20) together:
[Equation]
At this point, we have to bound the individual terms in the right hand side of (21), following the same procedure proposed by Maurer et al.
Firstly, to bound L ( F (cid:48) ) , let y, y (cid:48) ∈ R KTn , where y =
( y ti ) with y ti ∈ R K and y (cid:48)
=
( y (cid:48) ti ) with y (cid:48)
ti ∈ R K .
We can write the following:
[Equation]
whence L ( F (cid:48) )
≤ L ( F ) .
Then, we bound:
[Equation]
Then, since it is possible to bound the Euclidean diameter using the norm of the supremum value in the set
[Equation]
we bound D ( S
[Equation]
Also
[Equation]
we bound O ( F (cid:48) ) :
[Equation]
whence O ( F (cid:48) ) ≤ √ TO ( F ) .
To minimize the last term, it is possible to choose y 0 = 0 , as f (0) = 0 , ∀ f ∈ F , resulting in min y ∈ Y G
( F ( y ))
=
G ( F (0)) = 0 .
Then, substituting in (21), and recalling that G ( S ) ≤ G ( S (cid:48) ) :
[Equation]
Now, the ﬁrst term A of (17) can be bounded substituting (25) in (18): ε avg ( ˆw , ˆ h, ˆf ) − 1 nT (cid:88) ti
[Equation]
≤
[Equation]
nT
[Equation]
nT
[Equation]
A union bound between A , B and C of (17) completes the proof:
[Equation]
C A DDITIONAL DETAILS OF EMPIRICAL EVALUATION
Experiments have been run on an NVIDIA R (cid:13) DGX Station TM and Intel R (cid:13) AI DevCloud.
C. M ULTI F ITTED Q -I TERATION
We consider Car-On-Hill problem with discount factor 0 .
95 and horizon 100 .
Running Adam optimizer with learning rate 0 .
001 and using a mean squared loss, we train a neural network composed of 2 shared layers of 30 neurons each, with sigmoidal activation function, as described in Riedmiller (2005).
We select 8 tasks for the problem changing the mass of the car m and the value of the discrete actions a (Table 1).
Figure 1(b) is computed considering the ﬁrst four tasks, while Figure 1(c) considers task 1 in the result with 1 task, tasks 1 and 2 for the result with 2 tasks, tasks 1 , 2 , 3 , andfor the result with 4 tasks, and all the tasks for the result with 8 tasks.
To run FQI and MFQI, for each task we collect transitions running an extra-tree trained following the procedure and setting in Ernst et al.
(2005), using an (cid:15) -greedy policy with (cid:15)
= 0 .
1 , to obtain a small, but representative dataset.
The optimal Q -function for each task is computed by tree-search 3 for 100 states uniformly picked from the state space, and the 2 discrete actions, for a total of 200 state-action tuples.
C. M ULTI D EEP Q -N ETWORK
The ﬁve problems we consider for this experiment are: Cart-Pole , Acrobot , Mountain-Car , Car-OnHill , and Inverted-Pendulum 4 .
The discount factors are respectively 0 .
95 , and 0 .
The horizons are respectively 500 , 1 , 000 , 1 , 000 , 100 , and 3 , 000 .
The network we use consists of
ReLu units for each w t , t ∈ { 1 , . .
, T } block, with T = 5 .
Then, the shared block h consists of one layer with 80 ReLu units and another one with 80 sigmoid units.
Eventually, each f t has a number of linear units equal to the number of discrete actions a ( t ) i , i ∈ { 1 , . . .
, # A ( t ) } of task µ t which outputs the action-value Q t ( s, a ( t ) i ) =
y t ( s, a ( t ) i ) =
f t ( h ( w t ( s )) , a ( t ) i ) , ∀ s ∈ S ( t ) .
The use of sigmoid units in the second layer of h is due to our choice to extract meaningful shared features bounded betweenand 1 to be used as input of the last linear layer, as in most RL approaches.
In practice, we have also found that sigmoid units help to reduce task interference in multi-task networks, where instead the linear response of ReLu units cause a problematic increase in the feature values.
Furthermore, the use of a bounded feature space reduces the sup h, w (cid:107) h ( w ( ¯ X ))
(cid:107) term in the upper bound of Theorem 3, corresponding to the upper bound of the diameter of the feature space, as shown in Appendix B.
The initial replay memory size for each task is 100 and the maximum size is 5 , 000 .
We use Huber loss with Adam optimizer using learning rate 10 − 3 and batch size of 100 samples for each task.
The target network is updated every 100 steps.
The exploration is ε -greedy with ε linearly decaying from
[Table]
C. M ULTI D EEP D ETERMINISTIC P OLICY G RADIENT
The two set of problems we consider for this experiment are: one including Inverted-Pendulum ,
Inverted-Double-Pendulum , and Inverted-Pendulum-Swingup , and another one including HopperStand , Walker-Walk , and Half-Cheetah-Run 5 .
The discount factors are 0 .
99 and the horizons are 1 , 000 for all problems.
The actor network is composed of 600 ReLu units for each w t , t ∈ { 1 , . .
, T } block, with T = 3 .
The shared block h has 500 units with ReLu activation function as for MDQN.
Finally, each f t has a number of tanh units equal to the number of dimensions of the continuous actions a ( t ) ∈
A ( t ) of task µ t which outputs the policy π t ( s ) =
y t ( s ) =
f t ( h ( w t ( s ))) , ∀ s ∈ S ( t ) .
On the other hand, the critic network consists of the same w t units of the actor, except for the use of sigmoidal units in the h layer, as in MDQN.
In addition to this, the actions a ( t ) are given as input to h .
Finally, each f t has a single linear unit Q t ( s, a ( t ) )
=
y t ( s, a ( t ) )
=
f t ( h ( w t ( s ) , a ( t ) )) , ∀ s ∈ S ( t ) .
The initial replay memory size for each task is 64 and the maximum size is 50 , 000 .
We use Huber loss to update the critic network and the policy gradient to update the actor network.
In both cases the optimization is performed with Adam optimizer and batch size of 64 samples for each task.
The learning rate of the actor is 10 − 4 and the learning rate of the critic is 10 − 3 .
Moreover, we apply (cid:96) 2 -penalization to the critic network using a regularization coefﬁcient of 0 .
The target networks are updated with soft-updates using τ = 10 − 3 .
The exploration is performed using the action computed by the actor network adding a noise generated with an Ornstein-Uhlenbeck process with θ = 0 .and
σ = 0 .
Note that most of these values are taken from the original DDPG paper Lillicrap et al.
(2015), which optimizes them for the single-task scenario.