Lifelong Learning of Factored Policies via Policy Gradients
Abstract
Policy gradient methods have shown success in learning continuous control policies for highdimensional dynamical systems.
A major downside of such methods is the amount of exploration they require before yielding high-performing policies.
In a lifelong learning setting, in which an agent is faced with multiple consecutive tasks over its lifetime, reusing information from previously seen tasks can substantially accelerate the learning of new tasks.
We provide a novel method for lifelong policy gradient learning that trains lifelong function approximators directly via policy gradients, allowing the agent to beneﬁt from accumulated knowledge throughout the entire training process.
We show empirically that our algorithm learns faster and converges to better policies than single-task and lifelong learning baselines, and completely avoids catastrophic forgetting on a variety of challenging domains.
1. Introduction
Policy gradient (PG) methods have been successful in learning control policies on high-dimensional, continuous systems (Schulman et al., 2015; Lillicrap et al., 2016; Schulman et al., 2017).
However, like most methods for reinforcement learning (RL), they require the agent to interact with the world extensively before outputting a functional policy.
In some settings, this experience is prohibitively expensive, such as when training an actual physical system.
If an agent is expected to learn multiple consecutive tasks over its lifetime then we would want it to leverage knowledge from previous tasks to accelerate the learning of new tasks.
This is the premise of lifelong RL methods (Bou
Ammar et al., 2014; Kirkpatrick et al., 2017).
Most previous work in this ﬁeld has considered the existence of a central policy that can be used to solve all tasks the agent will encounter.
If the tasks are sufﬁciently related, this model serves as a good starting point for learning new tasks, and the main problem becomes how to avoid forgetting the knowledge required to solve tasks encountered early in the agent’s lifetime.
However, in many cases, the tasks the agent will encounter are less closely related, and so a single policy is insufﬁcient for solving all tasks.
A typical approach for handling this (more realistic) setting is to train a separate policy for each new task, and then use information obtained during training to ﬁnd commonalities to previously seen tasks and use these relations to improve the learned policy.
Note that this only enables the agent to improve policy performance after an initial policy has been trained.
Such methods have been successful in outperforming the original policies trained independently for each task, but unfortunately do not allow the agent to reuse knowledge from previous tasks to more efﬁciently explore the policy space, and so the learning itself is not accelerated.
We propose a novel framework for lifelong RL via PG learning that automatically leverages prior experience during the training process of each task.
In order to enable learning highly diverse tasks, we follow prior work in lifelong RL by searching over factored representations of the policyparameter space to learn both a shared repository of knowledge and a series of task-speciﬁc mappings to constitute individual task policies from the shared knowledge.
Our algorithm, lifelong PG: faster training without forgetting (LPG-FTW) yields high-performing policies on a variety of benchmark problems with a surprisingly low amount of experience needed per task, and avoids the problem of catastrophic forgetting (McCloskey and Cohen, 1989).
2. Related Work
A large body of work in lifelong RL is based on parameter sharing, where the underlying relations among multiple tasks are captured by the model parameters.
The key problem is designing how to share parameters across tasks in such a way that subsequent tasks beneﬁt from the earlier tasks, and the modiﬁcation of the parameters by future tasks does not hinder performance of the earlier tasks.
Two broad categories of methods have arisen that differ in the way they share parameters.
The ﬁrst class of lifelong RL techniques, which we will call single-model, assumes that there is one model that works for all tasks.
These algorithms follow some standard single-task learning (STL) PG algorithm, but modify the PG objective to encourage transfer across tasks.
A prominent example is elastic weight consolidation (EWC) (Kirkpatrick et al., 2017), which imposes a quadratic penalty for deviating from earlier tasks’ parameters to avoid forgetting.
This idea has been extended by modifying the exact form of the penalty (Li and Hoiem, 2017; Zenke et al., 2017; Nguyen et al., 2018; Ritter et al., 2018; Ebrahimi et al., 2020;
Titsias et al., 2020), but most of these approaches have not been evaluated in RL.
In order for single-model methods to work, they require one of two assumptions to hold: either all tasks must be very similar, or the model must be overparameterized to capture variations among tasks.
The ﬁrst assumption is clearly quite restrictive, as it would preclude the agent from learning highly varied tasks.
The second, we argue, is just as restrictive, since the over-parameterization is ﬁnite, and so the model can become saturated after a (typically small) number of tasks.
The second class, which we will call multi-model, assumes that there is a set of shared parameters, representing a collection of models, and a set of task-speciﬁc parameters, to select a combination of these models relevant for the current task.
A classical example is PG-ELLA (Bou Ammar et al., 2014; 2015; Isele et al., 2016), which assumes that the parameters for each task are factored as a linear combination of dictionary elements.
In a ﬁrst stage, these methods learn a policy for each task in isolation (i.e., ignoring any information from other tasks) to determine similarity to previous policies, and in a second stage, the parameters are factored to improve performance via transfer.
The downside of this is that the agent does not beneﬁt from prior experience during initial training, which is critical for efﬁcient learning in a lifelong RL setting.
Our approach, LPG-FTW, uses multiple models like the latter category, but learns these models directly via PG learning like the former class.
This enables LPG-FTW to be ﬂexible and handle highly varied tasks while also beneﬁting from prior information during the learning process, thus accelerating the training.
A similar approach has been explored in the context of model-based RL (Nagabandi et al., 2019), but their focus was discovering when new tasks were encountered in the absence of task indicators.
Other approaches store experiences in memory for future replay (Isele and Cosgun, 2018; Rolnick et al., 2019) or use a separate model for each task (Rusu et al., 2016; Garcia and Thomas, 2019).
The former is not applicable to PG methods without complex and often unreliable importance sampling techniques, while the latter is infeasible when the number of tasks grows large.
Meta RL (Duan et al., 2016; Finn et al., 2017; Gupta et al., 2018; Clavera et al., 2019) and multi-task RL (Parisotto et al., 2016; Teh et al., 2017; Yang et al., 2017; Zhao et al., 2017) also seek to accelerate learning by reusing information from different tasks, but in those settings the agent does not handle tasks arriving sequentially and the consequent problem of catastrophic forgetting.
Instead, there is a large batch of tasks available for training and evaluation is done either on the same batch or on a target task.
3. RL via Policy Gradients
In RL, the agent faces a Markov decision process (MDP)
(cid:104)X , U , T, R, γ (cid:105) , where X ⊆ R d is the set of states, U ⊆
R m is the set of actions, T : X × U × X
(cid:55)→ [0 , 1] is a probability distribution P ( x (cid:48)
| x , u ) of transitioning to state x (cid:48) after executing an action u in state x , R :
X × U (cid:55)→ R is the reward function that measures the goodness of each state-action pair, and γ ∈ [0 , 1) is the discount factor that reduces the importance of rewards far in the future.
A policy π : X × U (cid:55)→ [0 , 1] prescribes the behavior of the agent by specifying the probability P ( u | x ) of selecting an action u when on a state x .
The goal of the agent is to ﬁnd the policy π ∗ that maximizes the expected long-term returns
E (cid:2)(cid:80) ∞ i =
γ i R i (cid:3) , where R i
=
R ( x i , u i ) .
PG algorithms have shown success in solving continuous state-action RL problems by assuming that the policy π θ is a function approximator parameterized by θ ∈ R d (Schulman et al., 2015; Lillicrap et al., 2016; Schulman et al., 2017).
The search over policies then reduces to ﬁnding the set of parameters θ ∗ that optimizes the objective given by the long-term rewards: J ( θ ) =
E (cid:2)(cid:80) ∞
i =
γ i R i (cid:3) .
Different approaches use varied strategies for estimating the gradient ∇ θ J ( θ ) .
However, the common high-level idea is to use the current policy π θ to sample trajectories of interaction with the environment, and then estimating the gradient as the average of some function of the features and rewards encountered through the trajectories.
4. The Lifelong Learning Problem
We frame lifelong PG learning as online multi-task learning of policy parameters.
In this setting, the agent will be faced with a sequence of tasks Z (1) , . . .
, Z ( T max ) over its lifetime.
Each of these tasks will be an RL problem deﬁned by an
MDP Z ( t ) =
(cid:104)X ( t ) , U ( t ) , T ( t ) , R ( t ) , γ (cid:105) .
As described in
Section 3, each task’s policy is assumed to be parameterized by a vector θ ( t ) .
The goal of the lifelong learning agent is to ﬁnd the set of policy parameters (cid:8) θ (1) , . . .
, θ ( T max ) (cid:9) that maximizes the overall performance across all tasks:
[Equation]
We [Equation] assume knowledge of the total number of tasks, the order in which these tasks T ← 0 , L ← initializeL( d, k ) loop t ← getTask() if isNewTask( t ) then s ( t ) ← initializeSt( k )
T ← T +else
A ← A − 2 (cid:16) s ( t ) s ( t ) (cid:62) (cid:17) ⊗ H ( t ) b ← b − s ( t ) ⊗ (cid:0) − g ( t ) + 2 H ( t ) α ( t ) (cid:1) end if for i = 1 , . . . , N do T ← getTrajectories ( Ls ( t ) )
{+s ( t ) ← PGStep ( T , L , s ( t ) , µ ) if i mod M == 
+}
then α ( t ) ← Ls ( t ) g ( t ) , H ( t ) ← gradAndHess ( α ( t ) )
A t ← A + 2 (cid:16) s ( t ) s ( t ) (cid:62) (cid:17) ⊗ H ( t ) b t ← b + s ( t ) ⊗ (cid:0) − g ( t ) + 2 H ( t ) α ( t ) (cid:1) vec( L ) ← (cid:0) 1 T A t − 2 λ
I (cid:1) − 1 (cid:0) 1 T b t (cid:1) end if end for
A ← A t , b ← b t end loop will arrive, or the relations between the different tasks.
Upon observing each task, the agent will be allowed to interact with the environment for a limited time, typically insufﬁcient for obtaining optimal performance on the task without exploiting relevant information from prior tasks.
During this time, the learner will strive to discover any relevant information from the current task to 1) relate it to previously stored knowledge in order to permit transfer and 2) store any newly discovered knowledge for future re-use.
At any time, the agent may be evaluated on any previously seen task.
Crucially, this formulation requires the agent to perform well on all tasks (and not just the ﬁnal one), so it must strive to retain knowledge from all early tasks.
5. Lifelong Policy Gradient Learning
Our framework for lifelong PG learning uses factored representations.
The central idea is assuming that the policy parameters for task t can be factored into θ ( t )
≈
Ls ( t ) , where L ∈ R
d × k is a shared dictionary of policy factors and
s ( t ) ∈
R k are task-speciﬁc coefﬁcients that select components for the current task.
We further assume that we have access to some base PG algorithm that, given a single task, is capable of ﬁnding a parametric policy that performs well on the task, although not necessarily optimally.
Upon encountering a new task t , LPG-FTW, given as Algorithm 1, will use the base learner to optimize the taskspeciﬁc coefﬁcients s ( t ) , without modifying the knowledge base L .
This corresponds to searching for the optimal policy that can be obtained by combining the factors of L .
Every
M (cid:29) 1 steps, the agent will update the knowledge base L with any relevant information collected from t up to that point.
This allows the agent to search for policies with an improved knowledge base in subsequent steps.
Concretely, the agent will strive to solve the following opti- mization during the training phase:
[Equation]
where J ( t ) ( · ) is any PG objective and the (cid:96) 1 norm encourages sparsity.
The agent will then optimize the following second-order approximation to the multi-task objective to incorporate new knowledge into the dictionary:
[Equation]
[-. , N do T ← getTrajectories ( Ls ( t ) ) s ( t ) ← PGStep ( T , L , s ( t ) , µ ) if i mod M == 
 then α ( t ) ← Ls ( t ) g ( t ) , H ( t ) ← gradAndHess ( α ( t ) ) A t ← A + 2 (cid:16) s ( t ) s ( t ) (cid:62) (cid:17) ⊗ H ( t ) b t ← b + s ( t ) ⊗ (cid:0) −-] g
[Equation]
where g (ˆ t ) and H (ˆ t ) are the gradient and Hessian of J (ˆ t ) ( θ ) evaluated at α (ˆ t )
=
L ˆ t − 1 s (ˆ t ) :
[Equation]
Unlike Bou Ammar et al.
(2014), we do not compute the approximation around a single-task optimum, enabling LPGFTW to update L without ﬁnding the (often infeasible) optimum.
In order to solve this optimization, we ﬁnd:
[Equation]
and
[Equation]
where ⊗ denotes the Kronecker tensor product.
Then, the solution is given by vec( L t ) =
A − 1 b .
Notably, these can be computed incrementally as each new task arrives, so that
L can be updated without preserving data or parameters from earlier tasks.
In Equation 1, the agent leverages knowledge from all previous tasks while training on task t , by searching for θ ( t )
T ← 0 , L ← empty( d, 0 ) while T < k do t ← getTask() s ( t ) ← initializeSt( k )
T ← T +for i = 1 , . .
. , N do T ← getTrajectories( Ls ( t ) )
s ( t ) , (cid:15) ( t )
← PGStep
( T , L , s ( t ) , (cid:15) ( t ) , µ ) end for
L ← addColumn( L , (cid:15) ( t ) )
α ( t ) ← Ls ( t ) + (cid:15) ( t ) g
( t ) , H ( t ) ← gradAndHess( α ( t ) )
A ← A + 2 (cid:16) s ( t ) s ( t ) (cid:62) (cid:17) ⊗ H ( t ) b ← b + s ( t ) ⊗ (cid:0)
− g
( t ) + 2 H
( t ) α ( t ) (cid:1) end while in the span of L t − 1 .
This makes LPG-FTW fundamentally different from prior methods that learn each task’s parameter vector in isolation and subsequently combine prior knowledge to improve performance.
One potential drawback is that, by restricting the search to the span of L t − 1 , we might miss other, potentially better, policies.
However, any set of parameters far from the space spanned by L t − 1 would be uninformative for the multi-task objective, since the approximations to the previous tasks would be poor near the current task’s parameters and vice versa.
In Equation 2, LPG-FTW approximates the loss around the current set of parameters α ( t ) via a second-order expansion and ﬁnds the L t that optimizes the average approximate cost over all previously seen tasks, ensuring that the agent does not forget the knowledge required to solve them.
Time complexity
LPG-FTW introduces an overhead of
O ( k × d ) per PG step, due to the multiplication of the gradient by L (cid:62) .
Additionally, every M (cid:29) 1 steps, the update step of L takes an additional O ( d 3 k 2 ) .
If the number of parameters d is too high, we could use faster techniques for solving the inverse of A in Equation 2, like the conjugate gradient method.
We could also approximate the
Hessian with a Kronecker-factored (KFAC) or diagonal matrix.
While we didn’t use these approximations, they work well in related methods (Bou Ammar et al., 2014; Ritter et al., 2018), so we expect LPG-FTW to behave similarly.
However, note that the time complexity of LPG-FTW is constant w.r.t.
the number of tasks seen, since the cost is computed incrementally.
This applies to diagonal matrices, but not to KFAC matrices, which require storing all Hessians and recomputing the cost for every new task, which is infeasible for large numbers of tasks.
5.1. Knowledge Base Initialization
The intuition we have built holds only when a reasonably good L matrix has already been learned.
But what happens at the beginning of the learning process, when the agent has not yet seen a substantial number of tasks?
If we take the na¨ıve approach of initializing L at random, then the s ( t ) ’s are unlikely to be able to ﬁnd a well-performing policy, and so updates to L will not leverage any useful information.
One common alternative is to initialize the k columns of L with the STL solutions to the ﬁrst k tasks, α ( t ) | kt =1 .
This is evocative of the Forgy method of initializing the k -means cluster centroids with points from the data set (Anderberg, 1973).
However, when the α ( t ) ’s are sub-optimal, this method prevents tasks 2 – k from leveraging information from the earlier tasks, impeding them from achieving potentially higher performance.
Moreover, several tasks might rediscover information, leading to wasted resources in terms of both learning and capacity of L .
We propose an initialization method (Algorithm 2) that enables early tasks to leverage knowledge from previous tasks and prevents the discovery of redundant information.
The algorithm starts from an empty dictionary and adds error vectors (cid:15) ( t ) for the initial k tasks.
For each task t , we modify the optimization in Equation 1 for learning s ( t ) by adding (cid:15) ( t ) as additional learnable parameters, which will ﬁnd knowledge of task t not contained in L and then will be incorporated as a column of L : [Equation] t ) , (cid:15) [Equation]
t )
[Equation]
5.2. Base Policy Gradient Algorithms
Now, we show how two STL PG learning algorithms can be used as the base learner of LPG-FTW.
5.2.1. E PISODIC REINFORCE
Episodic REINFORCE updates parameters as:
[Equation]
where g θ is the policy gradient, given by:
[Equation]
and A ( x , u ) is the advantage function.
In order to incorporate REINFORCE into LPG-FTW, we would update the s ( t ) ’s as:
[Equation]
with the gradient given by :
[Equation]
The Hessian for the update of L in Equation 2 is given by H = 12 E (cid:2)(cid:80)
∞
i =0 ∇ θ , θ (cid:62) log π θ ( x i , u i )
A ( x i , u i ) (cid:3) , which evaluates to H = −
[Equation]
2 E (cid:2)(cid:80) ∞ [Equation] = [Equation]
xx (cid:62) A ( x
[Equation]
u i [Equation] (cid:3) in the case where the policy is a linear Gaussian (i.e., π θ = N ( θ (cid:62) x , σ ) ).
One major drawback of this is that the
Hessian is not negative deﬁnite, so Equation 2 might move the policy arbitrarily far from the original policy used for sampling trajectories.
5.2.2. N ATURAL P OLICY G RADIENT
The natural PG (NPG) algorithm allows us to get around this issue.
We use the formulation followed by Rajeswaran et al.
(2017), which at each iteration optimizes max θ g (cid:62) θ j − 1 ( θ − θ j − 1 ) subject to the quadratic constraint (cid:107) θ − θ j − 1 (cid:107) 2 F θ j −≤ δ , where
F θ = E (cid:2) ∇ θ log π θ ( x , u ) ∇ θ log π θ ( x , u ) (cid:62) (cid:3) is the approximate Fisher information of π θ (Kakade, 2002).
The base learner would then update the policy parameters at each iteration as:
[Equation]
with η θ = (cid:113) δ/ ( g (cid:62)
[Equation]
To incorporate NPG as the base learner in LPG-FTW, at each step we solve max s g (cid:62) s ( t ) j − 1 ( s − s ( t ) j − 1 ) subject to (cid:107) s − s ( t ) j − 1 (cid:107) 2 F s ( t ) j − 1 ≤ δ , which gives us the update:
[Equation]
We compute the Hessian for Equation 2 as H = − 1 η θ F θ j −by using the equivalent soft-constrained problem:
[Equation]
This Hessian is negative deﬁnite, and thus encourages the parameters to stay close to the original ones, where the approximation is valid.
6. Experimental Evaluation
We evaluated our method on a range of complex continuous control domains, showing that LPG-FTW achieves a substantial increase in learning speed and a dramatic reduction in catastrophic forgetting.
Baselines We compared against STL, which does not transfer knowledge across tasks, using NPG as described in
Section 5.2.2.
We then chose EWC (Kirkpatrick et al., 2017) from the single-model family, which places a quadratic penalty for deviating from earlier tasks’ parameters.
Finally, we compared against PG-ELLA (Bou Ammar et al., 2014), which factorizes the policy parameters like LPG-FTW, but ﬁrst uses STL to search for the policy parameters of each task and subsequently factorizes the learned parameters, limiting PG-ELLA’s speed to that of STL.
All lifelong algorithms used NPG as the base learning method.
Evaluation procedure We chose the hyper-parameters of
NPG to maximize the performance of STL on a single task, and used those hyper-parameters for all agents.
For EWC, we searched for the regularization parameter over ﬁve tasks on each domain.
For LPG-FTW and PG-ELLA, we ﬁxed all regularization parameters to 10 − 5 and the number of columns in L to k =5 , unless otherwise noted.
In LPGFTW, we used the simplest setting for the update schedule of L , M = N .
All experiments were repeated over ﬁve trials with different random seeds for parameter initialization and task ordering.
6.1. Empirical Evaluation on OpenAI Gym MuJoCo Domains
We ﬁrst evaluated LPG-FTW on simple MuJoCo environments from OpenAI Gym.
We selected the HalfCheetah,
Hopper, and Walker-2D environments, and created two different evaluation domains for each: a gravity domain, where each task corresponded to a random gravity value between 0 .
5 g and 1 . 5 g , and a body-parts domain, where the size and mass of each of four parts of the body (head, torso, thigh, and leg) was randomly set to a value between 0 .
5 × and 1 .
5 × its nominal value.
These choices led to highly diverse tasks, as we show in our evaluation.
We generated tasks using the gym-extensions (Henderson et al., 2017) package, but modiﬁed it so each body part was scaled independently.
We created T max = 20 tasks for HalfCheetah and Hopper domains, and T max = 50 for Walker-2D domains.
The agents were allowed to train on each task for a ﬁxed number of iterations before moving on to the next.
For these simple experiments, all agents used linear policies.
For the Walker2D body-parts domain, we set the capacity of L to k = 10 , since we found empirically that it required a higher capacity.
The hyper-parameters for NPG were manually selected by running an evaluation on the nominal task for each domain (without gravity or body part modiﬁcations).
We tried various combinations of the number of iterations, number of trajectories per iteration, and step size, until we reached a learning curve that was fast and reached proﬁciency.
Once these hyper-parameters were found, they were used all life-
[Figure]
performance in ﬁve domains and equivalent performance in the remaining one. EWC is faster and converges to higher performance than LPG-FTW in some domains, but completely fails to learn in others. Shaded error bars denote standard error over ﬁve random task orderings and parameter initializations
[Figure]
, and even after all tasks have been trained performance is maintained.
PG-ELLA always performed worse than
STL. EWC suffered from catastrophic forgetting in ﬁve domains, in two resulting in degradation below initial performance. Error bars denote standard error over ﬁve random task orderings and parameter initializations.
long learning algorithms.
For LPG-FTW, we chose typical hyper-parameters and held them ﬁxed through all experiments, forgoing potential additional beneﬁts from a hyperparameter search.
The only exception was the number of latent components used for the Walker-2D body-parts domain, as we found empirically that k
= 5 led to saturation of the learning process early on.
For PG-ELLA, we kept the same hyper-parameters as used for LPG-FTW, since they are used in exactly the same way for both methods.
Finally, for EWC, we ran a grid search over the value of the regularization term, λ , among { 1 e − 7 , 1 e − 6 , 1 e − 5 , 1 e − 4 , 1 e − 3 } .
The search was done by running ﬁve consecutive tasks for
H C _ G
H C _ B P
H o _ G
H o _ B P
W _ G
W _ B P
[Figure]
r e w a r d
[Figure] [Figure] [Figure]
ﬁfty iterations over ﬁve trials with different random seeds.
We chose λ independently for each domain to maximize the average performance after all tasks had been trained.
To make comparisons fair, EWC used the full Hessian instead of the diagonal Hessian proposed by the authors.
Figure 1 shows the average performance over all tasks as a function of the NPG training iterations.
LPG-FTW consistently learned faster than STL, and obtained higher ﬁnal performance on ﬁve out of the six domains.
Learning the task-speciﬁc coefﬁcients s ( t ) directly via policy search increases the learning speed of LPG-FTW, whereas PG-ELLA is limited to the learning speed of STL, as shown by the shared learning curves.
EWC was faster than LPG-FTW in reaching high-performing policies in four domains, primarily due to the fact that the policies are completely shared across tasks, which enables EWC to have starting policies with high performance.
However, EWC failed to even match the STL performance in two of the domains.
We hypothesize that this is due to the fact that the tasks are highly varied (particularly in the body-parts domains, since there are four different axes of variation), and the single shared policy is unable to capture a policy that works in all domains.
Results in Figure 1 consider only how fast the agent learns a new task using information from earlier tasks.
PG-ELLA and LPG-FTW then perform an update step (Equationfor LPG-FTW) where they incorporate knowledge from the current task into L .
The third bar from the left per each algorithm in Figure 2 shows the average performance after this step, revealing that LPG-FTW maintained performance, whereas PG-ELLA’s performance decreased.
This is because LPG-FTW ensures that the approximate objective is computed near points in the parameter space that the current basis L can generate, by ﬁnding α ( t ) via a search over the span of L .
A critical component of lifelong learning algorithms is their ability to avoid catastrophic forgetting.
To evaluate the capacity of LPG-FTW to retain knowledge from earlier tasks, we evaluated the policies obtained from the knowledge base L trained on all tasks, without modifying the s ( t ) ’
s. The rightmost bar in each algorithm in
Figure 2 shows the average ﬁnal performance across all tasks.
LPG-FTW successfully retained knowledge of all tasks, showing no signs of catastrophic forgetting on any of the domains.
The PG-ELLA baseline suffered from forgetting in all domains, and EWC in all but one of the domains.
Moreover, the ﬁnal performance of LPG-FTW was the best among all baselines in all but one domain.
One important question in the study of lifelong RL is how diverse the tasks used for evaluation are.
To measure this in the OpenAI Gym MuJoCo domains, we evaluated each task’s performance using the ﬁnal policy trained by LPGFTW on the correct task and compared it to the average performance using the policies trained on all other tasks.
Figure 3 shows that the policies do not work well across different tasks, demonstrating that the tasks are diverse.
Moreover, the most highly-varying domains, Hopper and
Walker-2D body-parts , are precisely those for which EWC struggled the most, suffering from catastrophic forgetting, as shown in Figure 2 in the paper.
This is consistent with the fact that a single policy does not work across various tasks.
In those domains, LPG-FTW reached the performance of
STL with a high speedup while retaining knowledge from early tasks.
6.2. Empirical Evaluation on More Challenging Meta-World Domains
Results so far show that our method improves performance and completely avoids forgetting in simple settings.
To showcase the ﬂexibility of our framework, we evaluated it on Meta-World (Yu et al., 2019), a substantially more challenging benchmark, whose tasks involve using a simulated Sawyer arm to manipulate various objects in diverse ways, and have been shown to be notoriously difﬁcult for state-of-the-art multi-task and meta learning algorithms.
We added an experience replay (ER) baseline that uses importance sampling over a replay buffer from all previous tasks’ data to encourage knowledge retention, with a 50-50 replay rate as suggested by Rolnick et al.
We chose the
NPG hyper-parameters on the reach task, which is the simplest task from the benchmark.
For LPG-FTW and PGELLA, we ﬁxed the number of latent components as k = 3 .
All algorithms used a Gaussian policy parameterized by a multi-layer perceptron with two hidden layers of 32 units
[Figure]
Metaworldmt
[Figure]
and tanh activation.
Given the high diversity of the tasks considered in this evaluation [Figure] we allowed all algorithms to use task-speciﬁc output layers, in order to specialize policies to each individual task.
We chose typical values for LPG-FTW for k , λ , and µ , and reused those for PG-ELLA.
We used fewer latent components ( k = 3 ), since MT10 contains only T max = 10 tasks
and we considered that using more than three policy factors would give our algorithm an unfair advantage over singlemodel methods.
For EWC, we ran a grid search for the regularization hyper-parameter in the same way as for the previous experiments.
For ER, we used a 50-50 ratio of experience replay as suggested by Rolnick et al.
and ensured that each batch sampled from the replay buffer had the same number of trajectories from each previous task.
LPG-FTW, PG-ELLA, and EWC all had access to the full
Hessian, and we chose for EWC not to share the variance across tasks since the outputs of the policies were taskspeciﬁc.
We ran all Meta-World tasks on version 1.5 of the
MuJoCo physics simulator, to match the remainder of our experimental setting.
We used the robot hand and the object location (6-D) as the observation space for all tasks.
Note that the goal, which was kept ﬁxed for each task, was not given to the agent.
For this reason, we removed 2 tasks from
MT50 that use at least 9-D observations— stick pull and stick push — for a total of T max = 48 tasks.
The top row of Figure 4 shows average learning curves across tasks.
LPG-FTW again was faster in training, showing that the restriction that the agent only train the s ( t ) ’s for each new task does not harm its ability to solve complex, highly diverse problems.
The difference in learning speed was particularly noticeable in MT50, where single-model methods became saturated.
To our knowledge, this is the ﬁrst time lifelong transfer has been shown on the challenging Meta-World benchmark.
The bottom row of Figureshows that LPG-FTW suffered from a small amount of forgetting on MT10.
However, on MT50, where L trained on sufﬁcient tasks for convergence, our method suffered from no forgetting.
In contrast, none of the baselines was capable of accelerating the learning, and they all suffered from dramatic forgetting, particularly on MT50, when needing to learn more tasks.
Conclusion
We proposed a method for lifelong PG learning that enables
RL learners to quickly learn to solve new tasks by leveraging knowledge accumulated from earlier tasks.
We showed empirically that our method, LPG-FTW, does not suffer from catastrophic forgetting, and therefore permits learning a large number of tasks in sequence.
Our proposed method for lifelong PG learning enables RL learners to quickly learn to solve new tasks by leveraging knowledge accumulated from earlier tasks.
Additionally, our algorithm, LPG-FTW, does not suffer from catastrophic forgetting, and therefore permits learning a large number of tasks in sequence.