Lifelong Learning using Eigentasks: Task Separation, Skill Acquisition, and Selective Transfer
Abstract
We introduce the eigentask framework for lifelong learning.
An eigentask is a pairing of a skill that solves a set of related tasks, paired with a generative model that can sample from the skill’s input space.
The framework extends generative replay approaches, which have mainly been used to avoid catastrophic forgetting, to also address other lifelong learning goals such as forward knowledge transfer.
We propose a wake-sleep cycle of alternating task learning and knowledge consolidation for learning in our framework, and instantiate it for lifelong supervised learning and lifelong RL.
We achieve improved performance over the stateof-the-art in supervised continual learning, and show evidence of forward knowledge transfer in a lifelong RL application in the game Starcraft 2.
1. Introduction
The goal of lifelong learning (Chen & Liu, 2016; Silver et al., 2013) is to continuously learn a stream of machine learning tasks over a long lifetime, accumulating knowledge and leveraging it to learn novel tasks faster (positive forward transfer)
(Fei et al., 2016) without forgetting the solutions to previous tasks (negative backward transfer).
The learning problem is non-stationary and open-ended.
The learner does not know the number or distribution of tasks, it may not know the identity of tasks or when the task changes and it must be scalable to accommodate an ever-increasing body of knowledge within a ﬁnite model.
These characteristics make lifelong learning particularly challenging for deep learning approaches, which are vulnerable to catastrophic forgetting in the presence of non-stationary data.
Many approaches to continual learning (Parisi et al., 2019;
Nguyen et al., 2018) and lifelong learning have been studied with a generative model of its inputs.
Our lifelong learning agents operate in a wake-sleep cycle, solving a stream of tasks during the wake phase, and consolidating new task knowledge during the sleep phase using generative replay.
and applied to computer vision (Hayes et al., 2018; Liu et al., 2020) and reinforcement learning (RL) (Ammar et al., 2015; Tessler et al., 2016) among other areas.
Recent work comprises many complementary approaches including learning a regularized parametric representation (Kirkpatrick et al., 2016; 2018), transfer of learned representation (Lee et al., 2019), meta-learning (Nagabandi et al., 2019), neuromodulation (Masse et al., 2018), dynamic neural network architectures (Li et al., 2019; Rusu et al., 2016), and knowledge consolidation using memory (van de Ven & Tolias, 2019).
This paper advances the state of the art in the generative replay approach to lifelong learning (van de Ven & Tolias, 2019; Shin et al., 2017).
In this approach, a generative model of the data distribution p ( x, y ) = p ( x ) p ( y | x ) is learned and used for data augmentation i.e., replay data from old tasks when learning a new task.
We refer to p ( x ) as the generator (e.g., of images) and p ( y | x ) as a skill (e.g., a classiﬁer).
Prior work has focused on generative replay as a way to mitigate catastrophic forgetting, i.e., avoiding negative backward transfer.
However, forward transfer – the ability to leverage knowledge to quickly adapt to a novel but related task – is an equally important lifelong learning problem that generative replay has not addressed.
A structured form of replay is one of the functions of mammalian sleep (Krishnan et al., 2019; Louie & Wilson, 2001).
Recent evidence in biology (McClelland et al., 2020) shows that only a few selected experiences are replayed during sleep, in contrast to the typical replay mechanisms in machine learning.
Our proposed approach bridges these gaps.
Our main contribution is a framework that combines generative memory with a set of skills that span the space of behaviors necessary to solve any given task.
The framework consists of a set of generator-skill pairs that partition a stream of data into what we call eigentasks .
Each generator models a subset of the input space, and the corresponding skill encodes the appropriate outputs for the inputs in the generator’s support set.
The likelihood of an input according to each generator is used to retrieve the appropriate skill, making the eigentask model a content-addressable memory for skills and enabling forward transfer.
Eigentasks can be seen as a combination of generative memory with mixture-of-experts (MoE) models (Makkuva et al., 2018;
Tsuda et al., 2020).
The MoE component facilitates forward transfer, while the generative component avoids forgetting.
We develop a concrete instantiation of eigentasks called Open World Variational Auto Encoder (OWVAE) that uses a set of VAEs (Kingma & Welling, 2014) as generators. OWVAE partitions data into eigentasks using out-of-distribution detection based on a likelihood ratio test in the latent space of the generator.
We present a loss function for end-to-end learning of eigentasks that incorporates the losses of the generators and skills, weighted by the likelihood ratio.
We show experimentally that OWVAE achieves superior performance comapred to state-of-the-art (SOTA) generative memory (GM) approaches on a new benchmark that contains a mix of MNIST and FashionMNIST datasets, and comparable performance in the splitMNIST benchmark.
OWVAE’s superior performance is attributed to task disentanglement (Achille et al., 2018) and conﬁrmed visually by comparing the samples generated by each VAE.
Our second contribution is a sampling strategy that improves the quality of generative replay by using the conﬁdence of the predictions output by the paired skills to reject outof-distribution samples.
Our experiments show improved continual learning and reduction in forgetting when using rejection sampling as compared to accepting all generated examples.
Our third contribution is a lifelong RL algorithm that leverages the OWVAE for exploration of new tasks.
The OWVAE generates advice by recalling options (Sutton et al., 1999) that are applicable to the current task, based on the eigentask partitioning of old tasks.
This advice is incorporated into an off-policy learner whose behavior policy is a mixture of the options and the policy being learned.
We apply our lifelong RL algorithm to the challenging video game Starcraft 2 (SC2). On a sequence of “mini-games” (Vinyals et al., 2017) as tasks, our approach shows positive forward transfer when the new task is of the same type as one of the old tasks.
In most mini-games we observe a jump start in the accumulated reward, and in one mini-game, forward transfer outperforms the single-task learner by 1.5x with 10x fewer samples due to better exploration.
2. Background
In lifelong learning, an agent is faced with a never-ending sequence of tasks .
Each task i is deﬁned by the tuple
T i =
( X i , Y i , P i , L i ) , where X i is the input space, Y i is the output space,
P i ∈ Prob ( X i ) is the input distribution, and L i ( x , y ) :
X ∗ i × Y ∗
i
(cid:55)→ R gives the loss for outputting y =
{ y 1 , . . .
, y k } for inputs x = { x 1 , . . .
, x k } .
In this paper we consider the case where all the input and output spaces are the same: X i ≡ X , Y i ≡ Y , and our deﬁnition of a task is T i =
( P i , L i ) .
We denote the (countably inﬁnite) set of tasks by T .
Since a task T i is totally characterized by its index i , we use tasks and their indices interchangeably.
A task sequence is a sequence of tuples (( [Equation] [Equation] [Equation]
, n [Equation] drawn from the task distribution W , where each i k ∈ T is a task index and the corresponding n k is the number of samples from P i k that the agent sees before the transition to the next task.
The lifelong learning agent’s objective is to learn a single hypothesis h :
X
(cid:55)→ Y that minimizes the expected per-sample loss wrt the task distribution,
[Equation]
where X n are n instances sampled IID from P .
Equation 1 deﬁnes the optimal hypothesis wrt the task distribution.
Note that the task distribution W is unknown to the agent, and X n does not contain the task index.
Lifelong learning (Eq. 1) is strictly harder than multi-task learning, where instances have known task IDs.
2.1. Backward and Forward Transfer
The notion of knowledge transfer is central to evaluating success in lifelong learning.
In this paper, we approach lifelong learning in an episodic setting where the agent learns in a series of learning epochs of ﬁxed length (not necessarily aligned with task boundaries).
In this setting, we can deﬁne the key metrics of forward and backward transfer in a manner that is agnostic to the task boundaries.
Let N ji be the total number of samples seen from task i through the end of epoch j .
We deﬁne a loss restricted to tasks actually seen by the learner through epoch j as
[Equation]
Backward Transfer (BT) describes the difference in performance on old tasks before and after one or more epochs of learning.
Let h j : k be the hypothesis obtained after training sequentially on the data from epochs j, j + 1 , . . .
, k , where h j is shorthand for h j : j .
The one-step backward transfer is,
[Equation]
Negative BT corresponds to forgetting knowledge learned in previous episodes, and positive BT could indicate successful knowledge consolidation between tasks.
Forward Transfer (FT) describes the difference in loss on new tasks with and without training on previous tasks.
[Equation]
The terms within parentheses restrict the loss to new tasks in the j th episode.
Positive FT indicates transfer of knowledge or skills between tasks, and typically corresponds to jump start performance and lower sample complexity.
2.2. Reinforcement Learning
The ﬂexibility of our eigentask framework allows it to be applied in supervised learning (SL), unsupervised learning, and reinforcement learning (RL).
For simplicity, we describe lifelong RL in the ﬁnite Markov decision process (MDP) setting.
An MDP is a tuple ( S , A , P, R ) , where S is a ﬁnite set of states, A is a ﬁnite set of actions, P : S ×
A (cid:55)→ Prob ( S ) is the transition function, R :
S × A (cid:55)→ R is the reward function.
The objective is a policy π : S (cid:55)→ Prob ( A ) that maximizes the value function V π ( s ) =
E S t ∼ P,A t ∼ π
[ (cid:80) ∞ t =1 γ t R
( S t , A t )] .
Tasks in lifelong RL are MDPs, with common state and action spaces, but P and R may differ.
The loss is the negative return, L ( x , y )
= − (cid:80) (
x,y ) ∈ ( x , y ) R ( x, y ) .
3. Eigentask Framework
Lifelong learning agents must balance plasticity vs. stability: improving at the current task while maintaining performance on previously-learned tasks.
The problem of stability is especially acute in neural network models, where na¨ıvely training an NN model on a sequence of tasks leads to catastrophic forgetting of previous tasks.
A proven technique for avoiding forgetting is to mix data from all previous tasks with data from the current task during training, thereby reverting the streaming learning problem back to an ofﬂine learning problem.
The generative memory (GM) approach accomplishes this with a generative model of the data distribution p ( x, y ) factorized as a generator p ( x ) over the input space X and a discriminator or skill p ( y | x ) conditioned on the input x .
A skill can correspond to a classiﬁer in supervised learning (SL) or to a policy or option in RL.
To achieve selective transfer, we propose to use multiple generator-skill pairs to disentangle streaming data into “canonical tasks” that we call eigentasks .
Informally, eigentasks partition the joint input-output space such that all inputs within an eigentask use the same skill.
Eigentasks capture task similarity deﬁned in terms of the combination of generative and skill losses.
The use of multiple generators corresponds to a mixture model.
Prior work on mixtures of VAEs or GANs (Dilokthanakul et al., 2016; Zhang et al., 2017; Rao et al., 2019) cluster inputs based on perceptual similarity alone or create one task per label.
On the other hand, mixtures-of-experts (Tsuda et al., 2020) capture skill similarity alone.
The eigentask framework combines the MoE concept with generative replay, to avoid forgetting and realize selective transfer.
3.1. Eigentask Deﬁnition
Formally, an eigentask is a tuple E =
( g, f ) comprising a generator g ( (cid:15) ) that deﬁnes a distribution over inputs X as a function of random noise (cid:15) , and a skill f : X (cid:55)→ Prob ( Y ) that maps inputs to outputs.
An eigentask model M = ( E n , τ ) consists of a set of eigentasks E and a similarity function τ : X (cid:55)→ Prob ( E ) whose output is a probability vector over the n eigentasks.
Typically, g , f and τ are parameterized functions such as DNNs.
We write τ as a function of the current input, but in general it may be a function of the entire input history.
We ﬁrst describe the loss function for end-to-end learning of eigentasks in the ofﬂine setting.
The extension to the streaming section is discussed in the next section.
Given a dataset { X, Y } , the general loss function for eigentask learning is,
[Equation]
where L gen and L disc denote generative and discriminative losses for the generator and skill, respectively.
The loss in (5) is agnostic to the choice of generator (e.g., VAE or GAN), skill (e.g. classiﬁer, policy, options), and similarity function τ .
Note that the generator-skill pairs are independent, to avoid interference between eigentasks.
This is in contrast to recent approaches that learn a shared embedding across tasks (e.g. Achille et al., 2018).
In our experiments, L disc is a cross-entropy loss for classiﬁcation problems and an RL loss such as the policy gradient for RL problems.
We develop a concrete instantiation of eigentasks called
Open World Variational Auto Encoder (OWVAE) that uses a set of VAEs (Kingma & Welling, 2014) as generators, with latent space denoted by z , encoder q φ ( z | x ) , decoder p θ ( x | z ) and prior z ∼ N (0 , I ) .
Figure 2 shows an OWVAE with two eigentasks.
We use reconstruction error between the input and decoder output as the generative loss L gen .
We propose to use a likelihood ratio test to deﬁne τ ( x ) .
1 We use the likelihood of decoder i generating the observed data x for some z , that is approximated using encoding z i .
[Equation]
where subscript i denotes eigentask i , z i =
[-7 uses the probability of decoder i generating the observed data x for some z , and Eq. 
-]
q φ i ( z | x ) is the encoding, Φ is the density of standard gaussian, σ denotes softmax. The assumption is valid when the decoder weights are the inverse of the encoder weights.
The loss function for
OWVAE is in Eq.
7, min
[Equation]
where L V AE is the standard VAE loss,
[Equation]
At test time, given an input x we calculate τ ( x ) by computing a forward pass through the encoder.
We sample the index of the decoder and skill to use according to the categorical distribution given by τ ( x ) .
In the simplest case, we pass the decoder output as the input to the skill.
In our experiments, we observed that using the mid-level features (the encoder features before projecting to latent space) led to the best accuracy of the skill, matching an observation by van de Ven & Tolias (2018).
Note that the above sampling method is conditional on x , and that the OWVAE does not allow direct sampling from the learned mixture because the mixing coefﬁcient τ ( x ) depends on the input.
Generative replay can be used to learn an OWVAE incrementally but requires direct sampling of old tasks.
Section 4 proposes a strategy for direct sampling.
3.2. The Wake-Sleep Cycle
Our lifelong learning agents operate in a wake-sleep cycle, solving a stream of tasks during the wake phase, and consolidating new task knowledge during the sleep phase using generative replay.
In the wake phase, the learner’s goal is to maximize FT wrt correctly outputs on incoming inputs.
In supervised learning that might mean simply outputting the correct label according to the new task, while in reinforcement learning the agent needs to explore the new task and maximize reward.
During the wake phase, the learner converts the streaming input to batched data, by storing new Algorithm 1 General Wake-Sleep Cycle Input: OWVAE M , buffer B , # sleep iterations N Initialize M ; set B to empty.
while True do
Initialize task learner T repeat { Wake Phase }
Classiﬁcation (Section 4): store new instance in bufferRL(Section 5): Update T, B with Alguntil B is full
{+Create copy M 
+}
= M for N iterations do { Sleep Phase }
Fetch batch b from B
Generate replay b M ∼ M 2 using Alg
Update M using Eq.
7 on b ∪ b M end for
Set B to empty end while task examples in a short-term buffer along with any intermediate solutions of wake phase learning.
Periodically, the learner enters a sleep phase, where the objective is memory consolidation over all tasks with minimal negative BT.
We use generative replay to incorporate new task batches in to an eigentask model that is continuously updated.
A general wake-sleep cycle is shown in Algorithm 1 where sleep phase is activated whenever the buffer is full.
We describe instantiations of the wake-sleep cycle for supervised learning and RL in the next two sections.
4. Lifelong Supervised Learning
Wake phase : In this work, we use a trivial wake phase for supervised classiﬁcation.
We simply store the new task examples (instance and label) to the buffer.
The OWVAE could be leveraged in the wake phase, e.g. augment new task data with selective replay of similar tasks, or using the OWVAE skills as hints for knowledge distillation.
For example, hints led to positive FT when the new task was a noisy version of old tasks (noise added to labels and pixels).
We have not yet investigated these possibilities completely as they are speciﬁc to the scenario.
Algorithms for learning from streaming data (e.g. Hayes et al., 2018; Smith et al., 2019) can be used to update the [-batch ( X new , Y new ) ∪ (cid:83) ni =
-]task learner.
Sleep phase : As mentioned in Section 3.1, sampling from an OWVAE is conditional on input x because the mixing coefﬁcients τ ( x ) are a function of x .
Some eigentasks may have received little or no training (e.g. when old tasks are few and similar).
These untrained generators will generate noise that must not be used to augment new task data.
To mitigate this problem, we developed a rejection sampling strategy using the conﬁdence of the skill associated with
Input: OWVAE M , batch size K , threshold δ
Initialize deque D [ y ] for each label y of size K repeatfor Eigentask i in M do Generate z i ∼ N (0 , I ) , x i ∼ p θ i ( x | z i )
y i = arg max p ψ i ( y | x i ) , conf i = max p ψ i ( y | x i ) if conf i > δ : Push x i , y i to D [ y i ] end for until Each D [ y ] is full or MAX TRIES Return (cid:83) y D [ y ] the generator.
Algorithm 2 shows the sampling strategy for OWVAE. We reject a sample ( x, y ) if the conﬁdence of the associated skill is below a threshold δ . We reject overrepresented labels and generate label-balanced replay.
These two reﬁnements to the sampling process were critical in achieving high accuracy on continual learning benchmarks.
In addition, rejection sampling improved the accuracy of the basic GM approach as well.
5. Lifelong Reinforcement Learning
In the lifelong RL setting, tasks are MDPs (Section 2.2), the eigentask skills are policies, and the associated generators generate states where the policies should be applied.
Wake Phase : One of the key determinants of RL performance is the efﬁciency of exploration.
Without any prior knowledge, RL algorithms typically explore randomly in the early stages of learning.
Our approach (see Algorithm 3) is to use the skill corresponding to the most relevant eigentask to aid in exploration.
An off-policy RL algorithm is used for training because the exploration policy is deﬁned by these skills. The behavior policy is a mixture of the target policy and the mixture of skills induced by the OWVAE’s τ -function.
Let π denote the target policy and let p ψ i be the i th skill.
Given a state s , the behavior policy b ( s ) is, b ( s ) = (cid:26) π ( s ) w.p. 1 − µ ( t ) p ψ i ( s ) w.p. µ ( t )
[Equation]
s [Equation] i each eigentask i (8) where µ ( t ) ∈ [0 , 1] is a “mixing” function that decays over time so that eigentask usage is gradually reduced and replaced by π .
We use importance weighting as implemented by the off-policy actor-critic algorithm VTrace (Espeholt et al., 2018), but our approach is compatible with any offpolicy algorithm.
In the last step of the wake phase, trajectories from the target policy are stored in the buffer, to be later consolidated in the sleep phase.
Sleep Phase :
In the sleep phase, the goal is to consolidate the ﬁnal target policy π into the eigentask skills p ψ i .
Our approach is based on policy distillation (Rusu et al., 2015),
Input: OWVAE M , Buffer B , Off-policy learner A , MDP M
Initialize policy π , µ ( t ) repeat s = state from M , get τ ( s ) from OWVAE using Eq. Sample action a using b ( s ) and τ ( s ) as in Eq.
Execute a in M .
Observe next state s (cid:48) and reward r Add ( s, a, s (cid:48) , r ) to RL training set
Update π using A .
Decrease µ ( t ) until Sample budget reached Reset M to initial state
Execute π to generate set of ( s, a, s (cid:48) , r )
Add ( s, a, s (cid:48) , r ) to buffer B that transforms the problem to a supervised learning problem. The sleep phase proceeds in the same manner as for supervised learning tasks (Section 4).
6. Experiments
We validate the idea of eigentasks on unsupervised, supervised classiﬁcation and RL tasks.
We show experimentally that OWVAE achieves superior performance comapred to state-of-the-art (SOTA) generative memory (GM) approaches RtF (van de Ven & Tolias, 2019) and DGR (Shin et al., 2017) on a new benchmark that contains a mix of MNIST and FashionMNIST datasets, and comparable performance in the splitMNIST benchmark for continual learning.
OWVAE’s superior performance is attributed to task disentanglement (Achille et al., 2018) and conﬁrmed visually by comparing the samples generated by each VAE.
We demonstrate our lifelong RL algorithm on the Starcraft(SC2) mini-games benchmark (Vinyals et al., 2017).
We demonstrate that our approach compares favorably to the baselines of single-task and multi-task RL.
6.1. Illustration: synthetic problem
In this section, we illustrate the OWVAE model on a synthetic but challenging problem for current GM approaches.
The problem is inspired by the Wisconsin Card Sorting task (Tsuda et al., 2020), where the tasks cannot be distinguished by “perceptual similarity”, but can be distinguished by “skill similarity”, so a mixture-of-experts model can solve the tasks (Tsuda et al., 2020).
We test whether an OWVAE can separate the tasks and achieve a high accuracy.
Consider two binary classiﬁcation tasks whose input space is an isotropic Gaussian in two dimenstions, and the labels for the two tasks are ﬂipped, e.g. task-0 labels are given by ( X ≥ 0) , and task-1 labels are ( X < 0) .
The data distribution is shown in Figure 3a.
Any classiﬁer that achieves an accuracy of p ∈ [0 , 1] on task-0 must have accuracy 1 − p on task-1, and thus average accuracy of 0 . 5 .
Figure 3b shows that an OWVAE with two eigentasks is able to achieve high accuracy ( > 0 . 7 ).
Figure 3c shows the reason: a meaningful τ has been learned that separates the tasks successfully into two eigentasks.
Interestingly, the learned eigentasks are ( X ≥ 0) and ( X < 0) , i.e. label-0 of task-0 is combined with label-1 of task-1 within one eigentask, and label-1 of task-0 is combined with label-0 of task-1 within another eigentask.
In contrast, current GM methods will not work because learning either task causes forgetting of the other.
6.2. Continual Learning for Supervised Classiﬁcation
We use the class-incremental learning (Class-IL) setting introduced in (van de Ven & Tolias, 2019).
In this setting, new classes or groups of classes are presented incrementally to the learning alorithm.
We use the standard splitMNIST problem and compare to the SOTA.
In splitMNIST, the MNIST dataset is split into ﬁve tasks with each task having two of the original classes.
Further, we create a new benchmark combining MNIST and Fashion-MNIST datasets.
The combined MNIST and Fashion-MNIST classes are split evenly into ten tasks.
Each task introduces two new classes, one MNIST digit and one fashion article.
In this new benchmark, we establish a new SOTA by showing that current replay based approaches are inferior to OWVAE.
In both benchmarks, each new task is trained for 500 iterations with a batch size of 32.
All compared SOTA approaches use a two layer perceptron with 650 neurons for the encoder and decoder, and a latent dimension of 100 .
We compare OWVAE against eight continual learning approaches spanning regularization, replay, and replay with exemplars (Table 1).
In addition, we show two baselines of single-task learner (lower bound - only learns on current task) and ofﬂine multi-task learner (upper bound - knows all tasks).
Training is done on the standard train sets and results are reported on the standard test sets. The OWVAE uses two eigentasks, each with the same architecture as SOTA but with 400 neurons only, so that the OWVAE has the same number of parameters as SOTA.
Within the OWVAE, the inputs to the skill are the mid-level features of the corresponding encoder, i [Table]
e. activations in the last layer of the encoder.
As in Section 4, no wake phase is used and sleep phase uses 500 iterations with a batch size of 32, and the threshold δ is set to 0 . 5 (as in Alg. 2). To study the impact of rejection sampling, we perform an ablation study over different augmentation strategies: (1) BaseAug: all examples generated during replay are accepted, (2) BAug: rejection sampling to create label-balanced replay, (3) VAug: rejecting low conﬁdence examples, and (4) VBAug: combining (2) and (3) (as in Alg 2)
[Table] The average accuracies over all tasks at the end of training are shown in Table 1. As observed in prior work, the class-IL
[Table]
setting is hard for the regularization approaches like EWC, as well as LwF; their performance is very low, comparable to the single task learner (they learn and immediately forget each task).
Our approach (Replay+Eigentasks) has accuracy comparable to other replay-based methods (except LwF) on splitMNIST.
However, on split(MNIST+FashionMNIST), our approach has a higher accuracy (about 4% higher).
Unsurprisingly, using exemplars within replay improves accuracy on both benchmarks.
Exemplars could be integrated into OWVAE in future work.
The ablation study shows that VBAug (as in the combination of rejection using both conﬁdence and label) leads to the most improvement (2-3% on SplitMNIST, 19% on split(MNIST+FashionMNIST)).
The combination VBAug performs better than BAug, BaseAug and VAug, whereas
VAug by itself seems to decrease the performance wrt
BaseAug.
Interestingly, VBAug and BAug improved the performance of the basic GM approach (as in BaseAug vs ET1: 2-3% improvement on splitMNIST, 5-7% improvement on split(MNIST+FashionMNIST)).
A detailed breakdown of accuracy per-task over time is shown in the appendix.
Figure 4 shows the task separation learned by OWVAE. The ﬁgure visualizes VAE reconstructions for each task.
It shows that the ﬁrst eigentask has learned all the MNIST digits and able to reconstruct them, whereas the second eigentask has [Figure] [Figure]
[Figure]
learned all the fashion articles. The ﬁrst eigentask has also learned some fashion articles whereas the second eigentask has not learned any digits. The blurry and noisy images are removed from replay by our rejection sampling strategy.
6.3. Starcraft
We use the Starcraft 2 learning environment (SC2)
(Vinyals et al., 2017).
SC2 is a rich platform in which diverse RL tasks can be implemented.
We used their “mini-games” as the task set for our experiments.
Our policy architecture is a slight modiﬁcation of their FullyConv architecture.
We use the 17 64x64 feature maps extracted by SC2LE.
6.3.1. E IGENTASK L EARNING
In the OWVAE, we use only two feature maps namely “unit type” (identity of the game unit present in each pixel) and “unit density” (average number of units per pixel).
An example of these feature maps is shown in the appendix.
The eigentasks for SC2 are learned in an unsupervised and ofﬂine manner.
We ﬁrst collected a dataset by executing a random policy in each mini-game and recording the feature maps per frame.
Whenever possible, we also collected similar data by running a scripted agent.
We then trained an OWVAE incrementally with three eigentasks in a fully unsupervised manner. The setup is similar to the continual learning experiment for splitMNIST (each mini-game is seen for 500 iterations etc.).
SC2 mini-games are perceptually different, so the OWVAE is able to separate the tasks into meaningful eigentasks despite the unsupervised training.
As shown in Figure 5, by looking at the relative values of the OWVAE τ -function, we see that the ﬁrst eigentask grouped all combat tasks together, but incorrectly grouped a navigation task (possibly due to the unsupervised training).
The second eigentask learned the BuildMarines task alone, whereas the third eigentask grouped the resource gathering tasks together.
For each task type, we observe one eigentask clearly dominating the τ values, while no single eigentask dominates always.
These task groupings can be conﬁrmed by looking at the VAE reconstructions shown in appendix.
6.3.2. F ORWARD P OLICY T RANSFER
We focus on the wake phase of our lifelong RL algorithm (Alg 3), and examine forward transfer due to efﬁcient exploration. We manually set the OWVAE skills selected from the set of trained single-task policies.
These policies are separated into groups, based on the task separation observed from the unsupervised OWVAE training above. Each skill is assigned a policy from the corresponding group.
Training uses the VTrace learning rule (Espeholt et al., 2018) in an A2C implementation heavily adapted from code published by Ring (2018).
We examined transfer from skills from source tasks that are either similar or dissimilar to the target task (Table 2).
The main results with forward transfer are summarized in Figure 6.
The plots include two baselines: the performance of a single task policy trained on the target task, and a multi-task policy trained on batches containing experience from all 6 tasks. In order to demonstrate efﬁcient exploration, the vertical axis shows mean per-episode return obtained by the behavior policy during training.
[Figure]
[Table]
Tasks
Combat
DefeatRoaches, DefeatZerglingsAndBanelings
Navigation
MoveToBeacon, CollectMineralShards
Hybrid
FindAndDefeatRoaches
Economy
CollectMineralsAndGas
Experiment (6a) examines transfer to the DefeatZerglingsAndBanelings task.
In the Transfer-Similar Task condition, the OWVAE skills are policies trained on CollectMineralShards and DefeatRoaches.
The DefeatRoaches task is another combat task, and thus is similar to the target task.
In the Transfer-Disimilar Task condition, the OWVAE skills are CollectMineralShards and MoveToBeacon.
In the Transfer-Similar condition, our approach yielded both good performance from the start of training and substantially better asymptotic performance.
Interestingly, our approach even surpassed the asymptotic performance of single-task and multi-task learning, clearly showing the impact of better exploration transferred through the OWVAE skills.
Furthermore, the asymptotic performance is also better than the best published performance for the FullyConv policy architecture (Vinyals et al., 2017) by about 1.5x while using 10x fewer RL iterations.
In the Transfer-Dissimilar condition, our approach still resulted in better initial performance than single task training, but converged to the asymptotic performance of the single task policy at a slower rate.
In experiment (6b), we study transfer to the MoveToBeacon task.
Because of the way we trained the OWVAE, the MoveToBeacon task gets clustered with two Combat tasks, and not with the more similar CollectMineralShards task.
As a result, the OWVAE τ function selects a combat skill to use for transfer to MoveToBeacon.
When we use the default schedule for the behavior policy mixing rate µ ( t ) , transfer from the inappropriate skill hinders learning.
However, using a different mixing schedule that gives more weight to the target policy allows the agent to overcome the effect.
Finally, experiment (6c) investigates transfer to the FindAndDefeatZerglings task.
This is an interesting target task because it combines elements of Combat and Navigation tasks, but is not highly similar to either of those categories.
We evaluated two different skill sets (either
CollectMineralShards+DefeatRoaches or CollectMineralShards+MoveToBeacon) but transfer had no clear positive or negative effect for this target task.
7. Discussion and Future Work
We introduced the eigentask framework for lifelong learning, which combines generative replay with mixture-of-experts style skill learning.
We use the framework in a wake-sleep cycle where new tasks are solved in the wake phase and experiences are consolidated into memory in the sleep phase.
We applied it to both lifelong supervised learning and RL problems.
We developed reﬁnements to the standard generative replay approach to enable selective knowledge transfer.
Combined with the rejection sampling trick, we achieved SOTA performance on continual learning benchmarks.
In lifelong RL, we demonstrated successful forward transfer to new tasks in Starcraft 2, and exceeded the best published performance on one of the Starcraft 2 tasks.
Our immediate goal in future work is to close the wakesleep loop in lifelong RL.
We have demonstrated success for components of the approach, but not for the full eigentask framework.
We are interested in adding change-point detection to improve on the likelihood ratio test, and hierarchical eigentasks that could be more compact and more efﬁcient.
Finally, we want to incorporate task similarity measures that account for history, to separate RL tasks that have similar observations but different dynamics.
Acknowledgements
This material is based on work supported by the Lifelong
Learning Machines (L2M) program of the Defense Advanced Research Projects Agency (DARPA) under contract HR0011-18-C-0051. Any opinions, ﬁndings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reﬂect the views of DARPA, the Department of Defense or the U.S. Government.
We use a policy distillation approach for policy consolidation.
As a proof-of-concept, we conducted an experiment using policy distillation to combine two SC2 policies – for the CollectMineralShards and DefeatRoaches tasks – into a single policy. In the experiment, we use real observations from both tasks for distillation, rather than sampling one set of observations from a generative model. Figure 7 compares the performance of the distilled policy to the performance of single task policies. Distillation is able to preserve the control knowledge embodied in both policies while compressing them into a single policy, and about 100x fewer training batches are required for distillation than were required originally to learn the policies being distilled.
Policy distillation thus provides an effective and efﬁcient means of knowledge consolidation for our framework. [Figure] B. Supplementary Experimental Results
[Figure]
BuildMarines
CollectMineralShards
DefeatZerglingsAndBanelings
MoveToBeacon
CollectMineralsAndGas
DefeatRoaches
[Figure]