C OMPOSITIONAL L ANGUAGE C ONTINUAL L EARNING
A BSTRACT
Motivated by the human’s ability to continually learn and gain knowledge over time, several research efforts have been pushing the limits of machines to constantly learn while alleviating catastrophic forgetting (Kirkpatrick et al., 2017b).
Most of the existing methods have been focusing on continual learning of label prediction tasks, which have ﬁxed input and output sizes.
In this paper, we propose a new scenario of continual learning which handles sequence-to-sequence tasks common in language learning.
We further propose an approach to use label prediction continual learning algorithm for sequence-to-sequence continual learning by leveraging compositionality (Chomsky, 1957).
Experimental results show that the proposed method has signiﬁcant improvement over state of the art methods.
It enables knowledge transfer and prevents catastrophic forgetting, resulting in more than 85% accuracy up to 100 stages, compared with less than 50% accuracy for baselines in instruction learning task.
It also shows signiﬁcant improvement in a machine translation task.
This is the ﬁrst work to combine continual learning and compositionality for language learning, and we hope this work will make machines more helpful in various tasks.
The code is available at: https://github.com/yli1/CLCL .
I NTRODUCTION
Continual Learning is a key element of human intelligence that enables us to accumulate knowledge from a never ending stream of data.
From machine learning perspective, there is no guarantee that information accessed at a current task to be revisited later in future tasks.
This leads to what is known as Catastrophic Forgetting (McCloskey & Cohen, 1989; McClelland et al., 1995); signiﬁcant drop in previously obtained knowledge of an AI system as it learns new information and gets less/no exposure to old information.
Several approaches have been proposed to bridge the gap between machine and human continual learning skills with catastrophic forgetting being the central problem.
Existing continual learning methods have focused mostly on classiﬁcation tasks (e.g. (Rebufﬁ et al., 2017; Lopez-Paz & Ranzato, 2017; Shin et al., 2017; Li & Hoiem, 2016;
Shmelkov et al., 2017; Triki et al., 2017; Li & Hoiem, 2016;
Triki et al., 2017; Rusu et al., 2016; Lee et al., 2017; Elhoseiny et al., 2018; Kirkpatrick et al., 2017c; Zenke et al., 2017; Chaudhry et al., 2018)).
In this paper, we propose a new scenario of continual learning which handles sequence-to-sequence tasks common in language learning.
Continual language learning is an open question which has not been studied extensively in machine learning and NLP domains.
It may facilitate a variety of applications in NLP systems.
For example, it enables a robot to keep on learning new tasks via natural language instruction, a conversational agent to adapt to new conversation topics quickly, and a neural machine translation system to expand its vocabulary continually.
Humans learn language by leveraging systematic compositionality , the algebraic capacity to understand and produce large amount of novel combinations from known components (Chomsky, 1957;
Montague, 1970).
Compositional generalization is critical in human cognition (Minsky, 1986;
Lake et al., 2017).
It also helps humans acquire language from a small amount of data, and expand vocabulary sequentially (Biemiller, 2001).
In contrast to humans’ such ability, state-of-the-art continual learning approaches do not achieve the expected generalization.
Table 1 and Figure 3 show the performance of state-of-the-art approaches (Kirkpatrick et al., 2017a;
Aljundi et al., 2018) when tested in instruction learning and machine translation tasks.
This highlights the lack of generalization of these approaches, designed after classiﬁcation tasks, on sequence generation language tasks and the importance of studying the design of continual learning methods for language learning.
[Table]
Modeling continual language learning with improved compositional understanding is at the heart of this paper.
More concretely, we address the challenge of open and growing vocabulary problem with continual learning.
It requires optimizing over two objectives.
First, previously learned knowledge should be transferred and combined with new knowledge.
Second, the learned model should resist catastrophic forgetting (Kirkpatrick et al., 2017b), where a model adapted to a new distribution no longer works on the original one.
To achieve these objectives, we use compositionality to separate semantics and syntax of an input sentence, so that we can convert label prediction algorithm to sequence to sequence algorithm for continual learning.
The contributions of this paper can be summarized as follows.
• We propose a new scenario of continual learning which handles sequence-to-sequence tasks common in language learning.
• We propose an approach to use label prediction continual learning algorithm for sequenceto-sequence continual learning by leveraging compositionality.
To our knowledge, this is the ﬁrst work for applying compositionality to continual learning of sequence-to-sequence tasks, targeting at both knowledge transfer to later stages and catastrophic forgetting prevention on previous stages.
• Experiments show that the proposed method has signiﬁcant improvement over multiple state-of-the-art baselines in both knowledge transfer and catastrophic forgetting prevention with almost 85% accuracy up to 100 stages on language instruction task.
It also shows signiﬁcant improvement in a machine translation task.
R ELATED W ORK
Our work is closely related to compositionality and continual learning or lifelong learning.
Here, we brieﬂy review some related work in these areas.
Compositionality Compositional generalization is critical in human cognition (Minsky (1986);
Lake et al.
, and it helps humans acquire language from a small amount of data, and expand vocabulary sequentially (Biemiller (2001)).
Therefore, researchers have been studying how to enable human-level compositionality in neural networks for systematic behaviour (Wong & Wang, 2007; Brakel & Frank, 2009), counting ability (Rodriguez & Wiles, 1998;
Weiss et al., 2018) and sensitivity to hierarchical structure (Linzen et al., 2016).
Recently, people proposed multiple related tasks (Lake & Baroni, 2018; Loula et al., 2018;
Lake et al., 2019) and methods (Lake & Baroni, 2018; Loula et al., 2018; Kliegl & Xu, 2018) with different kinds of RNN models and attention mechanisms.
Though these methods enable generalization when the training and test sentences have small difference, it has been an open problem (Yang et al., 2019) to reach human-level compositionality generalization.
More recently, Li et al.
(2019) proposed an entropy regularization method that achieves high performance on several NLP tasks.
In this paper, we study compositionality from continual learning angle.
By leveraging the compositional learning approach, we propose the con- tinual learning algorithm by encoding compositionality into DNN.
To our knowledge, our work is the ﬁrst to apply compositionality to continual learning in DNN.
Continual learning Continual learning or lifelong learning involves multiple stages.
Each stage has a set of classes and corresponding data, and the training can only access the data in the current stage.
Based on the way for overcoming catastrophic forgetting, continual learning work may be categorized into data-based and model-based approaches.
In data-based approaches , some methods store previous data either with replay buffer (Rebufﬁ et al., 2017; Lopez-Paz & Ranzato, 2017) or generative model (Shin et al., 2017); other approaches (Li & Hoiem, 2016;
Shmelkov et al., 2017;
Triki et al., 2017), employ the new task data to estimate and preserve the model behavior on previous tasks, mostly via a knowledge distillation loss as proposed in Learning without Forgetting (Li &
Hoiem, 2016).
These approaches are typically applied to a sequence of tasks with different output spaces.
To reduce the effect of distribution difference between tasks, (Triki et al., 2017) propose to incorporate a shallow auto-encoder to further control the changes to the learned features, while (Aljundi et al., 2017) train a model for every task (an expert) and use auto-encoders to help determine the most related expert at test time given an example input.
In model-based approaches , some methods dynamically increase model size for the growing information (Rusu et al., 2016; Xu &
Zhu, 2018); other methods (Fernando et al., 2017; Lee et al., 2017; Kirkpatrick et al., 2017c; Zenke et al., 2017) focus on the parameters of the network.
The key idea is to deﬁne an importance weight ω k for each parameter θ k in the network indicating the importance of this parameter to the previous tasks.
When training a new task, network parameters with high importance are discouraged from being changed.
In Elastic Weight Consolidation , (Kirkpatrick et al., 2017c) estimate the importance weights Ω based on the inverse of the Fisher Information matrix.
(Zenke et al., 2017) propose
Synaptic Intelligence , an online continual model where Ω is deﬁned by the contribution of each parameter to the change in the loss, and weights are accumulated for each parameter during training.
Memory Aware Synapses (Aljundi et al., 2018) measures Ω by the effect of a change in the parameter to the function learned by the network, rather than to the loss.
This allows to estimate the importance weights not only in an online fashion but also without the need for labels.
Finally, Incremental
Moment Matching (Lee et al., 2017) is a scheme to merge models trained for different tasks.
Modelbased methods seem particularly well suited for our setup, given that we work with an embedding instead of disjoint output spaces.
In this paper, we propose a method with minimal increase of model structure in each stage, and we leverage compositionality with explainable mechanisms that align with human learning.
C ONTINUAL L EARNING WITH C OMPOSITIONALITY 3. P ROBLEM DEFINITION
Conventional continual learning algorithms are designed after ﬁxed size input and label classiﬁcation output.
However in many tasks, such as language, both input and output are sequences and bridging this gap between continual learning and sequence-to-sequence models is at the heart of our work.
We facilitate more accurate continual sequence-to-sequence artiﬁcial learner by proposing an approach that can leverage Label Prediction Continual Learning ( LP-CL ) compositionally into Sequence-toSequence Continual Learning ( S2S-CL ) .
LP-CL: Label Prediction Continual Learning In LP-CL, we consider a word to label mapping problem, with input word x and corresponding output label y .
In initial learning stage, y takes one of K classes: y ∈ V init =
{ c 1 , c 2 , . . .
, c K } .
In continual learning stage, y takes a new class: y ∈ V cont =
{ c K +1 } .
In test, y takes all previous classes: y ∈ V init ∪ V cont .
For example, in language instruction task, input x is a primitive word, and output Y is the corresponding action symbol; in word-level machine translation, input x is an English content word, and output Y is the corresponding French word.
In initial training stage, we have multiple input word and output symbol pairs.
In continual learning stage, we have a new input and output word pair.
We train a model in initial training stage, and do not use the initial training data any longer in the rest of training stages.
We then switch to the data in continual learning stage, and continually update the model.
In test stage, we evaluate whether model can predict labels from both initial and continual learning stages.
We denote label prediction continual learning model (LP-CL) as P ( y | x ; θ ) .
S2S-CL: Sequence to Sequence Continual Learning
For sequence to sequence continual learning (S2S-CL), we consider sequential input X
= x 1 , x 2 , . . .
, x n and output Y = y 1 , y 2 , . .
. , y m .
Each output label y
i , i ∈ { 1 , . . .
, m } is from the corresponding label set in label prediction problem.
We want to make a model P ( Y | X ) for sequence to sequence continual learning.
Our goal is to facilitate better Sequence to Sequence Continual Learning
(S2S-CL) capability quantiﬁed as P ( Y | X ) by leveraging access and joint-learning with Label Prediction Continual Learning (LP-CL) model, P ( y | x ; θ ) .
3. U SE LP-CL A LGORITHM FOR S2S-CL WITH C OMPOSITIONALITY
The core idea of this work is to use compositionality to separate semantics and syntax, so that we can convert label prediction algorithm to sequence to sequence algorithm for continual learning.
In
Kirkpatrick et al.
(2017a), continual learning can be probabilistically deﬁned as follows.
[Equation]
Here, log P ( D T | ψ ) is the negative of loss function in task T , and log P ( ψ |D 1 ··· T − 1 ) is regularization related to parameters learned during tasks 1 · · · T − 1 .
In this work, we have two parts of parameters ψ
= θ, φ for semantics θ and syntax φ processing.
With compositionality (Li et al., 2019), we make θ and φ conditionally independent given D 1 ··· T − 1 .
[Equation]
We assume syntax φ does not change after the initial stage, so we realize regularization log P ( φ |D 1 ··· T − 1 ) by freezing φ during learning in task T .
We use label prediction continual learning algorithm for regularization log P ( θ |D 1 ··· T − 1 ) .
Please see Figure 1 for illustration.
We derive the proposed approach based on the above ideas.
To use label prediction algorithm in sequence to sequence problem, we need to extract label prediction problem from sequence to sequence problem.
Language is generally composed of semantics p and syntax f , so that we decompose an input sequence to them with compositionality.
In language instruction task, for example, input X is a word sequence, and output Y is a label sequence.
We consider X has two types of information: which labels are present ( X p ), and how the labels should be ordered ( X f ).
Y is constructed by the output label types ( Y p ), and output label order ( Y f )
(Eq. 1).
We then use chain rule (Eq. 2).
With compositionality, Y f functionally depends only on X f , and given Y f , Y p depends only on X p (Eq. 3).
For intuitive example, in language instruction, order of output actions depends only on input function words (syntax), and given the order, each output action (semantics) depends only on the corresponding input primitive.
In machine translation, output order depends only on input part-of-speech information (syntax), and given the order, each output word label (semantics) depends only on the corresponding input word.
[Equation]
We use LP continual learning for S2S continual learning (our goal) by decomposing output sequence to labels.
We assume that the labels y p 1 , . . .
, y pm are conditionally independent given output syntax
Y f and input semantic information X p (Eq. 4) (we design model in this way).
We then use total probability (Eq. 5).
We further design that x pi depends only on y fj and X p (attention mechanism), and with label prediction component, y pj depends only on x pi (Eq. 6).
[Equation]
P ( x pi | y fj , X p ) is an operation to apply attention map y fj on a sequence of value vectors X p , so that it does not have parameters.
Let θ be the parameter for label prediction module P ( y pj | x pi ; θ ) , and φ be the parameter for attention map generator P ( Y f | X f ; φ ) .
[Equation]
Since we assume continual learning stage does not contain new syntax patterns, we can freeze φ during continual learning stage.
θ is the parameter for label prediction module.
Therefore, we can use label prediction continual learning model (LP-CL) to enable compositional sequence to sequence continual learning (S2S-CL) as we detail in the next subsection.
3. D ISENTANGLE S EMANTIC AND S YNTACTIC R EPRESENTATIONS
Our S2S-CL approach is inspired by the idea of decomposing syntactic and semantic representations with compositionality (Li et al., 2019).
Note that it is not a continual learning approach but shows how compositionality can be encoded in sequence-to-sequence models.
The method disentangles syntactic and semantic 1 representations by using two representations.
One generates attention maps, and the other maps attended word to action.
It reduces entropy of the representations.
Suppose there are input x and output y .
x contains a sequence of words, where each input word is from an input vocabulary of size U .
y contains a sequence of output symbols, where each output symbol is from an output vocabulary of size V .
Both vocabularies contain an end-of-sentence symbol which appears at the end of x and y , respectively.
The model output ˆ y is a prediction for y .
Suppose both input words x 1 , . . .
, x n and output symbols y 1 , . .
, y m are in one-hot representation.
[Equation]
To disentangle information, an input sentence x is converted to semantic representation p and syntactic representation f .
Speciﬁcally, each word is encoded with two embeddings.
[Equation]
Then, they are concatenated to form two representations for the entire input sequence, i.e., p
[Equation]
Entropy regularization is introduced to achieve disentanglement by regularizing the L 2 norm of the representations L regularize =
L 2 ( p ) + L 2 ( f ) , and then adding noise to the representations.
[Equation]
is fed to a sequence-to-sequence module for decoding.
At each step j , the decoder generates b j ∈ R n , and attention map a j is obtained with Softmax.
With the attention map, weighted average v j on noised semantic representations p (cid:48) is computed.
Then it is fed to a fully connected one-layer network f predict to get score l j , and a Softmax is used to compute the output distribution ˆ y j .
The decoding ends if arg max ˆ y j is an end-of-sentence symbol.
[Equation]
The cross entropy of y and ˆ y is used as prediction loss L predict , and the ﬁnal loss L is the combination of prediction loss and entropy regularization loss.
λ is regularization weight.
[Equation]
3. L ABEL P REDICTION A LGORITHM FOR C ONTINUAL L ANGUAGE L EARNING
For language problems, it is natural to use non-parametric algorithm as label prediction continual learning algorithm, because input words and output actions are usually associated with embeddings.
In each stage, since the original method uses two embeddings E r ∈ R k r ( r ∈ { p, f } ) for a word and one embedding W for an action, we append the new semantic e p , syntactic e f and action w embeddings (Fig. 2).
We freeze the old embedding parameters and only learn the newly added ones in the stage.
E XPERIMENTS
We evaluate the proposed method in a continual learning task with multiple stages.
The ﬁrst stage is a standard process in which we train a model with combinations of multiple words in various sentence structures.
In each continual stage, we add a new input word and corresponding new output symbol.
The training dataset contains only one sample, whose input is a sentence with the new word, and output is a sequence with the new symbol.
For each stage, we can only use the data in that stage, and have no access to data in previous or future stages.
We have two objectives in continual learning.
We want previously learned knowledge to be transferred and combined with new knowledge (transfer learning), and an updated model to work on previous data (catastrophic forgetting prevention).
We evaluate transfer learning by testing whether the model works on data where the new word appears with old ones ( Transfer ).
We evaluate catastrophic forgetting prevention by testing whether the model works on data that only contain words up to the last stage ( Forget ).
We are also interested in preventing long-term catastrophic forgetting, because it is more difﬁcult than preventing short-term one.
Thus, we test whether the new model works on the evaluation dataset in the initial stage ( Long-forget ).
Baselines.
We designed baseline methods for compositionality Sequence-to-Sequence continual learning to validate our approach since, to our knowledge, this is the ﬁrst work for continual learning of natural language instructions and machine translation.
We applied standard sequenceto-sequence model ( Standard ) to our continual setting, and also the compositional generalization method ( Compositional ) (Li et al., 2019).
We also compare with state of the art continual learning baselines.
To ﬁt in the experimental setting, we focus on those that do not use replay buffer, and require minimum model structure extension, so that we added EWC (Kirkpatrick et al., 2017a) and MAS (Aljundi et al., 2018) as comparable baselines due to their popularity and competitive performance in label prediction setting.
The detailed implementation of the baseline and proposed methods can be found in Appendix B.
Metric.
We use accuracy as metric for both instruction learning and machine translation experiments.
A prediction is correct if and only if it is completely identical to the ground truth.
We run all experiments for ﬁve times with different random seeds
[Figure]
Aljundi et al., 2018).
The proposed method is signiﬁcantly better than all baselines. Please refer to Figure 3 and Figurein Appendix for details.
Instruction Learning We ﬁrst experiment on instruction learning task using SCAN dataset (Lake & Baroni, 2017).
The task is summarized in Table 2 in Appendix.
The details of dataset generation is in Appendix A.
The results are in Figure 3 (left) and Table 1 (more details in Table 3 in Ap- pendix).
The proposed method has signiﬁcantly better results than the baselines.
It maintains high accuracy up to 100 stages for both transferring knowledge from previous stages to future stages, and catastrophic forgetting prevention.
On the other hand, baseline methods drop performance over time.
Methods without compositionality (EWC, MAS) reduces quickly, maybe because they are not designed for transferring knowledge, and since the representations are entangled, all parameters are quickly changed, causing catastrophic forgetting.
Compositional method is better, but still drops, maybe because the parameters for syntax are changed over time.
This experiment shows the advantage of the proposed method over baselines.
Machine Translation We also investigated whether the proposed approach works for other continual language learning problems.
As an example, we conduct a proof-of-concept experiment for machine translation.
We modiﬁed the English-French translation task in (Lake & Baroni, 2018).
In each continual learning stage, we add an additional English-French word pair, in the format (“I am
ENGLISH”, “je suis FRENCH”).
Neither English word nor French word appears in previous stages.
This pair is used as training data in the stage, but test data contains other patterns.
Appendix A provides more details on dataset and model conﬁguration.
The result is shown in Figure 3 (right) and Table 4 in Appendix.
It shows that the proposed approach has stable and signiﬁcantly higher performance than baselines.
For Transfer and Forget evaluation, the baseline methods drop quickly.
However, for Long-forget evaluation, they keep positive accuracy over time.
This means the baseline methods have the ability to learn knowledge and remember for long time, but they are not as strong as the proposed method.
This experiment shows that the proposed approach has promise to be applied to real-world tasks.
D ISCUSSIONS 5. A TTENTION M AP V ISUALIZATION
We hope to use compositionality for continual learning, so we want to ﬁnd whether the model works in the expected mechanism.
We visualize activations of attention maps on the evaluation data in the ﬁrst continual stage (Figure 4).
[Figure]
o pp o s i t e r i g h t <
E O S >
RTURN RTURN JUMP RTURN
JUMP RTURN JUMP <
EOS> (a) Transfer.
l oo k
o
pp o
s i t
e l e f t t w i
c
e a n d t u r n l e f t < E O
S >
LTURN LTURN
LOOK LTURN LTURN LOOK
LTURN <
EOS> (b) Forget.
w a l k
o
pp o
s i t
e r i g h t t w i
c
e a f t e r r un l e f t < E O S >
LTURN RUN
RTURN RTURN
WALK RTURN
RTURN WALK <EOS> (c) Long-forget.
The visualization shows that, for each output action, the attention is on the corresponding input word.
Also, for the output end-of-sentence symbol, it is on the input end-of-sentence symbol.
It is consistent with the original work, and the way humans apply compositionality.
This indicates that the proposed method may be applicable to other tasks where humans use compositionality.
5. E MBEDDING V ISUALIZATION
We visualize how the new embedding parameters ﬁt in the space with predeﬁned dimensions, and accommodate with previously learned parameters.
The visualization of attention maps explains the syntactic information [Figure] and we are also interested in semantic information.
We use t-SNE (Maaten & Hinton, 2008) to project high dimensional embeddings to two dimensional space for visualization.
Our analysis focuses on semantic embedding, because it reﬂects how new information is encoded in the model.
Since action embedding shares much information with semantic embedding [Figure] and syntactic embedding is not supposed to contain new information because grammar does not change over stages
[Figure]
we leave them in Appendix D
[Figure] Figure 5 shows two phases in the continual learning experiment.
The ﬁrst phase is from the ﬁrst stage to around stage 50, where the new embeddings explore outside space.
The second phase is the rest of the stages [Figure] where the embeddings squeeze into the explored space, maybe because exploring becomes expensive with the dense population under regularization.
This may be an explanation for the performance decrease in the later stages of instruction learning experiment [Figure]
C ONCLUSION
In this paper [Figure] we propose an approach to use label prediction continual learning algorithm for sequence-to-sequence continual learning problem by leveraging compositionality.
To our knowledge, this is the ﬁrst work to combine continual learning and compositionality for sequence-tosequence learning.
Experiments show that the proposed method has signiﬁcantly better results than baseline methods, and it maintains almost more than 85% accuracy for both transfer learning and catastrophic forgetting prevention up to 100 stages.
The results demonstrate that language compositionality helps continual learning of natural language instruction both efﬁciently and effectively.
We hope this work will advance the communication between humans and machines and make machines more helpful in various tasks.
A D ATASET P REPARATION
We extend the grammar in SCAN dataset (Lake & Baroni, 2017) to generate data.
For the initial stage, we generate data with three action words (walk, look, run).
We then randomly and uniformly divide the data to two disjoint sets.
We use one set for initial stage training data (6,601 samples), and reserve the other set as initial dataset to evaluate catastrophic forgetting in continual stages (Forget, 6,602 samples).
The reserved data is also used to evaluate long-term catastrophic forgetting (Longforget).
In each continual stage, we generate data that contain a new action word (jump for the ﬁrst continual stage).
The data contain a one-word sentence (jump: JUMP), and we use it as training data (1 sample).
We use other data in the generated data as evaluation dataset (Transfer, 7,706 samples for the ﬁrst continual stage).
We then add Transfer to Forget for the next stage.
We set a limitation of 100,000 samples for each evaluation dataset.
When it has more samples than that, we select the amount uniformly at random after generating all data once.
Machine translation dataset is generated similarly from the translation dataset in SCAN dataset (Lake & Baroni, 2017).
We use the original training data as the initial training data, and the original test data as the initial test data.
[Table]
B I MPLEMENTATION
We use bidirectional LSTM as encoder, and unidirectional LSTM with attention as decoder for sequence to sequence architecture.
The ﬁrst and last states of encoder are concatenated as initial state of decoder.
The state size is h = 32 for encoder, and 2 h = 64 for decoder.
We also use k p = 64 , k f = 8 , α = 0 . 1 .
For EWC (Kirkpatrick et al., 2017a) and MAS (Aljundi et al., 2018), we use 10 for parameter regularization weight.
In initial stage, batch size is 512, and we run training 5,000 steps.
In each continual stage, batch size is 1, as each continual stage only contains one sample, and we run training 1,000 steps.
We have 100 continual stages.
We implement all methods with TensorFlow (Abadi et al., 2016).
C D ETAILED E XPERIMENT R ESULTS
More results and standard deviation for continual learning can be found in Table 3 for instruction learning and Table 4 for machine translation experiments.
D E MBEDDING V ISUALIZATION
Syntax embeddings in Figure 6 show that the embeddings in continual learning stages do not explore outside space, but live in the space explored in the initial stage.
Action embeddings in Figure 7 show two phases for exploration, similar with semantic embeddings.
This ﬁts expectation, because the compositionality and training encourage semantic embeddings and action embeddings to be similar.
[Table] [Table]
[Figure] [Figure]