Continual Learning for Natural Language Generation in Task-oriented Dialog Systems (2020)

Abstract: Natural language generation (NLG) is an essential component of task-oriented dialog systems. Despite the recent success of neural approaches for NLG, they are typically developed in an offline manner for particular domains. To better fit real-life applications where new data come in a stream, we study NLG in a “continual learning” setting to expand its knowledge to new domains or functionalities incrementally. The major challenge towards this goal is catastrophic forgetting, meaning that a continually trained model tends to forget the knowledge it has learned before. To this end, we propose a method called ARPER (Adaptively Regularized Prioritized Exemplar Replay) by replaying prioritized historical exemplars, together with an adaptive regularization technique based on Elastic Weight Consolidation. Extensive experiments to continually learn new domains and intents are conducted on MultiWoZ-2.0 to benchmark ARPER with a wide range of techniques. Empirical results demonstrate that ARPER significantly outperforms other methods by effectively mitigating the detrimental catastrophic forgetting issue.

## introduction as an essential part of task-oriented dialog systems @xcite , the task of natural language generation (nlg) is to produce a natural language utterance containing the desired information given a semantic representation (socalled dialog act). existing nlg models @xcite @xcite are typically trained offline using annotated data from a single or a fixed set of domains. however, a desirable dialog system in real-life applications often needs to expand its knowledge to new domains and functionalities. therefore, it is crucial to develop an nlg approach with the capability * correspondence author of continual learning after a dialog system is deployed. specifically, an nlg model should be able to continually learn new utterance patterns without forgetting the old ones it has already learned. the major challenge of continual learning lies in catastrophic forgetting @xcite . namely, a neural model trained on new data tends to forget the knowledge it has acquired on previous data. we diagnose in section 4.4 that neural nlg models suffer such detrimental catastrophic forgetting issues when continually trained on new domains. a naive solution is to retrain the nlg model using all historical data every time. however, it is not scalable due to severe computation and storage overhead. to this end, we propose storing a small set of representative utterances from previous data, namely exemplars, and replay them to the nlg model each time it needs to be trained on new data. methods using exemplars have shown great success in different continual learning @xcite @xcite and reinforcement learning @xcite tasks. in this paper, we propose a prioritized exemplar selection scheme to choose representative and diverse exemplar utterances for nlg. we empirically demonstrate that the prioritized exemplar replay helps to alleviate catastrophic forgetting by a large degree. in practice, the number of exemplars should be reasonably small to maintain a manageable memory footprint. therefore, the constraint of not forgetting old utterance patterns is not strong enough. to enforce a stronger constraint, we propose a regularization method based on the well-known technique, elastic weight consolidation (ewc @xcite ). the idea is to use a quadratic term to elastically regularize the parameters that are important for previous data. besides the wide application in computer vision, ewc has been recently applied to the domain adaptation task for neural machine translation @xcite . in this paper, we combine ewc with exemplar replay by approximating the fisher information matrix w.r.t. the carefully chosen exemplars so that not all historical data need to be stored. furthermore, we propose to adaptively adjust the regularization weight to consider the difference between new and old data to flexibly deal with different new data distributions. to summarize our contribution, (1) to the best of our knowledge, this is the first attempt to study the practical continual learning configuration for nlg in task-oriented dialog systems; (2) we propose a method called adaptively regularized prioritized exemplar replay (arper) for this task, and benchmark it with a wide range of state-of-the-art continual learning techniques; (3) extensive experiments are conducted on the multiwoz-2.0 @xcite dataset to continually learn new tasks, including domains and intents using two base nlg models. empirical results demonstrate the superior performance of arper and its ability to mitigate catastrophic forgetting. our code is available at https://github.com/mifei/ continual-learning-for-nlg 2 related work continual learning. the major challenge for continual learning is catastrophic forgetting @xcite , where optimization over new data leads to performance degradation on data learned before. methods designed to mitigate catastrophic forgetting fall into three categories: regularization, exemplar replay, and dynamic architectures. methods using dynamic architectures @xcite increase model parameters throughout the continual learning process, which leads to an unfair comparison with other methods. in this work, we focus on the first two categories. regularization methods add specific regularization terms to consolidate knowledge learned before. @xcite introduced the knowledge distillation @xcite . another direction is to regularize parameters crucial to old knowledge according to various importance measures @xcite @xcite . exemplar replay methods store past samples, a.k.a exemplars, and replay them periodically. instead of selecting exemplars at random, @xcite incorporated the herding technique @xcite to choose exemplars that best approximate the mean feature vector of a class, and it is widely used in castro et al. ( 2018); wu et al. (2019); hou et al. (2019); zhao et al. (2019) ; @xcite . @xcite proposed to store samples that the model is least confident. @xcite demonstrated the effectiveness of exemplars for various continual learning tasks in computer vision. catastrophic forgetting in nlp. the catastrophic forgetting issue in nlp tasks has raised increasing attention recently @xcite . @xcite ; @xcite identified the detrimental catastrophic forgetting issue while fine-tuning elmo @xcite and bert @xcite . to deal with this issue, @xcite proposed to replay pre-train data during fine-tuning heavily, and @xcite proposed an improved adam optimizer to recall knowledge captured during pretraining. the catastrophic forgetting issue is also noticed in domain adaptation setups for neural machine translation @xcite @xcite and the reading comprehension task @xcite . @xcite firstly studied the continual learning setting for dialog state tracking in task-oriented dialog systems. however, their setting is still a onetime adaptation process, and the adopted dataset is small. @xcite recently applied progressive network @xcite for the semantic slot filling task from a continual learning perspective similar to ours. however, their method is based on a dynamic architecture that is beyond the scope of this paper. @xcite proposed a boolean operation of "conceptor" matrices for continually learning sentence representations using linear encoders. @xcite combined continual learning and language systematic compositionality for sequence-to-sequence learning tasks. ## model in this section, we first introduce the background of neural nlg models in section 3.1, and the continual learning formulation in section 3.2. in section 3.3, we introduce the proposed method arper. ## conclusion