Overcoming Catastrophic Forgetting During Domain Adaptation of Seq2seq Language Generation (2022)

Abstract: Seq2seq language generation models that are trained offline with multiple domains in a sequential fashion often suffer from catastrophic forgetting. Lifelong learning has been proposed to handle this problem. However, existing work such as experience replay or elastic weighted consolidation requires incremental memory space. In this work, we propose an innovative framework, RMR_DSEthat leverages a recall optimization mechanism to selectively memorize important parameters of previous tasks via regularization, and uses a domain drift estimation algorithm to compensate the drift between different do-mains in the embedding space. These designs enable the model to be trained on the current task while keep-ing the memory of previous tasks, and avoid much additional data storage. Furthermore, RMR_DSE can be combined with existing lifelong learning approaches. Our experiments on two seq2seq language generation tasks, paraphrase and dialog response generation, show thatRMR_DSE outperforms SOTA models by a considerable margin and reduces forgetting greatly.

## introduction seq2seq language generation is the essential framework for many tasks such as machine translation, summarization, paraphrase, question answering, dialog response generation. in these applications, models are typically trained offline using annotated data from a fixed set of domains. however, in real-world applications, it is desirable for the system to expand its knowledge to new domains and functionalities, that is, it has the capability of human-like lifelong learning (lll) @xcite of acquiring new utterance patterns without forgetting what it has already learned. neural networks struggle to learn continuously and experience catastrophic forgetting (cf) when optimized on a sequence of learning problems @xcite . some past work in lll has demonstrated that discriminative models can be incrementally learnt for a sequence of tasks @xcite @xcite . in contrast, under generative settings there has been limited research. recent work in this area includes @xcite @xcite . existing work in lll adopts the replay based methods @xcite , such as latent replay, or regularization based methods @xcite , such as elastic weight consolidation (ewc) @xcite , synaptic intelligence @xcite . although they can reduce cf, they have some limitations. the replay-based methods require storing samples from previous tasks, and regularization methods often view all the model parameters as equally important and regularize them to the same extent. in addition, those approaches do not explicitly address the data distribution shift that causes the cf problem. the semantic gap between the embedding spaces of two domains is a leading reason of cf @xcite . as illustrated in figure 1 , each data point and their cluster centers trained in task 1 are shifted after training for task 2. @xcite proposed to compensate this gap without using any exemplars via domain shift. however, that study focused on classification tasks. in this work, we propose a novel method, regularized memory recall mechanism with additional domain shift estimation (rmr_dse), to alleviate cf in continuous seq2seq language generation. the first rmr component improves the regularization-based method through adaptive regularization. we convert fisher information matrix deployed in ewc to a tunable hyperparameter constrained by a vocabulary-related hyperparameter. further, we add a regularizer derived from the gradients of the generative function to tune model parameters. the second dse component compensates the representation difference from the two domains by estimating the semantic gap between them. we obtain embeddings for the current task's data using the previous and current models, and group embeddings from previous models into clusters. semantic shifts are computed for each cluster, and then used during inference time on previous test data to adjust its semantic representation to match the current model. our main contributions are: ## proposed method in this section, we introduce our proposed framework rmr_dse, as illustrated in figure 2 . in the lll scenario, models are trained for a sequence of domains (or tasks). for the first task, the model can be trained from scratch or using pretrained models. starting from the second model, parameters are initialized with the previous model. our rmr_dse method is a combination of regularization and domain shift estimation (dse). for the first part, we incorporate the mechanism of ewc to obtain a regularized memory recall mechanism (rmr) to optimize model training. for the second dse part, we design an algorithm to integrate both k-means and mean shift. a shift of embedding representations is estimated using the previous and current models, and it is compensated to reduce forgetting when we evaluate the current model on test data from the previous domain. although rmr_dse is a generic mechanism, we evaluate on two seq2seq language generation tasks in this work. the underlying models in seq2seq language generation can be any models, including transformers, lstm or variational auto-encoders. the following describes the rmr and dse components in details. ## experiments on paraphrase generation to test rmr_dse's generalization, we apply it to two datasets that both follow the seq2seq generation setup but are quite different tasks. in this section, we focus on the paraphrase generation task. ## discussions in this section, we provide some additional analyses and observations of this work. first, we take a closer look at the domain shift ## conclusion in this work, we introduce rmr_dse, a generic lll framework for addressing forgetting in seq2seq language generation learning. our experimental results have shown that it outperformed state-of-the-art method by a large margin in two neural seq2seq language generation tasks, paraphrase generation and dialog response generation.