Analyzing the Forgetting Problem in Pretrain-Finetuning of Open-domain Dialogue Response Models (2021)

Abstract: In this work, we study how the finetuning stage in the pretrain-finetune framework changes the behavior of a pretrained neural language generator. We focus on the transformer encoder-decoder model for the open-domain dialogue response generation task. Our major finding is that after standard finetuning, the model forgets some of the important language generation skills acquired during large-scale pretraining. We demonstrate the forgetting phenomenon through a set of detailed behavior analysis from the perspectives of knowledge transfer, context sensitivity, and function space projection. As a preliminary attempt to alleviate the forgetting problem, we propose an intuitive finetuning strategy named “mix-review”. We find that mix-review effectively regularizes the finetuning process, and the forgetting problem is alleviated to some extent. Finally, we discuss interesting behavior of the resulting dialogue model and its implications.

## introduction large-scale unsupervised pretraining @xcite @xcite @xcite has recently been shown to greatly boost the performance of natural language processing (nlp) models. on a high level, the pretrain-finetune framework can be viewed as a simple two-stage procedure: (1) use large-scale unsupervised text data to pretrain the model; (2) use target task data to finetune the model. recently, multiple works @xcite @xcite have reported that pretrained language models (lm) have implicitly stored large amounts of "world knowledge" in its parameters, and are able to answer common-sense questions. while these studies are encouraging, during the finetuning stage the model is usually trained on a dataset that is very different from the pretraining data, which leads to the potential danger that the model could forget precious skills gained during pretraining. this is an important question for open-domain dialogue response generation, which is the focus of our work, because the knowledge acquired during pretraining can greatly help make the dialogue interaction more engaging or informative. in figure 1 , we show that during finetuning, the model's performance on the pretraining data drastically degrades. while this drop is concerning, it does not necessarily mean that the skills from pretrained model are not well transferred to the end dialogue task, because the model should be evaluated in a dialogue setting. to better answer the question about how finetuning changes the pretrained model's behavior, in this work we conduct a set of behavior analysis from the perspectives of knowledge transfer, context sensitivity, and function space projection. our major finding is that in the finetuning stage, data separation causes the model to forget some of the important language generation skills acquired during pretraining. we also show that the forgetting problem can be alleviated by mixing pretraining and target-task data during finetuning. ## model formulation in this work we study the pretrain-finetune framework from the viewpoint of neural language generation (nlg). in particular, we focus on the opendomain dialogue response task, for the following reasons: (1) there is high similarity between the target dialogue response task (conditional nlg) and the pretraining language modeling (lm) objective, so we expect that language generation skills learnt during pretraining can be well transferred to the down-stream target task. (2) the sequence-tosequence (seq2seq) nature of the model allows us to characterize the model's generation behavior in various ways (e.g., context sensitivity). end-to-end dialogue response generation @xcite can be formulated as a sequence-tosequence (seq2seq) task: given a dialogue context (previous utterances), the model is asked to generate a high-quality response. in this work we adopt the encoder-decoder model architecture @xcite , which is widely used in nlg applications like dialogue response generation @xcite , machine translation @xcite , etc. in particular, we use the transformer model @xcite , which has currently become the most popular encoderdecoder model architecture @xcite . we use the same configuration as @xcite , which has 6 encoder/decoder layers, 16 attention heads, with an embedding dimension of 1024 and a feed-forward dimension of 4096. during standard finetuning, the adam optimizer (kingma and ba, 2014) is used to minimize the negative log-likelihood (nll) of the reference target sentence y given the input context x in the data distribution (denoted as p data ): given a trained seq2seq model, to generate a response for some contextual input, one needs to choose a decoding or sampling method. recent research @xcite @xcite has shown that a strategy called top-k sampling, in which the next word is sampled from the top k most probable choices, is a better choice than the traditional beam-search decoding, due to better diversity. our preliminary experiments (appendix a) have also verified this claim in the open-domain dialogue response setting. as a result, in this work, unless otherwise mentioned, we use top-k sampling as the default decoding method. in particular, we set k to 30 (we find it to work well in preliminary experiments). ## the pretrain-finetune framework in this section we first review the pretrain-finetune framework for encoder-decoder models. we discuss the language generation skills the model can acquire during pretraining, and more importantly, how we check whether the skills are "forgotten" during finetuning. finally, as a preliminary attempt to alleviate the forgetting problem, we propose the mix-review finetuning strategy. ## experiment results in this section, we conduct a set of detailed behavior analysis, characterising how different training strategies change the model's behavior. in particular, we aim to answer the crucial question about whether the model forgets precious language generation skills during standard finetuning, and whether mix-review helps the model remember the skills. we first present perplexity results for different finetuning methods in comparing to standard finetuning, mix-review further gives solid improvement. the gain is due to its strong regularization effect (which we study in the next three sections). however, the performance gap between mix-review and wd(✓ pre ) is not significant. we believe the reason is that the benefit (e.g., knowledge transfer) from alleviate the forgetting problem is not be well demonstrated in single-turn response evaluation, because the context is limited to the narrow scope of the specific datasets. we address this concern with multi-turn human evaluation in the next section. ## implications and discussion the sensitivity to dialogue context and the ability to transfer knowledge from pretraining opens the possibility of a data-driven knowledgable chatbot. in next, we discuss the malicious response problem for open-domain dialogue models. as shown by @xcite , it is relatively difficult to trigger the dialogue models trained from scratch to output malicious responses. however, as shown in finally, we discuss some limitations of our work. first, the mix-review strategy we proposed is a simple and preliminary attempt to alleviate the forgetting, and its performance is far from perfect. as shown in appendix c, in a lot of cases, the generation from mix-review is still boring or noninformative. next, the three datasets considered in this work are open-domain dialogue datasets, and they are not knowledge-intensive. it would be interesting, as future work, to check the forgetting problem for knowledge-grounded datasets such as topical-chat @xcite . on the other hand, knowledge-grounded nlg model @xcite @xcite has been an important and exciting research topic. these studies usually involve additional retrieval modules or external knowledge bases to provide the model with relevant information. in contrast to these works, we study whether the model can conduct knowledgeable dialogues by itself. forgetting as discussed in section 3.2, in contrast to the "catastrophic forgetting" problem in sequential learning @xcite , the performance drop on pretraining data is not necessarily bad for the nlp pretrain-finetune framework, and its implications have not been properly studied. in our analysis, we confirm the "forgetting" of important language generation skills during standard finetuning. the proposed mixreview strategy is similar to the pseudo-rehearsal algorithm in sequential learning @xcite , with the difference being that we assume we still have access to the pretraining data. ## conclusion in this work, we attempt to answer to question of whether during finetuning, the model has forgotten some of the useful nlg skills acquired during large-scale pretraining. through a set of detailed behavior analysis, we find the answer is, to some extent, yes. for example, the finetuned model fails to give detailed information about some knowledge terms, while the pretrained model can. as a preliminary attempt to alleviate the forgetting problem, we propose the mix-review finetuning method, and find it to be effective. our analysis shows that under the surface of the performance boost for standard metrics, large-scale pretraining changes the model's generative behavior in various profound ways. more importantly, the behavior change is influenced by the nature of data itself. for example, we demonstrate that we can discuss news with the dialogue model finetuned by mix-review, even when the target dataset is not about news (dailydialog). we believe that this opens the possibility of a completely data-driven way to customize a language generator.