Different Strokes for Different Folks: Investigating Appropriate Further Pre-training Approaches for Diverse Dialogue Tasks (2021)

Abstract: Loading models pre-trained on the large-scale corpus in the general domain and fine-tuning them on specific downstream tasks is gradually becoming a paradigm in Natural Language Processing. Previous investigations prove that introducing a further pre-training phase between pre-training and fine-tuning phases to adapt the model on the domain-specific unlabeled data can bring positive effects. However, most of these further pre-training works just keep running the conventional pre-training task, e.g., masked language model, which can be regarded as the domain adaptation to bridge the data distribution gap. After observing diverse downstream tasks, we suggest that different tasks may also need a further pre-training phase with appropriate training tasks to bridge the task formulation gap. To investigate this, we carry out a study for improving multiple task-oriented dialogue downstream tasks through designing various tasks at the further pre-training phase. The experiment shows that different downstream tasks prefer different further pre-training tasks, which have intrinsic correlation and most further pre-training tasks significantly improve certain target tasks rather than all. Our investigation indicates that it is of great importance and effectiveness to design appropriate further pre-training tasks modeling specific information that benefit downstream tasks. Besides, we present multiple constructive empirical conclusions for enhancing task-oriented dialogues.

## introduction pre-trained models, e.g., bert @xcite , roberta @xcite and gpt2 @xcite , have been widely used in many nlp tasks. these models are pretrained on the large-scale general text corpus, such as wikipedia or books, with self-supervised train-ing objectives. fine-tuning these models on downstream tasks can achieve excellent performance. recently, @xcite proposed a domain-adaptive pre-training method, they further pre-training the roberta on a large corpus of unlabeled domain-specific text, e.g., biomedical papers and computer science papers, before fine-tuning on downstream tasks and achieved strong performance. besides, they proved that it is also helpful to continue pre-training on the task-specific text. @xcite applied this method to task-oriented dialogue and proposed a new self-supervised pretraining objective on dialogue corpus. despite they achieved performance improvements, the improvements on different downstream tasks vary a lot, some tasks even obtain no improvement, which indicates that different downstream tasks may need different further pre-training tasks. to investigate this issue, we carry out experiments in the area of task-oriented dialogue. we choose one popular pre-training language model, bert @xcite as our base model, and construct a large scale domain-specific dialogue corpus which consists of nine task-oriented datasets for further pre-training @xcite . we also select four core task-oriented dialogue tasks, intent recognition, dialogue action prediction, response selection, and dialog state tracking as the downstream tasks used in fine-tuning phase. we aim to explore the following questions: 1) in the area of task-oriented dialogue, can further pre-training using the masked language model improve the performance of all downstream tasks? 2) do different further pre-training tasks have different effects on different downstream tasks? 3) which factors affect whether a further pre-training task can achieve improvement on a certain downstream task? 4) does combining different further pre-training tasks benefits more downstream tasks? to answer these questions, we design five selfsupervised pre-training tasks according to different characteristics of the downstream tasks. specifically, we first use specially designed pre-training tasks to further pre-training bert on the domainspecific corpus, obtaining multiple new pre-trained models, denoted as bert's variants. then, we finetune these variants on all downstream tasks and observe the effect of different pre-training tasks on different downstream tasks. from experiment results, we figure out that: 1) further pre-training with masked language model does not achieve improvements for all downstream tasks, it is necessary to design special further pre-training tasks according to the characteristics of dialogue data. 2) different pre-training tasks do have different effects on different downstream tasks, and there is a need to design a specific pre-training task for a certain downstream task. 3) model's ability and structure are two key factors influencing effectiveness of the further pre-training on a certain downstream task. 4) training two further pre-training tasks in a multi-task paradigm does not lead to incremental performance improvements on downstream tasks. the main contribution of our work is to give a set of empirical principles about how to design effective further pre-training tasks for enhancing the task-oriented dialogue. the key points of the design are to make the model structures of the pretraining task and the downstream task similar and let the model learn the abilities required by downstream tasks in the pre-training phase while maintaining the masked language model's training. we release the source code at the github repo. 1 2 background 2.1 pre-trained models two-stage training. large pre-training models, such as bert @xcite , roberta @xcite , gpt2 @xcite , xlnet @xcite t5 @xcite , are trained on massive general domain text with self-supervised training objectives, like masked language model @xcite ) and permutation language model @xcite . these models learned strong and general word representations, fine-tuning these pre-trained models on downstream tasks is proved to be effective. ## approaches in this section, we firstly present the three-stage training framework, then introduce five expressly designed further pre-training tasks and downstream tasks. at last, we present some heuristic analysis on the relations between the tasks in the further pre-training and the fine-tuning stage. ## results and discussion in this section, we collect experimental results and analyse the effects of different further pre-training tasks on different downstream tasks. ## conclusion in this work, we study how to make further pretraining more effective to downstream tasks in the area of the task-oriented dialog. firstly, we notice that further pre-training using mlm objective does not improve all downstream tasks, then we designed multiple pre-training tasks for dialog data, finding that different pre-training tasks benefit different downstream tasks. further, we observe that ability and structure are key factors influencing whether a pre-training task is helpful to a downstream task. these finds can be used as empirical principles to design pre-training tasks.