When does Further Pre-training MLM Help? An Empirical Study on Task-Oriented Dialog Pre-training (2021)

Abstract: Further pre-training language models on in-domain data (domain-adaptive pre-training, DAPT) or task-relevant data (task-adaptive pre-training, TAPT) before fine-tuning has been shown to improve downstream tasks’ performances. However, in task-oriented dialog modeling, we observe that further pre-training MLM does not always boost the performance on a downstream task. We find that DAPT is beneficial in the low-resource setting, but as the fine-tuning data size grows, DAPT becomes less beneficial or even useless, and scaling the size of DAPT data does not help. Through Representational Similarity Analysis, we conclude that more data for fine-tuning yields greater change of the model’s representations and thus reduces the influence of initialization.

## introduction pre-trained models such as bert @xcite and gpt2 @xcite have been used in a wide range of nlp tasks and achieved superior performance. these models usually follow the pre-train and fine-tune paradigm, which adopts unsupervised pre-training on large-scale corpora and supervised fine-tuning for downstream task adaption. however, the pre-training corpora are in the general domain, while the data of downstream tasks fall in more task-specific domains. to bridge the data distribution gap, further pretraining has been applied and shows consistent improvements @xcite . according to the training data used in this process, @xcite termed domain-adaptive pre-training (dapt), which uses the data in the same domain of the target task and task-adaptive pre-training (tapt), which uses much less unlabeled training data from the target task than dapt. they found that dapt masked lm leads to performance gains under both high-and low-resource settings and tapt is beneficial with or without dapt. dapt has shown effectiveness for task-oriented dialog modeling. @xcite further pretrained bert on 9 task-oriented dialog corpora and outperformed bert on four downstream tasks, especially in the few-shot setting. @xcite further pre-trained gpt-2 on 13 dialog corpora ranging from chitchats to task-oriented dialogs, leading to better results on three task-oriented datasets. however, does further pre-training always help? @xcite performed dapt on 700m opendomain dialogs and tapt, but the resulting model only outperforms bert in 4 out of 7 task-oriented dialog datasets. we also observe that replacing bert with tod-bert-mlm @xcite ) that is further pre-trained mlm on 101k task-oriented dialogs does not always bring a significant difference on downstream tasks. so far, however, there has been little discussion about when and why further pre-training on in-domain data can boost the performance on a downstream task and how the dapt data size can affect this. in this paper, we conduct an empirical study on the effect of further pre-training bert base on task-oriented dialogs. our experiments are organized around the following research questions: ## conclusion in this work, we conduct an empirical study to investigate the effect of further pre-training mlm on task-oriented dialogs. different from earlier findings @xcite , neither dapt nor tapt always improves performances on downstream tasks in our experiments. in the low-resource setting, however, dapt is more helpful, and the size of dapt data needed to perform best increases. through rsa, we find that as the fine-tuning data grows, the impact of model initialization fades away, which could be the explanation. we also show that although further pre-training can improve the model's lm ability on downstream datasets, this may not contribute much to downstream tasks under pre-train and fine-tune paradigm, calling for novel pre-training objectives and effective ways to use pre-trained models.