Task-adaptive Pre-training and Self-training are Complementary for Natural Language Understanding (2021)

Abstract: Task-adaptive pre-training (TAPT) and Self-training (ST) have emerged as the major semi-supervised approaches to improve natural language understanding (NLU) tasks with massive amount of unlabeled data. However, it’s unclear whether they learn similar representations or they can be effectively combined. In this paper, we show that TAPT and ST can be complementary with simple TFS protocol by following TAPT -> Finetuning -> Self-training (TFS) process. Experimental results show that TFS protocol can effectively utilize unlabeled data to achieve strong combined gains consistently across six datasets covering sentiment classification, paraphrase identification, natural language inference, named entity recognition and dialogue slot classification. We investigate various semi-supervised settings and consistently show that gains from TAPT and ST can be strongly additive by following TFS procedure. We hope that TFS could serve as an important semi-supervised baseline for future NLP studies.

## introduction deep neural networks @xcite often require large amounts of labeled data to achieve state-of-the-art performance @xcite . however, acquiring high-quality labels is a costly process, which inspires research on methods that can effectively utilize unlabeled data to improve performance @xcite . large pretrained language models like bert @xcite , roberta @xcite and t5 @xcite can learn general language understanding abilities from large-scale unlabeled corpora and have reduced this annotation cost. in this paradigm, large neural networks are first pre-trained on massive amounts of unlabeled data in a self-supervised manner and then finetuned on large amount of labeled data for downstream tasks, which has led * now at google research. to large improvements for natural language understanding on standard benchmarks @xcite . however, their success still relies on large amount of data during finetuning stage. for example, @xcite shows that bert only achieves 6.4% joint goal accuracy with 1% finetuning data for dialogue state tracking task, a core component of task-oriented dialogue systems, making it far behind its full counterpart 45.6%. this data-intensive finetuning poses several challenges for many real-world applications, where collecting large amount of labeled data is not only expensive and time-consuming, but also infeasible in many cases due to data access and privacy constraints @xcite . semi-supervised learning @xcite provides a plausible solution to address aforementioned data hungry issue by making effective use of freely available unlabeled data. one of the most popular semi-supervised learning algorithms is selftraining @xcite . in self-training, a teacher model is first trained on available labeled data and then used to generate pseudo labels for unlabeled data. the original hand-annotated labeled data and the pseudo-labeled data are combined to train a student model. the student model is assigned as a teacher model in next round and the teacher-student training procedure is repeated until convergence or reaching maximum rounds. self-training utilizes unlabeled data in a task-specific way during pseudo labeling process @xcite and has been successfully applied to a variety of tasks, including image recognition @xcite , automatic speech recognition @xcite ), text classification @xcite , sequence labeling @xcite and neural machine translation @xcite . recently, task-adaptive pre-training (tapt) (gururangan et al., 2020) was further proposed, which can adapt pre-trained language models, e.g. bert and roberta to unlabeled in-domain training set to improve performance @xcite . the intuition of tapt is that datasets curated to capture specific tasks of interest tend to cover only a subset of the text available within the broader domain and continued pretraining on the task dataset itself or data relevant to the task can be helpful @xcite . tapt tends to adapt its linguistic representation by utilizing the unlabeled data in a task-agnostic way @xcite . with the recent success of task-adaptive pre-training and self-training in natural language understanding (nlu), a research question arises: are task-adaptive pre-training (tapt) and selftraining (st) complementary for natural language understanding (nlu)? in this paper, we show that tapt and st can be complementary with simple tfs protocol by following @xmath0. tfs protocol follows three steps: ## experiments here we conduct comprehensive experiments and analysis on different nlu datasets to demonstrate the effectiveness of tfs. ## conclusion in this paper, we demonstrate that tapt and st are complementary for nlu tasks with tfs by following @xmath1. our extensive experiments in various semi-supervised setups across six popular datasets show that they are not only complementary but also strongly additive with tfs protocol. we further show that tfs outperforms st through (1) a better initialization from tapt (2) more accurate predictions from tapt finetuned models. we hope that tfs could serve as an important semi-supervised baseline for future nlp studies.