Self-training Improves Pre-training for Natural Language Understanding (2021)

Abstract: Unsupervised pre-training has led to much recent progress in natural language understanding. In this paper, we study self-training as another way to leverage unlabeled data through semi-supervised learning. To obtain additional data for a specific task, we introduce SentAugment, a data augmentation method which computes task-specific query embeddings from labeled data to retrieve sentences from a bank of billions of unlabeled sentences crawled from the web. Unlike previous semi-supervised methods, our approach does not require in-domain unlabeled data and is therefore more generally applicable. Experiments show that self-training is complementary to strong RoBERTa baselines on a variety of tasks. Our augmentation approach leads to scalable and effective self-training with improvements of up to 2.6% on standard text classification benchmarks. Finally, we also show strong gains on knowledge-distillation and few-shot learning.

## introduction self-training is a semi-supervised method which uses a teacher model, trained using labeled data, to create synthetic labels for unlabeled examples @xcite . these synthetic labels are then used to train a student model. this approach is called self-training when the student model has a similar or higher capacity than the teacher, and knowledge distillation @xcite when the student model is smaller than the teacher. self-training has been successfully applied to a variety of tasks, including image recognition @xcite @xcite , automatic speech recognition @xcite @xcite , sequence generation @xcite , and parsing @xcite ). an alternative semi-supervised technique is pretraining @xcite @xcite , which has led to large improvements for natural language understanding compared to purely supervised learning. in that case, models are first trained on an auxiliary task, such as language modeling, followed by fine-tuning on the task of interest. a natural question is the following: do pretraining and self-training capture the same information, or are they complementary? recently, @xcite studied this question in the context of image recognition, showing that selftraining was helpful, even in addition to pretraining. however, their study mostly considers supervised pre-training, in which models were trained on imagenet classification. moreover, in cases where large amounts of supervised data were available for the downstream task, pre-training was not helpful, even without self-training. this is in contrast to natural language understanding for which language modeling pre-training is a very strong baseline that leads to large improvements for all the tasks we consider. an important ingredient for self-training, and semi-supervised learning in general, is the unannotated data and the fact that it comes from the same domain as the downstream task. existing work, such as uda @xcite , selftraining @xcite and back-translation for machine translation @xcite @xcite , assumes the existence of unannotated data in the same domain as the downstream task. this assumption limits the broad application of such semi-supervised methods, in particular in the case of low-resource downstream tasks. a second important question is thus: how can we obtain large amounts of unannotated data from specific domains? in this paper, we propose a data augmentation method, sentaugment, to build datasets of "indomain" data for a given task from data crawled on the web. web data covers many domains, and is available in large quantities. we use a large bank of web documents and construct sentence embeddings @xcite @xcite @xcite ) that allow us to retrieve domain-specific unannotated sentences, which are similar to the existing training set of the downstream tasks. our sentence embedding model is optimized for similarity search, trained with a triplet loss on ground-truth paraphrases, parallel sentences as well as as hard negatives @xcite . we train a teacher model using the labeled task data and then further use it to synthetically label the retrieved sentences, and train the final model based on this synthetic dataset. experiments show that sentaugment is effective for self-training, knowledge distillation and few-shot learning. the approach is generally applicable to new problems, leading to improvements on a variety of domains and tasks such as hate-speech and movie review classification over a strong roberta @xcite baseline. to the best of our knowledge, this is the first study showing that self-training is complementary to a strong pre-training baseline for natural language understanding. specifically, we make the following contributions: ## approach our sentaugment approach retrieves task-specific in-domain unsupervised data from a large bank of sentences which is used for self-training, where the teacher model -a roberta-large model finetuned on the downstream task -synthetically labels it. the synthetic labeled data is finally used to train the output student model (see figure 1 ). we give more details on our approach in what follows. ## experimental setup next, we give details on how we build the bank of sentences, what downstream tasks we use for evaluation and we describe our training procedure for semi-supervised learning. ## analysis and results in this section, we first report results on selftraining, knowledge-distillation and few-shot learning with our best approach. we then provide an analysis of the key factors that makes self-training with sentaugment work in the context of natural language understanding. ## analysis of similarity search in this section, we present the results of our sen-taugment sentence embedding (sase) method on semantic textual similarity (sts) benchmarks and present examples of retrieved sentence based on large-scale similarity search. ## conclusion recent work in natural language understanding has focused on unsupervised pretraining. in this paper, we show that self-training is another effective method to leverage unlabeled data. we introduce sentaugment, a new data augmentation method for nlp that retrieves relevant sentences from a large web data corpus. self-training is complementary to unsupervised pre-training for a range of natural language tasks and their combination leads to further improvements on top of a strong roberta baseline. we also explore knowledge distillation and extend previous work on few-shot learning by showing that open domain data with sentaugment is sufficient for good accuracy.