To Pretrain or Not to Pretrain: Examining the Benefits of Pretrainng on Resource Rich Tasks (2020)

Abstract: Pretraining NLP models with variants of Masked Language Model (MLM) objectives has recently led to a significant improvements on many tasks. This paper examines the benefits of pretrained models as a function of the number of training samples used in the downstream task. On several text classification tasks, we show that as the number of training examples grow into the millions, the accuracy gap between finetuning BERT-based model and training vanilla LSTM from scratch narrows to within 1%. Our findings indicate that MLM-based models might reach a diminishing return point as the supervised data size increases significantly.

## introduction language modeling has emerged as an effective pretraining approach in wide variety of nlp models. multiple techniques have been proposed, including bi-directional language modeling @xcite , masked language models @xcite , and variants of denoising auto-encoder approaches @xcite @xcite . today, it is rare to examine a leaderboard without finding the top spots occupied by some variant of a pretraining method. 1 the future of nlp appears to be paved by pretraining a universal contextual representation on wikipedia-like data at massive scale. attempts along this path have pushed the frontier to up 10× to the size of wikipedia @xcite . however, the success of these experiments is mixed: although improvements have been observed, the downstream task is usually data-limited. there is evidence that large-scale pretraining does not always lead to state-of-the-art results @xcite , especially on tasks such as machine translation, where abundance of training data, and the 1 https://super.gluebenchmark.com/leaderboard existence of strong augmentation methods such as back translation might have limited the benefit of pretraining. this paper examines the pretraining benefits of downstream tasks as the number of training samples increases. to answer this question, we focus on multi-class text classification since: (i) it is one of most important problems in nlp with applications spanning multiple domains. (ii) large sums of training data exists for many text classification tasks, or can be obtained relatively cheaply through crowd workers @xcite . we choose three sentiment classification datasets: yelp review (yel, 2019), amazon sports and electronics review @xcite , ranging in size from 6 to 18 million examples. 2 we finetune a roberta model @xcite ) with increments of the downstream dataset, and evaluate the performance at each increment. for example, on the yelp dataset whose size is 6 million, we train the models on subsets of the data with each subset size being in the sequence (60k, 600k, 1.8m, 3m .., 6m). for comparison, we also train a vanilla bilstm, and another bilstm which uses pretrained roberta token embeddings. we observe that when both models are trained on 1% of the data, the gap between bilstm and roberta models is at its peak, but as the training dataset size increases, the bilstm model accuracy keeps on increasing whereas roberta's accuracy remain mostly flat. as the dataset size increases, the accuracy gap shrinks to within 1%. our study suggests that collecting data and training on the target tasks is a solution worth considering, especially in production environments where accuracy is not the only considered factor, rather inference latency is often just as crucial. we benchmarked the inference latency of the these models on both cpu and gpu for different batch sizes, and as expected, we observe at least 20× speedup for the bilstm compared to the roberta. this paper provides new experimental evidence and discussions for people to rethink the mlm pre-training paradigm in nlp, at least for resource rich tasks. ## related works scaling the number of training examples has long been identified as source of improvement for machine learning models in multiple domains including nlp @xcite , computer vision @xcite and speech @xcite . previous work has suggested that deep learning scaling may be predictable empirically @xcite , with model size scaling sub-linearly with training data size. @xcite concluded that accuracy increases logarithmally with respect to training data size. however, these studies have focused on training models in the the fully supervised setting, without pretraining. one closer work is @xcite where it is shown that randomly initialized standard computervision models perform no worse than their ima-genet pretrained counterparts. however, our work focuses on text classification. we do not examine the benefit of pretraining, at large, rather we focus on the benefit of pretraining for resource rich tasks. another concurrent work that is still under review, in (nakkiran and sutskever, 2020) observes that, in some translation task such as iwslt14, small language models exhibit even lower test loss compared to the large transformer model when the number of training samples increases. ## discussion our findings in this paper indicate that increasing the number of training examples for 'standard' models such as lstm leads to performance gains that are within 1 percent of their massively pretrained counterparts. due to the fact that there is no good large scale question answering dataset, it is not clear if the same findings would hold on this type of nlp tasks, which are more challenging and semantic-based. in the future work, we will run more experiments if there are some other large scale open datasets. despite sentiment analysis being a crucial text classification task, it is possible, though unlikely, that the patterns observed here are limited to sentiment analysis tasks only. the rationale behinds that is that pretrained lstms have kept up very well with transformer-based counterparts on many tasks @xcite . one way to interpret our results is that 'simple' models have better regularization effect when trained on large amount of data, as also evidenced in the concurrent work (nakkiran and sutskever, 2020).the other side of the argument in interpreting our results is that mlm based pretraining still leads to improvements even as the data size scales into the millions. in fact, with a pretrained model and 2 million training examples, it is possible to outperform an lstm model that is trained with 3× more examples. ## conclusion finetuning bert-style models on resource-rich downstream tasks is not well studied. in this paper, we reported that, when the downstream task has sufficiently large amount of training exampes, i.e., millions, competitive accuracy results can be achieved by training a simple lstm, at least for text classification tasks. we further discover that reusing the token embeddings learned during bert pretraining in an lstm model leads to significant improvements. the findings of this work have significant implications on both the practical aspect as well as the research on pretraining. for industrial applications where there is a trade-off typically between accuracy and latency, our findings suggest it might be feasible to gain accuracy for faster models by collecting more training examples.