Downstream Datasets Make Surprisingly Good Pretraining Corpora (2023)

Abstract: For most natural language processing tasks, the dominant practice is to finetune large pretrained transformer models (e.g., BERT) using smaller downstream datasets. Despite the success of this approach, it remains unclear to what extent these gainsare attributable to the massive background corpora employed for pretraining versus to the pretraining objectives themselves. This paper introduces a large-scale study of self-pretraining, where the same (downstream) training data is used for both pretraining and finetuning.In experiments addressing both ELECTRA and RoBERTa models and 10 distinct downstream classification datasets, we observe that self-pretraining rivals standard pretraining on the BookWiki corpus (despite using around 10x–500x less data), outperforming the latter on 7 and 5 datasets, respectively. Surprisingly, these task-specific pretrained models often perform well on other tasks,including the GLUE benchmark. Besides classification tasks, self-pretraining also provides benefits on structured output prediction tasks such as span based question answering and commonsense inference, often providing more than 50% of the performance boosts provided by pretraining on the BookWiki corpus. Our results hint that in many scenarios, performance gains attributable to pretraining are driven primarily by the pretraining objective itself and are not always attributable to the use of external pretraining data in massive amounts. These findings are especially relevant in light of concerns about intellectual property and offensive content in web-scale pretraining data.

## introduction for training predictive models operating on natural language data, the current best practice is to pretrain models on large unlabeled upstream corpora to optimize self-supervised objectives, for example, masked language modeling (mlm); the resulting weights are then used to initialize models that are subsequently trained (finetuned) on the labeled downstream data available for the task at hand. large-scale pretrained models typically provide significant performance boosts when compared to models trained directly on the downstream task (with random initializations) @xcite @xcite . upstream corpora tend to be significantly larger than the downstream corpora and the success of this approach is often attributed to its ability to leverage these massive upstream corpora @xcite . for example, the seminal bert model @xcite was pretrained using the bookwiki corpus which is a combination of english wikipedia and bookscorpus @xcite , totaling 13gb of plain text. subsequent models have moved on to web-scale data. for example, xlnet @xcite , roberta @xcite , and t5 @xcite )), were trained on 158gb, 160gb and 750gb of data, respectively. as upstream corpus size and downstream performance have gone up, popular attempts at explaining these gains have focused on themes of "knowledge transfer" from the upstream corpus, attributing them to shared linguistic structure, semantics @xcite , and facts about the world @xcite . however, since the introduction of large-scale pretraining corpora occurred together with the invention of self-supervised pretraining objectives (e.g. masked language modeling @xcite and replaced token detection @xcite ), it remains unclear to what extent large-scale corpora are integral to these leaps in performance. for several tasks, especially summarization, recent works achieved surprising performance gains in settings where the upstream corpus is created synthetically with arbitrary symbols, but the pretraining objective is designed to capture some of the structure of the task @xcite . in this work, we ask just how much of pretraining's benefits could be realized in the absence of upstream corpora by pretraining directly on the downstream corpora (with the same self-supervised objectives). we find that this approach, which we call self-pretraining, often rivals the performance boosts conferred by off-the-shelf models pretrained on large upstream corpora (figure 1 ), even outperforming them on 7 out of 10 datasets. prior research has shown that additional self-supervised pretraining of off-the-shelf models using the downstream data can give further gains @xcite . our study goes further, showing that even when starting from random initializations, and without using any external data beyond the downstream data itself, self-pretraining can rival standard practices. since self-pretraining requires the same data that must already be available for finetuning, the benefits of pretraining in this case cannot be attributed to transfer of knowledge from the upstream corpus. instead, these benefits can only be attributed to the pretraining objective, which is possibly able to learn some inductive biases better than the finetuning objective (e.g. linguistic knowledge @xcite ), or perhaps simply initializes network parameters such that their statistics lead to better optimization during finetuning @xcite . while similar observations were made in the computer vision community (el-nouby et al., 2021), we argue that it is especially important to establish these phenomena in the language domain, for which building on self-supervised pretrained models is now a ubiquitous practice. to understand differences in predictions with different pretraining strategies (i.e., between selfpretrained and off-the-shelf models), we analyse the errors made by these models on the same downstream data (sec. 6). despite similar performance of these models, we find that self-pretrained and off-the-shelf models make significantly less corre-lated errors when compared to two independently finetuned models pretrained with either strategy. we find that models pretrained on one downstream dataset often perform surprisingly well when finetuned to other downstream datasets (sec. 5), including the glue benchmark. even though the downstream datasets in our study come from a wide variety of domains (e.g., news, online forums, tweets), we find that pretraining on any of these downstream datasets delivers significant performance gains on most datasets (greater than half of off-the-shelf model's gains in 88% of cases) irrespective of domain. however, the best performance on a downstream dataset is usually achieved by the model pretrained on that dataset itself. in addition to classification tasks, we also experiment with tasks such as span-based question answering, named entity recognition, and grounded commonsense inference (sec. 8). self-pretraining delivers around 40-80% of the performance boost compared to models pretrained on the bookwiki corpus across electra and roberta models. hence, self-pretraining can perform better than finetuning randomly initialized models even for tasks that require prediction of more complex structured output than a single label, and for tasks whose solution relies on commonsense knowledge. our contributions can be summarized as follows: pretraining on downstream data in nlp task-adaptive pretraining (tapt @xcite ) consists of taking off-the-shelf pretrained models like bert and roberta and engaging in further pretraining on the downstream datasets before finetuning them to the task at hand. tapt has been shown to improve performance of offthe-shelf models in a variety of works @xcite @xcite . another way in which downstream data has been used is for retrieval to create a small pretraining corpus for efficient pretraining @xcite . by contrast, our work pretrains models only on the downstream dataset, enabling a head-to-head comparison between the performance of off-the-shelf and self-pretrained models, and (in some situations) challenging the necessity of upstream corpora altogether. claims about knowledge transfer many works claim that pretraining extracts generally useful knowledge from the upstream corpus such as linguistic patterns @xcite @xcite and facts @xcite , and that this accounts for the performance gains that they enjoy on downstream tasks. several works, e.g., in the probing literature @xcite @xcite , demonstrate that from the internal representations of a model, it is easy (e.g., via linear models) to predict certain linguistic features or real-world facts. however, these studies do not clarify the mechanism by which these observations relate to performance gains on downstream tasks. @xcite recognizes this limitation, stating "the observation of a (linguistic) pattern does not tell us how it is used". our work suggests that to the extent that such knowledge extraction plays a role in pretraining's benefits, sufficient knowledge is often present in the downstream dataset and need not be transferred from huge upstream corpora. challenges to the knowledge transfer narrative multiple previous works have questioned whether knowledge transfer can fully account for the efficacy of pretraining. improvements in performance on downstream nlp tasks have resulted from pretraining on other modalities like music and code @xcite , sequences of meaningless symbols @xcite @xcite , and language denatured via shuffling of words @xcite . on the other hand, models pretrained on language have shown improved performance on tasks dealing with other modalities such as image classification @xcite and reinforcement learning for games @xcite . by contrast, we show that without surplus upstream data of any modality, self-pretraining alone can often perform comparably or even better than standard pretraining with a large upstream corpus. in a similar vein with these papers, our work suggests that a large portion of pretraining's success may come from alternative, unexplored mechanisms which have more to do with the pretraining objective than knowledge transfer from upstream corpora. ## experimental setup our experiments center around the electra model @xcite and the roberta-base model @xcite . on the broadest set of experiments, for which we can only afford to train one model, we employ electra because it performs better than roberta given comparable compute budgets @xcite . in particular, we use the small variant of electra (14 million parameters), which performs similarly to bertbase on glue (difference of ≈2 points) while training much faster @xcite . however, we replicate many of these results on the larger roberta-base model revealing similar results and thus establishing the generality of our findings. during pretraining, a text sequence is fed into the model with some tokens masked out. while mlmonly models like roberta only have a generator network that predicts the content of the masked tokens, electra has an additional discriminator module that predicts if those predictions were correct. both the generator and the discriminator ## self-pretraining performance in our first set of experiments, we compare selfpretraining's performance with other pretraining techniques. for each dataset, we pretrain an elec-tra model on text from its training split and then finetune it on the same training data using the associated labels. to create a pretraining corpus from a downstream dataset, we concatenate the input text from each of the examples, assembling them in random order. we evaluate the performance of each finetuned model on the corresponding dataset's test split. for qqp and mnli we just use the validation split because test set labels are private. for all datasets, we evaluate performance by accuracy, except for sentiment140 and jigsaw toxicity, for which we use pearson correlation and microaveraged auc scores, respectively (these are not multi-class classification problems). notably, all self-pretrained models deliver significant performance boosts on their respective datasets ( we also evaluated the alternate pretraining technique tapt as described in @xcite . in this technique, we take the off-the-shelf electra model, which has already been pretrained on the upstream bookwiki corpus, and further pretrain it on the downstream dataset for 100 epochs. self-pretraining outperforms tapt on 6 datasets, notably including the two datasets where it outperformed the off-the-shelf models by the greatest benefit margin -agnews and yahoo answertopics. interestingly, tapt performs worse than off-the-shelf model on the same 3 datasets where self-pretraining performs worse than off-theshelf model (except sentiment140). none of the three pretraining approaches seem to be uniformly better than any other. finally, we also evaluate the self-pretrained models on the glue benchmark and report results on the dev set foot_1 . the performance of the models on their pretraining dataset does not correlate strongly with its glue score. the glue score also does not monotonically go up with increasing dataset size, indicating that the data domain makes some difference. for example, the amazon polarity corpus scores just 66.14 on glue despite being about 1.4gb in size, while agnews which is 27mb in size, scores 74.30. the highest glue score is achieved by pretraining on yahoo answertopics. ## cross dataset finetuning in this set of experiments, we investigated if the models pretrained on a dataset are only useful for that specific task, or are they useful across the whole spectrum of tasks that we consider. we took each model pretrained on a dataset in our testbed and finetuned and evaluated it on all other datasets in the testbed. the performance benefits provided in all cases are shown as a heatmap in figure 2 . we found that for almost all downstream datasets, pretraining on any other dataset provides significant advantage (figure 2 ). in most cases, pretraining on the downstream dataset itself performs the best. among datasets where self-pretraining performs better than off-the-shelf model (i.e. the diagonal entry is greater than 1), pretraining on datasets of larger size does not help further. however, for the datasets where self-pretraining's benefit is much less than 100% (i.e. mnli and discovery), pretraining on a larger dataset (e.g., yahoo answertopics) performs better than self-pretraining. among all the pretrained models, a few models perform consistently good or bad across different downstream datasets (figure 2 ). for example, the model pretrained on yahoo answertopics gets the highest average score of 0.90 across all datasets, while the paws-pretrained model gives the lowest aggregate score of 0.64. similarly, there are downstream datasets that are benefited consistently by either a large or a small margin by pretraining on different datasets. for example, performance on qqp and paws receives huge boosts by pretraining on most datasets. in contrast, performance on sentiment140 is mostly low , even dropping below 20% for 3 pretrained models. we perform an ablation to investigate that given a fixed dataset to finetune on, is it better to pretrain on the exact same data (i.e., using the same set of inputs), or is it better to pretrain on different data with an identical distribution. to test this hypothesis, we divided the training splits of the downstream datasets randomly into two equal subsets (denoted as a and b). we pretrained one model on each subset and then finetuned them on both subsets separately. the validation and test sets used for finetuning are the same as in the original dataset. we do not see any consistent benefits with pretraining and finetuning on the same dataset ( ## difference in outputs of self-pretrained and off-the-shelf models since self-pretrained models and off-the-shelf models perform similarly in terms of classification accuracy, a natural question to ask is: do these models make errors on the same set of inputs? to answer this question, we investigate the difference in predictions made by models pretrained with different strategies across all multi-class classification tasks. in particular, given model f a and f b , we compute error inconsistency, defined as follows: compared to error inconsistency between two models with the same pretraining dataset, we ob-serve that models trained with different pretraining datasets have high error inconsistency in predictions ( we make the following observations: first, as expected we observe that ensembling improves model performance as compared to a single model ( ## ablations with other pretraining architectures we conducted our experiments so far with electra-small architecture because it is faster to pretrain than other popular models, yet delivers good downstream performance @xcite (e.g. comparable to bert-base on glue benchmark). here, we conduct experiments with a larger model and a different pretraining objective to test the efficacy of self-pretraining more broadly. we experiment with the roberta model which uses the masked language modeling objective, rather than electra's objective. we use the roberta-base architecture, which has a much larger parameter count of 110 million, compared to electra-small's 14 million. due to resource constraints, we pretrained the roberta models for fewer iterations as in @xcite . we pretrain a roberta-base model on the book-wiki corpus for the same number of iterations. our results show that self-pretraining performs comparably to pretraining on bookwiki corpus, delivering over 85% of pretraining benefit on 9 out of 10 datasets, and outperforming the model pretrained on bookwiki corpus ( ## performance on structured prediction and commonsense nli while the bulk of our experiments were on a variety of classification tasks, we also experiment with some tasks beyond simple classification. we experiment with three types of tasks: (i) span based question answering, (ii) named entity recognition (ner), and (iii) grounded commonsense inference. for question answering we use the squad dataset @xcite (v1.1) and report the f1-score. for ner, we use the conll-2012 ner task which uses annotations from ontonotes v5.0 @xcite involving 18 kinds of named entities. to measure performance, we use the overall f1 score. we use the seqeval library for evaluation ( https: //github.com/chakki-works/seqeval ). we include swag @xcite and hel-laswag @xcite for multiple-choice sentence completion. for electra-small models, we see that for each of these datasets self-pretraining achieves more than 70% pretraining benefit, and for robertabase model the benefit is 40-80% ( ## conclusion and future work in this work, we showed that pretraining models only on text from the downstream dataset performs comparably to pretraining on a huge upstream corpus for a wide variety of datasets. the errors made by such self-pretrained models on the downstream tasks are significantly different from the ones made by the off-the-shelf models pretrained on upstream corpora. our results suggest that the importance of learning from surplus upstream data for improving downstream task performance may have been overestimated. crucially, our experiments also do not show that upstream data does not help at all or knowledge transfer does not occur, but simply questions to what extent it is responsible for downstream gains. for example, the impressive zeroshot performance very large language models such as gpt-3 @xcite clearly suggests knowledge transfer is involved. one direction of future work would be to investigate how the performance of self-pretraining compares of pretraining on upstream corpora as the model sizes go up by orders of magnitude. we found that the quantity and quality of data required for pretraining to provide significant benefit (over a randomly initialized model trained only with a supervised loss) is quite low. downstream datasets which are tiny in comparison to typical upstream corpora, still function as useful pretraining corpora for getting performance gains across a wide range of datasets. since self-pretraining does not involve any upstream corpus, it prevents exposure of the model to potentially undesirable contents in the large upstream corpus, while still delivering large performance benefits. research has demonstrated the negative influence of web-sourced pretraining corpora on models, such as generating toxic language @xcite or reflecting racial biases in predictions @xcite . for use cases that require avoding such issues, selfpretraning can provide a viable alternative to standard pretraining. in future work, we hope to compare how self-pretrained models and off-theshelf models perform on these negative measures such as toxicity and social biases.