When Do You Need Billions of Words of Pretraining Data? (2021)

Abstract: NLP is currently dominated by language models like RoBERTa which are pretrained on billions of words. But what exact knowledge or skills do Transformer LMs learn from large-scale pretraining that they cannot learn from less data? To explore this question, we adopt five styles of evaluation: classifier probing, information-theoretic probing, unsupervised relative acceptability judgments, unsupervised language model knowledge probing, and fine-tuning on NLU tasks. We then draw learning curves that track the growth of these different measures of model ability with respect to pretraining data volume using the MiniBERTas, a group of RoBERTa models pretrained on 1M, 10M, 100M and 1B words. We find that these LMs require only about 10M to 100M words to learn to reliably encode most syntactic and semantic features we test. They need a much larger quantity of data in order to acquire enough commonsense knowledge and other skills required to master typical downstream NLU tasks. The results suggest that, while the ability to encode linguistic features is almost certainly necessary for language understanding, it is likely that other, unidentified, forms of knowledge are the major drivers of recent improvements in language understanding among large pretrained models.

## introduction pretrained language models (lms) like bert and roberta have become ubiquitous in nlp. new models require massive datasets of tens or even hundreds of billions of words @xcite to improve on existing models on language understanding benchmarks like glue @xcite . much recent work has used probing methods to evaluate what these models do and do not *equal contribution we fit an exponential curve which we scale to have an initial value of 0 and an asymptote at 1. classifier and mdl probing mainly test models' encoding of linguistic features; blimp tests model's understanding of linguistic phenomena; lama tests factual knowledge; superglue is a suite of conventional nlu tasks. with these questions in mind, we evaluate and probe the minibertas @xcite , a group of roberta models pretrained on 1m, 10m, 100m, and 1b words, and roberta base @xcite pretrained on about 30b words, using five methods: first we use standard classifier probing on the edge probing suite of nlp tasks @xcite to measure the quality of the syntactic and semantic features that can be extracted by a downstream classifier with each level of pretraining. second, we apply minimum description length (mdl) probing @xcite to the edge probing suite, with the goal of quantifying the accessibility of these features. third, we test the models' knowledge of various syntactic phenomena using unsupervised acceptability judgments on the blimp suite @xcite . fourth, we probe the models' world knowledge and commonsense knowledge using unsupervised language model knowledge probing with the lama suite @xcite . finally, we fine-tune the models on five tasks from superglue @xcite to measure their ability to solve conventional nlu tasks. for each evaluation method, we fit an exponential learning curve to the results as a function of the amount of pretraining data, shown in figure 1 . we have two main findings: first, the results of classifier probing, mdl probing, and unsupervised relative acceptability judgement (blimp) show that the linguistic knowledge of models pretrained on 100m words and 30b words is similar, as is the description length of linguistic features. second, roberta requires billions of words of pretraining data to effectively acquire factual knowledge and to make substantial improvements in performance on dowstream nlu tasks. from these results, we conclude that there are skills critical to solving downstream nlu tasks that lms can only acquire with billions of words of pretraining data. future work will likely need to look beyond core linguistic knowledge if we are to better understand and advance the abilities of large language models. ## methods we probe the minibertas, a set of 12 roberta models pretrained from scratch by @xcite on 1m, 10m, 100m, and 1b words, the publicly available roberta base @xcite , which is pretrained on about 30b words, 1 and 3 roberta base models with randomly initialized parameters. descriptions of the five evaluation methods appear in the subsequent sections. 2 in each experiment, we test all 16 models on each task involved. to show the overall trend of improvement, we use non-linear least squares to fit an exponential learning curve to the results. 3 we upsample roberta base results in regression in order to have an equal number of results for each data quantity. we use a four-parameter exponential learning curve used to capture diminishing improvement in performance as a function of the number of practice trials @xcite : we plot the results in a figure for each task, where the y-axis is the score and the x-axis is the amount of pretraining data. 5 for some plots, we use min-max normalization to adjust the results into the range of [0, 1], where 0 and 1 are the inferred values of p 0 and p ∞ , respectively. 6 ## classifier probing we use the widely-adopted probing approach of @xcite , @xcite , and otherswhich we call classifier probing-to test the extent to which linguistic features like part-of-speech and coreference are encoded in the frozen model representations. we adopt the ten probing tasks in the 1114 0.0 0.5 1.0 part-of-speech dependencies constituents relations (semeval) srl none 1m 10m 100m 1b 30b 0.0 0.5 1.0 sem. proto role 1 none 1m 10m 100m 1b 30b sem. proto role 2 none 1m 10m 100m 1b 30b ontonotes coref. none 1m 10m 100m 1b 30b entities none 1m 10m 100m 1b 30b winograd coref. performance (normalized) overall learning curve task learning curve overall results task results bert-large task performance in each subplot we also plot the overall edge-probing performance, which we calculate for each miniberta as its average f1 score on the 10 edgeprobing tasks (after normalization). for context, we also plot bert large performance for each task as reported by @xcite . task formulation and training following tenney et al., we use attention pooling to generate representation(s) of the token span(s) involved in the task and train an mlp that predicts whether a given label correctly describes the input span(s). we adopt the "mix" representation approach described in the paper. to train the probes, we use the same hyperparameters used in tenney et al. and tune the batch size and learning rate. 8 results we plot results in figure 2 . from the single-task curves we conclude that most of the feature learning occurs @xmath0. based on the best-fit curve, we can estimate that 90% of the attainable improvements in overall performance are achieved @xmath1. most plots show broadly similar learning curves, which rise sharply with less than 1m words of pretraining data, reach the point of fastest growth (in the log domain) around 1m words, and are nearly saturated with 100m words. the most notable exception to this pattern is the winograd task, which only rises significantly between 1b and 30b words of pretraining data. 9 as the winograd task is designed to test commonsense knowledge and reasoning, the results suggest that these features require more data to encode than syntactic and semantic ones, with the caveat that the dataset is smaller than the other edge probing tasks, and results on winograd tasks are highly sensitive to factors such as task formulation @xcite . we observe some general differences between different types of tasks. figure 3 shows the aggregated learning curves of syntactic, semantic, and commonsense tasks. the syntactic learning curve rises slightly earlier than the semantic one and 90% of the improvements in syntactic learning can be made with about 10m words, while the semantic curve still rises slightly after 100m. this is not surprising, as semantic computation is generally thought to depend on syntactic representations @xcite . the commonsense learning curve (for winograd coref. only) rises far later, and is projected to continue to rise long after syntactic and semantic features stop improving. ## minimum description length probing in this experiment, we study the minibertas with mdl probing @xcite , with the goal of revealing not only the total amount of feature information extracted by the probe, but also the effort taken by the probe to extract the features. mdl measures the minimum number of bits needed to transmit the labels for a given task given that both the sender and the receiver have access to the pretrained model's encoding of the data. a well-trained decoder model can help extract labels from the representations and thus reduce the number of bits needed to transmit the labels. since the model itself will also need to be transmitted, the total description length is a sum of two terms: the data codelength is the number of bits needed to transmit the labels assuming the receiver has the trained decoder model, i.e. the cross-entropy loss of the decoder. the model codelength is the number of bits needed to transmit the decoder parameters. we follow voita and titov's online code estimation of mdl, where the decoder is implicitly transmitted. as in section 3, we train decoders using the same hyperparameter settings and task definitions as @xcite . 10 results we plot the online code results in figure 4 . the overall codelength shows a similar trend to edge probing: most of the reduction in feature codelength is achieved with fewer than 100m words. mdl for syntactic features decreases even sooner. results for winograd are idiosyncratic, probably due to the failure of the probes to learn the task. the changes in model codelength and data codelength are shown on the bar plots in figure 4 . we compute the data codelength following voita and titov (2020) using the training set loss of a classifier trained on the entire training set, and the model codelength is the total codelength minus the data codelength. the monotonically decreasing data codelength simply reflects the fact that the more data rich roberta models have smaller loss. when it comes to the model codelength, however, we generally observe the global minimum for the randomly initialized models (i.e., at "none"). this is expected, and intuitively reflects the fact that a decoder trained on random representations would provide little information about the labels, and so it would be optimal to transmit a very simple decoder. on many tasks, the model codelength starts to decrease when the pretraining data volume exceeds a certain amount. however, this trend is not consistent across tasks and the effect is relatively small. ## unsupervised grammaticality judgement we use the blimp benchmark @xcite to test models' knowledge of individual grammatical phenomena in english. blimp is a challenge set of 67 tasks, each containing 1000 minimal pairs of sentences that highlight a particular morphological, syntactic, or semantic phenomena. minimal pairs in blimp consist of two sentences that differ only by a single edit, but contrast in grammatical acceptability. a language model classifies a minimal pair correctly if it assigns a higher probability to the acceptable sentence. since roberta is a masked language model (mlm), we measure pseudo log-likelihood @xcite to score sentences @xcite . ## unsupervised language model knowledge probe lama is a test suite introduced by @xcite to test lms' factual knowledge. it contains over 50,000 cloze statements converted from subjectrelation-object triples or question-answer pairs extracted from four datasets: googlere, foot_2 tre-x (elsahar et al., 2018), conceptnet @xcite , and squad @xcite . the google-re and t-rex tasks are each divided into three sub-tasks. ## fine-tuning on nlu tasks superglue is a benchmark suite of eight classification-based language-understanding tasks @xcite . we test each miniberta on five superglue tasks on which we expect to see significant variation at these scales. 12 the hyperparameter search range used for each task is described in the appendix. ## discussion factual knowledge may play a large role in explaining superglue performance. this hypothesis is backed up by results from the winograd edge-probing task (figure 2 ) and the lama tasks (figure 6 ), which suggest that most of the im-provements in the model's world and commonsense knowledge are made with over 100m words. however, the lama learning curve shows signs of slowing between 1b and 30b words, the super-glue curve does not. another possible explanation is that linguistic features encoded by a model may not be easily accessible during fine-turning. @xcite found that roberta can learn to reliably extract many linguistic features with little pretraining data, but requires billions of words of pretraining data before it uses those features preferentially when generalizing. in light of warstadt et al.'s findings, we had initially hypothesized that feature accessibility as measured by mdl might show a shallower or later learning curve than standard classifier probing. 13 our findings do not support this hypothesis: figure 1 shows no substantial difference between the classifier probing mdl probing curves. however, we do not totally rule out the possibility that linguistic feature accessibility continues to improve with massive pretraining sets. there are potential modifications to voita and titov's approach that could more faithfully estimate feature accessibility. first, although roberta is actually fine-tuned in most applications, we and voita and titov measure mdl taking the outputs of the frozen roberta model as input to a trainable mlp decoder. it may be more relevant to measure mdl by fine-tuning the entire model @xcite . second, mdl actually estimates the information content of a particular dataset, rather than the feature itself. @xcite propose an alternative to mdl that measures feature complexity in a way that does not depend on the size of the dataset. ## related work probing neural network representations has been an active area of research in recent years @xcite . with the advent of large pretrained transformers like bert @xcite , numerous papers have used classifier probing methods to attempt to locate linguistic features in learned representations with striking positive results @xcite . however, another thread has found problems with many probing methods: classifier probes can learn too much from training data @xcite and can fail to distinguish features that are extractable from features that are actually used when generalizing on downstream tasks @xcite @xcite . moreover, different probing methods often yield contradictory results @xcite . there have also been a few earlier studies investigating the relationship between pretraining data volume and linguistic knowledge in language models. studies of unsupervised acceptability judgments find fairly consistent evidence of rapid improvements in linguistic knowledge up to about 10m words of pretraining data, after which improvements slow down for most phenomena. van they measure roberta's preference for linguistic features over surface features during fine-tuning on ambiguous classification tasks. @xcite find large improvements in knowledge of subject-verb agreement and reflexive binding up to 10m words, and little improvement between 10m and 80m words. @xcite find that gpt-2 trained on 42m words performs roughly as well on a syntax benchmark as a similar model trained on 100 times that amount. other studies have investigated how one model's linguistic knowledge changes during the training process, as a function of the number of updates @xcite . @xcite also investigate how performance on superglue (and other downstream tasks) improves with pretraining dataset size between about 8m and 34b tokens. in contrast to our findings, they find that models with around 500m tokens of pretraining data can perform similarly on downstream tasks to models with 34b words. however, there are many differences in our settings that may lead to this divergence. for example, they pretrain for a fixed number of iterations (totaling 34b token updates), whereas the minibertas we use were pretrained with early stopping. they also use prefix prompts in their task formulations, and adopt an encoder-decoder architecture and thus their model has roughly twice the number of parameters of the largest model we evaluate. there is also some recent work that investigates the effect of pretraining data size of other languages. @xcite pretrain bert-based language models on 10mb, 100mb, 500mb, 1gb, 2gb, and 4gb of french text and test them on a question answering task. they find that the french mlm pretrained on 100mb of raw text has similar performance to the ones pretrained on larger datasets on the task, and that corpus-specific selfsupervised learning does not make a significant difference. @xcite also show that french mlms can already learn a lot from small-scale pretraining. concurrent work @xcite probes roberta models pretrained on different numbers of iterations using a set of probing tasks similar to ours. they find that linguistic abilities are acquired fastest, world and commonsense knowledge learning takes more iterations, and reasoning abilities are never stably acquired. both studies show that linguistic knowledge is easier to learn than factual knowledge. ## conclusion we track several aspects of roberta's ability as pretraining data increases. we find that ability in syntax and semantics largely saturates after only 10m to 100m words of pretraining data-on par with the data available to human learners-while learning factual knowledge requires much more data. we also find that scaling pretraining data size past billions of words significantly improves the nlu performance, though we cannot fully explain what abilities drive this improvement. answering this question could be a stepping stone to more data-efficient models.