Probing Across Time: What Does R o BERT a Know and When? (2021)

Abstract: Models of language trained on very large corpora have been demonstrated useful for natural language processing. As fixed artifacts, they have become the object of intense study, with many researchers “probing” the extent to which they acquire and readily demonstrate linguistic abstractions, factual and commonsense knowledge, and reasoning abilities. Recent work applied several probes to intermediate training stages to observe the developmental process of a large-scale model (Chiang et al., 2020). Following this effort, we systematically answer a question: for various types of knowledge a language model learns, when during (pre)training are they acquired? Using RoBERTa as a case study, we find: linguistic knowledge is acquired fast, stably, and robustly across domains. Facts and commonsense are slower and more domain-sensitive. Reasoning abilities are, in general, not stably acquired. As new datasets, pretraining protocols, and probes emerge, we believe that probing-across-time analyses can help researchers understand the complex, intermingled learning that these models undergo and guide us toward more efficient approaches that accomplish necessary learning faster.

## introduction current nlp approaches lean heavily on language models trained on very large corpora @xcite @xcite @xcite . many researchers have sought to interpret what kinds of knowledge are acquired during this "pretraining" phase @xcite @xcite . extending @xcite , we systematically conduct probing across the pretraining iterations, to understand not just what is learned (as explored in numerous past analyses of fixed, already-trained * equal contribution. models), but also when. in this work, we aim to inform future work on more efficient pretraining (e.g., fewer iterations are needed to acquire some kinds of knowledge) and on understanding dependencies among different kinds of knowledge. specifically, we apply a probing across time framework to the widely used roberta masked language model @xcite . we reproduce the pretraining of roberta and apply a suite of probes at many checkpoint iterations across pretraining ( §3). our rich probe suite covers a diverse range of desirable knowledge types: linguistic properties @xcite , factual knowledge @xcite , and commonsense @xcite and basic reasoning capabilities @xcite . our main finding is that linguistic information tends to be acquired fast, factual and commonsense knowledge slower, and reasoning abilities are largely unlearned. we next apply probing across time to instances of roberta trained on text from varying domains and with varying amounts of data ( §4). our experiments show that the learning order and learning patterns of different types of knowledge generally hold regardless of the data variation. however, different data choices do have an impact on the learning speed and the final performance. our findings suggest that the inclusion of data in more diverse domains is more important than the quantity alone. finally, we compare probes across time with research benchmark task performance across time ( §5). we find that most of these benchmark tasks (e.g., sst-2, @xcite require a relatively small number of pretraining steps to achieve high performance, which is similar to the fast learning patterns shown by linguistic probes. some other tasks that are designed to test more complex knowledge (e.g., @xcite benefit from longer pretraining time, aligning well with our findings for the correspond-ing type of probes. we expect that, as new pretrained models and new probes emerge, probing-across-time analyses can help synthesize evidence for models' capabilities. we release our code, as well as all the pretraining checkpoints at https://github.com/ leo-liuzy/probe-across-time to benefit future research. ## probing across time the great success of pretrained language models has motivated researchers to characterize what kinds of knowledge they encode. probing seeks to determine how much is known by the pretrained model, so its representations are used without finetuning. few or zero additional parameters are estimated to target the probing task so that success on the probing task is attributable to pretraining alone. past probing work has applied probing analysis after pretraining is complete and compares different models (e.g., bert vs. glove). we extend probing to different parameter states (i.e., training checkpoints) over the same model's pretraining trajectory. by observing how probe performance changes over time, we hope to understand not just what the pretrained language model knows, but also when. to that end, we adopt a diverse set of probes aiming at different types of knowledge ( §2.1) and apply those probes at different iterates of the estimated model across pretraining. this requires pretraining from scratch and checkpointing intermediate parameter estimates ( §2.3). considering our computational budget, we choose roberta foot_0 as our case study because of its popularity in supporting downstream probes and tasks. we leave exploration of different objectives and architectures, e.g., gpt-2 @xcite , as future work. we also set up baselines as additional relative references to stand for the best and the worst expected results from pretraining ( §2.2). see §a for implementation details and the computational cost. ## learning patterns in this section, we use our reproduced roberta to ask and answer, at which stage does the model acquire each kind of knowledge? in addition to plotting probe performance across time (i.e., parameter updates during learning; see fig. 1 ), a useful measurement is the number of updates required to reach x% of the maximum performance achieved by our model across all iterations. we denote this measurement by "learning progress-x%" for @xmath0, 95, 97}; it is indicated by the bottom horizontal bars in each probe's plot in fig. 1 . we consider each type of knowledge in turn. all of the following discussion is supported by fig. 1 . ## varying the pretraining corpus our experiments on the pretraining trajectory in the previous section demonstrate that, when acquired, different types of knowledge are learned at different stages. on the other hand, previous work @xcite demonstrates that language domain and data size are important factors for a pretrained model's performance in downstream tasks. thus, the question arises: do our observations on the pretraining trajectory hold regardless of the training corpus? in this section, we ask and answer how do domains and corpus sizes affect the learning trajectory? domains with the same setting as §3, we pretrain roberta with three controlled domains: english wikipedia, realnews, and stories. we downsample realnews and stories to be roughly the same size as wikipedia (12 gb). the first row of fig. 2 compares probing performance across time over varying domains. on one hand, the general learning pattern from the previous section persists regardless of the domain: linguistic knowledge is acquired faster than the other types. on the other hand, we find that the change in the pretraining domain affects the final performance of all knowledge types but to different extents. lkt is generally less affected than blimp, probably due to the additional training of lkt's classifier on the probing data. although the model slowly acquires both factual and commonsense knowledge regardless of corpus domains, factual knowledge is much more affected by domain (in fact, most affected among all). lama especially shows very slow learning on the stories domain, implying that factual knowledge might be very sparse in stories. the fact that the included factual tests-google-re, t-rex and squad-are sourced from wikipedia might explain why the factual tests show roberta trained on wikipedia learns faster than the one trained on realnews. though many existing works @xcite @xcite used story data to study commonsense, pretraining the model on stories still performs worst on our commonsesense probes (conceptnet in lama and most probes in cat). on the olmpics probes, we keep observing large fluctuation on different domains, but do see more signs of learning on the wikipedia and realnews domains than on stories. see §b.3 for more detailed plots. corpus sizes to investigate the impact of corpus size, we experimented on downsampled english wikipedia (4 gb), the original english wikipedia (12 gb), and a combination of wikipedia with downsampled realnews and downsampled stories (36 gb in total). the second row of fig. 2 compares probing performance over varying corpus sizes. in general, the learning order and learning patterns of different types of knowledge that we discuss in §3 still hold for all data sizes. comparing different data sizes, we find that the biggest corpus (with more diverse inclusion of domains) generally learns faster and results in an ultimately better roberta in all tested knowledge. on the other hand, interestingly, comparing the downsampled english wikipedia (4 gb) and original english wikipedia (12 gb), simply increasing the corpus size without changing domains does not improve the final probing performance substantially in all categories. this is even true for lama, which tests factual knowledge relevant to wikipedia. this suggests that diversity of data might be more important for pretraining than quantity. see §b.4 for more detailed plots. ## experiments on research benchmarks figure 2: performance of probes across time, comparing pretraining domains and corpus sizes. this plot reveals that the learning order and patterns of different knowledge types generally hold regardless of the data variation, with higher impact from corpus domains. performance for each probing package is averaged over its 4-5 selected tasks and smoothed with a coefficient of 0.5 for better visualization (same as the first column in fig. 1 ). ## ethical concerns we estimate our carbon costs in appendix §a. we recognize the possibility that changes to hyperparameters might lead to different conclusions and leave it to future work to balance the costs of such exploration with the value of a more detailed understanding of hyperparameter impact on learning over time. we note that the robustness of learning patterns is partly supported by results from @xcite , since they probed models trained with different hyperparameters and observed similar linguistic patterns as we did. ## conclusion we have shown how probing across time reveals when, during pretraining iterations, a masked language model acquires various kinds of knowledge. roberta, our case study model, is shown to learn linguistic knowledge faster than factual and commonsense knowledge, but struggle to learn reasoning abilities. we explored variation due to corpus domain and size, and related our findings to research benchmark tasks. as models evolve and new probes emerge, we believe our probing across time framework can serve as a general framework to inform progress on both fronts. yukun zhu, ryan kiros, richard s. zemel, ruslan salakhutdinov, raquel urtasun, antonio torralba, and sanja fidler. 2015. aligning books and movies: towards story-like visual explanations by watching movies and reading books. in proc. of iccv.