  Models of language trained on very large corpora have been demonstrated
useful for NLP. As fixed artifacts, they have become the object of intense
study, with many researchers "probing" the extent to which linguistic
abstractions, factual and commonsense knowledge, and reasoning abilities they
acquire and readily demonstrate. Building on this line of work, we consider a
new question: for types of knowledge a language model learns, when during
(pre)training are they acquired? We plot probing performance across iterations,
using RoBERTa as a case study. Among our findings: linguistic knowledge is
acquired fast, stably, and robustly across domains. Facts and commonsense are
slower and more domain-sensitive. Reasoning abilities are, in general, not
stably acquired. As new datasets, pretraining protocols, and probes emerge, we
believe that probing-across-time analyses can help researchers understand the
complex, intermingled learning that these models undergo and guide us toward
more efficient approaches that accomplish necessary learning faster.
