How much pretraining data do language models need to learn syntax? (2021)

Abstract: Transformers-based pretrained language models achieve outstanding results in many well-known NLU benchmarks. However, while pretraining methods are very convenient, they are expensive in terms of time and resources. This calls for a study of the impact of pretraining data size on the knowledge of the models. We explore this impact on the syntactic capabilities of RoBERTa, using models trained on incremental sizes of raw text data. First, we use syntactic structural probes to determine whether models pretrained on more data encode a higher amount of syntactic information. Second, we perform a targeted syntactic evaluation to analyze the impact of pretraining data size on the syntactic generalization performance of the models. Third, we compare the performance of the different models on three downstream applications: part-of-speech tagging, dependency parsing and paraphrase identification. We complement our study with an analysis of the cost-benefit trade-off of training such models. Our experiments show that while models pretrained on more data encode more syntactic knowledge and perform better on downstream applications, they do not always offer a better performance across the different syntactic phenomena and come at a higher financial and environmental cost.

## introduction the use of unsupervised pretrained language models in the context of supervised tasks has become a widely spread practice in nlp, with transformer-based models such as bert @xcite and roberta @xcite achieving outstanding results in many well-known natural language understanding benchmarks such as glue @xcite and squad @xcite . consequently, several studies investigate the types of knowledge learned by * work partially done during internship at amazon ai. bert, how and where this knowledge is represented and what the best methods to improve it are; see, e.g., @xcite . there is evidence that, among other information (e.g., part-of-speech, syntactic chunks and roles @xcite @xcite , morphology in general @xcite , or sentence length @xcite ), bert representations implicitly embed entire syntax trees @xcite . language models are traditionally assessed by information-theoretical metrics such as perplexity, i.e., the probability of predicting a word in its context. the general wisdom is that the more pretraining data a model is fed, the lower its perplexity gets. however, large volumes of pretraining data are not always available and pretraining is costly, such that the following questions need to be answered: (i) do we always need models pretrained on internetscale corpora? (ii) as the models are pretrained on more data, and their perplexity improves, do they encode more syntactic information and offer a better syntactic generalization? (iii) do the models with more pretraining perform better when applied in downstream tasks? to address these questions, we explore the relation between the size of the pretraining data and the syntactic capabilities of roberta by means of the minibertas models, a set of 12 roberta models pretrained from scratch by @xcite on quantities of data ranging from 1m to 1b words. in particular: we observe that models pretrained on more data encode a higher amount of syntax according to hewitt and manning (2019b)'s metrics, but do not always lead to a better syntactic generalization. indeed, we find that models pretrained on less data perform equally good or even better than those pretrained on more data on 3 out of 6 syntactic test suites. when applied to downstream tasks, the models pretrained on more data perform generally better. however, the analysis of the trade-off between the cost of training a model and its performance shows that small performance gains come at a high economical and environmental cost that should be considered when developing new models. in what follows, section 2 provides some background on the syntactic assessment of language models, model costs, and the works related to ours. section 3 describes our experimental setup, introducing the minibertas models and the syntactic tests as well as the downstream applications we explore. section 4 presents the outcome of our experiments. section 5 offers a cost-benefit analysis of the pretraining of the different models, and section 6 summarizes the implications that our work has for the use of pretrained language models. ## results in this section, we explore the impact of the size of pretraining data on the syntactic information encoded by roberta from three different angles. ## cost-benefit analysis for the sake of a more holistic view on the quality of the models, we perform a cost-benefit analysis of the performance gains in the different tasks, with an estimate of the financial and environmental cost of developing the models. as the resources used to train the minibertas are not publicly available, we rely on the data provided in @xcite to estimate the cost of developing each individual model based on the costs of roberta, trained on 30b words, in proportion to the amount of words used to train each family of models. model family cost co 2 e (lbs) pos dep. parsing paraphrase id. roberta-base-1b @xmath05075 582.5 95.53 (+1.11%) 83.97 (+4.04%) 87.57 (+2.79%) roberta-base-10m @xmath150 5.825 91.69 (base) 65.45 (base) 79.44 (base) to develop each minibertas models, warstadt et al. run the pretraining 10 times for the bigger family (roberta-base-1b), and 25 times for the other three families (roberta-base-100m, roberta-base-10m and roberta-med-small-1m) with varying hyperparameters. therefore, to compute the cost of developing each family of models, we multiply the cost of training a single model by the number of pretraining runs needed to obtain it. ## discusion and conclusions our experiments shed light on the impact of pretraining data size on the syntactic capabilities of roberta. our results indicate that models pretrained with more data encode better syntactic information (as measured by hewitt and manning's structural probes) and offer a higher syntactic generalization over the different syntactic phenomena covered by the tests assembled in @xcite . moreover, models pretrained with more data seem to be more robust to the presence of modifiers in the syntactic tests, i.e,. intervening content inserted before the critical region. as was already observed in @xcite , there is no simple relationship between the perplexity of the models and the syntaxgym score: the variance in intrafamily sg score is not explained by the perplexity differences. when zooming in on the different test circuits, probing different linguistic phenomena, we observe that there is a high variability between the scores achieved by the models of the same family, with no single model for any family performing best across all tests. while the family pretrained with more data outperforms all the models of the other families on 3 out of 6 circuits, it offers a surprisingly low performance in gross syntactic state, clearly outperformed by the smaller models. we also compare the performance of the different models fine-tuned on pos tagging, dependency parsing and paraphrase identification, observing that models with more training data offer a better performance, and the model with the smaller architecture (roberta-med-small-1m) performs remarkably worse. however, while the amount of training data between families grows exponentially, we observe that the performance grows at a much slower rate, suggesting that there may be a limit to the knowledge that a roberta model can acquire solely from raw pretraining data. we complement our findings with a financial and environmental cost-benefit analysis of pretraining models on different amounts of data. we show that while models pretrained on more data encode more syntactic information and perform generally better on downstream applications, small performance gains come at a huge financial and environmental cost. thus, when developing and training new models we should weigh between the benefit of making models bigger and pretraining them on huge datasets and the costs this implies, prioritizing computationally efficient hardware and algorithms. a question that still needs to be addressed by future work is whether it is possible to complement information-theoretical metrics such as perplexity with metrics measuring specific types of knowledge, e.g., syntax, in order to develop and select more robust and efficient models to solve natural language understanding tasks.