A Systematic Assessment of Syntactic Generalization in Neural Language Models (2020)

Abstract: While state-of-the-art neural network models continue to achieve lower perplexity scores on language modeling benchmarks, it remains unknown whether optimizing for broad-coverage predictive performance leads to human-like syntactic knowledge. Furthermore, existing work has not provided a clear picture about the model properties required to produce proper syntactic generalizations. We present a systematic evaluation of the syntactic knowledge of neural language models, testing 20 combinations of model types and data sizes on a set of 34 English-language syntactic test suites. We find substantial differences in syntactic generalization performance by model architecture, with sequential models underperforming other architectures. Factorially manipulating model architecture and training dataset size (1M-40M words), we find that variability in syntactic generalization performance is substantially greater by architecture than by dataset size for the corpora tested in our experiments. Our results also reveal a dissociation between perplexity and syntactic generalization performance.

## introduction a growing body of work advocates that assessment of neural language models should include both information-theoretic metrics, such as perplexity, as well as targeted linguistic evaluation. benchmarks such as glue @xcite have demonstrated that neural language models trained on naturalistic corpora for next-word prediction learn representations that can yield remarkable performance on many semantic tasks. targeted syntactic evaluations have shown that these models also implicitly capture many syntactic generalizations, ranging from subject-verb agreement materials and code can be found at https://github. com/cpllab/syntactic-generalization . to long-distance filler-gap dependencies @xcite @xcite . this paper aims to bring targeted evaluations of syntactic performance to scale, complementing similar developments in semantic evaluation @xcite . because the most widespread currency of evaluation for language models is perplexity-how well, on average, a model predicts a word in its contexta primary focus of this paper is the relationship between a model's perplexity and its performance on targeted syntactic evaluations. as perplexity improves, can we expect more human-like syntactic generalization? how do training dataset size and model architecture jointly affect syntactic generalization? and what picture of models' syntactic generalization emerges when evaluation is brought to scale, across dozens of controlled syntactic tests? in this paper we offer initial answers to these questions, systematically assessing the syntactic generalization abilities of neural language models on 34 targeted test suites (33 adapted from previously published work, and 1 novel) covering a wide range of syntactic phenomena. test suites are written using a standard format that allows for flexible predictions which more closely resemble those used in psycholinguistic studies, specifically allowing for predictions about interactions among multiple testing conditions. performance on each test suite is reported as a syntactic generalization (sg) score. we group test suites into six syntactic circuits based on the linguistic representations needed to achieve high performance on each suite. we train four classes of neural models and one baseline n-gram model on four datasets derived from a newswire corpus, consisting of 1, 5, 14, and 42 million tokens. while previous work has compared model architectures for a fixed dataset size (e.g. @xcite and network sizes for a fixed architecture (e.g. @xcite , our controlled regime allows us to make an apples-to-apples comparison across model architectures on a range of sizes. in addition, we evaluate several off-the-shelf models which were trained on datasets ranging up to 2 billion tokens. our results address the three questions posed above: first, for the range of model architectures and dataset sizes tested, we find a substantial dissociation between perplexity and sg score. second, we find a larger effect of model inductive bias than training data size on sg score, a result that accords with @xcite . models afforded explicit structural supervision during training outperform other models: one structurally supervised model is able to achieve the same sg scores as a purely sequence-based model trained on ∼100 times the number of tokens. furthermore, several transformer models achieve the same sg score as a transformer trained on ∼200 times the amount of data. third, we find that architectures have different relative advantages across types of syntactic tests, suggesting that the tested syntactic phenomena tap into different underlying processing capacities in the models. ## methods we designed a controlled paradigm for systematically testing the relationship between two design choices -model class and dataset size -and two performance metrics -perplexity and syntactic generalization capacity. section 3.1 describes the test suites collected for our evaluation, and sections 3.2 and 3.3 describe the datasets and model classes investigated. ## results figure 1 shows the average accuracy of all models on the complete set of sg test suites. asterisks denote off-the-shelf models. all neural models achieve a sg score significantly greater than a random baseline (dashed line). however, the range within neural models is notable, with the bestperforming model (gpt-2-xl) scoring over twice as high as the worst-performing model (lstm). also notable are the controlled gpt-2 and rnng models, which achieve comparable performance to transformer-xl and jrnn, despite being trained on significantly smaller data sizes. test perplexity we now return to the three major issues presented in section 1. in 4.1 we present evidence that sg score is dissociated from perplexity. in 4.2 we argue that model architecture accounts for larger gains in sg score than amount of training data. and in 4.3 we show that this cross-architecture difference is due largely to variance on a handful of key test suites. ## discussion this work addresses multiple open questions about syntactic evaluations and their relationship to other language model assessments. our results dissociate model perplexity and performance in syntactic generalization tests, suggesting that the two metrics capture complementary features of language model knowledge. in a controlled evaluation of different model classes and datasets, we find model architecture plays a more important role than training data scale in yielding correct syntactic generalizations. our circuit-level analysis reveals consistent failure on licensing but inconsistent behavior on other circuits, suggesting that different syntactic circuits make use of different underlying processing capacities. in addition to the insight these results provide about neural nlp systems, they also bear on questions central to cognitive science and linguistics, putting lower bounds on what syntactic knowledge can be acquired from string input alone. targeted syntactic evaluation is just one in a series of complementary methods being developed to assess the learning outcomes of neural language processing models. other methods include classifying sentences as grammatical or ungrammatical @xcite , decoding syntactic features from a model's internal state @xcite , or transfer learning to a strictly syntactic task such as parsing or pos tagging @xcite . as each task brings an explicit set of assumptions, complementary assessment methods can collectively provide greater insight into models' learning outcomes. although this paper, together with @xcite , report what is to our knowledge the largestscale targeted syntactic evaluations to date, we emphasize that they are only first steps toward a comprehensive understanding of the syntactic capabilities of contemporary language models. this understanding will be further advanced by new targeted-evaluation test suites covering a still wider variety of syntactic phenomena, additional trained models with more varied hyperparameters and randomization seeds, and new architectural innovations. humans develop extraordinary grammatical capabilities through exposure to natural linguistic input. it remains to be seen to just what extent contemporary artificial systems do the same.