BL i MP : The Benchmark of Linguistic Minimal Pairs for E nglish (2020)

Abstract: We introduce The Benchmark of Linguistic Minimal Pairs (BLiMP),1 a challenge set for evaluating the linguistic knowledge of language models (LMs) on major grammatical phenomena in English. BLiMP consists of 67 individual datasets, each containing 1,000 minimal pairs—that is, pairs of minimally different sentences that contrast in grammatical acceptability and isolate specific phenomenon in syntax, morphology, or semantics. We generate the data according to linguist-crafted grammar templates, and human aggregate agreement with the labels is 96.4%. We evaluate n-gram, LSTM, and Transformer (GPT-2 and Transformer-XL) LMs by observing whether they assign a higher probability to the acceptable sentence in each minimal pair. We find that state-of-the-art models identify morphological contrasts related to agreement reliably, but they struggle with some subtle semantic and syntactic phenomena, such as negative polarity items and extraction islands.

## introduction current neural networks for sentence processing rely on unsupervised pretraining tasks like language modeling. still, it is an open question how the linguistic knowledge of state-of-the-art language models (lms) varies across the linguistic phenomena of english. recent studies (e.g., @xcite @xcite have explored this question by evaluating lms' preferences between minimal pairs of sentences differing in grammatical acceptability, as in example 1. however, each of these studies uses a different set of metrics, and focuses on a small set of linguistic paradigms, severely limiting any possible bigpicture conclusions. we introduce the benchmark of linguistic minimal pairs (shortened to blimp), a linguistically motivated benchmark for assessing the sensitivity of lms to acceptability contrasts across a wide range of english phenomena, covering both previously studied and novel contrasts. blimp consists of 67 datasets automatically generated from linguist-crafted grammar templates, each containing 1,000 minimal pairs and organized by phenomenon into 12 categories. validation with crowdworkers shows that blimp faithfully represents human preferences. we use blimp to study several pretrained lms: transformer-based lms gpt-2 @xcite and transformer-xl @xcite , an lstm lm trained by @xcite , and an n-gram lm. we evaluate whether the lm assigns a higher probability to the acceptable sentence in each minimal pair to determine which grammatical distinctions lms are sensitive to. this gives us indirect evidence about each model's linguistic knowledge and allows us to compare models in a fine-grained way. we conclude that current neural lms appear to acquire robust knowledge of morphological agreement and some syntactic phenomena such as ellipsis and control/ raising. they show weaker evidence of knowledge about argument structure, negative polarity item licensing, and the semantic properties of quantifiers. all models perform at or near chance on extraction islands. overall, every model we we conclude that whereas models like gpt-2 appear to have significant linguistic knowledge, this knowledge is concentrated in some specific domains of english grammar. we use blimp to uncover several linguistic phenomena where even state-of-the-art language models clearly lack human-like knowledge, and to bring into focus those areas of grammar that future studies evaluating lms should investigate in greater depth. ## results and discussion an lm's overall accuracy on blimp is simply the proportion of the 67,000 minimal pairs in which the model assigns a higher probability to the acceptable sentence. we report the results for all models and human evaluation in the 5-gram model's poor performance-overall and on every individual category-indicates that blimp is likely not solvable from local co-occurrence statistics alone. because we evaluate pretrained models that differ in architecture and training data, we can only speculate about what drives these differences (though see § 6.3 for a controlled ablation study on the lstm lm). the results seem to indicate that access to training data is the main driver of performance on blimp for the neural models we evaluate. this may explain why transformer-xl and the lstm lm perform similarly in spite of differences in architecture, as both are trained on approximately 100m tokens of wikipedia text. relatedly, gpt-2's advantage may come from the fact that it is trained on roughly two orders of magnitude more data. possibly, lstms trained on larger datasets could perform comparably to gpt-2, but such experiments are impractical because of the inefficiency of training lstms at this scale. ## conclusion and future work we have shown ways in which blimp can be used as tool to gain evidence about both the overall and fine-grained linguistic knowledge of language models. like the glue benchmark @xcite , blimp assigns a single overall score to an lm that summarizes its general sensitivity to minimal pair contrasts. it also provides a breakdown of lm performance by linguistic phenomenon, which can be used to draw more concrete conclusions about the kinds of grammatical features learned acquired by a given model. this kind of information is a linguistically motivated evaluation of lms that can complement common metrics like perplexity. furthermore, the extent to which humans resemble data-driven learners like language models is debated in linguistics and cognitive science (see e.g., @xcite . in some domains, we may require the aid of innate knowledge to acquire phenomenon-specific knowledge resembling that tested in blimp. by evaluating whether selfsupervised learners like lms acquire human-like grammatical acuity in a particular domain, we gather indirect evidence as to whether this phenomenon is a necessary component of humans' innate knowledge. another aim of blimp is to serve as a guide for future work on the linguistic evaluation of lms. it is particularly interesting to better understand those empirical domains where current lms appear to acquire some relevant knowledge, but still fall short of human performance. the results from blimp suggest that-in addition to relatively well-studied phenomena like fillergap dependencies, npis, and binding-argument structure remains one area where there is much to uncover about what lms learn. more generally, as language modeling techniques continue to improve, it will be useful to have large-scale tools like blimp to efficiently track changes in what these models do and do not know about grammar.