Grammaticality and Language Modelling (2020)

Abstract: Ever since Pereira (2000) provided evidence against Chomsky’s (1957) conjecture that statistical language modelling is incommensurable with the aims of grammaticality prediction as a research enterprise, a new area of research has emerged that regards statistical language models as “psycholinguistic subjects” and probes their ability to acquire syntactic knowledge. The advent of The Corpus of Linguistic Acceptability (CoLA) (Warstadt et al., 2019) has earned a spot on the leaderboard for acceptability judgements, and the polemic between Lau et al. (2017) and Sprouse et al. (2018) has raised fundamental questions about the nature of grammaticality and how acceptability judgements should be elicited. All the while, we are told that neural language models continue to improve. That is not an easy claim to test at present, however, because there is almost no agreement on how to measure their improvement when it comes to grammaticality and acceptability judgements. The GLUE leaderboard bundles CoLA together with a Matthews correlation coefficient (MCC), although probably because CoLA’s seminal publication was using it to compute inter-rater reliabilities. Researchers working in this area have used other accuracy and correlation scores, often driven by a need to reconcile and compare various discrete and continuous variables with each other. The score that we will advocate for in this paper, the point biserial correlation, in fact compares a discrete variable (for us, acceptability judgements) to a continuous variable (for us, neural language model probabilities). The only previous work in this area to choose the PBC that we are aware of is Sprouse et al. (2018a), and that paper actually applied it backwards (with some justification) so that the language model probability was treated as the discrete binary variable by setting a threshold. With the PBC in mind, we will first reappraise some recent work in syntactically targeted linguistic evaluations (Hu et al., 2020), arguing that while their experimental design sets a new high watermark for this topic, their results may not prove what they have claimed. We then turn to the task-independent assessment of language models as grammaticality classifiers. Prior to the introduction of the GLUE leaderboard, the vast majority of this assessment was essentially anecdotal, and we find the use of the MCC in this regard to be problematic. We conduct several studies with PBCs to compare several popular language models. We also study the effects of several variables such as normalization and data homogeneity on PBC.

## background the three currently most popular means of evaluating a neural language model are: (1) perplexity, an information-theoretic measure that was in use long before neural networks became the preferred means of implementing language models; (2) task performance profiles, in which derivative aspects of a language model's predictions are embedded in a so-called "downstream" task, with all other aspects of the implementation held constant; and the purpose of this paper is ultimately to find a task-independent means of testing how well language model probabilities might serve as grammaticality regression scores. using evidence from targeted linguistic evaluations, we argue for the point-biserial correlation as at least the basis of such a task-independent measure, and then use the pbc to examine several neural models along with some important variables that affect both their evaluation and the data that we evaluate on. borrowing a convention from linguistic theory, @xcite coined the use of "minimal pairs" as input to language models in order to test these fine-grained variations. for example: a. the manager that the architects like doubted himself. b. *the manager that the architects like doubted themselves. these pairs deal with referential agreement in specific syntactic environments. if a model assigns the grammatical string in a pair a higher score than the ungrammatical string, then we say that the model made the correct prediction on that pair. having evaluated the model over a large number of these pairs, we can compute an accuracy score, relative to a 50% random baseline. @xcite have taken exception to the design of many such evaluations on that grounds that: (1) a number of english nouns are stereotypically gendered, which conditions pronoun choice, and (2) the unigram probabilities of reflexive pronouns are different, which biases the probabilities that models assign to sentences that contain them. to circumvent these shortcomings, they generalized the pairs to larger sets of strings in which multiple nouns were used in multiple positions so that lexical choice and order could be permuted across sets. they also introduced distractors, grammatical strings that contain material irrelevant, or distracting, to the determination of the sentence's grammaticality. one set that they use, for example, is: hu et al. (2020) also evaluate models with accuracies. because there are three classes of string, rather than two, a model is said to have made the correct prediction if the ungrammatical data receive a lower score than both the baseline and distractor data. but because there are more than three strings, they do not compare individual scores from the candidate model, but rather the three means that result from averaging the conditional pronoun probabilities of the baseline, distractor and ungrammatical strings, respectively. this alternative design not only provided better accuracies than were achieved by @xcite , but the inclusion of distractors in the design lowers the random baseline from 50% to 33.3% accuracy. @xcite conclude that current neural language models are learning more about the licensing of reflexive anaphora than was previously thought. ## theoretical exceptions in a typical psycholinguistics experiment, we would give human subjects a task to perform during which they would be presented with a stimulus that was labelled as either baseline, distractor or ungrammatical. the effect of the stimulus on the task could be measured by time to completion, the number of correct tokens retrieved during a fixed interval of time, etc. regardless, the task would almost certainly be chosen so that samples of its corresponding measure of success would be normally distributed. so a within-subjects mean of these quantities is entirely justifiable. the situation is somewhat less clear with the scores that are returned by a neural language model. ignoring for the moment that @xcite are interested in conditional pronoun probabilities and not sentence probabilities, the scores are generally not regarded as measures of success on a task per se -there is no actual task here, apart from achieving a high rank in the evaluation. legitimate task performance profiles are defined over separate downstream tasks, such as those in the glue leaderboard @xcite . it is rather more difficult to think of downstream tasks that depend on conditional pronoun probabilities, however. note that for @xcite , the ratio of conditional pronoun probabilities of a set of stimuli was the same as the ratio of their total sentence probabilities because the reflexive pronoun is always the last word of the sentence, and the contexts preceding the pronouns are always identical. several papers by @xcite , have argued instead that sentence probabilities can justifiably be interpreted as gradient grammaticality scores, rejecting the longstanding assumption in generative linguistics that grammaticality is a binary judgement. it is also possible to regard sentence probabilities as summaries of group behaviour, such as relative frequencies of binary grammaticality judgements across multiple individual participants, with no claim of gradience implied for any single participant. this in turn raises the very old question of whether neural networks in fact have any cognitive plausibility, which has recently started to be debated again @xcite . sample distributions of means converge to a normal distribution even if the underlying population distribution is not normal itself, and so whether an average would be justified in this group interpretation would depend to a great extent on the sizes of the sets of strings (relatively small, as we have seen) as well as how skewed the underlying distribution was. ## aggregated point biserial correlations our proposed alternative involves two changes. first, we propose using a point biserial correlation between the output probability of a language model and binary grammaticality judgements. second, we propose calculating pbcs not on a set-by-set basis, but for all probabilities generated by a fixed model using all of the contexts of a fixed experiment. to consider figure 1 again, the model on the left has a pbc of 0.25, whereas the model on the right has one of 0.73. correlations such as the pbc range between -1 and 1, where 1 is perfect correlation, 0 is no correlation, and -1 is perfect anti-correlation. our choice of pbc is perhaps the less controversial of these two changes, as our motivation for doing so is mainly due to the fact that it is the standard measure for correlating a continuous or interval random variable with a discrete random variable. our decision to "aggregate" data by ignoring the boundaries between the controlled, minimal sets that have become so widely accepted as a part of targeted syntactic evaluations is perhaps counterintuitive. but as long as the necessary distractors, permutations and lexical alternations that avoid bias appear somewhere in the context of the experiment, they will be compared to each other, although along with additional comparisons that were not made when accuracy was averaged over sets. those additional comparisons, however, will merely corroborate the model's (in)ability to more robustly distinguish between well-formed and non-well-formed strings, and the experiment itself does restrict the variability of those comparisons to a great extent. in our experience, aggregating makes the evaluation more resilient to choices of normalizers such as slor @xcite , its results are in closer accord to our intuitive judgements, and, as expected, it handles sample bias better. both accuracy (30-100%) and aggregate pbc (-0.01-0.81) vary widely from experiment to experiment in @xcite data, and yet the average of per-set pbcs tends to be less dispersed. the experiments in figure 1 , for example, have microaveraged pbcs of 0.77 (left) and 0.89 (right). it could therefore be argued that the effect size of the dependent variable that @xcite were attempting to measure is not as large as the choice of minimal set. aggregation would then also be an effective means of utilizing the available range of correlation values. total (weighted) accuracy and baseline-plusdistractor vs. ungrammatical pbc have a spearman's correlation of 0.876 (@xmath0.5×10 -25 ) across @xcite experiments and models. ## task-independent grammaticality classification the famous "colorless green ideas sleep furiously" (cgisf) example @xcite posited a seemingly irreconcilable divide between formal linguistics and statistical language modelling, arguing that every sequence of words not attested in the collective memory of a language's use would be considered equally "remote" by a putative instance of the latter, regardless of whether the sequence was grammatical (cgisf) or ungrammatical. the example was presented briefly and informally in order to reject statistical language modelling as an alternative approach to the one advocated and developed in greater detail by @xcite . it was only presented with one other example, the reverse of the sentence, i.e., "furiously sleep ideas green colorless", in order to draw a contrast between two nonsensical sequences, only one of which (cgisf) is grammatical. @xcite provides an attempt at a refutation by constructing a statistical language model based upon an agglomerative markov process @xcite , and then observing that cgisf is assigned a probability by the model which is roughly 200 000 times greater than the probability assigned to its reversal. there has nevertheless been some scepticism expressed about the ensuing euphoria among computer scientists -mainly by linguists. @xcite notes that the trigram model from @xcite assigns different rankings to 10 different permutations of cgisf, depending on the training corpus (e.g., the wall street journal corpus versus an example training corpus taken from lau et al. ( 2015) ). can the scores assigned to these sequences be reliably construed as a regression scale of grammaticality (or perhaps acceptability), if they are so fickle? chowdhury and zamparelli (2018) also express concern about the ability of neural language models to generalize to more abstract grammatical phenomena than subject-verb agreement. what we will present in this section is a more thorough appraisal of how well statistical language models perform as instruments of grammaticality testing overall, using pbcs. previous research on grammaticality/acceptability and language models has mainly designed experiments using naturally occurring english sentences, and modifies those sentences based on various individual linguistic phenomena to manually introduce a specific source of ungrammaticality into the sentences. notable exceptions include cola as well as the linguistic inquiry (li) corpus of grammatical and ungrammatical sentences collected by @xcite and @xcite , and used in @xcite . both are based upon examples found in linguistics publications. @xcite @xcite create ungrammatical sentences by roundtrip translating natural english sentences. we will use both cola and the li corpus. ## experiment results cola point-biserial correlation test