The Acceptability Delta Criterion: Testing Knowledge of Language using the Gradience of Sentence Acceptability (2021)

Abstract: Any test that promises to assess Human Knowledge of Language (KoL) for any statistically-based Language Model (LM) must meet three requirements: (1) comprehensive coverage of linguistic phenomena; (2) replicable and statistically-vetted human judgement data; and (3) test the LM’s ability to track the gradience of sentence acceptability. To this end, we propose here the LI-Adger dataset: a comprehensive collection of 519 sentence types (4177 sentences) spanning the field of current generative linguistics, accompanied by attested and replicable human acceptability judgements (Sprouse & Almeida, 2012; Sprouse et al. 2013; Sprouse & Almeida, 2017). Finally, we posit the Acceptability Delta Criterion (ADC), an evaluation metric that tests how well a LM can track changes in human acceptability judgements across minimal pairs instead of testing whether the LM assigned a greater likelihood to the expert-labeled acceptable sequence of a minimal pair (S_1 > S_2). We benchmark six different BERT (Devlin et al. 2018) models and a baseline trigram model with the ADC. Although the best performing BERT model scores 94%, and the trigram scores 75% classification accuracy under the traditional metric, performance drops precipitously to 38% for BERT and 30% for the trigram model under the ADC. Adopting the ADC reveals how much harder it is for LMs to track the gradience of acceptability across minimal pairs. With this work, we propose and provide the three necessary requirements for a comprehensive linguistic analysis and test of the apparently Human KoL exhibited by LMs that we believe is currently missing in the field of Computational Linguistics.

## introduction assessing human knowledge of language (kol) in statistically-based language models (lms) generally involves assuming some fundamental prop-erty or computation occurring in the human language faculty and arguing that an often poorly understood, statistical, and typically connectionist model, also encodes that property or uses that computation. this quickly becomes a problematic task because understanding the human language faculty has been conventionally posed as a problem to be solved at a causal level removed from the algorithmic and computational implementation levels. put in more abstract terms, assessing the kol of a lm requires inferring some abstract operation inside a human black box based on input-output analysis and determining whether a second, statistical black box is somehow also performing the same operation by some other means. the issue is made even more challenging by changes in either field that consequently change our assumptions surrounding the human language faculty or the black boxes used in machine learning (ml). this, in turn, immediately impacts any claims relating the two by some abstract property, linguistic or otherwise, that is required for the evaluation of lms. if any concrete progress is to be made when it pertains to kol in lms, then the design of the tests we perform and their conclusions must be based on the same empirical data from current input-output analyses of the human language faculty that have subsequently been used to build the linguistic theories that attempt to characterize and explain human kol. we take concrete steps toward designing such a test of kol for lms by positing the necessary components required to build upon the same foundation of empirical data as in the field of generative linguistics. first, we posit a gold standard in empirical data: the li-adger dataset, a collection of linguistic phenomena representative of the field of linguistics, accompanied by human acceptability judgements in the form of magnitude estimation (me) data, all collected by @xcite @xcite . altogether, the dataset has an attested maximum false positive (type 1 error) rate between 1-12% and is well above the 80% threshold for statistical power (<20% false negatives, or type 2 errors) @xcite . the reliability of the li-adger dataset is such that, if the linguistic theories were somehow proven to be incorrect and reformulated, it would not be because of the data, but because of incorrect theorizing; any tractable theory of linguistics must account for the attested empirical phenomena observed in the li-adger dataset @xcite . to complement this data, we propose the acceptability delta criterion (adc), a proof of concept metric that enforces the gradience of acceptability in its evaluation of model performance, and adopts the attested continuous human judgements as the groundtruth labels that lms must approximate in order to demonstrate kol. finally, we use two distinct bidirectional encoder representations from transformers (bert; @xcite ) models: the outof-the-box bert trained using the masked language modeling (mlm) objective (bert mlm ), and bert cola , which is fine-tuned using the corpus of linguistic acceptability (cola; @xcite . as an additional baseline check, we include a trigram model trained using the british national corpus (bnc) to test whether the adc would constitute a meaningful advantage in statistical significance (fewer type 1 errors) over the traditional method of comparing probabilities over isolated, minimally-differing sequences of words. henceforth we refer to the traditional metric as the blimp criterion 1 @xcite , that will be contrasted against the new measure proposed here, the acceptability delta criterion (adc). under the blimp criterion, bert cola , correctly evaluates 2213 out of 2365 (∼94%) minimal pairs in the li-adger dataset; that is, for those 2213 minimal pairs, bert cola gives a higher score to the sentence in the minimal pair deemed by experts to be the acceptable one of the pair. to put the performance of bert cola into perspective, the trigram model's output, when normalized for sequence length using the syntactic log-odds ratio (slor; @xcite is able to correctly evaluate 1781 out of 2365 (∼75%) 1 although this has been the traditional metric to test kol in the literature @xcite @xcite ; among others), we name this metric with the same acronym as that of the benchmark for linguistic minimal pairs by @xcite mainly for conciseness and ease of use. minimal pairs. considering the coverage of phenomena in the li-adger dataset, we may interpret these results in one of two ways: either metrics such as the blimp criterion lead to tests with low levels of statistical significance (with a high rate of false positives), or a basic trigram model using slor encodes the kol necessary to account for 75% of the phenomena in generative linguistics. precisely this dilemma is what the adc is meant to solve. evaluating the models on how well they track the gradience of acceptability by adopting the adc at its strictest level (with δ = 0.5, where δ indicates the number of standard deviation units a model's scores are allowed to deviate from the corresponding human judgements) leads to a notable drop in performance, particularly for the trigram model. bert cola only correctly scores 726 out of 2365 (∼31%) minimal pairs, whereas the trigram model with slor correctly scores 712 out of 2365 (∼30%). the adc is therefore a much harder task, because the difference in likelihood acceptability score between the two sentences in each minimal pair is what is being evaluated. ## related work the success of neural language models at different natural language tasks, such as next sentence prediction (nsp), machine translation (mt) and question answering (qa), among others foot_0 , has made it a popular endeavor to assess the potential kol encoded in the learned representations of the language models and how that kol may be contributing to their performance. human kol, due to its abstract, deliberately acomputational nature, can only be assessed via proxies, generally by probing language acquisition or use. at present, the studies of lms' kol that rely on an input-output analysis of the system tend to focus on probing their weak generative capacity: i.e, testing whether a given lm can discern whether a particular sequence of words is or is not in the set of sentences generated by some presumed corresponding grammar, typically by comparing the probabilities the lm assigns to different but related sequences of words. contrast this with testing the lm's strong generative capacity by evaluating whether it has assigned the correct syntactic structure, or series of candidate syntactic structures in the case of ambiguous or homonymous sequences, to a sequence of words @xcite . @xcite have taken seminal steps toward evaluating lms beyond their weak generative capacity by positing the benchmark of linguistic minimal pairs for english (blimp). they automatically generated 67 datasets of 1000 minimal pairs each from grammar templates that span 12 linguistic phenomena. they designed the templates to contrast in grammatical acceptability by isolating specific phenomena in syntax, morphology or semantics. in doing so, the authors intend to mirror what a working linguist uses to probe kol in native speakers of a language. because such principles generally appeal to grammatical constraints, they go beyond simple weak generative capacity. while the tradition of using minimal pairs in linguistics dates back nearly 100 years (e.g. bloomfield 1933, among many others), the concept of using minimal pairs to evaluate nn models is not entirely new either @xcite @xcite ; to name a few). however, the creators of blimp take the idea to a much larger scale and propose a single metric for evaluation, which we have named the blimp criterion out of convenience. for a given minimal pair m i consisting of an acceptable sentence s i,1 and an unacceptable sentence s i,2 , if a lm evalutes p (s i,1 ) > p (s i,2 ), then the lm has met the blimp criterion for m i . the authors of blimp thus score a lm on the blimp dataset according to the percentage of all the minimal pairs for which it was able to fulfill the blimp criterion. this, of course, can be broken down into further analyses of the 12 linguistic phenomena they sought to represent in the dataset. because the blimp dataset relies on templates, the choice and design of the linguistic phenomena represented in the data is inherently limited, and therefore unrepresentative of syntax or linguistics more broadly. secondly, the templated approach produces semantically implausible sentences, such as sam ran around some glaciers as noted by @xcite in their paper. @xcite have shown that semantic implausibility is a very strong confounding factor when eliciting human acceptability judgements, even in a forced choice (fc) task such as the one used to collect the human judgement data in blimp. finally, the blimp criterion treats sentence acceptability as a functionally categorical phenomenon, which we assert to be gradient by nature. our proposed test of human kol in lms takes this minimal pair approach to the next level by addressing the three gaps we have outlined above. we (1) expand linguistic coverage to be representative of the field with the li-adger dataset, (2) further control for confounding factors such as semantic implausibility when eliciting human acceptability judgements by using the examples constructed by hand by @xcite @xcite (3) enforce the gradient nature of sentence acceptability judgements with the adc. ## why bert is the benchmark. we chose bert as the model to test due to the growing body of research attributing ever greater human kol to bert. @xcite have shown high matthews correlation coefficient (mcc) scores between the expert acceptability labels for the sentences in the corpus of linguistic acceptability (cola; @xcite and bert cola models' predictions. they have gone on to show with a grammatically annotated cola analysis set that bert cola models exhibit very strong positive mcc scores on multiple syntactic features. for example, they claim bert exhibits strong knowledge of complex or noncanonical argument structures such as ditransitives and passives, and has a distinct advantage over baseline performance on sentences with long-distance dependencies such as questions. finally, @xcite used the raw pseudo-log-likelihood (pll; @xcite @xcite scores from the out-of-the-box bert mlm to evaluate its kol using the blimp benchmark and found it to correctly predict 84.8% of the minimal pairs in blimp, thereby beating gpt-2 by 4.2% and almost reaching the human baseline at 88.6%. we take the information provided here and the overarching body of research surrounding bert foot_1 as the baseline level of performance to be expected from the model. we hypothesize that bert is capable of exhibiting reasonable levels of gradience when it calculates sequence likelihoods or acceptability scores across minimal pairs when compared to the human me data. ## the li-adger dataset is the test set. the li-adger dataset is a collection of two separate datasets. the first consists of a randomly selected sample of 150 pairwise phenomena (300 sentence types) from linguistic inquiry (li) 2001-2010 collected by @xcite . each pairwise phenomenon includes 8 hand-constructed, semantically plausible and lexically matched minimal pairs such that most of the contribution of lexical information to the acceptability of the sentences would be distributed equally to the pair. the second set of sentences is an exhaustive selection of 219 sentence types from @xcite 's core syntax textbook (198 directly from the textbook + 21 as additional controls) that form 105 multi-condition phenomena collected by @xcite . much like the li dataset, 8 tokens of each sentence type were created by hand by the original authors such that the structural properties of the condition were maintained but the lexical items varied. for the purposes of the li-adger dataset as a whole, we have split each multi-condition phenomenon into minimal pairs by taking each possible combination of acceptable and unacceptable sentences in the condition as a valid minimal pair, when properly lexically matched. we include an explicit example of the arrangement with a multicondition phenomenon from chapter 8 (functional categories iii) of the core syntax textbook in appendix a. the adger dataset, in virtue of being sampled from the core syntax textbook, that constructs a theory of syntax from the ground up on the basis of key examples, can be taken to have a reasonably good coverage of the field of syntax. we augment this with the li dataset, sampled from the 111/114 articles published in linguistic inquiry about us english syntax from 2001-2010 (out of the total 308 articles published during that time). therefore, to the extent that the adger core syntax textbook and li2001-2010 are representative of the data in the field, so is the li-adger dataset. @xcite . just as important as the coverage of linguistic phenomena represented in the li-adger dataset is the foundation of human judgement data built upon it. @xcite collected magni-tude estimation (me) and yes-no (yn) judgement data from a total of 440 native participants for the 469 data points they sampled from the adger core syntax textbook. after conducting three different statistical analyses on the data (standard frequentist tests, linear mixed-effects models, and bayes factor analyses), they found that the maximum replication failure rate between formal and informal judgements (i.e. formal vs. informal data collection methods) was 2 percent @xcite . @xcite took those analyses even further with their sample of 148 pairwise phenomena from li2001-2010. they collected data for the li sentences using the 7-point likert scale (ls) task, me and forced choice (fc) and vetted this dataset via 5 different statistical analyses (descriptive directionality, one-and two-tailed null hypothesis tests, bayes factor analysis and linear mixed effect models). they estimated a minimum replication rate for journal data of 95 @xmath0, sprouse and almeida (2017) sampled 50 pairwise phenomena from the li dataset in a complementary study that determined the statistical power of formal linguistics experiments by task and average effect size and recommend setting the threshold for well-powered experiments at 80% statistical power. they find that the fc task would reach the 80% power threshold and detect 70% of the phenomena published in li2001-2010 with just ten participants, assuming each provides only one judgement per phenomenon. with fifteen participants, fc would detect 80% of the phenomena. because the me task has less statistical power than fc, it requires at least thirty to thirty-five participants to reach the same 80% coverage of li2001-2010 as fc @xcite . because the sample sizes for the li-adger datsets are much larger (104 participants per condition for the li sentences and 40 for the adger sentences), we do not forfeit any statistical power by using me data in spite of the higher statistical power of the fc task. on the contrary, the me task will allow us not only to perform the same type of functionally categorical acceptability comparison as the blimp criterion, but also allow us to make comparisons between every condition in the dataset. taken together, the human me data that accompany the li-adger dataset are therefore reliable, replicable and statistically powerful. this collection of empirical data has the added benefit of being theory-agnostic; if linguistic theories were to fundamentally change in the future, the significance and validity of the data would remain unchanged because this statistically vetted empirical evidence would still remain to be accounted for. ## the acceptability delta criterion (adc) is the loss function. thanks to the me data associated with each sentence in the li-adger dataset, we can now make direct acceptability comparisons, not just between the two sentences of a minimal pair, but also across minimal pairs, and even across phenomena. the reason why is precisely the gradient nature of acceptability: violations of certain linguistic phenomena elicit much stronger responses from the participants than others, as observed in the differences in me scores across minimal pairs and across linguistic phenomena in the dataset. we include an example of such a difference in appendix b. acceptability judgement experiments carry as a necessary underlying assumption that acceptability is a percept that arises in response to linguistic stimuli. collecting data about the percept requires then that the subject report that perception of acceptability @xcite @xcite . consequently, acceptability judgements are a behavioral response that may vary in intensity, much like brightness, loudness, temperature, pain, etc. the degree of this response is inherently informative, in particular because acceptability is the behavioral output of the grammatical system, to which neither speakers nor linguists have direct access. we include one example of this variability in intensity and how it is reflected in the human me judgements in appendix b. to this end, we propose the acceptability delta criterion (adc). it is founded on the principle that, if we are to ascribe any inferred knowledge of one black box (the human language faculty) to another black box (neural language models) based solely on an input-output analysis of both systems, then the response of both systems must agree both categorically and in magnitude. in other words, for a minimal pair whose change in human acceptability rating is nearly night and day, a language model with comparable kol must output a similarly drastic change in acceptability rating across the same minimal pair. to make this example more concrete: suppose we have a language model l with output function f that takes in a sequence of words x i and outputs a score y i . the first step in computing the adc is to understand the range of values output by the language model l over the 4179 li-adger sentences: @xmath1, y 2 , ..., y 4179 ]. with the full range of values, we apply a z-score transformation to each of the values in y by subtracting the mean of y from each of the values and then dividing them by the standard deviation of y . this will yield the set of z-score transformed predictions @xmath2, z 2 , ..., z 4179 ]. notice that because this is a purely linear transformation, it preserves the relationships between the data points. in addition, the resulting set of predictions z represents a standardized form of y , where each prediction z i is expressed in standard deviation units of y i from the mean of y (schütze and @xcite . now that we have grounds for making the comparison foot_2 and a value for how acceptable the model l finds a sequence of words x i in terms of standard deviation units z i , we can begin to compare the degree of this acceptability response to the human judgement data, also expressed in standard deviation units. for a given minimal pair m i consisting of an acceptable sentence s i,1 and an unacceptable sentence s i,2 , we will have 4 pieces of information: two human z-score transformed acceptability judgements h i,1 and h i,2 , and two language model scores z i,1 and z i,2 . we turn these into two concrete points of comparison: a human acceptability delta ∆h @xmath3,1 -h i,2 and a language model acceptability delta ∆lm @xmath4,1 -z i,2 . in this new formulation, no information has been lost. recall that the blimp criterion is met for the minimal pair m i when the language model scores the acceptable sentence higher than the unacceptable one, i.e. ∆lm @xmath5. with the fully defined delta values as well as a reformulated blimp criterion in terms of the delta values, we may at last proceed to define the adc. let δ be a scalar value indicating the number of maximum allowed units of deviation between the human judgement delta ∆h i and the language model delta ∆lm i . using this δ value, we consider the adc to be met for the minimal pair m i when the following two conditions are met: the δ parameter in equation 2 can be adjusted to allow for larger or smaller amounts of deviation between the human and lm acceptability deltas. if δ is set to a large number, the adc functionally becomes the blimp criterion because it is dominated by equation 1. the main difference would be that, instead of comparing the expert labels to the lm's output, the human judgements would become the ground truth. for example, if δ is set to a very large number, and the human me data find the expert-labeled unacceptable sentence as more acceptable than the expert-labeled acceptable counterpart, then the lm is expected to follow the same monotonicity. we include a concrete example of the adc in action in appendix c. because the δ parameter determines how closely the lm's acceptability deltas must track the human judgement deltas, we adopt a first approximation of δ = 0.5, which we believe to be the proper value of δ in order to test for the gradience of acceptability. additionally, we adopt a value of δ = 1 in order to show how performance under the adc changes as the δ parameter is relaxed. we include results for both values of δ in the case study to be presented here. ## the kol experiment brings everything together. with the li-adger dataset as the test set, the human me data as the ground truth labels and the adc as the loss function, this kol experiment now only requires the model to test. in order to ensure we observe the best performance by bert in our benchmarks, we adopt two parallel approaches to the tests: a syntactic probing approach and a mlm approach with a variant of the cloze task @xcite . for the probing approach, we train a linear softmax classifier (probe) on top of bert's hidden layer outputs using the corpus of linguistic acceptability (cola; @xcite . we use the huggingface transformers library @xcite to fine-tune three pre-trained versions of bert in order to be comprehensive in our coverage: 10 random seeds of bert base-uncased , 20 random seeds of bert large-uncased , and 20 random seeds of bert large-cased . for further control we attempt to replicate the mean mcc scores per linguistic phenomenon obtained by @xcite on their later published cola analysis set. for the purposes of transparency and the replicability of our own experiment, we include the results of our models' performance on the cola analysis set in appendix d. for each of the three bert cola we selected the random restart that yielded the highest mcc score on the cola test set as the model to test using the adc. finally, we need to adapt the categorical output of the fine-tuned bert models (c j where @xmath6, 0}) to a more gradient form in order to ensure the models have the best chance of tracking sentences through the acceptability gradient. we rely on @xcite 's formulation of the probabilities 5 of the categorical labels as the softmax of bert's final hidden layer output h i : therefore, the output of the fine-tuned bert models can be defined as: out of convenience, we will switch the categorical labels from {0, 1} to {-1, 1}. this allows us to multiply the model's confidence in a particular label (p (c j |h i )) by the label itself, written explicitly below: with this formulation, we can easily retrieve both the predicted categorical label (sign(out i )) and the model's confidence in that label (|out i |). reformulations aside, because probing relies on training an additional classifier on top of the latent, albeit poorly understood, representations of neural lms, it is extremely difficult to control for confounding variables, such as the information being introduced into the system by training the probing classifier in the first place @xcite . additionally, d'amour et al. ( 2020 ) have found substantial evidence indicating that these overparametrized neural lms by nature exploit different sets of spurious correlations according to their random initialization in spite of exhibiting very similar performance on i.i.d. test sets. this poses a unique set of difficulties for the use of probes for any assessment of kol in such lms. we mitigate these concerns with the second, parallel approach to our experiment: we use the publicly available model checkpoints for bert base-uncased , bert large-uncased , and bert large-cased , which were originally trained using the masked language modeling (mlm) and next sentence prediction (nsp) objectives @xcite . because mlm is one of the tasks used to pre-train bert in the first place, we use it to test the models in their out-of-the-box state. by masking each token in a sentence s i sequentially and recovering the log likelihood of the original token, we are able to calculate a pseudo-log-likelihood (pll) score for the sentence. @xcite have shown that bert's pll scores are able to outperform gpt-2 on the blimp dataset under the traditional metric, as well as on other natural language benchmarks, potentially due to bert's ability to better leverage the left and right context of each masked token when calculating its likelihood. this altogether strongly favors pll scores as the ideal metric to test the out-of-the-box bert models with the adc. now we have 6 distinct bert models to test: the three best performing random restarts of bert base-uncased , bert large-uncased , and bert large-cased when fine-tuned using cola (collectively bert cola ), and the three unadulterated, publicly available model checkpoints for which we will calculate pll scores (collectively bert mlm ) foot_3 . to this assortment, we add the tri-gram model trained by @xcite using the british national corpus (bnc). we will use both its raw negative log-likelihood scores and syntactic log-odds ration (slor; @xcite scores. finally, we test four metrics: the original blimp criterion, and the adc with δ values of 0.5, 1.0 and 5.0. the goal with the last δ = 5.0 is to observe how model performance approaches that of the blimp criterion when the adc is dominated by equation 1. the results of these tests are presented in ## results we summarize the results of our proposed kol benchmark with the bert and trigram models in to further compare how each of the models' acceptability deltas (∆lm i ) behave with respect to the ground truth (∆h i ), we plot in figure 1 a heatmap of the pearson's correlation coefficient (pcc) matrix for all 8 models and the human judgements. the models that were best able to track the human judgement deltas through the full spectrum of acceptability were the bert mlm mod-bert large-cased , such as gpt-2 or gpt-3 to the case study presented here. ## discussion the first column of however, the adc does raise the concern of whether we are fundamentally changing the results by adopting human acceptability judgements as ground-truth instead of the expert labels. we evaluated all 8 models with ever greater δ values in order to observe how much this should concern us, and as observed in the rightmost column of ## contributions with this work, we have identified three quintessential requirements to a test of human knowledge of language in statistically-based language models: it must (1) exhibit comprehensive coverage of linguistic phenomena, (2) support attested and replicable human judgement data, and (3) test lms' ability to track different linguistic phenomena across the full range of the acceptability gradient. additionally, we advance a test of kol for lms that meets all three requirements. first, we present the li-adger dataset as a gold standard test dataset: a comprehensive, empirically attested collection of 519 pairwise and multicondition phenomena collected by @xcite from linguistic inquiry (li) 2001-2010, and by sprouse and almeida (2012) from @xcite core syntax textbook. to complement the test dataset, we present statistically powerful, replicable and validated human magnitude estimation (me) data collected by @xcite and @xcite as the ground truth labels we expect lms with human kol to approx-imate. finally, in order to tie the li-adger dataset and the human me data together, we present the acceptability delta criterion (adc), a metric that tests lms for human kol by requiring lms to track the validated human judgements through the gradient spectrum within a specified margin (δ) as the acceptability values change across minimal pairs. our three main contributions with this work when taken together create a comprehensive and powerful input-output analysis of human kol for lms. with further ongoing work, the test will empower us to see a fine-grained analysis of which phenomena a lm is able to account for in its output from how well it predicts the acceptability deltas around them. it is our hope that researchers will rely on the li-adger dataset for its coverage of empirically attested linguistic phenomena, embrace the paradigm of human judgements as the groundtruth labels that lms are expected to approximate, and, beyond that, adopt the adc as we take our understanding of kol in lms to the next level. alex warstadt, alicia parrish, haokun liu, anhad mohananey, wei peng, sheng-fu wang, and samuel r. bowman. 2020. blimp: the benchmark of linguistic minimal pairs for english. transactions of the association for computational linguistics, 8:377-392. alex warstadt, amanpreet singh, and samuel r bowman. 2019. neural network acceptability judgments. transactions of the association for computational linguistics, 7:625-641. ethan wilcox, roger levy, takashi morita, and richard futrell. 2018. what do rnn language models learn about filler-gap dependencies? arxiv preprint arxiv:1809.00042. thomas wolf, lysandre debut, victor sanh, julien chaumond, clement delangue, anthony moi, pierric cistac, tim rault, remi louf, morgan funtowicz, joe davison, sam shleifer, patrick von platen, clara ma, yacine jernite, julien plu, canwen xu, teven le scao, sylvain gugger, mariama drame, quentin lhoest, and alexander rush. 2020. transformers: state-of-the-art natural language processing. in proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38-45, online. association for computational linguistics.