UnNatural L anguage I nference (2021)

Abstract: Recent investigations into the inner-workings of state-of-the-art large-scale pre-trained Transformer-based Natural Language Understanding (NLU) models indicate that they appear to understand human-like syntax, at least to some extent. We provide novel evidence that complicates this claim: we find that state-of-the-art Natural Language Inference (NLI) models assign the same labels to permuted examples as they do to the original, i.e. they are invariant to random word-order permutations. This behavior notably differs from that of humans; we struggle to understand the meaning of ungrammatical sentences. To measure the severity of this issue, we propose a suite of metrics and investigate which properties of particular permutations lead models to be word order invariant. For example, in MNLI dataset we find almost all (98.7%) examples contain at least one permutation which elicits the gold label. Models are even able to assign gold labels to permutations that they originally failed to predict correctly. We provide a comprehensive empirical evaluation of this phenomenon, and further show that this issue exists in pre-Transformer RNN / ConvNet based encoders, as well as across multiple languages (English and Chinese). Our code and data are available at https://github.com/facebookresearch/unlu .

## introduction of late, large scale pre-trained transformer-based @xcite models-such as roberta @xcite , bart @xcite , and gpt-2 and -3 @xcite -have exceeded recurrent neural networks' performance on many nlu tasks @xcite . several papers have even suggested that transformers pretrained on a language modeling (lm) objective can capture syntactic informa- ## related work researchers in nlp have realized the importance of syntactic structure in neural networks going back to @xcite ). an early hand annotation effort on pascal rte @xcite suggested that "syntactic information alone was sufficient to make a judgment" for roughly one third of examples @xcite . anecdotally, large generative language models like gpt-2 or -3 exhibit a seemingly humanlike ability to generate fluent and grammatical text @xcite . however, the jury is still out as to whether transformers genuinely acquire syntax. models appear to have acquired syntax. when researchers have peeked inside transformer lm's pretrained representations, familiar syntactic structure @xcite @xcite @xcite , or a familiar order of linguistic operations @xcite , has appeared. there is also evidence, notably from agreement attraction phenomena @xcite that transformerbased models pretrained on lm do acquire some knowledge of natural language syntax @xcite @xcite @xcite @xcite . results from other phenomena @xcite ) such as npi licensing @xcite ) lend additional support. the claim that lms acquire some syntactic knowledge has been made not only for transformers, but also for convolutional neural nets @xcite , and rnns @xcite @xcite @xcite -although there are many caveats (e.g., @xcite @xcite @xcite . models appear to struggle with syntax. several works have cast doubt on the extent to which nli models in particular know syntax (although each work adopts a slightly different idea of what "knowing syntax" entails). for example, @xcite argued that the knowledge acquired by models trained on nli (for at least some popular datasets) is actually not as syntactically sophisticated as it might have initially seemed; some transformer models rely mainly on simpler, nonhumanlike heuristics. in general, transformer lm performance has been found to be patchy and variable across linguistic phenomena @xcite @xcite @xcite . this is especially true for syntactic phenomena @xcite @xcite @xcite , where transformers are, for some phenomena and settings, worse than rnns @xcite . from another angle, many have explored architectural approaches for increasing a network's sensitivity to syntactic structure @xcite . @xcite showed that learning jointly to perform nli and to parse resulted in parse trees that match no popular syntactic formalisms. furthermore, models trained explicitly to differentiate acceptable sentences from unacceptable ones (i.e., one of the most common syntactic tests used by linguists) have, to date, come nowhere near human performance @xcite . insensitivity to perturbation. most relatedly, several concurrent works @xcite @xcite @xcite investigated the effect of word order permutations on transformer nns. pham et al. ( 2020 ) is very nearly a proper subset of our work except for investigating additional tasks (i.e. from the glue benchmark of @xcite ) and performing a by-layer-analysis. @xcite also relies on the glue benchmark, but additionally investigates other types of "destructive" perturbations. our contribution differs from these works in that we additionally include the following: we (i) outline theoretically-informed predictions for how models should be expected to react to permuted input (we outline a few options), (ii) show that permuting can "flip" an incorrect prediction to a correct one, (iii) show that the problem isn't specific to transformers, (iv) show that the problem persists on out of domain data, (v) offer a suite of flexible metrics, and (vi) analyze why models might be accepting permutations (bleu and pos-tag neighborhood analysis). finally, we replicate our findings in another language. while our work @xcite only permutes data during fine-tuning and/or evaluation, recently sinha et al. explored the sensitivity during pre-training, and found that models trained on n-gram permuted sentences perform remarkably close to regular mlm pre-training. in the context of generation, @xcite crafted linguistically relevant perturbations (on the basis of part-of-speech tagging and dependency parsing) to evaluate whether permutation hinders automatic machine translation models. relatedly, but not for translation, @xcite investigated a smaller inventory of perturbations with emphasis on phrasal boundaries and the effects of n-gram perturbations on different layers in the network. nli models are very sensitive to words. nli models often over-attend to particular words to predict the correct answer @xcite . @xcite show that some short sequences of non-human-readable text can fool many nlu models, including nli models trained on snli, into predicting a specific label. in fact, @xcite observed that for one of three test sets, bert loses some accuracy in wordperturbed sentences, but that there exists a subset of examples for which bert's accuracy remains intact. if performance isn't affected (or if permutation helps, as we find it does in some cases), it suggests that these state-of-the-art models actually perform somewhat similarly to bag-of-words models @xcite . ## our approach as we mentioned, linguists generally take syntactic structure to be necessary for humans to know what sentences mean. many also find the nli task to a very promising approximation of human natural language understanding, in part because it is rooted in the tradition of logical entailment. in the spirit of propositional logic, sentence meaning is taken to be truth-conditional @xcite @xcite . that is to say that understanding a sentence is equivalent to knowing the actual conditions of the world under which the sentences would be (judged) true @xcite . if grammatical sentences are required for sentential inference, as per a truth conditional approach @xcite , then permuted sentences should be meaningless. put another way, the meanings of highly permuted sentences (if they exist) are not propositions, and thus those sentences don't have truth conditions. only from their truth conditions of sentences can we tell if a sentence entails another. in short, the textual entailment task is technically undefined in our "unnatural" setting. since existing definitions don't immediately extend to unnatural language inference (unli), we outline several hypothetical systematic ways that a model might perform, had it been sensitive to word order. we hypothesize two models that operate on the first principles of nli, and one that doesn't. in the first case, model a deems permuted sentences meaningless (devoid of truth values), as formal semantic theories of human language would predict. thus, it assigns "neutral" to every permuted example. next, model b does not deem permuted sentences meaningless, and attempts to understand them. humans find understanding permuted sentences difficult (see our human evaluations in §7). model b could also similarly struggle to decipher the meaning, and just equally sample labels for each example (i.e., assigns equal probability mass to the outcome of each label). finally, we hypothesize a non-systematic model, model c, which attempts to treat permuted sentences as though they weren't permuted at all. this model could operate similarly as bag-of-words (bow), and thus always assign the same label to the permuted examples as it would to the un-permuted examples. if the model failed to assign the original gold label to the original unpermuted examples, it will also fail to assign the original gold label to its permutations; it will never get higher accuracy on permuted examples than on unpermuted ones. we find in our experiments that the state-of-theart transformer-based nli models (as well as pre-transformer class of models) do not perform like any of the above hypothetical models. they perform closest to model c, but are, in some cases, actually able to achieve higher accuracy on permuted examples. to better quantitatively describe this behaviour, we introduce our suite of permutation acceptance metrics that enable us to quantify how accepting models are of permuted sentences. ## methods constructing the permuted dataset. for a given dataset d having splits d train and d test , we first train an nli model m on d train to achieve comparable accuracy to what was reported in the original papers. we then construct a randomized version of d test , which we term as dtest such that: for each example (p i , h i , y i ) ∈ d test (where p i and h i are the premise and hypothesis sentences of the example respectively and y i is the gold label), we use a permutation operator f that returns a list ( pi , ĥi ) of q permuted sentences (p i and ĥi ), where q is a hyperparameter. f essentially permutes all positions of the words in a given sentence (i.e., either in premise or hypothesis) with the restriction that no words maintain their original position. in our initial setting, we do not explicitly control the placement of the words relative to their original neighbors, but we analyze clumping effects in §5. dtest now consists of |d test |×q examples, with q different permutations of hypothesis and premise for each original test example pair. if a sentence s (e.g., h i ) contains w words, then the total number of available permutations of s are (w -1)!, thus making the output of f a list of (w-1)! q permutations in this case. for us, the space of possible outputs is larger, since we permute p i and h i separately (and ignore examples for which any |s|≤ 5). ## results we present results for two types of models: we drop examples from test sets where we are unable to compute all unique randomizations, typically these are examples with sentences of length of less than 6 tokens. 2 models accept many permuted examples. we find ω max is very high for models trained and evaluated on mnli (in-domain generalization), reaching 98.7% on mnli dev. and test sets (in roberta, compared to a of 90.6% ( bold marks the highest value per metric (red shows the model is insensitive to permutation). for almost all examples in d test such that model m predicts the gold label. we also observe high ω rand at 79.4%, showing that there are many examples for which the models outperform even a random baseline in accepting permuted sentences (see appendix e for more ω values.) evaluating out-of-domain generalization with anli dataset splits resulted in an ω max value that is notably higher than a (89.7% ω max for roberta compared to 45.6% a). as a consequence, we encounter many flips, i.e., examples where the model is unable to predict the gold label, but at least one permutation of that example is able to. however, recall this analysis expects us to know the gold label upfront, so this test can be thought of as running a word-order probe test on the model until the model predicts the gold label (or give up by exhausting our set of q permutations). for out-of-domain generalization, ω rand decreases considerably (36.4% ω rand on a1), which means fewer permutations are accepted by the model. next, recall that a classic bag-of-words model would have p @xmath0. no model performs strictly like a classic bag of words although they do perform somewhat bow-like (p @xmath1, figure 5 ). we find this bow-likeness to be higher for certain non-transformer models, (infersent) as they exhibit higher p c (84.2% for infersent compared to 70.7% for roberta on mnli). models are very confident. the phenomenon we observe would be of less concern if the correct label prediction was just an outcome of chance, which could occur when the entropy of the log probabilities of the model output is high (suggesting uniform probabilities on entailment, neutral and contradiction labels, recall model b from §3). we first investigate the model probabilities for the transformer-based models on the permutations that lead to the correct answer in figure 2 . we find overwhelming evidence that model confidences on in-distribution datasets (mnli, snli) are highly skewed, resulting in low entropy, and it varies among different model types. bart proves to be the most skewed transformer-based model. this skewness is not a property of model capacity, as we observe distilbert log probabilities to have similar skewness as roberta (large) model, while exhibiting lower a, ω max , and ω rand . for non-transformers whose accuracy a is lower, the ω max achieved by these models are also predictably lower. we observe roughly the same relative performance in the terms of ω max (figure 5 and appendix similar artifacts in chinese nlu. we extended the experiments to the original chinese nli dataset @xcite , and reused the pre-trained roberta-large and infersent (non-transformer) models on ocnli. our findings are similar to the english results ( other results. we investigated the effect of sentence length (which correlates with number of possible permutations; appendix a), and hypothesisonly randomization (models exhibit similar phenomenon even when only hypothesis is permuted; appendix c). ## analyzing syntactic structure associated with tokens a natural question to ask following our findings: what is it about particular permutations that leads models to accept them? since the permutation operation is drastic and only rarely preserves local word relations, we first investigate whether there exists a relationship between permutation acceptance scores and local word order preservation. concretely, we compare bi-gram word overlap (bleu-2) with the percentage of permutations that are deemed correct (figure 3 ). 3 although the probability of a permuted sentence to be predicted correctly does appear to track bleu-2 score (figure 3 ), the percentage of examples which were assigned the gold label by the transformer-based models is still higher than we would expect from permutations with lower bleu-2 (66% for the lowest bleu-2 range of 0 -0.15), suggesting preserved relative word order alone cannot explain the high permutation acceptance rates. thus, we find that local order preservation does correlate with permutation acceptance, but it doesn't fully explain the high permutation acceptance scores. we now further ask whether ω is related to a more abstract measure of local word relations, i.e., part-of-speech (pos) neighborhood. many syntactic formalisms, like lexical functional grammar (kaplan and @xcite , head-drive phrase structure grammar @xcite or lexicalized tree adjoining grammar @xcite , are "lexicalized", i.e., individual words or morphemes bear syntactic features telling us which other words they can combine with. for example, "buy" could be associated with (at least) two lexicalized syntactic structures, one containing two noun phrases (as in kim bought cheese), and another with three (as in lee bought logan cheese). we speculate that our nli models might accept permuted examples at high rates, because they are (perhaps noisily) reconstructing the original sentence from abstract, word-anchored information about common neighbors. to test this, we pos-tagged d train using 17 universal part-of-speech tags (using spacy, honnibal et al. 2020). for each w @xmath2, we compute the occurrence probability of pos tags on tokens in the neighborhood of w i . the neighborhood is specified by the radius r (a symmetrical window r tokens from w @xmath3. we denote this sentence level probability of neighbor pos tags for a word w i as ψ r {w i ,s i } ∈ r 17 (see an example in figure 7 in the appendix). sentence-level word pos neighbor scores can be averaged across d train to get a type level score ψ r {w i ,d train } ∈ r 17 , ∀w @xmath4. then, for a sentence s @xmath5, for each word w @xmath6, we compute a pos mini-tree overlap score: concretely, β k {w i ,s i } computes the overlap of topk pos tags in the neighborhood of a word w i in s with that of the train statistic. if a word has the same mini-tree in a given sentence as it has in the training set, then the overlap would be 1. for a given sentence s i , the aggregate β k {s i } is defined by the average of the overlap scores of all its words: in figure 4 . we observe that the pos tag minitree hypothesis holds for transformer-based models, roberta, bart and distilbert, where the percentage of accepted pairs increase as the sentences have higher overlap with the un-permuted sentence in terms of pos signature. for non-transformer models such as infersent, convnet, and bilstm models, the pos signature ratio to percentage of correct permutation remains the same or decreases, suggesting that the reasoning process employed by these models does not preserve local abstract syntax structure (i.e., pos neighbor relations). ## human evaluation we expect humans to struggle with unli, given our intuitions and the sentence superiority findings (but see @xcite ). to test this, we presented two experts in nli (one a linguist) with permuted sentence pairs to label. 4 concretely, we draw equal number of examples from mnli matched dev set (100 examples where roberta predicts the gold label, d c and 100 examples where it fails to do so, d f ), and then permute these examples using f. the experts were given no additional information (recall that it is common knowledge that nli is a roughly balanced 3-way classification task). unbeknownst to the experts, all permuted sentences in the sample were actually accepted by the roberta (large) model (trained on mnli dataset). we observe that the experts performed ## training by maximizing entropy we propose an initial attempt to mitigate the effect of correct prediction on permuted examples. as we observe in §5, model entropy on permuted examples is significantly lower than expected. neural networks tend to output higher confidence than random for even unknown inputs (gandhi and lake, 2020), which might be an underlying cause of the high permutation acceptance. an ideal model would be ambivalent about randomized ungrammatical sentences. thus, we train nli models baking in the principle of mutual exclusivity (gandhi and lake, 2020) by maximizing model entropy. concretely, we fine-tune roberta on mnli while maximizing the entropy (h) on a subset of n randomized examples ((p i , ri ), for each example (p, h) in mnli. we modify the loss function as follows: using this maximum entropy method (@xmath7, we find that the model improves considerably with respect to its robustness to randomized sentences, all while taking no hit to accuracy ( ## future work & conclusion we show that state-of-the-art models do not rely on sentence structure the way we think they should: nli models (transformer-based models, rnns, and convnets) are largely insensitive to permutations of word order that corrupt the original syntax. we also show that reordering words can cause models to flip classification labels. we do find that models seem to have learned some syntactic information as is evidenced by a correlation between preservation of abstract pos neighborhood information and rate of acceptance by models, but these results do not discount the high rates of permutation acceptance, and require further verification. coupled with the finding that humans cannot perform unli at all well, the high rate of permutation acceptance that we observe leads us to conclude that current models do not yet "know syntax" in the fully systematic and humanlike way we would like them to. a few years ago, manning (2015) encouraged nlp to consider "the details of human language, how it is learned, processed, and how it changes, rather than just chasing state-of-the-art numbers on a benchmark task." we expand upon this view, and suggest one particular future direction: we should train models not only to do well on clean test data, but also to not to overgeneralize to corrupted input.