Variation and generality in encoding of syntactic anomaly information in sentence embeddings (2021)

Abstract: While sentence anomalies have been applied periodically for testing in NLP, we have yet to establish a picture of the precise status of anomaly information in representations from NLP models. In this paper we aim to fill two primary gaps, focusing on the domain of syntactic anomalies. First, we explore fine-grained differences in anomaly encoding by designing probing tasks that vary the hierarchical level at which anomalies occur in a sentence. Second, we test not only models’ ability to detect a given anomaly, but also the generality of the detected anomaly signal, by examining transfer between distinct anomaly types. Results suggest that all models encode some information supporting anomaly detection, but detection performance varies between anomalies, and only representations from more re- cent transformer models show signs of generalized knowledge of anomalies. Follow-up analyses support the notion that these models pick up on a legitimate, general notion of sentence oddity, while coarser-grained word position information is likely also a contributor to the observed anomaly detection.

## introduction as the nlp community works to understand what is being learned and represented by current models, a notion that has made sporadic appearances is that of linguistic anomaly. analyses of language models have often tested whether models prefer grammatical over ungrammatical completions (e.g. @xcite , while analyses of sentence embeddings have probed for syntax and semantics by testing detection of sentence perturbations @xcite . such work tends to exploit anomaly detection as a means of studying linguistic phenomena, setting aside any direct questions about encoding of anomaly per se. however, models' treatment of anomaly is itself a topic that raises * this work was done while the first author was a visiting student at the university of chicago. important questions. after all, it is not obvious that we should expect models to encode information like "this sentence contains an anomaly", nor is it obvious which types of anomalies we might expect models to pick up on more or less easily. nonetheless, anomalies are easy to detect for humans, and their detection is relevant for applications such as automatic error correction @xcite , so it is of value to understand how anomalies operate in our models, and what impacts anomaly encoding. in the present work we seek to fill this gap with a direct examination of anomaly encoding in sentence embeddings. we begin with fine-grained testing of the impact of anomaly type, designing probing tasks with anomalies at different levels of syntactic hierarchy to examine whether model representations better support detection of certain types of anomaly. then we examine the generality of anomaly encoding by testing transfer performance between distinct anomalies-here our question is, to the extent that we see successful anomaly detection, does this reflect encoding of a more general signal indicating "this sentence contains an anomaly", or does it reflect encoding of simpler cues specific to a given anomaly? we focus on syntactic anomalies because the hierarchy of sentence structure is conducive to our fine-grained anomaly variation. (sensitivity to syntactic anomalies has also been studied extensively as part of the human language capacity @xcite , strengthening precedent for prioritizing it.) we apply these tests to six prominent sentence encoders. we find that most models support nontrivial anomaly detection, though there is substantial variation between encoders. we also observe differences between hierarchical classes of anomaly for some encoders. when we test for transferability of the anomaly signal, we find that for most encoders the observed anomaly detection shows little sign of generality-however, transfer performance in bert and roberta suggests that these more recent models may in fact pick up on a generalized awareness of syntactic anomalies. follow-up analyses support the possibility that these transformer-based models pick up on a legitimate, general notion of syntactic oddity-which appears to coexist with coarser-grained, anomalyspecific word order cues that also contribute to detection performance. we make all data and code available for further testing. foot_0 ## related work this paper builds on work analyzing linguistic knowledge reflected in representations and outputs of nlp models @xcite @xcite . some work uses tailored challenge sets associated with downstream tasks to test linguistic knowledge and robustness @xcite @xcite @xcite @xcite . other work has used targeted classification-based probing to examine encoding of specific types of linguistic information in sentence embeddings more directly @xcite @xcite @xcite @xcite . we expand on this work by designing analyses to shed light on encoding of syntactic anomaly information in sentence embeddings. a growing body of work has examined syntactic sensitivity in language model outputs @xcite @xcite @xcite , and our agree-shift task takes inspiration from the popular number agreement task for language models @xcite @xcite . like this work, we focus on syntax in designing our tests, but we differ from this work in focusing on model representations rather than outputs, and in our specific focus on understanding how models encode information about anomalies. furthermore, as we detail below, our agree-shift task differs importantly from the lm number agreement tests, and should not be compared directly to results from those tests. our work relates most closely to studies involving anomalous or erroneous sentence information @xcite @xcite . some work investigates impacts from random shuffling or other types of distortion of input text @xcite or of model pre-training text @xcite on downstream tasks-but this work does not investigate models' encoding of these anomalies. @xcite present and test with the cola dataset for general acceptability detection, and among the probing tasks of @xcite there are three that involve analyzing whether sentence embeddings can distinguish erroneous modification to sentence inputs: somo, bshift, and coordinv. yin et al. (2020) also generate synthetic errors based on errors from nonnative speakers, showing impacts of such errors on downstream tasks, and briefly probing error sensitivity. more recently, @xcite conduct anomaly detection with various anomaly types at different layers of transformer models, using training of gaussian models for density estimation, and finding different types of anomaly sensitivity at different layers. we build on this line of work in anomaly detection with a fine-grained exploration of models' detection of word-content-controlled perturbations at different levels of syntactic hierarchy. our work is complementary also in exploring generality of models' anomaly encoding by examining transfer performance between anomalies. ## syntactic anomaly probing tasks to test the effects of hierarchical location of a syntactic anomaly, we create a set of tasks based on four different levels of sentence perturbation. we structure all perturbations so as to keep word content constant between original and perturbed sentences, thus removing any potential aid from purely lexical contrast cues. our first three tasks involve reordering of syntactic constituents, and differ in hierarchical proximity of the reordered constituents: the first switches constituents of a noun phrase, the second switches constituents of a verb phrase, and the third switches constituents that only share the clause. our fourth task tests sensitivity to perturbation of morphological number agreement, echoing existing work testing agreement in language models @xcite . ## experiments we generate probing datasets for each of the anomaly tests described above. we then apply these tasks to examine anomaly sensitivity in a number of generic sentence encoders. 4 datasets each of the above perturbations is used to create a probing dataset consisting of normal sentences and corresponding modified sentences, labeled as normal and perturbed, respectively. within each probing task, each normal sentence has a corresponding perturbed sentence, so the label sets for each task are fully balanced. each probe is formulated as a binary classification task. normal sentences and their corresponding perturbed sentences are included in the same partition of the train/dev/test split, so for any sentence in the test set, no version of that sentence (neither the perturbed form nor its original form) has been seen at training time. we draw our normal sentences from @xcite (premise only). perturbations of those sentences are then generated as our perturbed sentences. probing we analyze sentence embeddings from these prominent sentence encoders: infersent (conneau et al., 2017), skip-thoughts @xcite , gensen @xcite , bert @xcite , and roberta @xcite . the first three models are rnnbased, while the final two are transformer-based. to test the effectiveness of our control of word content, we also test bag-of-words (bow) sentence embeddings obtained by averaging of glove @xcite embeddings. this allows us to verify that our probing tasks are not solvable by simple lexical cues (c.f. ettinger et al., 2018), thus better isolating effects of syntactic anomalies. we train and test classifiers on our probing tasks, with sentence embeddings from the above encoders as input. the classifier structure is a multilayer perceptron (mlp) classifier with one hidden layer. foot_3 ## anomaly detection results fig. 1 shows anomaly detection performance for the tested encoders. we can see first that for three of our four tasks-all reordering tasks-our bow baseline performs perfectly at chance, verifying elimination of lexical biases. bow on the agree-shift task is just above chance, reflecting (expected) slight bias in morphological variations. comparing between tasks, we see that verb-ob yields highest overall performance while subn-obn yields the lowest. as mentioned above, a particularly informative comparison is between verb-ob and mod-noun, which both involve swapping sequentially adjacent content, but at different hier-archical levels. we see that encoders consistently show stronger performance on verb-ob than mod-noun, suggesting that the broader hierarchical domain of verb-ob may indeed make anomalies more accessible for encoding. the only anomaly that affects a broader span of the sentence is subn-obnbut we see that this is instead one of the most challenging tasks. we suspect that this is attributable to the fact that, as described above, detecting this anomaly may require extra world knowledge and common sense, which certain encoders may have less access to. foot_4 it is not unexpected, then, that bert and roberta, with comparatively much larger and more diverse training data exposure, show a large margin of advantage on this challenging subn-obn task relative to the other encoders. agree-shift patterns roughly on par with mod-noun, though notably infersent (and skipthoughts) detects the agreement anomaly much more readily than it does the mod-noun anomaly. comparing between encoders, we see clear stratification in performance. infersent shows the least anomaly awareness, performing for half of the tasks at chance level with the bow baseline. gensen and skip-thoughts, by contrast, consistently occupy a higher performance tier, often falling not far behind (but never quite on par with) the highest level of performance. the latter distinction is reserved for bert and roberta, which show the strongest anomaly sensitivity on all tasks. all models show stronger performance on verb-ob than mod-noun, but the hierarchical difference between these tasks seems to have particularly significant impact for infersent and skip-thoughts, with cues relating to verb-ob seemingly encoded by in-fersent, but cues relating to mod-noun seeming to be absent. mod-noun also yields the largest margin of difference between gensen and skip-thoughts. since skip-thoughts is one objective of gensen, this suggests that the additional gensen objectives provide an edge particularly for the finer-grained information needed for mod-noun. bert and roberta emerge soundly as the strongest encoders of anomaly information, with roberta also consistently outperforming bert. while this is in line with patterns of downstream task performance from these models, it is noteworthy that these models also show superior perfor-mance on these anomaly-detection tasks, as it is not obvious that encoding anomaly information would be relevant for these models' pre-training objectives, or for the most common nlu tasks on which they are typically evaluated. ## investigation on generality of anomaly encoding the above experiments suggest that embeddings from many of these encoders contain signal enabling detection of the presence of syntactic anomalies. however, these results cannot tell us whether these embeddings encode awareness that an "error" is present per se-the classifiers may simply have learned to detect properties associated with a given anomaly, e.g., agreement mismatches, or occurrence of modifiers after nouns. in this sense, the above experiments serve as finer-grained tests of levels of hierarchical information available in these embeddings, but still do not test awareness of the notion of anomaly in general. in this section we take a closer look at anomaly awareness per se, by testing the extent to which the sensitivities identified above are specific to individual anomalies, or reflective of a more abstract "error" signal that would apply across anomalies. we explore this question by testing transfer performance between different anomaly types. ## discussion in the experiments above, we have taken a closer look at the nature of syntactic anomaly encoding in sentence embeddings. using fine-grained variation in types of syntactic anomalies, we show differences in patterns of anomaly detection across encoders, suggesting corresponding differences in the types of anomaly information encoded by these models. while the margins of difference in anomaly-specific sensitivity are less dramatic between small rnn-based models and larger transformer models, when we examine the generality of the detected anomaly signal, we find that only bert and roberta show signs of higher-level anomaly awareness, as evidenced by non-trivial transfer performance between anomalies. what might be driving the anomaly encoding patterns indicated by our results? explicit syntactic training does not appear to be necessary. gensen is the only model that includes an explicit syntactic component in its training (constituency parsing), which could help to explain that model's comparatively strong performance on the individual anomaly detection tasks. however, it is noteworthy that gensen performs mostly on par with skipthoughts, which constitutes just one of gensen's objectives, and which uses only prediction of adjacent sentences. bert and roberta, the only models to show signs of more generalized anomaly encoding, have no explicit syntactic training at all. however, various findings have suggested that these types of models do develop syntactic sensitivity as a result of their more generalized training objectives @xcite @xcite . we can imagine various ways in which objectives involving prediction of words in context, as used by bert and roberta, could encourage learning of a generalized notion of syntactic anomaly. it may be the case that oddities occur together, in which case they could be mutually predictive and therefore of value for optimizing a prediction-based objective. more generally, anomalous sentences are likely identifiable as less probable, or more difficult to generate coherent predictions for. this relationship between anomaly and sentence probability raises a related question: is this a problem for our conclusions here? could models simply be identifying anomalous sentences as less probable, without any actual notion of syntactic anomaly? in nlp models, assessment of text probabilities is closely related to assessment of text naturalness and acceptability. for this reason, teasing apart general sensitivity to probability versus genuine awareness of syntactic grammaticality phenomena is a recurring challenge when testing syntactic knowledge in language modelsand these things are similarly potentially entangled in our analyses here. to an extent this entanglement is inevitable and unproblematic: we necessarily expect syntactic anomalies to lower the probability of a string, and we can expect some awareness of syntactic anomaly to be important for assigning lower probability to an ungrammatical string. we can imagine, for instance, a situation in which a language model has no sensitivity to what constitutes good versus anomalous syntax, and thus assigns probability solely on the basis of word co-occurrence or other unrelated indicators of naturalness. in this sense, although it is not difficult to imagine how the close relationship between anomaly and sentence probability could be an explanation for findings that suggest anomaly awareness in these models, this does not change the fact that model representations may end up with genuine, detectable encoding of generalized anomaly information as a byproduct of probabilistic training-and this genuine anomaly encoding may be what we are detecting with our tests here. however, future work can examine further the relationship between syntactic anomaly and model perplexities, to explore whether these embeddings could show signs of anomaly sensitivity while in fact exclusively encoding confounding probabilistic information unrelated to syntactic anomaly. our content-word-only analysis provides one source of evidence on the relative importance of genuine syntactic sensitivity in success on our syntactic-related anomaly tasks. this test aims to tease apart the extent to which success on our tasks requires processing of finer-grained syntactic information, versus the extent to which models can succeed based on more superficial content word position information. we find in this analysis that most encoders do benefit from the finer-grained syntactic information provided by function words, supporting an important role for more advanced syntactic sensitivity in these tasks-however, we also find that substantial proportions of the observed detection accuracy can indeed be achieved with content words alone. to the best of our knowledge, we are the first to report this finding. this leaves us with two key takeaways. first, to an extent there is good reason to believe that a reasonable amount of genuine syntactic sensitivity is involved in the highest levels of success on our anomaly tasks. second, success on syntactic anomaly tasks can also be nontrivially inflated by use of more superficial cues. that is to say, as usual, these datasets have a habit of enabling learning from simpler cues than are intended. this takeaway highlights a need for caution in interpreting detection of reordering anomalies as evidence of deeper syntactic encoding per se @xcite -especially in probing datasets that use naturally-occurring data without exerting controls for confounding cues. while these results have shed light on potential encoding of generalized syntactic anomaly knowledge in pre-trained models, there are many further questions to pursue with respect to these models' handling and understanding of such anomalies. we will leave for future work the problem of understanding in greater detail how model training may contribute to encoding of a generalized awareness of anomalies in text, how a genuine notion of syntactic anomaly could be further disentangled from general probability sensitivity, and how one could exploit models' awareness of anomaly for improving model robustness on downstream pipelines. ## conclusion we have undertaken a direct study of anomaly encoding in sentence embeddings, finding impacts of hierarchical differences in anomaly type, but finding evidence of generalized anomaly encoding only in bert and roberta. follow-up analyses support the conclusion that these embeddings encode a combination of generalized and anomaly-specific cues in these embeddings, with models appearing to leverage both finer-grained and coarser-grained information for anomaly detection. these results contribute to our understanding of the nature of encoding of linguistic input in embeddings from recent models. future work can further explore the relationship between naturalness-oriented training and cultivation of abstract anomaly awareness, and how these insights can be leveraged for more robust and human-like processing of language inputs. alex warstadt, amanpreet singh, and samuel r bowman. 2019. neural network acceptability judgments. transactions of the association for computational linguistics, 7:625-641. aaron steven white, pushpendre rastogi, kevin duh, and benjamin van durme. 2017. inference is everything: recasting semantic resources into a unified evaluation framework. in proceedings of the eighth international joint conference on natural language processing (volume 1: long papers), pages 996-1005. adina williams, nikita nangia, and samuel bowman. 2018. a broad-coverage challenge corpus for sentence understanding through inference. in proceedings of the 2018 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long papers), pages 1112-1122. yi yang, wen-tau yih, and christopher meek. 2015. wikiqa: a challenge dataset for open-domain question answering. in proceedings of the 2015 conference on empirical methods in natural language processing, pages 2013-2018. fan yin, quanyu long, tao meng, and kai-wei chang. 2020. on the robustness of language encoders against grammatical errors. in proceedings of the 58th annual meeting of the association for computational linguistics.