How is BERT surprised? Layerwise detection of linguistic anomalies (2021)

Abstract: Transformer language models have shown remarkable ability in detecting when a word is anomalous in context, but likelihood scores offer no information about the cause of the anomaly. In this work, we use Gaussian models for density estimation at intermediate layers of three language models (BERT, RoBERTa, and XLNet), and evaluate our method on BLiMP, a grammaticality judgement benchmark. In lower layers, surprisal is highly correlated to low token frequency, but this correlation diminishes in upper layers. Next, we gather datasets of morphosyntactic, semantic, and commonsense anomalies from psycholinguistic studies; we find that the best performing model RoBERTa exhibits surprisal in earlier layers when the anomaly is morphosyntactic than when it is semantic, while commonsense anomalies do not exhibit surprisal at any intermediate layer. These results suggest that language models employ separate mechanisms to detect different types of linguistic anomalies.

## introduction transformer-based language models (lms) have achieved remarkable success in numerous natural language processing tasks, prompting many probing studies to determine the extent of their linguistic knowledge. a popular approach is to formulate the problem as a multiple-choice task, where the lm is considered correct if it assigns higher likelihood to the appropriate word than an inappropriate one, given context @xcite @xcite . the likelihood score, however, only gives a scalar value of the degree that a word is anomalous in context, and cannot distinguish between different ways that a word might be anomalous. it has been proposed that there are different types of linguistic anomalies. chomsky we investigate several patterns: first, surprisal at lower layers corresponds to infrequent tokens, but this effect diminishes towards upper layers. second, morphosyntactic violations begin to trigger high surprisals at an earlier layer than semantic violations. in this paper, we introduce a new tool to probe for surprisal at intermediate layers of @xcite , roberta @xcite , and xlnet @xcite , formulating the problem as density estimation. we train gaussian models to fit distributions of embeddings at each layer of the lms. using blimp @xcite for evaluation, we show that this model is effective at grammaticality judgement, requiring only a small amount of in-domain text for training. figure 1 shows the method using the roberta model on two example sentences. we apply our model to test sentences drawn from blimp and 7 psycholinguistics studies, exhibiting morphosyntactic, semantic, and commonsense anomalies. we find that morphosyntactic anomalies produce out-of-domain embeddings at earlier layers, semantic anomalies at later layers, and no commonsense anomalies, even though the lm's final accuracy is similar. we show that lms are internally sensitive to the type of linguistic anomaly, which is not apparent if we only had access to their softmax probability outputs. our source code and data are available at: https://github.com/spoclab-ca/ layerwise-anomaly. ## model we use the transformer language model as a contextual embedding extractor (we write this as bert for convenience). let l be the layer index, which ranges from 0 to 12 on all of our models. using a training corpus {w 1 , • • • , w t }, we extract contextual embeddings at layer l for each token: for evaluating the layerwise surprisal of a new sentence @xmath0, • • • , t n ], we similarly extract contextual embeddings using the language model: the surprisal of each token is the negative log likelihood of the contextual vector according to the multivariate gaussian: ## levels of linguistic anomalies we turn to the question of whether lms exhibit different behaviour when given inputs with different types of linguistic anomalies. the task of partitioning linguistic anomalies into several distinct classes can be challenging. syntax and semantics have a high degree of overlap -there is no widely accepted criterion for distinguishing between ungrammaticality and semantic anomaly (e.g., @xcite gives a survey of current proposals), and @xcite challenges this dichotomy entirely. similarly, @xcite noted that semantic anomalies depend somewhat on world knowledge. within a class, the anomalies are also heterogeneous (e.g., ungrammaticality may be due to violations of agreement, wh-movement, negative polarity item licensing, etc), which might each affect the lms differently. thus, we define three classes of anomalies that do not attempt to cover all possible linguistic phenomena, but captures different levels of language processing while retaining internal uniformity: ## conclusion we use gaussian models to characterize outof-domain embeddings at intermediate layers of transformer language models. the model requires a relatively small amount of in-domain data. our experiments reveal that out-of-domain points in lower layers correspond to low-frequency tokens, while grammatically anomalous inputs are out-of-domain in higher layers. furthermore, morphosyntactic anomalies are recognized as out-ofdomain starting from lower layers compared to syntactic anomalies. commonsense anomalies do not generate out-of-domain embeddings at any layer, even when the lm has a preference for the correct cloze completion. these results show that depending on the type of linguistic anomaly, lms use different mechanisms to produce the output softmax distribution. commonsense -urbach and kutas commonsense -chow et al. commonsense -federmeier and kutas commonsense -warren et al. semantic -osterhout and mobley semantic -osterhout and nicol semantic -warren et al. semantic -pylkkänen and mcelree semantic -blimp (animacy) morphosyntax -osterhout and nicol morphosyntax -blimp (det-noun) morphosyntax -blimp (subject-verb) 0 1 2 3 4 5 6 7 8 9 10 11 12 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 layer surprisal gap figure 7: surprisal gap plot using xlnet.