Probing for Understanding of E nglish Verb Classes and Alternations in Large Pre-trained Language Models (2022)

Abstract: We investigate the extent to which verb alternation classes, as described by Levin (1993), are encoded in the embeddings of Large Pre-trained Language Models (PLMs) such as BERT, RoBERTa, ELECTRA, and DeBERTa using selectively constructed diagnostic classifiers for word and sentence-level prediction tasks. We follow and expand upon the experiments of Kann et al. (2019), which aim to probe whether static embeddings encode frame-selectional properties of verbs. At both the word and sentence level, we find that contextual embeddings from PLMs not only outperform non-contextual embeddings, but achieve astonishingly high accuracies on tasks across most alternation classes. Additionally, we find evidence that the middle-to-upper layers of PLMs achieve better performance on average than the lower layers across all probing tasks.

## introduction we investigate the extent to which verb alternation classes are represented in word and sentence embeddings produced by pre-trained language model (plm) embeddings @xcite . as first comprehensively cataloged by @xcite , verbs pattern together into classes according to the syntactic alternations in which they can and cannot participate. for example, (1) illustrates the causativeinchoative alternation. break can be a transitive verb in which the subject of the sentence is the agent and the direct object is the theme, as in example (1a). it can also alternate with the form in (1b), in which the subject of the sentence is the theme and the agent is unexpressed. however, (2) demonstrates that cut cannot participate in the same alternation, despite its semantic similarity. janet broke the cup. b. the cup broke. jack sprayed paint on the wall. b. jack sprayed the wall with paint. the alternations in which a verb may participate is taken to be a lexical property of the verb (e.g. @xcite @xcite . moreover, we hypothesize that the alternations should be observable within large text corpora, and are therefore available during the pretraining procedure for plms such as bert @xcite . in contrast, ungrammatical examples such as ( 2b ) and ( 4b ) should be virtually absent from the training data. this leads us to hypothesize that plm embeddings should encode whether particular verbs are allowed to participate in syntactic frames of various alternation classes. our research questions are as follows: through our series of experiments, we find that plm embeddings indeed encode information about verb alternation classes at both the word and sentence level. while performance is relatively consistent on the word-level task for the four plms we analyze, we find that electra @xcite significantly outperforms the other models for the sentence-level task. furthermore, we find evidence suggesting that middle-to-upper layers encode more information about verb alternation classes since they consistently improve upon the lower layers across all tasks. the rest of the paper is organized as follows: after a brief review of related literature in section 2, we present datasets and models that are relevant to our experiment in sections 3 and 4. we then present two experiments to answer our research questions in sections 5 and 7. section 6 presents an additional control task @xcite to test whether our linear probes are selective for the given tasks. finally, we offer a discussion in section 8 and overall conclusions in section 9. ## related work our work follows @xcite , who attempt to predict verb class membership and sentence grammaticality judgments on the basis of glove embeddings @xcite and embeddings derived from the 100m-token british national corpus with a single-directional lstm @xcite . for the sentence-level task, they further process the input embeddings using a sentence encoder trained on a "real/fake" sentence classification task. varying multi-layer perceptron (mlp) architectures are used for the classification step. because their primary research focus has to do with how neural language models inform learnability (in the sense of human language acquisition), they intentionally use smaller language models derived from "an amount of data similar to what humans are exposed to during language acquisition" and avoid models trained on "several orders of magnitude more data than humans see in a lifetime" (p. 291). as described in section 5, we depart and build upon @xcite by examining the embedding representations of plms instead of static embeddings. we then use an intentionally simple and selective linear diagnostic classifier to probe the representations, as our research questions focuses on the plm embeddings themselves. we note that @xcite achieves only modest performance in prediction accuracy and mcc, and only for a limited number of verb classes. while this is a valuable result for their research goals, our hypothesis is that plms will achieve better performance due to a combination of their contextual representations, complex architectures, and larger training corpora. to our knowledge, attempting to predict verb alternation class membership along the lines of levin 1993 from plm representations is novel. however, two very closely related lines of work include the experiments of @xcite , which respectively evaluate the performance of various plms on the cola @xcite and blimp @xcite benchmarks, which include acceptability judgment examples from a wide variety of linguistic phenomena (including verb argument structures). we distinguish our experiments from these papers in two major ways. first, we attempt to directly probe the linguistic knowledge of individual plm embedding layers with a classification probe instead of specifically finetuning the models to a specific task. second, we limit our focus to verb alternation classes and present detailed analysis about patterns and trends across different alternations and their corresponding syntactic frames. ## data in our experiments, we use two dataset created by @xcite . one is the lexical verbframe alternations dataset (lava), which is based on the verbs and alternation classes defined in @xcite . it contains a mapping of 516 verbs to 5 alternation classes, which are further subdivided into two syntactic frames for each alternation. the broad categories of the alternation classes are: spray-load, causative-inchoative, dative, there-insertion, and understood-object. ## models in addition to bert, we perform experiments on several recent transformer-based plms including roberta @xcite , deberta @xcite , and electra @xcite which vary from bert in a few ways including modifications to bert's tokenization and pre-training procedure and the size of their training corpus. to make comparisons between each model fair, we use the base architectures for each model which have 12 layers, 12 attention heads, and a hidden layer size of 768. foot_1 ## control task for each verb v i in lava with a binary label y i denoting whether v i can participate in the syntax frame sl-with, we independently sample a control behavior c(v) by randomly assigning a binary "membership" value to v i based on the empirical membership distribution of verbs that participate in the sl-with syntax frame. the control task is the function that maps each verb, v i , to the label specified by the control behavior c(v i ): following the experiment design of @xcite , we compare the selectivity of a linear probe, an multi-layer perceptron with 1-hidden layer (mlp-1), and an mlp with 2-hidden layers (mlp-2) where the selectivity of a model is defined by the difference between its accuracy on the real task (i.e. predicting verb membership for the sl-with frame) and the control task. in addition, we explore several "complexity control" methods including limitation of feature dimensionality, reducing the number of training examples, and increasing regularization. ## discussion on the word level prediction task, all plms achieve strong correlation (> 0.7 mcc) across all syntactic frames with the strongest performance in the "there" frame (1.00, achieved by bert and electra) and the weakest performance on the "non-reflexive" frame (0.794, achieved by roberta). when look- for the sentence-level experiments, we see a similar outcome wherein the upper-middle plm layers achieve the best performance on average. however, we observe that there is much more variation in performance between each plm. electra and bert are relatively consistent, since their best performing layer for all alternation classes either come from the middle or upper layers. in contrast, the lower layers of roberta achieve the best performance on the dative alternation, and both roberta and deberta achieve the best perfor-mance on the understood-object alternation from the first layer. these anomalies can potentially be explained by the claim that different alternation classes require different types of linguistic knowledge (i.e. syntactic v.s. semantic) which are encoded in different plm layers. however, the consistently strong performance of the upper layers for bert and electra across all alternation classes provides counter evidence against the claim. electra is the best performing model overall on the sentence-level acceptability task, achieving the best mcc and accuracy on four of the five alternation classes (all except dative). unsurprisingly, electra also excels on the combined dataset compared to the other models (0.165 mcc over the second-best performing model, deberta). while it is difficult to attribute the model's success to a specific property, one hypothesis is that its gener-ator/discriminator architecture closely resembles the fava task of identifying acceptable sentences from linguistic minimal pairs. this idea is reinforced by the authors as well, who note that the model's relatively strong performance on cola potentially stems from the fact that the acceptability judgment task of cola "closely matches elec-tra's pre-training task of identifying fake tokens" @xcite . while we are optimistic about our results, there are several limitations to our experiments. first, we only analyze five different alternation classes which is a small subset of the 83 classes presented in @xcite . in addition -although our control task ensures that our classifier probe is relatively selective for the first experiment and bert, it may not necessarily generalize well to the second experiment, other syntactic frames, and other models. in the future, we hope to expand our selectivity experiments to a wider array of syntactic frames and models. ## conclusion and future work overall, our results support the hypothesis that plm contextual embeddings encode linguistic information about verb alternation classes at both the word and sentence level. for the frame-selectional verb classification task, all plms achieve significant improvement upon the reference cola-style embeddings from @xcite , especially for frames in which the cola-style embeddings obtain weak correlation (i.e. "locative", "reflexive", and "non-reflexive"). also, it is clear that model performance tends to improve from lower to upper layers, which can be seen the most easily from the mean performance across layer figures. for the sentence acceptability task, we arrive at similar conclusions, albeit with greater distinction in results between different models and layers. while there are numerous factors that may be responsible for the improved performance from plms, we hypothesize that the improvement can largely be attributed to the attention-based encodings of transformer models since we only saw modest improvements in performance from the reference embeddings when using the bottom "static" layers for each plm. in terms of future work, there are several interesting avenues that we hope to explore. from the data perspective, it would certainly be worthwhile to test whether our insights and conclusions extends to the dozens of alternations described in @xcite that are not present in the lava and fava datasets. there are also several interesting adaptations that can be made to our experiment methodology. for example, instead of just analyzing the base architecture for each plm, we could also analyze small and large variants to directly evaluate the effect of scaling training data and model size within the same model. moreover, while we attempt to control the probe confounder problem by building a selective probe, there is no guarantee that the classifier probes do not pick up on arbitrary signals in the training data that lead to non-meaningful improvements in performance. two promising alternative approaches that mitigate this risk include unsupervised evaluation of minimal pairs as shown in @xcite and "amnesic probing", which tests whether a property that can be extracted from a probe is actually relevant to task importance @xcite .