Implicit representations of event properties within contextual language models: Searching for “causativity neurons” (2021)

Abstract: This paper addresses the question to which extent neural contextual language models such as BERT implicitly represent complex semantic properties. More concretely, the paper shows that the neuron activations obtained from processing an English sentence provide discriminative features for predicting the (non-)causativity of the event denoted by the verb in a simple linear classifier. A layer-wise analysis reveals that the relevant properties are mostly learned in the higher layers. Moreover, further experiments show that appr. 10% of the neuron activations are enough to already predict causativity with a relatively high accuracy.

## introduction and motivation in natural language processing (nlp), machine learning models based on artificial neural networks have achieved impressive results in recent years, due to large amounts of available training data and powerful computing infrastructures. contextual language models (lms) such as elmo @xcite , bert @xcite , and xlnet @xcite have particularly contributed to this. however, it is oftentimes not clear which kinds of generalizations these models make, i.e., what exactly they learn. in this respect, neural networks suffer from a lack of transparency and interpretability. recent research has started to investigate these questions. since the successful use of neural word embeddings and lms (e.g., @xcite @xcite ) for a range of nlp/nlu tasks, it is clear that lms capture meaning to a certain degree, in particular lexical meaning. concerning syntactic information, work on different 1 our datasets are available at https://github.com/eseyffarth/ predicting-causativity-iwcs-2021 types of language models, in particular rnns and transformer-based contextual language models, has shown that these models learn morphology @xcite , syntactic structure and syntactic preferences to a certain degree (see @xcite @xcite @xcite @xcite . in this paper, we expand the question of what linguistic properties these models learn towards whether pretrained contextualized models capture more abstract semantic properties, in particular properties that contribute to the structure of the semantic representation underlying a given sentence. more concretely, we investigate whether an lm such as bert represents whether a sentence denotes a causative event or not. if this was the case, we would expect a systematic difference between for instance bert's neuron activations for (1-a) and for (1-b). note that the two sentences share almost no lexical elements, so the neuron activations are expected to be mostly different. our research question is focused on whether there are systematic activation patterns that can be observed that are common to all instances of causative sentences, and others that are common to all instances of noncausative sentences, independent of sentence content. one of the common approaches to probe neural network models is to use a probing classifier. given a linguistic property of interest, the idea is to extract contextualized activations of units (words/phrases/sentences) relevant to the property. a classifier is then trained to learn the property by using the extracted activations as features. the performance of the classifier is taken to approximate the degree to which the language model learned the linguistic property. we also use probing classifiers and probe the model as a whole, its individual layers and its neurons with respect to causativity. we use the neurox toolkit @xcite to conduct the probing experiments. we experiment using two 12-layer pretrained models, bert @xcite and xlnet @xcite , as well as a distilled version of bert, distilbert @xcite . our findings and contributions are as follows: we create a novel dataset of sentences with verbs that are labeled for causativity/non-causativity. using this dataset for probing, we show that this abstract semantic property is learned by the pretrained models. it is better represented in the higher layers of the model and, furthermore, there is a subset of appr. 10% of the neurons that encodes the property in question. ## related work a number of interpretation studies have analyzed representations of pre-trained models and showed that they learn linguistic information such as part of speech tagging, semantic tagging and ccg tagging @xcite @xcite . a typical procedure to analyze representation is a post-hoc analysis using a probing classifier. it has been shown that word-level concepts are learned at lower layers while sentence-level concepts are learned at higher layers @xcite . @xcite extended the layer-level analysis towards individual neurons of the network. they proposed linguistic correlation analysis (lca) to identify neurons with respect to a linguistic property. durrani et al. (2020); dalvi et al. (2020) later used lca to analyze pre-trained models in the context of linguistic learning and redundancy in the network respectively. in this work, we also aim to analyze pre-trained models at model-, layer-and neuron-level using post-hoc analysis methods. different from others, we concentrate on an abstract, structure-building semantic property, namely causativity of events. our focus is on lexical causatives, that is, verbs whose lexical meaning has a causative aspect @xcite . in dowty's aspect calculus, such verbs are analyzed as [φ cause ψ], where φ and ψ are sentences and causation is a "two-place sentential connective", notably even for sentences that only contain a single verb phrase. thus, john killed bill is decomposed as in (2) @xcite . ## method over the last years, there has been an increasing interest in assessing linguistic properties encoded in neural representations. a common method to reveal these linguistic representations employs diagnostic classifiers or probes @xcite . a common diagnostic classifier is a linear classifier trained for the underlying linguistic task, using the activations generated from the trained neural network model as features. the performance of the classifier is used as a proxy to measure the amount of linguistic information present in the activations. we also use a linear classifier for probing. consider a pre-trained neural network model m with l layers: {l 1 , l 2 , . . . , l l }, where each layer l i is of size h. given a dataset @xmath0, s 2 , ..., s t } consisting of t sentences, the contextualized embedding of sentence s j at layer l i is z i @xmath1. in pretrained models like bert, a special token [cls] is appended with every training instance during training. the token is later optimized for sentence embedding during transfer learning @xcite . we consider the representations of @xcite for sentence embedding in this study. the [cls] representation extracted from various layers is used as input features to the probing classifier. model-level probing: to assess to what extent a linguistic property is learned in the model, we first take the sentence representations of all layers as features for linear classification, i.e., all z i j for layer-level probing: here, we question how much individual layers of a model represent our property of interest. we train a linear classifier on the activations of each individual layer. the performance of each layer serves as a proxy to how much information it encodes with respect to our property. neuron-level probing: while the layer-level probing tells about how much linguistic information is learned in a layer, it does not tell about the learning of individual neurons in the network. it is possible that while a particular layer performs best in the layer-level probing, the best neurons learning about the linguistic property are spread across many layers. in neuron-level probing, we aim to identify the most salient neurons across the network that learn the linguistic property at hand. we follow the linguistic correlation analysis method (lca) of @xcite to conduct this analysis. given representations of the model as in the model-level probing, lca trains an elas-ticnet @xcite classifier, and provides a salient list of neurons with respect to the linguistic property. elasticnet provides a balance between selecting very focused localized features and distributed features (here: neurons). equation (2) gives the loss function: ## data to prepare our datasets, we create different sets of verbs that are labeled for (non)causativity, and then use them as seeds to collect sentences from a corpus to be used as input to the classifier. ## discussion as shown in our datasets are randomly collected from a larger corpus with no regard for verb frequency. this results in datasets where some verbs occur only once or twice, some are never seen in the training data, and some are more common. our goal is to determine whether the classifiers successfully learn to predict (non)causativity, independently of specific verb lemmas. the results reported so far are all averaged over all verbs in a dataset, illustrating that some models are more successful on the classification task than others (e.g. bert achieving higher accuracy scores than the other models on the first two datasets). additionally, it is also worth exploring the accuracy of the classifiers for individual verbs, particularly those that are most likely to be mislabeled by any of the classifiers. broadly, the frequently mislabeled verbs fall in three categories: 1. presumed errors due to parsing mistakes and subsequent errors in the gold data; 2. errors due to incorrect labels of ambiguous verbs in the gold data; 3. errors due to an ambiguity between full verb, light verb, and auxiliary verb. presumed errors due to parsing mistakes and subsequent errors in the gold data most of the frequently-mislabeled verbs in d tr 5 fall into this category. these verbs occur only a few times each, indicating that they do not represent a deeper structural issue with the classifiers; for instance, sentences with the root verb mark occasionally appear incomplete in encow, as exemplified in ( 6 ). the verb sound is labeled as a causative verb in our gold data (e.g., "to sound the bells"), but appears often in another word sense, as exemplified in (7-a). in these sentences, the verb does not have a direct object as expected; the reason for their inclusion in our datasets is an incorrect dependency parse in encow. in other words, the causative gold label is assigned by mistake. a similar case is mean; as with sound, many instances do not involve a direct object at all, as exemplified in (7-b), but are included because of an incorrect parse. (7) a. that sounds so scary !!! (encow-05-11095175) b. you mean screw justice ? errors due to incorrect gold labels of ambiguous verbs in d tr , face is the most mislabeled causative verb. the presumed causative label for this verb comes from the vn class confront-98, which contains verbs such as target or combat. however, the mislabeled examples from the dataset seem to evoke a weaker, more passive sense of face, as in (9-a), where human annotators might not assign a causative label. in these cases, the label assigned by the classifier is actually correct, while the gold label is not. the mislabeled instances of cover in d all are, similarly to face, an artefact of verb polysemy and should in fact not be regarded as causative sentences, as exemplified in (9-b). in d tr , all classifiers occasionally label instances of noncausative leave as causative, particularly xl-net. leave is a member of the vn classes become-109.1-1-1, escape-51.1-1-1, fulfilling-13.4.1, future having-13.3, keep-15.2, and others. while not all of these classes license basic intransitive sentences of the type included in our datasets, this illustrates the polysemy of leave, which might be an explanation for the relatively high number of mislabeled instances in our experiments. generally, in d all , noncausative alternating verbs are among the most mislabeled verbs. since the dataset contains different numbers of verbs of each type, this may be a sparsity effect more than an effect of these verbs being more difficult to label. this question will be approached with new datasets in future work. the reason for most errors of this type is that our datasets were created automatically with the help of a lexical resource. in order to avoid such polysemy issues, a version of the datasets with human annotations would be necessary. errors due to an ambiguity between full verb, light verb, and auxiliary verb finally, the verbs have and be are the most mislabeled nonalternating noncausative verbs in d all . these verbs appear in light verb constructions, as auxiliary verbs, and in a range of word senses that can be causative or noncausative. the examples in (11) illustrate why the classifiers are struggling to label such sentences as noncausative. note that in all cases, the maltparser annotations provided alongside encow mark a form of have as the root verb. to summarize, many of the "errors" of the classifiers are actually not errors but incorrect labels in the gold data. this means that the classifiers might be better in predicting causativity than assessed by our evaluation. ## conclusion we set up a series of classification experiments with a range of datasets to determine whether large language models learn implicit representations of causativity, a linguistic property that is not necessarily represented syntactically or morphologically in english. we compare classifiers based on bert, distilbert, and xlnet, and find that all learn to predict causativity to a large extent. differences in classification accuracy are observed across different datasets (see we also show that causativity is represented rather in the higher layers of the models and, furthermore, that reducing each model to only the 10% of its neurons that are most correlated with the causativity property only leads to small differences in accuracy, sometimes an increase in accuracy due to the elimination of non-discriminative features. our error analysis suggests that many of the classification errors are actually labeling errors in the data, due either to a wrong parse of the sentence in our source corpus encow or to the polysemy of verbs that can be causative in certain readings but are not causative in some of the readings mislabeled in the dataset. put differently, the classifiers were probably better in identifying causativity than their accuracy scores suggest. while our datasets were created with little manual effort and already led to good results, we are planning on pursuing possible improvements in the future in order to avoid these labeling errors as far as possible.