Does BERT Recognize an Agent? Modeling D owty’s Proto-Roles with Contextual Embeddings (2022)

Abstract: Contextual embeddings build multidimensional representations of word tokens based on their context of occurrence. Such models have been shown to achieve a state-of-the-art performance on a wide variety of tasks. Yet, the community struggles in understanding what kind of semantic knowledge these representations encode. We report a series of experiments aimed at investigating to what extent one of such models, BERT, is able to infer the semantic relations that, according to Dowty’s Proto-Roles theory, a verbal argument receives by virtue of its role in the event described by the verb. This hypothesis were put to test by learning a linear mapping from the BERT’s verb embeddings to an interpretable space of semantic properties built from the linguistic dataset by White et al. (2016). In a first experiment we tested whether the semantic properties inferred from a typed version of the BERT embeddings would be more linguistically plausible than those produced by relying on static embeddings. We then move to evaluate the semantic properties inferred from the contextual embeddings both against those available in the original dataset, as well as by assessing their ability to model the semantic properties possessed by the agent of the verbs participating in the so-called causative alternation.

## introduction in the last two decades, word embeddings have become one of the most widely used tools for the encoding of lexical meaning in computational models of language. different flavours of such models have been proposed, all of which have in common the idea of representing lexical elements as multidimensional vectors inferred from their context of occurrence (for a review, see @xcite . the last wave of word embeddings followed the transformer-based models breakthrough @xcite , that resulted in the development of the so-called contextual embeddings. these representations are generated by models like bert @xcite or gpts @xcite and derive their name by their ability to keep track of the different contexts in which a word occurs, giving different vector representations for the same word appearing with different surrounding neighbours (for review, see @xcite . this has been a major improvement over static embeddings obtained from models such as lsa @xcite , glove @xcite and word2vec @xcite , allowing this kind of representation to reach state-of-the-art performance in a great variety of natural language processing (nlp) tasks. notwithstanding their wide usage, mainly due to their great empirical successes, the community still struggles to understand what kind of information word embeddings are actually able to encode about language structure, and how they do it. the problem has been even sharpened with contextual embeddings, which are considered to be more entangled representations, usually bigger in dimensions than previous versions, and are obtained from deeper neural models, whose inner working is more complex. due to this fact, to better understand and explain what kind of structure these models are able to represent is becoming more and more desirable and several research lines started to spring out with this purpose. to the present days the encoding of syntactic knowledge in these model has been more studied than their ability to deal with semantic facets of language @xcite , but the number of studies in that direction, usually carried out by means of probing tasks developed on top of pre-trained architectures @xcite , is also growing @xcite . in the present paper we focus on the modelling of the semantic content of what dowty @xcite labelled as thematic proto-roles, that are clusters of entailment properties that an arguments derives solely by virtue of its role in the event described by a predicate. following previous work by @xcite we use a linear transformation mapping between embeddings of verbs represented in bert's space and a set of interpretable vectors derived from the universal decompositional semantics dataset on proto-roles properties @xcite , in which human ratings about argument properties have been annotated and collected. however, the present work differs from @xcite under several respects: i.) we deal with contextual embeddings, focusing on those yielded by bert, ii.) we experiment with representations at the token-level, iii.) we successfully apply spca as a de-noising technique, iv.) and we qualitatively address the phenomenon of the causative-inchoative alternation, for which the notion of semantic (proto-) role is crucially relevant. the goal of this paper is twofold: i.) to test whether the bert contextual embeddings of a verb encode semantic information concerning the proto-role properties held by its arguments, and ii.) to test whether this knowledge can be distilled by means of a linear mapping, thus leading the way to the development of full-scale systems able to extract this knowledge for a wide range of verbs. the following pages are organized as follows: in section 2, we quickly review the literature investigating the semantic content of vector representations, before discussing the notion of thematic role and its empirical foundations. we describe and test our method in sections 4 and 5, respectively, while section 6 is devoted to a general discussion of the merits and limitations of the use of the bert's embeddings to model proto-role information. ## the semantics of word embeddings notwithstanding the wide usage of word embeddings in nlp and related fields, the literature trying to characterize the semantic properties of these representation is quite scarce. concerning the efforts in trying to understand whether and to what extent thematic roles information is encoded in contex-tual embeddings, @xcite , (building on works by @xcite proposed a suite of classification tasks aimed at investigating how these representation encode sentence structure across a range of syntactic, semantic, local, and long-range phenomena. crucially, these authors report a very small improvement over non-contextual baselines. thematic role information seems to be recoverable with this strategy, but to an extent which is not that notable. ettinger (2020) tested bert on a suite of diagnostics drawn from human language experiments, among which the most relevant to our scopes is the semantic role sensitivity and event knowledge task, that tests the model ability to discern between good and bad sentence fillers on the basis on the required semantic role. results showed that the model is not that accurate at matching human predictions, even if some of the information appear to be encoded. @xcite developed a suite of probing tasks with the aim of assessing what kind of semantic information about the surrounding words is encoded in a contextual embedding. for instance, a task of this suite can implement the question "what does the embedding of the verb tell us about the animacy of the subject noun?" as a binary task for a mlp classifier trained and tested on the embeddings of a single word @xcite . relevant for our purposes, these scholars report that much information about subject's animacy can be recovered by inspecting the embedding of the verb. however, our work is more strongly related, both in goals and methods, to those by @xcite , utsumi (2020), @xcite and above all to @xcite . all these authors have used a linear transformation to learn a mapping between an embedding space and a space derived from human judgements. @xcite learned a mapping towards the short normalized descriptions (feature norm) collected by @xcite in order to learn to predict perceptual features for novel words. both utsumi (2020) and @xcite applied that strategy to decode word vectors in terms of the brain based semantic features collected in @xcite . finally, @xcite focused on finding fine-grained proto-role information by learning the mapping between several static embeddings and an entailment space based on the same ratings we use in this experiment, i.e. the white et al. @xcite dataset that is introduced in the next section. all in all, these works show that both contextual and static embeddings encode a wide range of (psycho)linguistic relevant information that can be inferred by means of a simple linear mapping. ## thematic proto-roles theories about the role played by the arguments of the verb at the syntax-semantics interface have come in many flavours. one of the most debated and controversial matter in theoretical linguistics concerns the definition of the notion of semantic role and the development of a reliable method for the identification of such categories. in contrast with the traditional view of semantic roles as discrete primitive semantic categories, david dowty proposed to reduce the total number of roles to two prototypical notions, which he called proto-agent and proto-patient @xcite . in dowty's view, roles are defined by a set of entailments determined by the meaning of the verb. some properties contribute to characterize an argument as proto-agent, while others as proto-patient, but they can be present in different degrees and in mixed configurations. these configurations correspond to classical intermediate roles such as experiencer, theme, and so on. this approach has some important advantages over classical views of semantic roles, which have always been in tension between the choice of the right number of roles and the right mapping between grammatical and semantic role. dowty's theory does not discard completely the possibility of identifying a different role for each different peculiar argument, placing a distinction between specific roles and linguistic roles. while the former are specific for each verb (e.g., to build has two main arguments: the builder and the buildee), the latter are generalizations aimed at capturing common traits about different specific roles. for example builder, killer, worker, seller are all instances of proto-agent, but all at a different degree. furthermore, concerning the selection of the argument, dowty sets a straightforward rule in his argument selection principle stating that: "in predicates with grammatical subject and object, the argument for which the predicate entails the greatest number of proto-agent properties will be lexicalized as subject of the predicate; the argument having the greatest number of proto-patient entailments will be lexicalized as the direct object." @xcite . this claim received empirical validation on a cognitive perspective by @xcite , among others. this scholar proved, through a series of experiments, not only that the hypothesis has psychological validity, but also that "speakers can make inferences about these properties from grammatical roles alone [...]" @xcite . inspired by these findings @xcite built a crowd-sourcing experiment to test the theory against a large amount of data and substantially confirmed the results by @xcite . the latter approach has been the precursor of the dataset by @xcite that we adopt here to build a semantic space based on human judgments. ## general methodology our ultimate goal is to probe the kind of distributional knowledge encoded in contextual embeddings in order to assess whether bert (and arguable other models of the same family) is able to encode proto-role semantic information. as such, we opted for a methodology that has been tested and proven in our reference literature @xcite @xcite . similarly to what has been done by @xcite , indeed, we created a linear mapping between a semantic space composed of bert embeddings and a vector space derived from the ratings collected in the white et al. ## conclusions although our strategy has been proven to be good at modeling the proto-roles phenomenon in bert embeddings to a certain extent, some intrinsic limitations of our work have to be taken into account. firstly, we used a linear regression (pls) model as a strategy to build the mapping, but, due to the complexity of the type of information enquired and that of the bert space, more complex, non-linear transformations, like a multi layer perceptron, might be a better choice for the task. secondly, the data we used are the best at our disposal, but they are not necessarily the best possible in absolute and might be further improved, by both revising the questions and the set of properties. thirdly, we obtained mixed results between the token and the type levels concerning which is the best modeled proto-role. notwithstanding these limitations, we have shown that fine-grained information about proto-roles properties of the arguments is recoverable inspecting the embeddings of the verbs yielded by bert. also, our results suggest that there might be a discrepancy between the properties of the two proto-roles and that proto-agent properties are better modeled and predicted. we have also been able to show how different proto-roles entailments can be predicted in verbs participating to the causativeinchoative alternation. additionally, we successfully employed spca to reduce the noise in our data, which might be a promising cue about future usages of this technique in the field of nlp. it is worth emphasizing that the main goal of this research is to test bert's ability to capture some crucial aspects of the verbal argument structure. even if there can be practical applications of our method (e.g., it can be used as a starting point for semantic role labeling or, crucially, semantic proto-role labeling; @xcite , our main interest is more theoretical and methodological. many of the probing tasks that are used today, indeed, do not focus on the prototypical nature of semantic roles, which is precisely a fundamental pillar and the major innovation of dowty's theory and of the present work. moreover, our analysis of the causative/inchoative alternation is just a first example of a series of tasks that we plan to develop to characterize the knowledge acquired by these models to explore key aspects of the syntax-semantic interface and of verb argument realization. finally, we will also extend our approach to other contextual embeddings models, like gpt @xcite and xlnet @xcite .