Don’t Invite BERT to Drink a Bottle: Modeling the Interpretation of Metonymies Using BERT and Distributional Representations (2020)

Abstract: In this work, we carry out two experiments in order to assess the ability of BERT to capture the meaning shift associated with metonymic expressions. We test the model on a new dataset that is representative of the most common types of metonymy. We compare BERT with the Structured Distributional Model (SDM), a model for the representation of words in context which is based on the notion of Generalized Event Knowledge. The results reveal that, while BERT ability to deal with metonymy is quite limited, SDM is good at predicting the meaning of metonymic expressions, providing support for an account of metonymy based on event knowledge.

## introduction metonymy is one of the most important sources of lexical polysemy and consists in the meaning shift of a noun that is used to refer to another entity to which it is related @xcite . for instance, bottle refers to a solid container in (1a), but in (1b) it stands for some liquid contained in it: metonymy is a productive and systematic process (e.g., all nouns denoting containers show the same polysemy as bottle, giving rise to the so-called container-for-content metonymic alternation). therefore, both linguistic @xcite @xcite and psycholinguistic @xcite studies contest the treatment of metonymy like a case of lexical ambiguity, and instead support the hypothesis that metonymic interpretations result from the inherently dynamic and generative nature of lexical representations that can acquire new meanings by integrating information activated by the textual and extralinguistic context. vector representations (aka word embeddings) produced by distributional semantic models (dsms) are particularly suitable for modeling contextual semantic effects, due to their "gradedness" and their dependence on the linguistic contexts @xcite . traditional dsms represent the content of lexical types through a single vector that "summarizes" their whole distributional history. things have recently changed with the introduction of deep neural architectures for language modeling like bert @xcite , whose word representations have helped achieving state-of-the-art results in a wide variety of supervised nlp tasks. these embeddings are intrinsically contextualized, in the sense that the model computes a different vector for each token occurrence of the same word, depending on the sentence in which the token appears. in this work, we test whether bert contextualized embeddings can be used to model the meaning shifts associated with metonymic uses of words. given its pervasiveness in everyday communication, we suggest that the extent to which metonymy is captured by bert is an important testbed to evaluate its actual ability to model natural language. in line with this goal, we require the model to induce the additional meaning of metonymic expressions by encoding it into the contextualized embeddings. we also compare bert performance with that achieved by the structured distributional model by @xcite , in which the context-sensitive nature of lexical meaning is instead captured by integrating a rich array of distributional knowledge about events and their typical participants @xcite . ## related work over the years, various methods to obtain contextualized representations of word meaning have been developed in different fields. research in distributional semantics @xcite has taken non-contextual representations of words as starting point from which contextualized vectors capable of modeling various types of meaning alternations are derived @xcite . however, these models have never been used to predict metonymic semantic shifts. lately, transformer language models (e.g., bert, gpt2, etc.) have stormed ai and nlp with a new generation of word embeddings that are expected to capture lexical meaning variation in context @xcite . in particular, the representations produced by bert @xcite have been used to create high performing models for many language understanding tasks, although their status as a linguistically sound model of meaning is debated @xcite . @xcite test bert on several cases of figurative language, but to the best of our knowledge bert ability to identify metonymy has never been addressed yet. ## models in bert, the embedding of a word is modified with contextual information through the self-attention mechanism of transformers @xcite . as is well known, bert is trained on two tasks: predicting randomly masked tokens (masked language model) and determining whether a sentence follows another sentence in a dataset (next sentence prediction). since our intent is to assess the model ability to understand metonymic meanings, we test the contextual embeddings themselves (bert-emb), rather than fine-tuning them in a supervised classification task (cf. @xcite for a similar approach). we also investigate if this ability is reflected in the probabilities that the model assigns to the masked metonymic word, thereby exploiting bert as a language model (bert-lm). the model against which we evaluate bert is inspired by the structured distributional model by @xcite , which is based on the notion of generalized event knowledge (gek; @xcite . gek is conceptual knowledge about real-world events and their participants, which has been shown to influence sentence processing by causing expectations regarding the upcoming input. for example, when the first part of a sentence starting with the words the police arrested is processed, expectations about the possible objects are generated based on knowledge about the typical patients of the event arrest (e.g., thief, burglar, etc.). since gek becomes activated quickly, it has been argued that the meaning assigned to words in context comes as a result of the interaction between lexical meaning and the expectations generated @xcite . the model presented here uses the graph-based distributional model of event knowledge introduced in @xcite to compute a contextualized representation of word meaning, which is obtained by integrating the lexical embedding of a word with a vector representation of the expectations activated by the context for the event role of the word. as in the model of @xcite , we approximate the expectations activated by a word w by selecting the words with the highest pointwise mutual information (pmi) with w in a corpus. 1 we use a parsed corpus to extract different expectations according to their syntactic roles, as a surface approximation of semantic roles (e.g., typical patients are derived from the verb direct objects). for example, given the sentence the guest tasted the bottle, we consider the most typical objects of w (the verb taste), which provide an approximation of the typical patients of the event expressed by the word. we take the direct objects since this is the function of the metonymic word w m in the sentence (i.e., the noun bottle). a key feature of the model is that the activated words are filtered according to their pmi association strength with the metonymic word w m . in our example, words for foods (e.g., fruit, meat, etc.) and drinks (e.g., wine, beer, etc.) are activated by taste, but the former are discarded because they have low pmi values with bottle. this process, which is an original innovation with respect to the chersoni et al. model, simulates the interaction between lexical information and active expectations as described by @xcite , and at the same time reproduces the associative processes involved in metonymy interpretation by updating the salience of expectations based on their relation with the metonymic word. finally, we calculate the centroid of the activated expectation vectors and the embedding of w m to obtain its contextualized representation. let w be the k words with the highest pmi with the verb w and w m the n words in w with the highest pmi with the metonymic word w m (for all experiments, we set @xmath0. the contextualized representation of the metonymic word -→ w m is built by summing the lexical vectors of the words in w m and the metonymic word w m . each element of the resulting vector is then divided by the number of words used for the creation of the vector (equivalent to n+1). this procedure extends the notion of mean to a vector space to produce context-adapted representations of word meaning which are comparable with lexical vectors. finally, -→ w m is defined as follows: