Comparing Probabilistic, Distributional and Transformer-Based Models on Logical Metonymy Interpretation (2020)

Abstract: In linguistics and cognitive science, Logical metonymies are defined as type clashes between an event-selecting verb and an entity-denoting noun (e.g. The editor finished the article), which are typically interpreted by inferring a hidden event (e.g. reading) on the basis of contextual cues. This paper tackles the problem of logical metonymy interpretation, that is, the retrieval of the covert event via computational methods. We compare different types of models, including the probabilistic and the distributional ones previously introduced in the literature on the topic. For the first time, we also tested on this task some of the recent Transformer-based models, such as BERT, RoBERTa, XLNet, and GPT-2. Our results show a complex scenario, in which the best Transformer-based models and some traditional distributional models perform very similarly. However, the low performance on some of the testing datasets suggests that logical metonymy is still a challenging phenomenon for computational modeling.

## introduction the phenomenon of logical metonymy is defined as a type clash between an event-selecting metonymic verb (e.g., begin) and an entity-denoting nominal object (e.g., the book), which triggers the recovery of a hidden event (e.g., reading). logical metonymies have been widely studied, on the one hand, in theoretical linguistics as they represent a challenge to traditional theories of compositionality @xcite . on the other hand, they received extensive attention in cognitive research on human sentence processing as they determine extra processing costs during online sentence comprehension @xcite , apparently related to "the deployment of operations to construct a semantic representation of the event" @xcite . 1 logical metonymy has also been explained in terms of the words-as-cues hypothesis proposed by jeffrey elman @xcite . this hypothesis relies on the experimental evidence that human semantic memory stores knowledge about events and their typical participants (see @xcite for an overview) and claims that words act like cues to access event knowledge, incrementally modulating sentence comprehension. the results obtained in a probe recognition experiment by @xcite , in line with this explanation, suggest that speakers interpret logical metonymies by inferring the most likely event the sentences could refer to, given the contextual cues. previous research in nlp on logical metonymy has often been influenced by such theoretical explanation @xcite @xcite . in our contribution, we propose a general comparison of different classes of computational models for logical metonymy. to begin with, we tested two approaches that have been previously introduced in the literature on the topic: probabilistic and distributional models @xcite . we also examined the structured distributional model (sdm) by @xcite , which represents sentence meaning with a combination of formal structures and distributional embeddings to dynamically integrate knowledge about events and their typical participants, as they are activated by lexical items. finally, to the best of our knowledge, we are the first ones to include the recent transformer language models into a contrastive study on logical metonymy. transformers @xcite are the dominant class of nlp systems in the last few years, since they are able to generate "dynamic" representations for a target word depending on the sentence context. as the interpretation of logical metonymy is highly sensitive to context, we deem that the contextual representations built by transformers might be able to integrate the covert event that is missing in the surface form of the sentence. all models are evaluated on their capability of assigning the correct interpretation to a metonymic sentence, that is, recovering the verb that refers to the correct interpretation. this task is hard for computational models, as they must exploit contextual cues to distinguish covert events with a high typicality (e.g., the pianist begins the @xmath0. ## evaluation results regarding the thematic fit models, we observe that there is no difference between the tf-add and tf-prod models, as they obtain similar scores. 6 one of the anonymous reviewers argues that the performance of the transformer-based models might be influenced by the prompt sentence and suggest more variations of the input sentences. we indeed tested several manipulations of the inputs before feeding them to the transformers, changing 1) the tense of the metonymic verb (using the past tense) and 2) the number of the direct object (we used the plurals of the dataset nouns). however, the results did not show any consistent trend. however, we need to point out that, when the system computes the intersection of the two lists of the top verbs for subjects and objects, sometimes the number of retrieved items is less than 20 (the model parameter for the verb embedding selection, cf. section 3.3.2). therefore, independently of the selected function, the verbs used to compute the prototypical vector are eventually all those belonging to the intersection. moreover, tf-models are often close to, and never significantly outperform the probabilistic baseline. among the distributional models, sdm is the one that obtains a considerable performance across all the datasets. this model performs close to roberta both in the traxler and in the ce dataset. this result is surprising, considering that sdm is trained just on a dump of wikipedia, while roberta is trained on 160 gb of text and implements advanced deep learning techniques. this outcome confirms that sdm, which has been designed to represent event knowledge and the dynamic construction of sentence meaning, is able to adequately model the typicality of events. this aspect has been suggested to be one of the core components of the language processing system @xcite @xcite . on the other hand, transformers also provided interesting results. roberta achieves the best score for the l&l dataset, reaching a statistical significance of the improvement over sdm (@xmath1.01). 9 more importantly, it is the only transformer that consistently obtains good results across all datasets, while the scores from other transformer models are highly fluctuating. we believe that the gigantic size of the training corpus is a factor that positively affects its performance. at the same time, gpt-2 achieves the highest score for mc dataset (0.87) (but the improvement over roberta and sdm does not reach statistical significance), although it performs significantly lower on the other benchmarks 10 . for the sake of completeness, we also report the overall performance of each model over the two tasks. results identify roberta and gpt-2 as the best models for the correlation and classification tasks, respectively. however, we wonder if the average score is a valid measure to identify the best model. these two models tend to have a wavering behavior, which results in large differences between the two datasets scores. specifically, roberta achieves 0.75 for the l&l dataset, but only 0.39 for the ce one, with 0.36 points of difference. similarly, gpt-2 reaches 0.89 scores for the mc dataset, but its performance goes down by 0.16. on the contrary, sdm behavior is more stable, with a smaller gap between the two datasets' scores (0.13 point difference for the correlation task and just 0.05 for the accuracy task). ## discussion and conclusions in this paper, we have presented a comparative evaluation of several computational models on the task of logical metonymy interpretation. we frame this problem as the retrieval of an event that is not overtly expressed in the surface form of the sentence. according to elman's words-as-cues framework, human subjects can infer the covert event in logical metonymy thanks to the generalized knowledge about events and participants stored in their semantic memory. hence, during sentence processing, words in the sentence create a network of mutual expectations that triggers the retrieval of typical events associated with lexical items and gen-erates expectations about the upcoming words (elman, 2014). to tackle the task of logical metonymy interpretation, computational models must be able to recover unexpressed relationships between the words, using a context-sensitive representation of meaning that captures this event knowledge. the most compelling outcome of the reported experiments is probably the performance of sdm, which achieves the best score for the tr and the ce datasets. these results demonstrate the significance of encoding event structures outside the embeddings (which are treated as nodes in a distributional graph), and the ability of the sdm compositional function to dynamically update the semantic representation for a sentence. however, the evaluation scores are not very high, especially in the correlation task. results reveal that the contextualized information used by computational models is useful to recall plausible events connected to the arguments, but this is still not sufficient. even transformer models, which currently report state-of-theart performances on several nlp benchmarks, are not performing significantly better than the sdm model, which is trained on a smaller corpus and without any advanced deep learning technique. error analysis highlights that they are able to identify the plausible scenarios in which the participants could occur, but they still struggle in perceiving different nuances of typicality. our experiments show how the logical metonymy task can be seen as a testing ground to check whether computational models encode common-sense event knowledge. future work might follow two directions. on the one hand, expanding the coverage of the graph could favourably increase the performance of sdm. on the other hand, transformer models could be tested with new experimental settings, such as the fine-tuning of the pre-trained weights on thematic fit-related @xcite @xcite or semantic role classification tasks @xcite @xcite .