Construction Artifacts in Metaphor Identification Datasets (2023)

Abstract: Metaphor identification aims at understanding whether a given expression is used figuratively in context. However, in this paper we show how existing metaphor identification datasets can be gamed by fully ignoring the potential metaphorical expression or the context in which it occurs. We test this hypothesis in a variety of datasets and settings, and show that metaphor identification systems based on language models without complete information can be competitive with those using the full context. This is due to the construction procedures to build such datasets, which introduce unwanted biases for positive and negative classes. Finally, we test the same hypothesis on datasets that are carefully sampled from natural corpora and where this bias is not present, making these datasets more challenging and reliable.

## introduction the automatic identification of metaphors in corpora is an active area of research in nlp @xcite @xcite @xcite , and consequently, previous works have proposed the construction, curation and testing of metaphor indentification datasets in various forms @xcite , also stemming from previous psycholinguistic research @xcite . a common feature of metaphor identification tasks is that they are usually framed as binary classification, where a potentially metaphorical expression (pme) and a context are provided as input, and a system has to determine whether the given pme is used metaphorically or not. for instance, dark is used metaphorically in the sentence 'the latest developments move us closer to a dark age', but not in 'i like dark colors'. while this setting is attractive for testing supervised systems, it also simplifies the task, introducing a real risk of not testing metaphoricity, but instead spurious correlations that might lead to an artificially correct solutions. analyses of this sort have often proven critical in understanding the relationship between a proposed task and whether performance on associated datasets can be directly linked to performance on the task itself. notable examples include the work of @xcite , who showed that, in the task of lexical relation modeling, the unreasonably high performance of supervised systems could be attributed to lexical memorization, that is, the presence of large number of prototypical cases in the dataset (e.g. animal for the hypernymy relation) that made it simple for models to "detect the relation" without having to consider both words in the pair. this issue has been identified also in word analogies @xcite @xcite and, beyond lexical semantics, in natural language inference (nli). in nli, given a hypothesis and a premise, a system must determine whether the premise entails, contradicts or is neutral with respect to the hypothesis. previous works have shown that supervised models could rely on superficial factors, e.g., in the snli dataset @xcite , hypothesis-only models are surprisingly competitive @xcite , a trend also observed in medical nli datasets @xcite . in this paper, we report experimental results which suggest that (1) language models can identify metaphorical expressions with great accuracy without even seeing the given expression; and that (2) a model only seeing the metaphorical expression with no context performs very competitively, in both cases close to the model with complete information. a crucial distinction in this phenomenon, however, is that this is only observed in datasets not sampled from a natural distribution, and that whenever such distribution is preserved, these shortcuts are not as effective. we thus propose a simple sampling procedure from a naturally-distributed corpus that mitigates this issue. ## metaphor identification datasets this section presents the list of existing englishlanguage metaphor identification datasets used in our experiments. all the datasets are summarized in ## evaluation the evaluation is aimed at understanding to what extent metaphor identification datasets are affected by construction or sampling biases. ## conclusion getting inspiration from previous studies analysing artifacts and biases from nlp datasets, we delved into the field of metaphor identification. by proposing baselines with incomplete information that hide the pme or the context, we show how the performance achieved by a supervised model is closed to the model that uses complete information. this highlights the type of bias that a model is picking up at training time, which differ from what a well-trained model would be expected to learn. finally, we show that this problem is generally observed in datasets that are artificially constructed, as carefully sampling from datasets (vuac_bo) stemming from a fully-annotated corpus alleviates this issue.