Leveraging a New S panish Corpus for Multilingual and Cross-lingual Metaphor Detection (2022)

Abstract: The lack of wide coverage datasets annotated with everyday metaphorical expressions for languages other than English is striking. This means that most research on supervised metaphor detection has been published only for that language. In order to address this issue, this work presents the first corpus annotated with naturally occurring metaphors in Spanish large enough to develop systems to perform metaphor detection. The presented dataset, CoMeta, includes texts from various domains, namely, news, political discourse, Wikipedia and reviews. In order to label CoMeta, we apply the MIPVU method, the guidelines most commonly used to systematically annotate metaphor on real data. We use our newly created dataset to provide competitive baselines by fine-tuning several multilingual and monolingual state-of-the-art large language models. Furthermore, by leveraging the existing VUAM English data in addition to CoMeta, we present the, to the best of our knowledge, first cross-lingual experiments on supervised metaphor detection. Finally, we perform a detailed error analysis that explores the seemingly high transfer of everyday metaphor across these two languages and datasets.

## introduction metaphor can broadly be defined as the interpretation of a concept belonging to one domain in terms of another concept from a different domain @xcite . metaphorical expressions are recurrent in natural language as a mechanism to convey abstract ideas through specific experiences related to the real, physical world or to send a stronger message in a discourse. there is a large body of work from various fields such as linguistics, psychology or philosophy that tried to provide a theoretical characterization of metaphor. some approaches are based on the semantic similarity shared between the domains involved @xcite , while others explain metaphorical uses of language in terms of violation of selectional preferences @xcite . other perspectives focus on the communicative impact of using a metaphorical expression in contrast to its literal counterpart @xcite . following previous work on metaphor detection in natural language processing (nlp) @xcite , our approach is based on the conceptual metaphor theory of @xcite . they do not conceive metaphors just as a cognitive-linguistic phenomenon commonly used in our everyday utterances. instead, metaphors are understood as a conceptual mapping that typically reshapes an entire abstract domain of experience (target) in terms of a different concrete domain (source). the high frequency of metaphors in everyday language has increased the popularity of research for this type of figurative language in the nlp field. one of the reasons behind is the fact that the automatic processing of metaphors is essential to achieve a successful interaction between humans and machines. in this sense, it is considered that other nlp tasks performance could benefit from metaphor processing, such as machine translation @xcite , sentiment analysis @xcite , textual entailment @xcite or hate speech detection @xcite , among others. however, the large majority of research on metaphor detection has been done for english, for which the public release of the vuam dataset within the figlang shared tasks from 2018 and 2020 marked a major milestone @xcite . in this paper we would like to contribute to research in multilingual and cross-lingual metaphor detection by presenting a new wide coverage dataset in spanish with annotations for everyday metaphorical expressions. in this context, the contributions of this work are the following: (i) a new publicly available dataset for metaphor detection in spanish from a variety of domains, cometa; (ii) an in-depth discussion of problematic cases and of adapting the mipvu method to annotate metaphor in spanish; (iii) a quantitative and qualitative analysis of the resulting cometa corpus; (iv) competitive baselines using 18 large monolingual and multilingual language models in monolingual and cross-lingual evaluation settings, showing that modern language models such as deberta @xcite perform similarly to models specifically trained for metaphor processing like melbert @xcite ; (v) error analysis shows that, for these languages and datasets, cross-lingual metaphor transfer is very high, mostly due to the presence of metaphorical usage of commonly used verbs; (vi) the cometa dataset, code and fine-tuned models are publicly available 1 to encourage research in multilingual and cross-lingual metaphor detection and to facilitate reproducibility of results. ## previous work metaphorical expressions can be conveyed through multiple linguistic structures and can be classified according to different criteria @xcite . a common distinction is that of conventional metaphors (as in example (1)), highly extended among speakers and lexicalized, and novel metaphors (example (2)), which are less frequent in everyday utterances (examples taken from rai and chakraverty (2020)). most work on metaphor detection has focused on english texts. the vu amsterdam metaphor corpus (vuamc or vuam) @xcite is the most extensive dataset with annotations for the characterization of linguistic metaphor. it consists of english texts labeled with several typologies of metaphor following the the vu metaphor identification procedure (mipvu), discussed in section 3.2.1. it was subsequently adapted to other languages @xcite . however, spanish was not included and, for the languages that were, this adaptation did not include the development of annotated corpora. first attempts to tackle metaphor detection in english were corpus-based (charteris @xcite @xcite . most recent approaches address the task as sequence labeling usually based on deep learning, neural networks and word embeddings @xcite . in addition, syntactic and semantic features (wordnet, framenet, verbnet, dependency analysis, morphology, etc.) are exploited in order to boost the performance of such models. the celebration of the 2018 and 2020 shared tasks @xcite around the detection of metaphors using the vuam dataset contributed to a huge jump in development and performance, although top results were achieved by classifying mostly conventional metaphors @xcite . others combine metaphor theories as features in addition to annotated data to feed pre-trained models based on the transformers architecture @xcite . for instance, the state-of-theart system melbert @xcite uses the metaphor identification procedure (mip) (pragglejaz, 2007) and selectional preferences @xcite @xcite . these theories argue that terms with matching semantic features tend to appear in the same context, and metaphors usually do not comply with this hypothesis. furthermore, the recently published model of miss roberta wilde @xcite benefits from dictionary definitions as an additional feature to train their model based on the architecture of melbert. due to the lack of labeled data to train supervised models, previous work addressing spanish metaphor processing has mainly been based on unsupervised approaches. however, as it is often the case for many other nlp tasks, unsupervised approaches obtain far lower results than supervised methods @xcite . other issues are that most work in spanish has focused either on a very specific type of conceptual metaphor @xcite , or on the characterization of metaphor in very domainspecific data @xcite . the development of cometa aims to compensate this lack of resources for the spanish language. to the best of our knowledge, it constitutes the largest dataset of general domain texts with metaphorical annotations in spanish that, despite not reaching the size of the vuamc corpus, can be used as a starting point, suitable to be extended and improved in the future, to further advance multilingual and cross-lingual methods for metaphor detection. ## dataset development in the following subsections we detail the creation process of our dataset cometa, including the data collection and annotation. ## evaluation in this section we present the experiments on metaphor detection in spanish and english. furthermore, we also report the results of the first supervised cross-lingual experiments for metaphor detection. the main objective of the cross-lingual evaluation setting was to examine which kind of metaphors carried more often across languages. ## conclusions and future work in this work we have created cometa, which to the best of our knowledge is the largest dataset with metaphor annotations in spanish composed of texts from various domains to be publicly available. we also discussed in detail the main issues that emerged during the annotation process for spanish. in order to evaluate the quality of cometa's annotations we carried out a series of experiments in both monolingual and cross-lingual environments, using the largest dataset with metaphor annotations in english, the vuam corpus, as reference point.. moreover, we set a new state of the art on the task of metaphor detection in english and set a strong baseline for the task in spanish, which hopefully will encourage researchers to continue with this line of work. the aim of this work is to lay the foundations for future development on metaphor detection in spanish and cross-lingually. regarding the dataset, a future line of work would introduce more finegrained tags that represent the different kinds of metaphorical expressions. this task should be performed by multiple annotators, in order to explore agreement over the whole dataset, as well as to observe if doubtful cases share any feature that could be leveraged for their identification. the presence of more fine-grained tags would also enable a deeper statistical analysis of cometa that could be exploited to study how metaphor manifests in spanish and whether there are similarities with the usage of metaphor in other languages. results obtained from our experiments encourage future research to continue with cross-lingual approaches. we hypothesize that these results may be due to the difference in size of the training data in both languages or the application of mipvu guidelines to spanish, which is not the language it was originally designed for. future experimental work is needed to test these interpretations, which could benefit from the extension of the annotations in cometa we just mentioned.