Evaluating the Development of Linguistic Metaphor Annotation in M exican S panish Popular Science Tweets (2024)

Abstract: Following previous work on metaphor annotation and automatic metaphor processing, this study presents the evaluation of an initial phase in the novel area of linguistic metaphor detection in Mexican Spanish popular science tweets. Specifically, we examine the challenges posed by the annotation process stemming from disagreement among annotators. During this phase of our work, we conducted the annotation of a corpus comprising 3733 Mexican Spanish popular science tweets. This corpus was divided into two halves and each half was then assigned to two different pairs of native Mexican Spanish-speaking annotators. Despite rigorous methodology and continuous training, inter-annotator agreement as measured by Cohen’s kappa was found to be low, slightly above chance levels, although the concordance percentage exceeded 60%. By elucidating the inherent complexity of metaphor annotation tasks, our evaluation emphasizes the implications of these findings and offers insights for future research in this field, with the aim of creating a robust dataset for machine learning.

## introduction computational approaches to metaphor date back at least to the 1980s, when artificial intelligence (ai) and natural language processing (nlp) became interested in the structure and mechanisms of the phenomenon @xcite . since then, there has been growing interest among researchers in understanding how computers can effectively process both linguistic and nonlinguistic metaphors. an instance of this progressive work has been the various workshops developed within the acl, the naacl and the emnlp, since 2007, on metaphor, in particular, and on figurative language, in general. broadly speaking, automatic metaphor processing has branched into three fundamental areas: metaphor identification or detection, metaphor interpretation, and metaphor generation @xcite . usually regarded as the 'first step', metaphor identification aims to automatically recognize linguistic expressions that convey metaphorical meaning within a text. for this task, supervised machine learning techniques trained on annotated datasets are often used to distinguish linguistic patterns indicative of metaphor. however, despite recent advances in figurative language processing (flp) focused on metaphor processing for english, the situation for the spanish language is quite different. although there are tools and models developed for automatic metaphor processing tasks in english, the same level of development and availability has not been reached for spanish. more precisely, our literature review has revealed a substantial gap regarding nlp approaches to metaphor in mexican spanish tweets within the realm of science communication. this represents a novel and unexplored area of research, where the intersection of metaphorical language and science popularization discourse in the context of mexican spanish on x (previously twitter) remains a largely unexplored territory. this study has the objective of analyzing the usage of linguistic metaphors through nlp techniques to provide an overview of metaphor identification and classification within short scientific communication posts on x in mexico. ## related work research and advances in automatic metaphor processing in spanish remain scarce to this day. specifically, if we focus solely on annotated corpora approaches for training supervised machine learning models, we have limited resources available. so far, the only publicly available annotated dataset on spanish linguistic metaphors is the corpus for metaphor detection in spanish (cometa) @xcite . this linguistic dataset represents the first documented effort to compile a collection of general domain texts for everyday metaphor detection in spanish. cometa also marks the first adaptation of the mipvu guidelines to this romance language, although during our literature review, it has not been possible to find the annotation guidelines used for the cometa. for english, besides specifically trained models for metaphor processing such as melbert @xcite and miss roberta wilde @xcite , the work of @xcite is remarkable, since it focuses on the generation of scientific metaphors. using gpt-3 as a base model, these authors developed metaphorian, a system that assists science writers in the creation of scientific metaphors. the metaphorian system allows users to search for, add and modify scientific metaphors, which is a valuable creative assistance tool for formulating difficult-to-explain scientific concepts in terms of more familiar concepts. ## corpus annotation it is important to clarify that the primary subject of this research is linguistic metaphor annotation, according to the theoretical-methodological foundation of mipvu, rather than conceptual metaphor analysis. nonetheless, we have resorted to some cmt notions in the annotation guide, similar to the approach used by @xcite , for didactic purposes in explaining metaphors to the annotators. moreover, given our selection of popular science as the genre of interest, our annotation focuses on identifying both scientific metaphors and everyday or colloquial metaphors in the corpus, which is appropriate as these texts bridge the specialized realm of science and the colloquial domain of language. in our annotation protocols, we center on identifying three types of linguistic metaphor across popular science tweets: direct (dm), indirect (im) and personification (pm). we define dm as an explicit comparison between the source domain and the target domain, characterized by three units: the source unit (label: 'md_fuente'), the target unit (label: 'md_meta) and the signal or cue (label: 'md_señal'). im is understood as an implicit comparison between the source domain and the target domain, consisting of only one unit -the source unit (label: 'md_indirecta') -since the target unit is elided. finally, we explain pm as the attribution of human or animate semantic features (label: 'personificador') to an inanimate or abstract object (label: 'pers_obj'). as far as we know, this is the only public effort to annotate linguistic metaphors specifically in mexican spanish. both the original guidelines in spanish and the english translation are accessible in our github repository. following this, we have annotated a corpus of 3733 popular science communication tweets. this dataset comprises mexican spanish tweets from 19 science communicators on x based in mexico, which were published from january 2020 to may 2023 and extracted with the x api. 1 it should be emphasized that the information on these user accounts was collected without specific preferences for a particular scientific domain, which led to a wide topic range in the corpus, from astronomy and general physics to genetics and history of science, among other areas. we gathered a group of 4 native mexican spanish-speaking annotators to conduct an initial annotation of the entire corpus. these annotators are all undergraduate linguistics students, aged between 18 and 25, 1 female and 3 male. to enhance annotation, we opted for the argilla platform as it supports token classification tasks on loaded datasets in spanish. subsequently, we divided the corpus into two halves, and assigned each half to a pair of annotators (1866 and 1867 tweets respectively), ensuring balanced coverage and consistency in the annotation process. this approach allowed us to efficiently distribute the workload while maintaining a rigorous and systematic approach to linguistic metaphor annotation. we trained this group of annotators to apply 6 labels corresponding to the 3 metaphor types. of 1 after its acquisition by elon musk, twitter was renamed 'x' and the texts published on it became known as 'posts'. however, since at the time of data collection, this platform was called twitter and its texts 'tweets', we have decided to preserve said term for this paper.