Cross-Lingual Metaphor Detection for Low-Resource Languages (2024)

Abstract: Research on metaphor detection (MD) in a multilingual setup has recently gained momentum. As for many tasks, it is however unclear how the amount of data used to pretrain large language models affects the performance, and whether non-neural models might provide a reasonable alternative, especially for MD in low-resource languages. This paper compares neural and non-neural cross-lingual models for English as the source language and Russian, German and Latin as target languages. In a series of experiments we show that the neural cross-lingual adapter architecture MAD-X performs best across target languages. Zero-shot classification with mBERT achieves decent results above the majority baseline, while few-shot classification with mBERT heavily depends on shot-selection, which is inconvenient in a cross-lingual setup where no validation data for the target language exists. The non-neural model, a random forest classifier with conceptual features, is outperformed by the neural models. Overall, we recommend MAD-X for metaphor detection not only in high-resource but also in low-resource scenarios regarding the amounts of pretraining data for mBERT.

## introduction song titles such as life is a highway are prominent examples of how we use metaphors in our everyday life. but songs are by far not their only habitats: on average and across domains, metaphors can be found in every third sentence @xcite . @xcite define a conceptual metaphor as "understanding one conceptual domain @xcite in terms of another conceptual domain @xcite " @xcite . in the above example, the domain life (a) is understood in terms of the domain journey (b). detecting whether or not a word or expression is a metaphorical linguistic expression (i.e. whether or not it is used metaphorically) is vital for many nlp applications, such as sentiment analysis, machine translation, information extraction, and dialog systems, cf. @xcite . metaphor detection (md) can further support automatic essay scoring @xcite , schizophrenia detection @xcite , and propaganda identification @xcite . many efforts have been made to tackle the task of metaphor detection (md), foot_0 and successfully so: close-to-human performance was reached by systems using large pretrained language models like bert @xcite for english datasets containing single sentences with a metaphorical expression @xcite . for a long time, @xcite were the only ones to perform md cross-lingually, namely for spanish, russian and farsi. only recently, @xcite and @xcite addressed metaphor detection in a multilingual setup with the same languages as @xcite . whereas @xcite focused on probing metaphoricity within the transformer layers, @xcite used a template-based prompt learning approach to md. these multilingual md approaches focus on languages where large amounts of data are available for pretraining. insights are missing, however, on whether or not large language models are also suitable for md in languages with small amounts of pretraining data. the current study addresses this bottleneck and compares neural and non-neural cross-lingual models for detecting metaphors in languages with varying degrees of pretraining data, including the lowresource language latin. our metaphor detection focuses on word-based classification, as in the following example from the metaphor dataset by @xcite : actions talk even louder than phrases. language # wikipedia articles english (source) ≈ 6.7m german ≈ 2.8m russian ≈ 1.9m latin ≈ 0.1m we define a binary classification task to detect whether or not the underlined target word is used metaphorically in the given context. for zero-and few-shot classification we apply multilingual bert (mbert) @xcite and the adaptation method mad-x @xcite , which have shown state-of-the-art results for e.g. named entity recognition and question answering. as our non-neural model, we apply a random forest classifier @xcite , as random forest classifiers generally perform well in low-resource scenarios @xcite . our model utilizes a vector space model and conceptual features (abstractness and supersenses) -similarly to the model introduced by @xcite . as for target languages, we investigate modelling performances for german, russian and latin, because the amount of data used to pretrain mbert varies greatly across these three languages (see contributions. the main contribution of this paper is a comparison and a series of insights regarding cross-lingual neural and non-neural models for md in languages with high-to-low degrees of pretraining data, i.e., german, russian and latin. more specifically, 1) we find that with default hyperparameters, zero-shot mbert performs best: results are above a majority vote baseline for all three target languages. 2) mad-x performs best when hyperparameter-tuning is carried out or large amounts of source language training data are used. the idea of "conceptual features" also inspired @xcite , who used abstractness scores, imageability scores and semantic supersenses as classification features. whereas @xcite focused on english data only, @xcite trained on english data and then evaluated the model cross-lingually on spanish, farsi and russian. their model represents the basis for the random forest classifier used in our experiments. @xcite focused on md for german particle verbs. they also used 1) abstractness and imageability ratings as well as 2) scores indicating the distributional fit of particle verbs with regard to base verb contexts. in addition, they used 3) unigram context words and 4) noun clusters as features. do @xcite were the first to use a neural model architecture for md, namely a multilayer perceptron with word embeddings. their approach performed comparable to existing models without requiring feature engineering. @xcite explored the relationship between metaphors and emotions by building several multi-task learning models. the best performing architecture made use of bert embeddings used as input to a multilayer perceptron or to additional attention layers. they reached state-of-the-art results in 2019 for both metaphor and vad prediction. @xcite transformed word-based metaphor detection into a reading comprehension problem; their approach, deepmet, was the most successful model in the 2020 metaphor detection shared task @xcite . @xcite fine-tuned bert for md. to perform word-based binary metaphor classification, they copied the input sentence and masked the target word. the original sentence and the masked copy were used as input for a sequence classification task. the bert model then predicted whether the two sentences appeared in the same context; if yes, they predicted a literal usage of the masked word; otherwise they predicted a metaphorical usage. they also performed sentence-level classification and sequential labelling of metaphorical expressions. their results showed an increase over previous state-of-the-art models. we use their word-based classification approach for the mbert-based classifiers in our experiments. while their focus was on english, we use it in a multilingual setup. @xcite exploited the fact that many datasets are based on the metaphor identification process (mip; pragglejaz group, 2007), where a word is annotated as metaphorical if its contextual meaning is dissimilar to its "more basic meaning" (among further criteria). while prior models (such as melbert by @xcite ) grounded on mip use decontextualized representations of the target word, @xcite successfully gathered the representation of the target word from sentences where it was used literally. cross-lingual representations. @xcite proposed a bootstrapping method to create bilingual vector spaces from non-parallel data. usually, a high-dimensional vector in a feature vector space uses context features as dimensions. for the proposed bilingual vector space, these features consisted of translation pairs. this method can be applied to any language pair. multilingual bert @xcite was pretrained on data from 104 languages. @xcite pointed out limitations of large multilingual pretrained language models by demonstrating that these models do not transfer knowledge well for low-resource target languages (i.e. languages with small pretraining corpora) and for distant language pairs. they showed that first fine-tuning on large amounts of data and then continuing fine-tuning with very few examples from the target language considerably improves results across all languages and tasks. the current paper investigates whether these findings also apply to md. @xcite tried to mitigate problems of multilingual language models targeting low-resource languages by using an adaptation method, i.e. by inserting small amounts of trainable weights into an existing pretrained model (see section 4). we also apply these multiple adapters for cross-lingual transfer (mad-x) to md in our experiments. ## datasets and preprocessing source language. we used the dataset from @xcite as our basic english training dataset. it is based on the tenten foot_2 web corpus, contains 222 instances, and is balanced. this basic training dataset was previously used by @xcite for evaluation. in the course of our experiments we augmented the amount of training data by adding the imbalanced dataset by @xcite , which consists of 1,639 instances. the augmented version comprises 1,861 instances. 4 target languages. @xcite also provide the russian dataset that we used for evaluation, which is balanced, consists of 240 instances, and is based on the tenten web corpus. for evaluation in german, we used the md dataset provided by @xcite , which is based on the web corpus decow14ax @xcite and where the target words are particle verbs. to balance the dataset, we reduced the original dataset from @xcite to 896 metaphorical and 896 literal instances. for our latin dataset we used the lexham figurative language of the new testament dataset @xcite , which is published in the logos foot_4 bible software. it shows passages from the new testament (we used the american standard version of the bible), and highlights the metaphors in each verse. we extracted 100 sentences, of which 50 were annotated as metaphorical and 50 were annotated as literal. as the metaphors were annotated in the english bible text, we then manually searched for the latin translations in the vulgate foot_5 . the first author of this paper, a classical philologist, ensured that the metaphors found in the english texts correspondingly occurred in the latin texts, i.e. that the american standard version did not introduce metaphors that were not present in the vulgate. foot_6 below we provide two example sentences for each dataset, together with the respective categorization into metaphorical vs. literal. the twentieth century saw intensive development of new technologies. → metaphorical we preprocessed all datasets such that the original sentence was available, as well as a copy of the original sentence, where we replaced the target word by the [mask]-token. these two sentences were then further preprocessed by the hugging-face foot_9 tokenizer pipeline. in addition, the random forest classifier required the target word (a verb) and its dependent subject and object as lemmas, which we annotated in cases where the information was missing. figure 1 illustrates an example of input and output across models. ## models for zero-shot and few-shot classification, we used mbert @xcite . for zero-shot classification, we fine-tuned the pretrained language model for md on the source language data and used this model for predictions in all three target languages. for few-shot classification, we first fine-tuned mbert on source-language training data, and then fine-tuned it again on a small amount of target language data (see @xcite . additionally, we applied mad-x (pfeiffer et al., 2020b), which consists of three types of adapters: language adapters, task adapters and invertible adapters. for this method, the pretrained model was frozen and two language adapters were trained on a masked language modelling task: one adapter was trained on unlabelled data from the source language, and one on unlabelled data from the target language. then, the source language adapter was inserted in addition to the task adapter, and the latter was trained on labelled data from the source language. finally, inference was performed by plugging in the target language adapter and the (language-agnostic) task adapter. the invertible adapters were plugged in simultaneously with the language adapters, but come with a slightly different architecture because they adapt the embeddings, while the language and task adapters were inserted into each transformer layer. for all neural models we utilized the word-based md method by @xcite , where the original sentence and a copy of that sentence with the masked target word were used as input for sequence classification. as our non-neural model we replicated the random forest classifier by @xcite . this model contains three feature types: 1) abstractness and imageability scores, which @xcite generated on the basis of the mrc ratings by @xcite , 2) supersenses, i.e., "coarse semantic categories", where a word can belong to several synsets in wordnet @xcite , each of which is associated with several supersenses. we created a feature vector with these supersenses as dimensions, e.g., the noun "head" occurs in 33 synsets, 3 of which are related to the supersense noun.body. the dimension corresponding to the supersense noun.body then receives 3/33 (example taken from @xcite . 3) further features were produced with the vector space model by @xcite . this model utilizes multilingual information in order to generate similar vectors for synonymous words. all these features were extracted from the target word -in our case, a verb -and from its dependent subject and object. for cross-lingual inference, the model relies on one-to-many translations: all translations were given for a target language word, and the scores obtained for the translations were averaged (see @xcite . for translation, we used word2word by @xcite . ## conclusion while research on md has focused on languages with comparably large amounts of data used for pretraining large language models, our experiments have shown that neural cross-lingual methods are suitable for languages with relatively large (russian and german) and small amounts of pretraining data (latin). especially mad-x performed very well, with the highest results across all experiments for german and latin using a small training dataset and tuned hyperparameters, and for russian using a large training dataset and default hyperparameters. zero-shot classification with mbert performed decently on a small training dataset and default hyperparameters across all three languages. fewshot classification with mbert as applied in our experiments was not successful, as it relies on validation data for shot-selection, which is not possible in the cross-lingual setup. the non-neural random forest classifier, even though it yielded competitive results for russian and german, was generally outperformed by the neural models -even for latin, where small amounts of data were used to pretrain the neural models. it is unclear, however, why performance was better for russian than for german across experiments. a qualitative analysis revealed a range of possible explanations, namely the inherent difficulty of particle verbs, idioms, and rich metaphorical contexts in the german dataset. whereas for the few-shot experiments we conducted sequential fine-tuning on source and target language data, @xcite showed that joint (instead of sequential) fine-tuning leads to few-shot models that yield higher results and are more robust in terms of hyperparameters (e.g. number of training epochs). we plan to employ this method for md in future work, because few-shot fine-tuning showed promising results but still depends on target-language validation data. another next step will be to compare our models' performance for latin to their performance for romance languages, in order to minimize the typological differences between the target languages. we will also investigate how the models presented in this paper perform in contrast to newer multilingual large language models such as mt5 18 @xcite .