Nut-cracking Sledgehammers: Prioritizing Target Language Data over Bigger Language Models for Cross-Lingual Metaphor Detection (2023)

Abstract: In this work, we investigate cross-lingual methods for metaphor detection of adjective-noun phrases in three languages (English, German and Polish). We explore the potential of minimalistic neural networks supported by static embeddings as a light-weight alternative for large transformer-based language models. We measure performance in zero-shot experiments without access to annotated target language data and aim to find low-resource improvements for them by mainly focusing on a k-shot paradigm. Even by incorporating a small number of phrases from the target language, the gap in accuracy between our small networks and large transformer architectures can be bridged. Lastly, we suggest that the k-shot paradigm can even be applied to models using machine translation of training data.

## introduction metaphors are a phenomenon of figurative language where meaning about a more abstract concept is expressed by applying it to a more concrete domain. according to cognitive linguistic theories by @xcite , they are systematic linguistic instantiations of so-called conceptual metaphors. an example of a conceptual metaphor is emotion is liquid, which manifests in expressions such as bubbly personality, his anger boiled over and overflowing joy. other definitions describe metaphors as novel usages of words, in which the semantic preference of the syntactic arguments is violated. as an example, to eat prefers animate subjects and edible objects. the metaphor the job ate his confidence violates this preference @xcite . previous studies show metaphors to make up a substantial portion of natural language foot_0 and heavily influence decision-making in discourse @xcite , making their detection a valuable topic in nlp. since conceptual metaphors are based around semantic concepts and not words, they are shared throughout similar cultures and can sometimes be directly translated (er griff mein argument an -he attacked my argument). in other cases, the same conceptual metaphor might exist in two languages, but is lexicalized differently. while a direct translation of seine stimmung war im keller -his mood was in the basement could most likely still be understood, a more conventional phrasing would be his mood plummeted or he was feeling down. therefore, metaphor detection across different languages is an interesting topic worth exploring. however, most annotated metaphor resources center on english. in this paper, we investigate the application of modern zero-shot methods without access to annotated target language data for cross-lingual metaphor detection of adjective-noun phrases in three different languages. we go on to soften the zero-shot limitation and measure how smaller feed-forward models can become competitive to transformer-based systems, by incorporating a small number of target language phrases into the training process. lastly, we apply the same few-shot paradigm to improve models which use machine-translated data and discuss the results. ## related work previous works about metaphor detection were mostly monolingual and supervised. they often leveraged additional resources, static word embeddings and in more recent experiments pre-trained transformer models @xcite , @xcite , @xcite ). the latter is currently the most commonly used option. in a shared task about metaphor detection in 2020 @xcite , more than half of all participants used some variation of a transformer architecture. recent concerns regarding the alternative usage of static word embeddings for metaphor detection were also voiced for theory based reasons (maudslay and teufel, 2022). work addressing cross-lingual metaphor detection includes @xcite , where semantic features and word vectors were used to transfer english metaphor knowledge about adjective-noun or verb-subject-object phrases into spanish, farsi and russian, @xcite , an unsupervised approach for a transfer from german to middle high german based on self-trained fasttext embeddings @xcite and sanchez-bayona and agerri (2022) who present first zero-shot results between their newly created spanish corpus and the english vuamc by @xcite by using xlm-roberta @xcite , a transformer based multilingual language model. this leaves many possible approaches to crosslingual transfer-learning for metaphor detection unexplored. a common way to allow for this transfer in other tasks is machine translation to convert training data from the source language into the target language or vice versa @xcite . @xcite provide a lightweight alternative to bigger transformer models, by aligning static fasttext embeddings across 44 languages through a cross-domain similarity local scaling criterion. while multilingual models do work in zero-shot scenarios, @xcite show the benefit of shifting to a k-shot scenario, in which small target language datasets of size k are incorporated into training. similarly, @xcite present findings which support that using a development set in the target language can improve performance by preventing catastrophic forgetting of multilingual knowledge during training. more recently, large language models such as chatgpt 2 , are used as zero-shot or few-shot in-context learning systems ( laskar et al. (2023), yuan et al. (2023) ). chatgpt is a model of the gpt-3.5 or gpt-4 series, which is trained through a reinforcement learning from human feedback component @xcite and also possesses multilingual knowledge. ## task and data this paper focuses on binary classification of the metaphoricity of adjective-noun tuples, since this setup had the most available data in several languages. in these phrases, the metaphoric meaning can stem from the conceptual transfer of either the 2 https://chat.openai.com/ ## experiments in this section, we describe a series of binary classification experiments of our collected phrases. each experiment described is conducted for all six possible combinations of training and test splits of our three available languages. we prioritize stability of our results over ideal hyperparameters and aim to ensure a fair comparison. therefore, in all following experiments, we incorporate early stopping, learning rate warm-up and report the average result of ten majority vote ensembles with seven seeds each. ## results in this section, we present the results of the experiments we conducted. for all of them, we report the accuracy on the test sets. ## discussion impact of machine translation by looking at our translated data, we try to explain why the translation based zero-shot bert experiments benefited more from the translations than the fasttext baselines. by our choice of method, we end up with translations of individual data points where the two word adjective-noun pair structure is lost (warmer milchschokoladenton to warm milk chocolate tone, crushed stone to schotter). by automatically postagging the translated test data with spacy (honnibal and montani (2017)), we measure these deviations in syntactic structure. depending on language pair and direction, they make up between 7.5% and 38.4% of the translations. in general, length difference of the translations is less of a problem for the scalable transformer models, than for our 600 dimensional, fixed length neural network. this can be circumvented by using recurrent neural networks. analogue to our findings of preferring simple models to large ones for this task, the same could possibly apply to the translation methods, since statistical or dictionary based methods could lead to less deviation in syntax and therefore to better results. this deviation can also explain how our models profit more from the authentic k-shot data, since it better represents the test data. we also observed the nmt already implicitly performing metaphor detection to better lexicalize concepts in the target language (schwieriger spagat to difficult balancing act instead of a less conventional difficult split for the conceptual mapping life is a sequence of motion). ## conclusion and future work the findings of this paper serve to reinforce the idea that larger language models are not always inherently superior at every task and should therefore not automatically be considered the default choice. we have shown how primitive fasttext models can be competitive with large transformer based language models for syntactically trivial but semantically complex tasks such as cross-lingual metaphor detection of adjective-noun phrases. furthermore, these small models can easily be enhanced to out-figure 5 : scatter plot of the different fasttext seeds for rising k and exemplary for en to pl. we distinguish between the f1 score of literals, metaphors and accuracy. the points are coloured for k and scattered by their percentage of seen adjectives of the test data through the k-shot data (ol%). line plots are provided for the average performance and adjective overlap when including the whole training data and the average performance for zero-shot. investigation of the peculiar behaviour of chat-gpt's dependence on the prompts, as seen in