Transfer Learning Parallel Metaphor using Bilingual Embeddings (2022)

Abstract: Automated metaphor detection in languages other than English is highly restricted as training corpora are comparably rare. One way to overcome this problem is transfer learning. This paper gives an overview on transfer learning techniques applied to NLP. We first introduce types of transfer learning, then we present work focusing on: i) transfer learning with cross-lingual embeddings; ii) transfer learning in machine translation; and iii) transfer learning using pre-trained transformer models. The paper is complemented by first experiments that make use of bilingual embeddings generated from different sources of parallel data: We i) present the preparation of a parallel Gold corpus; ii) examine the embeddings spaces to search for metaphoric words cross-lingually; iii) run first experiments in transfer learning German metaphor from English labeled data only. Results show that finding data sources for bilingual embeddings training and the vocabulary covered by these embeddings is critical for learning metaphor cross-lingually.

## introduction in the literature, figurative language is instantiated in many different ways. one of the most challenging tasks of figurative language detection, however, is metaphor identification. @xcite finds that up to almost 20% of words in a text are metaphorrelated. however, most work in the field is focused strongly on the english language. as such, early work on computational metaphor interpretation was performed by @xcite uses latent semantic analysis to adjust the meaning of a predicate p when it is applied to an argument a. in kintsch's theory the predicate is what we typically call a metaphor's source @xcite and the argument is its target (e.g., selfies @xcite go viral @xcite ). before word embeddings were used based on implementations such as word2vec, lsa helped to generate high-dimensional semantic spaces using singular value decomposition for dimension reduction. kintsch uses cosine similarity to compare a metaphorical predication (i.e., its numerical representation) to some of its semantic surroundings. today, semantic information mainly is encoded by word embeddings @xcite . @xcite recently presented work of metaphor prediction using an rnn classifier. together with different types of embeddings vectors, the authors perform neural metaphor detection, in a sequence labeling setup, and in a classification setup. one of the most famous works regarding the development of training and testing data sets is delivered by @xcite . the authors present a method for the identification of metaphor in language at the word-level based on methodological and empirical corpus-linguistic work in english and dutch. the method formulates manual instructions and is a refinement based on the metaphor identification procedure (mip) presented by @xcite . the extended annotation version (mipvu) is developed at vrije universiteit amsterdam (vua) and demonstrates case studies addressing metaphor in english and dutch news amongst others. while there is a lot of room for improvement in the field of metaphor detection and interpretation, especially languages other than english lack resources and successful algorithms. transfer learning (tl) is one way to overcome this issue. but work in this field is rare. @xcite use lexical semantic features of words participating metaphoric construction. the authors use transfer learning based on bilingual dictionaries to find metaphoric expressions across languages. their work supports the consensus that metaphors are rather conceptual. more recently, @xcite perform probing of metaphor-annotated data sets. next to other tasks, they also probe for cross-lingual performance using a multilingual pre-trained language model and a data set of four high-resource languages (english, russian, spanish, farsi). in this paper, we present different strategies to overcome resource gaps using transfer learning strategies. we start with a literature overview before we perform first experiments to assess these techniques for the german language. one example for instance-based tl is @xcite . for training, the authors only use information from relevant re-weighted instances in the target domain. the target samples are selected upfront based on the uncertainty (distance of sample x to the decision boundary) in a binary model trained on source and target samples. also, @xcite find relations between different instance distributions in source and target. they formulate requirements for instance distribution and classification function different in source and target. then, they solve for these differences using semisupervised instance-weighting. @xcite migrate knowledge-from labeled data-from a source feature space to a target feature space. the authors show that one can use for example labeled text data to train a model for image classification when image labels are rare. feature-based transfer: feature-based tl aims to reduce the gap between the marginal (p (x t ) ̸ = p (x s ), e.g., word frequencies) and conditional distributions (p (y t |x t ) ̸ = p (y s |x s ), typically ylabels) of source and target domain @xcite . in asymmetric feature-based tl, often a transformation ϕ s /ϕ t from source to target is employed @xcite , which especially works well when both domains share the same label spaces. in symmetric feature based tl, features are transferred from source and target respectively into a common space. @xcite transfer components across domains into a reproducing kernel hilbert space using maximum mean discrepancy as a distance measure. in the sub-space represented by that hilbert space, data properties are preserved and data distributions of different domains can still remain similar. this enables the training of classifiers in a source domain for use in a target domain. also @xcite consider the use of source domain and target domain data represented by heterogeneous features of different dimensions. two projection matrices help to transform data from source and target into a common subspace, and two feature mapping functions use these projections to augment the data in that new space. ## method overview in the following sections (i.e., sec. 4 to sec. 7), we present a procedure to tl for metaphor prediction. we start with a description of the metaphor corpus that we use as gold data. once it is completely translated and annotated for metaphor source words in the german translation (sec. 4), we can use it for other evaluation setups too. right now, we have 500 samples finished and use them for neighborhood retrieval (c.f., sec. 6) and classification testing (c.f., sec. 7) in section 5, we introduce the source data that we build our bilingual embeddings upon and describe a merging strategy of the parallel data. we also present different approaches to handle compound metaphor sources in the target language and how they affect the distance to the english language counterparts (in the embeddings space). the latter is performed on 500 samples of the metaphor corpus. in sec. 6, we discuss the training of the bilingual embeddings after we perform a retrieval of a metaphoric german language ## metaphor gold corpus the corpus: a first step is to create a gold corpus to have a test set available for all sorts of techniques, be it supervised, unsupervised or transfer. hence, we start from the corpus of @xcite . it origins from sources such as news articles, blog posts, and online forums. it consists of more than 1700 sentences using metaphoric language. the authors propose the use of conceptual schema to represent scenarios of metaphor usage. they recognize 70 source domains which again are grouped into 14 ontological categories. the corpus is manually validated and contains annotations for a metaphor's target, a metaphor's source, their associated linguistic and conceptual metaphors and the metaphors' lexical trigger. the linguistic metaphor annotation refers to terms from the sentence itself, so we can use this information to find the corresponding figurative label for a term. we prefer this corpus over the famous vua corpus @xcite , especially because we are also interested in seeing the effect of having training and testing data from different domains (see sec. 7 ). further, having a german-translated and annotated version in place, we can add more diverse data sets to the community. a last reason is that the entire data set (once mirrored to german) offers a good sample size for tuning and evaluating further neural-based classifiers. ## training cross-lingual embeddings before we develop a supervised training setup with our data at hand (next section), it is important to learn about the potential contexts offered by crosslingual embeddings. therefore, we first test different setups of cross-lingual embeddings to retrieve the distances between a metaphoric word in an english language text and its german counterpart in the target language. using 500 manually annotated samples of our (parallel) metaphor corpus, we now retrieve the german counter word given an english language metaphoric word in the embeddings spaces trained from different parallel data setups: figure 2 shows the distribution of the german metaphoric word among the nearest neighbors of a metaphoric word from the english data. all corpora except the small metaphor corpus show an inverted bell curve meaning that most of the metaphors have their german counterpart among the 100 nearest neighbors or beyond their 10,000 nearest neighbors. the metaphor data (blue) rather show a bell distribution of the metaphoric words in the target language. however, we only added the blue curve for comparison reasons. the distributions of english and german metaphoric words @xmath0, and hence, in the language's semantics. fig. 3a shows a scatter plot of the distribution of cosine similarities between the metaphors' sources in english and german respectively in the metaphor corpus. the vast majority of values is very close to one. this is especially because the vocabulary of this model is not big, and most words are in close neighborhood of the metaphor source word. fig. 3b shows a plot of cosine similarities distributed between the metaphors sources in english and german respectively in the metaphor corpus and polnear. as polnear is about ten times as big as metaphor, we can see the data points are moving more towards zero being not as similar anymore. fig. 3c shows cosine similarities distributed in metaphor and europarl 100k. we encounter a much lower oov-rate (min-freq of 2). we also see that the normalized sum for component combination achieves higher similarities with the english metaphor sources than the averages do. even though we do not have many data points here, we still learn that compounds are somewhat difficult to associate to the english source words. hence, we plan to test the impact of decomposed compounds in the test data once our entire gold corpus is finished 8 in the next section, we use our bilingually trained embeddings foot_8 in a tl-classification task. we find out experimentally that a mix of metaphor, poln-ear, and europarl with a minimum frequency of 5 and lower-cased embeddings sources covers most while applying the english-trained model in german (row 3 and 4). this might be the case since our testing data set is better balanced than the vua data set. on the other hand, metaphoric contexts are not as well represented for german as they are for english in the model, especially since we did not word-align the data even though positions of verbs differ in both languages. 10 looking into samples, we found that the especially low recall is caused by a lot of verbs not used very figuratively, such as "waffenrechte verteidigen/schützen" (defend/protect gun rights). we plan to investigate these issues in detail to refine our choices and methods for training bilingual embeddings. we did not use pre-trained bilingual embeddings even though existing work often comes with links to data and code (c.f, @xcite ; hermann and blunsom (2014)). however, these data often is difficult to collect as links are not available, broken or regeneration is laborious. further, often bilingual embeddings are trained on europarl which is not necessarily the domain, we can hope to find a lot of metaphoric language-also a problem in our approach as we use europarl data too. during our work we also learned that adding source data from the news domain to our embeddings data reduces distances in the embeddings space (c.f., fig. 3b ). ## next steps a next step is to predict metaphoric language in a target language using pre-trained transformer models and our gold data for fine-tuning for example in a classification task. for this task, the embeddings representation of a sentence and the metaphor's source word is given, and the metaphoric word of the target language needs to be predicted. another step might be applying tl methods of neural machine translation (e.g., kocmi and bojar 10 we also tested our approach using bilingual embeddings from upfront word-aligned @xcite data. however, test f1-score remains below 10%. we belief that the n:m relations of words make it difficult for the classifier to identify the semantic in the target language well enough. 2018 ). as we learned in sec. 2.2 usually a neural model is trained on a high-resource language pair and tested on a low-resource pair. in our setup, we could encounter this using a sequence labeling model trained on an english language metaphor corpus and combine it with a (pre-trained) translation model of english and german. as sources for the translation and evaluation part, we might also consider to use the parallel data from common crawl emnlp (2018). figure 4 shows an overview on the technique considering language model and tagging model probabilities as common translation setups do. ## conclusion in this paper, we presented an overview of transfer learning techniques structured in a twofold manner: i) types of transfer learning, and ii) transfer learning techniques from a task-oriented perspective. we presented first steps towards the application of modern transfer learning techniques towards metaphor prediction in german language text. the experiments make clear that successfully training bilingual embeddings depends on the vocabulary coverage of the source texts. we furthermore are in the process of annotating a parallel corpus (en-de) of metaphor starting from a pre-existing english language corpus, which we plan to use as a gold data set to test transformer-based models.