Dispersing the clouds of doubt: can cosine similarity of word embeddings help identify relation-level metaphors in S lovene? (2023)

Abstract: Word embeddings and pre-trained language models have achieved great performance in many tasks due to their ability to capture both syntactic and semantic information in their representations. The vector space representations have also been used to identify figurative language shifts such as metaphors, however, the more recent contextualized models have mostly been evaluated via their performance on downstream tasks. In this article, we evaluate static and contextualized word embeddings in terms of their representation and unsupervised identification of relation-level (ADJ-NOUN, NOUN-NOUN) metaphors in Slovene on a set of 24 literal and 24 metaphorical phrases. Our experiments show very promising results for both embedding methods, however, the performance in contextual embeddings notably depends on the layer involved and the input provided to the model.

## introduction in recent decades, metaphors have been recognized as a ubiquitous phenomenon in all types of discourse @xcite @xcite , and because of their central role in both language, thought and communication @xcite @xcite , they have been addressed by various fields and disciplines, from linguistics, neurolinguistics, psycholinguistics, cognitive linguistics, social science, and computer science. the main underlying mechanism of metaphor involves representing one domain in the terms of another @xcite @xcite . the represented domain, usually more abstract, is called the target domain, and the domain it is represented by is called the source domain, which is usually more concrete and based on physical experience. for example, in the expression political storm, we represent the target domain of politics in terms of the source domain of weather. for a metaphor to be apt @xcite , the domains have to share certain features or relations, but otherwise be sufficiently different from one another. on the one hand, this semantic difference can be observed between the metaphorically used word and its context. @xcite put forward the idea of metaphors as "selectional preference violations", that is, the context of the metaphorically used word is not the context this word would normally select. on the other hand, metaphorically used expressions also exhibit some form of polysemy in themselves. the contextual meaning of the metaphorically used word is different from its most basic meaning which is expressed in literal contexts. the latter is also the defining factor of the most frequently used procedure for manual metaphor identification in texts (mipvu, @xcite . these two facets of metaphors have often been used and explored in automatic metaphor identification approaches. various methods have been proposed that model language and meaning on the basis of the distributional hypothesis @xcite , according to which similar words have similar contexts. in these models, the meanings of words are determined by their relationships to other words in that same space, and similar words tend to have similar vectors and similar neighbourhoods. older approaches to metaphor modelling use distributional vectors created with the help of e.g. latent semantic analysis @xcite , while more recent ones use distributed word embeddings obtained through deep-learning @xcite . an important distinction can also be drawn depending on the level of metaphor processing: word-level, relation-level, or sentencelevel. on the word-level, the task is to determine the metaphoricity of a (or each) word. on the sentence-level, the whole sentence is classified for containing metaphor(s) or not. on the relationlevel, which we are concerned with in this exper-iment, the expressions under question are pairs of words that have a syntactic relation between a source and target term, for instance verb-object (break a promise) or adjective-noun constructions (deep thought). related to and sometimes overlapping with relation-level metaphors is the wider class of multi-word expressions (mwes), which include phraseological units such as idioms and proverbs, and other fixed expressions such as compounds and collocations @xcite . especially idioms can overlap with metaphoric expressions by their meaning non-compositionality by which the meaning of the whole cannot be directly derived from the meaning of its parts. some idioms may in fact even stem from metaphorical conceptualizations, e.g. to throw dust in someone's eyes. another shared characteristic can be lexicalization, that is, both mwes and conventional metaphors can be included in dictionaries, for example parent company. however, mwes mostly require some extent of syntactic fixedness, and, more importantly, they always require at least 2 constituent words, while metaphors can take form of a word, a phrase, or even a whole paragraph. in slovene, automatic figurative language processing is still in its early stages, with only a few semi-supervised @xcite and supervised automatic models proposed @xcite . the direct use of cosine similarity between the source and target word for metaphor identification in slovene has not yet been explored and could possibly allow unsupervised extraction of metaphorical candidates from text, avoiding the need for manually annotated data. the aim of the experiments presented here is two-fold: 1) to investigate the representation of metaphorical expressions in both static and dynamic embeddings and evaluate their use for metaphor identification, 2) to establish a baseline by which to distinguish between metaphor and nonmetaphor. ## related work metaphor identification has been approached from various perspectives, using or combining several tools and resources. state-of-the-art approaches for english and other more resourced languages use deep learning methods to train metaphor identification models on large annotated corpora in a supervised manner. because the focus of our work is on unsupervised classification and evaluation of word embeddings for this purpose, here we only report on some previous work in this same direction. one of the first unsupervised approaches is by @xcite to identify verbal metaphors in the bnc corpus. starting with a seed set of 62 metaphorical verb-object and verb-subject pairs, they apply unsupervised noun and verb clustering on vectors obtained from corpus frequencies in order to extend the range of target and source concepts. then, they search the bnc for metaphorical expressions using these two expanded lexicons and achieve a precision of 0.71. @xcite evaluate both static word2vec and distributional vectors on data from a behavioural study to test if they encapsulate metaphoricity, familiarity and meaningfulness. they test these features with a multiple regression analysis, to see if they are correlated with cosine similarity. for both vector types, their results show that low values of metaphoricity were predictors of high cosine similarity. @xcite also use word2vec embeddings trained on reference corpora for chinese and english to investigate their use for the identification of nominal metaphors (x is y). they devise a method that combines calculating the relatedness of words (x,y) via cosine similarity with checking for hyper-/hyponymy relation in wordnet. if the similarity is lower than a predefined threshold and the concepts have no taxonomic relationship in wordnet, the candidate is classified as a metaphor. they establish the threshold value of cosine similarity as the best overlap (convergence) between literal recall, metaphor recall and accuracy, and determine it to be at 0.235 for english and 0.575 for chinese. this also shows that the threshold varies greatly on the language involved and that language-specific baselines need to be determined. @xcite use cbow and skipgram embeddings and wordnet to predict the metaphoricity of a verb in a sentence. for each target word, they find the best-fit synonym, hypernym or hyponym in wordnet that matches the context by having the highest cosine similarity to the context vector of the sentence. then, they compute the cosine similarity between the best-fit word and the target word, and classify the target word as metaphor if the similarity is lower than a threshold of 0.6, which was pre-established on the basis of a development set. @xcite experiment with both vi-sual and linguistic embeddings in predicting phraselevel metaphors. they obtain both individual word embedddings and joint phrase embeddings based on the skipgram method, and investigate various combinations of measuring similarity via cosine distance. they obtain best results with their multimodal approach, while in linguistic embeddingsonly setting, computing the similarity between the words in the phrase outperforms computing similarities of phrase embeddings. in a semi-supervised manner, zayed et al. (2018) use a seed set of verb-noun phrases to determine the metaphoricity of the candidate verbs on the phrase level. first, they find the most similar verb in the seed set using cosine distance and word mover's distance, and compare the similarity of the candidate noun to the nouns associated with the most similar verbs in the seed set. they also compare glove and word2vec static embeddings methods, and achieve the best results using glove embeddings and cosine distance. more recently, @xcite tested the knowledge instilled in bert models by applying the "landmark method" introduced in @xcite , which tries to determine which properties are transferred from the source to the target domain. namely, metaphoricity relies on some common ground between the two domains which makes the comparison plausible. in their experiment, pedinotti et al. check whether the representations of metaphors are closer to these common ground 'landmarks' or to the literal properties of source domain words that are not relevant to the metaphor mapping. they conclude that metaphorically used words are consistently more similar to literal landmarks in the first few layers of bert embeddings. moreover, they observe a difference comparing conventional and creative expressions: while models achieve steadily better accuracy (in terms of wrong answers) for conventional metaphors, the accuracy actually drops in the later layers for creative metaphors. among unsupervised approaches to mwes, which are somewhat similar to relation-level metaphors, we can mention @xcite and @xcite . @xcite investigate english nominal compounds, where the head of the phrase is a noun (adjective-noun and noun-noun), and their syntactic counterparts in french and portuguese. to distinguish compositional from non-compositional (idiomatic) mwes, they measure the cosine similarity between the combined vectors of the parts and the vector of the compound. moreover, they investigate the influence of various variables: different distributional models, preprocessing methods, dimension sizes, and context sizes. they find that the models can successfully capture idiomaticity, with word2vec as the best performing model for english, while for french and portuguese, the ppmi-based models fared better. in addition, they find that models for the morphologically richer french and portuguese benefit from preprocessing steps such as lemmatization and stopword removal. in a more recent approach, @xcite investigate various contextual models for their representation of potentially idiomatic expressions, i.e. expressions that can be literal or idiomatic depending on the context, in english and portuguese. they measure the cosine similarity of the embeddings of idiomatic compounds with 1) the embeddings of their meaning-preserving compounds and 2) literal synonyms of the components. their experiments show the idiomatic phrases are closer to the literal synonyms than to their meaning-preserving paraphrases, leading to the conclusion that idiomaticity is not yet adequately captured by contextual models. ## results and discussion for the static embeddings, we first analyze the relationship and balance between recall and precision for the literal and metaphorical classes at different cosine similarity thresholds. as shown in figure 3 , the values converge and balance out with cosine similarity values between 0.42 and 0.49. then, for each of the scenarios (static, contextual, different layers of contextual embeddings, dif- in the next step, we computed the self-similarity of the word in different contexts. we only focused on the words that are used both literally and metaphorically in our dataset. the average selfsimilarities are depicted in figure 4 . not surprisingly, embeddings from the individual-and phraseinputs are very similar throughout the model, as the context is practically identical. the least similar, as expected, are embeddings from the individual inputs compared to those from sentence inputs. there seem to be observable differences in the average word self-similarity, especially when comparing the individual word embedding to its sentence embedding and the word's embeddings in the phrase and sentence contexts. however, the linear regression and anova tests show no significant relationship between the word's self-similarity in any of the layers and any of the settings: the absolute highest r 2 = 0.2431 (@xmath0.1) was achieved when comparing the embedding from the individual word to the embedding of the word in the sentence on the 4th layer. we assume that this is due to the design of contextualized models, which are intended to represent word meaning in wider contexts and fail to produce sensible representations when presented with narrower contexts. to further evaluate cosine similarity as a predictor of metaphoricity, we classify our examples according to the thresholds obtained from linear regression models with significance levels @xmath1.01. we report the results in ## conclusion in this paper, we presented the first experiment on unsupervised identification of metaphors on the phrase level in slovene with word embeddings. based on a dataset of 24 comparable pairs of metaphorical and literal phrases, we investigated the use of cosine similarity in both static and contextual embeddings. the results show that both methods achieve comparable results in terms of precision, recall and accuracy when comparing cosine similarities between the phrase's constituent words. in line with previous research, we also intuit that lower layers exhibit less contextualized information and are generally more suited to the task. however, in our experiments with self-similarity, where we compared the candidate word's embeddings in different contexts, the results show no statistical significance and cannot be used to determine a metaphorical shift in meaning. in conclusion, this preliminary experiment showed promising results for unsupervised metaphor identification, but will have to be evaluated on more data which may contain less clear-cut examples of metaphorical and literal language. future work includes testing the method on more examples and other embedding models. we also plan to investigate the use of psycholinguistic measures such as abstractness for relation-level metaphor identification, and evaluate the methods with respect to the syntactic type of construction used. another interesting avenue for further research could be investigating other methods for combining subword embeddings, which could potentially provide a better word representation for the purposes of metaphor identification.