Metaphor Understanding Challenge Dataset for LLM s (2024)

Abstract: Metaphors in natural language are a reflection of fundamental cognitive processes such as analogical reasoning and categorisation, and are deeply rooted in everyday communication. Metaphor understanding is therefore an essential task for large language models (LLMs). We release the Metaphor Understanding Challenge Dataset (MUNCH), designed to evaluate the metaphor understanding capabilities of LLMs. The dataset provides over 10k paraphrases for sentences containing metaphor use, as well as 1.5k instances containing inapt paraphrases. The inapt paraphrases were carefully selected to serve as control to determine whether the model indeed performs full metaphor interpretation or rather resorts to lexical similarity. All apt and inapt paraphrases were manually annotated. The metaphorical sentences cover natural metaphor uses across 4 genres (academic, news, fiction, and conversation), and they exhibit different levels of novelty. Experiments with LLaMA and GPT-3.5 demonstrate that MUNCH presents a challenging task for LLMs. The dataset is freely accessible at https://github.com/xiaoyuisrain/metaphor-understanding-challenge.

## introduction large language models (llms), such as bert @xcite , gpt-3 @xcite and llama @xcite , have become a common paradigm in natural language processing (nlp). several benchmarks have been proposed to investigate the capabilities of llms @xcite @xcite ; and comprehensive analyses have been conducted, evaluating their performance on a range of nlu tasks @xcite @xcite @xcite . the community has extensively examined llm performance on question answering, summarisation, sentiment analysis, natural language inference; a each metaphor sample has a * highlighted * word that is metaphorically used, and is accompanied by up to 5 crowdsourced paraphrases : substituting the highlighted word with one of the provided words should result in an apt paraphrase. for a selection of metaphor samples, we also provide expert annotation : a pair of correct and incorrect substitution words. metaphors are linguistic expressions based on conceptual mappings between a target and a source domain @xcite . the verb phrase to stir excitement, for example, is based on the conceptual metaphor feeling is liquid, with feeling (excitement) being the target domain and liquid (something that can be stirred) the source domain. the metaphor compares feeling with liquid, introducing vividness into the description of an otherwise intangible emotional impact. such cross-domain mappings are sets of systematic ontological correspondences, mapping concepts and their relational structure across distinct domains. performing this mapping is an essential part of reasoning involved in the interpretation of metaphorical language @xcite @xcite . humans use metaphors so naturally and frequently that they largely fly under our radar. in figure 2 : two tasks for munch: given a sentence containing a metaphorically used word, a model is prompted to 1) select correct paraphrases from two given candidates (paraphrase judgement), and 2) paraphrase the sentence by replacing the highlighted metaphorically used word (paraphrase generation). several corpora have been created that contain metaphor annotations at either word or sentence level. these include the vua corpus @xcite , the lcc metaphor datasets @xcite and the metaphor-emotion dataset of @xcite , among others. these datasets have been widely used to develop and evaluate automated metaphor identification systems (see @xcite for a survey), but they do not contain information of how the annotated metaphors are interpreted. on the other hand, several works developed datasets with a focus on interpretation, typically casting the problem as a paraphrasing task @xcite @xcite . yet, those datasets were often small in scale (containing 200-1000 instances) and were not designed to test the reasoning process by which a metaphor is interpreted, which remains an open question. this paper presents a novel metaphor understanding challenge dataset for llms (munch). it provides over 10k paraphrases for metaphorical sentences and 1.5k triples of a metaphorical sentence and two candidate paraphrases, which could be apt or inapt (for dataset examples see figure 1 ; for statistics see appendix a). the metaphorical sentences were extracted from vua texts, spanning four genres (academic, news, fiction, and conver-sation) and featuring metaphors at different levels of novelty. each metaphorical sentence contains a content word that is marked as metaphorically used. a candidate paraphrase replaces the metaphorical word with another word, so that the resulting sentence is the same as the reference sentence except for that one word, therefore representing a lexical substitution task. an apt paraphrase shows correct contextual interpretation of the metaphor while an inapt paraphrase uses a word that is related to a literal, source domain sense of the metaphorical word (see the examples of correct and incorrect substitution words in figure 1 ). such a setup of the task is inspired by the conceptual metaphor theory @xcite and allows us to investigate whether the model performs full metaphor interpretation by cross-domain mapping or rather resorts to more shallow lexical similarity. in order to investigate this in a more controlled fashion, we opted for a lexical substitution task. specifically, we test whether the model consistently chooses the correct target domain paraphrase (therefore, fully interpreting the metaphor) or rather bases its decisions on lexical similarity and chooses the inapt paraphrase that is similar in meaning to the literal use of the metaphorical word. we set up a fill-in-the-blank task to crowdsource apt paraphrases, and manually selected the best paraphrases using expert knowledge. we also manually created inapt paraphrases from wordnet synsets, so that the apt and inapt paraphrases reflect the target and source domains of the metaphors respectively. specifically, the inapt paraphrases are synonyms or hypernyms associated with the word's literal use (the source domain). using this dataset, we tested the metaphor understanding capabilities of llama-13b, 30b, and gpt-3.5 zero-shot in two tasks: paraphrase judgement and paraphrase generation, as illustrated in figure 2 . our results show that both tasks are challenging for the models. in particular, the models are prone to confuse the target and source domains of the metaphors, as they often fail to distinguish the inapt paraphrases from the apt paraphrases or reference sentences. our experiments also reveal that llms' metaphor understanding capabilities are associated with genre, metaphor novelty, and pos of the metaphorical word. the munch tasks thus allow us to gain insight into how llms process metaphors as well as how this remarkable ability can be improved in the future. vua has been widely used in studies on automated metaphor detection @xcite @xcite @xcite . however, the corpus does not specify the conceptual metaphors indicated by the mrws or provide annotation for interpreting the metaphors. the corpus is not directly applicable to automated metaphor interpretation. shutova (2010) defined automated metaphor interpretation as a paraphrasing task: given a metaphorical expression where a word is marked as metaphorically used, the model should replace this word with another word to render a literal paraphrase of the expression. for example, the verb phrase stir excitement, where stir is used metaphorically, should be paraphrased as provoke excitement. bizzoni and lappin (2018) created a metaphor paraphrase evaluation corpus (mpec), which provides correct and incorrect paraphrases for ∼200 short sentences containing metaphor use; paraphrases could greatly differ from the reference sentences. @xcite created the newsmet dataset, which consists of 1k verbal metaphors in news headlines as well as their literal equivalents; incorrect paraphrases are not provided. several recent studies approached metaphor understanding as an inference task. the impli dataset @xcite includes entailed and non-entailed sentences for ∼900 metaphorical sentences. the flute dataset @xcite provides entailment and contradiction pairs for 1500 metaphorical sentences (including 750 similes). fine-tuned transformer-based models @xmath0.8 accuracies in these 2 studies in predicting the class of a given sentence pair. recent studies also employed multiple-choice and generative tasks to assess llms' ability to rea-son with metaphorical language. the miqa benchmark @xcite uses such tasks to test whether models can distinguish metaphorical and literal uses of the same words; 150 conventional metaphors are involved. the fig-qa task @xcite includes 10k similes (a type of direct metaphor) and requires models to distinguish a pair of metaphors of opposite meanings. @xcite examined llms' figurative language understanding by asking them to generate text after encountering an idiom or simile. the munch dataset provides 3k samples of indirect metaphors, 10k correct paraphrases, and 1.5k incorrect paraphrases. it is therefore one of the largest datasets for paraphrasing of indirect metaphors. the candidate paraphrases are also systematically different from the ones in previous datasets, as we tailored the dataset for testing llms' understanding of metaphors as crossdomain mappings and correctly capturing the underlying relational structures. we summarise differences between munch and previous datasets and provide more details for the latter in appendix a. ## data collection: metaphor samples the metaphor samples in our dataset were selected from the publicly available metaphor corpus vua. each metaphor sample is a sentence containing a highlighted mrw, the metaphorical word to be interpreted; a paraphrase uses a single word to substitute the metaphorical word. we use two criteria for selecting metaphorical sentences: novelty of the metaphor and possibility of single-word substitution. we explicate our selection process below. the novelty criterion. we employed novelty scores from do @xcite to increase the proportion of novel metaphors in our dataset. scores range from -1 to 1. vua contains a large proportion of conventional metaphors: the metaphorical use of the word can be found in a dictionary of contemporary language use @xcite . as llms might have encountered enough data for such conventional metaphor uses during pre-training, the understanding of such metaphors should be relatively easy. to render a more challenging dataset, we excluded mrws with novelty scores below -0.3. metaphors with a novelty score higher than -0.3 could still be conventional: the crowd workers who provided the novelty annotations in do @xcite relied on their intuition instead of a dictionary like @xcite . and metaphorical uses included in dictionaries may still be considered novel by lay people. we chose -0.3 as the threshold in order to collect a large and diverse dataset as a starting point. ## annotation of apt paraphrases crowdsourcing task. we constructed a fill-inthe-blank task to crowdsource (apt) paraphrases for the metaphorical sentences. each task included 30 sentences to be paraphrased, so that the task can be finished within 30 minutes. under each sentence, the workers were presented with a copy of the sentence where the metaphorical word is replaced with a blank; they were asked to fill the blank with a single word so that the new sentence is a semantically and grammatically apt paraphrase of the reference sentence. if they were not able to paraphrase the sentence, they were asked to explain why it was difficult. examples of good and bad answers were provided in the instructions (see appendix b). the workers were recruited via prolific foot_0 . we set prescreening criteria to only include adult (@xmath1. the workers were asked to confirm within the task that they met these criteria. after giving consent to participate and reading the instructions, they were also required to correctly paraphrase a trial sentence in order to access the task. more details (worker's consent, the trial sentence) are given in appendix b. we released 99 tasks in total and collected 5 data points for each of the 2970 reference sentences. we received single-word substitutions for 2953 sentences (the other 17 are presented and explained in appendix b), and 61% of them got repeated answers-multiple workers submit the same paraphrase despite the question being open-ended. this confirms the reliability of our task. ## data analysis the dataset contains approximately the same number of samples from academic and news genres, and fewer samples from fiction and conversation, as shown in as we excluded mrws of novelty scores lower than -0.3, the metaphor samples exhibit a wider range of novelty scores above 0 than below 0 (see appendix d). meanwhile, a large proportion of the metaphor samples could be considered only slightly novel or conventional (novelty scores between -0.3 and 0.3). metaphor samples of the highest novelty scores can be from any of the four genres. despite their different proportion in the entire dataset ( we also examined the cosine similarity between the metaphorical words and apt and inapt substitutions. since the inapt paraphrases were based on the more basic meaning of the mrws (section 5), we expected inapt substitution words to be more similar to the metaphorical words than apt substitution words. we computed the cosine similarity scores using glove-wiki-gigaword-300 embeddings (pennington et al., 2014), accessed through the gensim python library. figure 3 shows the distribution of cosine similarity scores for 1006 triples, excluding the ones containing out-of-vocabulary words. surprisingly, the target-apt pairs tend to have higher cosine similarity scores than the targetinapt pairs. the plot suggests that the 3 pairs are distinguishable in terms of cosine similarity scores, with target-apt pairs being the most similar, and apt-inapt the least similar. this might be associated with the fact that our metaphorical sentences were sampled from vua, which, being representative of metaphor use in natural discourse, includes a large percentage of conventional metaphors. nonetheless, the majority of the cosine similarity scores are above 0, and the 3 pairs still share a wide range of similarity scores. the distribution plot is therefore also suggestive of the reliability of our dataset, as well as its potential challenge for llms. ## model evaluation we evaluated llama-13b, llama-30b, and gpt-3.5 (text-davinci-003) on two tasks: (task 1) paraphrase judgement, which requires a model to select correct paraphrases for a given reference sentence from given candidates; and (task 2) paraphrase generation, which asks a model to generate correct paraphrases for a given reference sentence. the paraphrase judgement task used the 1492 triples that include inapt paraphrases; the generation task used all 2953 metaphorical sentences. details regarding computational budget is given in appendix e. ## discussion paraphrase judgement. we looked into the type of errors the models made in paraphrase judgement. the number of each combination of expected and predicted answers for each model is in appendix f. we found that llama-30b and gpt-3.5 could ignore the semantic differences between a given sentence and an inapt paraphrase, as they tend to predict both candidates as correct when presented with one or more inapt paraphrases. llama-13b, on the other hand, tends to assume that the two given candidates always contain one apt and one inapt paraphrase. nonetheless, it did not seem capable of distinguishing the two, as it made a similar number of option a and option b predictions. ## conclusion we release a dataset of manually created apt and inapt paraphrases for metaphorical sentences and present two metaphor understanding tasks, which we demonstrate to be challenging for current llms. the errors the models make in the paraphrase generation task indicate various levels of misunderstanding of the metaphors. in the paraphrase judgement task, the models' accuracy was lower than the random baseline in the majority of the cases; a closer look at their errors reveals that the models had difficulty in detecting the inaptness of the inapt paraphrases. the experiments also show that the models performed better when being instructed to focus on the metaphorical word, and that genre and the pos and novelty of the metaphorical word are all potential factors that affect model performance.