Merely Judging Metaphor is Not Enough: Research on Reasonable Metaphor Detection (2024)

Abstract: Metaphor, as an advanced form of cognition, is challenging to understand their meaning. Current metaphor detection tasks only provide labels (i.e., metaphor or literal) without interpreting how to understand them. In this paper, we improve the metaphor detection task and explore the reason of metaphor. To the best of our knowledge, we are the first work to reason about metaphor using mainstream Large Language Models (LLMs). Specifically, we utilized ChatGPT3.5 to expand the mainstream datasets in current metaphor detection, including VUA ALL, TroFi, and MOH-X. We input the original sentence, target word, and usage (metaphor or literal) into ChatGPT, guiding it to generate corresponding metaphor reason. Then, we designed supervised baseline experiments (e.g., RoBERTa, GPT-2) and zero-shot experiments with LLMs (e.g., LLaMA3). For the results generated by the above experiments, we provided the case study. We devised four methods that include manual evaluation to evaluate the reason performance of the model, and discussed extensively the advantages and disadvantages of these evaluation methods. Our code is available at https://github.com/yc-cy/Metaphorical-Reasoning.

## introduction metaphor is essentially a cognitive mechanism that exists in human thinking, used to construct conceptual frameworks @xcite . in nlp, metaphor detection refers to determining whether a given target word is used metaphorically, given its context @xcite . considering an example of a metaphor detection task: "his voice is like heavenly music.". in this sentence, "heavenly music" is used metaphorically. by associating the term "heavenly music" with his voice, it conveys that his is "besiege" used metaphorically in "she was besieged by so many problems that she got discouraged."? yes, "besiege" is used metaphorically in this sentence. why? please give me a reason. to convey a sense of being overwhelmed or surrounded by difficulties . voice is extremely melodious. detecting and understanding metaphor is crucial for downstream nlp tasks, including information extraction @xcite , sentiment analysis @xcite , machine translation @xcite , and seamless human-computer interaction (rai and chakraverty, 2020). traditional metaphor detection methods include using dependency trees @xcite , employing prompt @xcite , and multi-task learning @xcite . however, current metaphor detection tasks face generalization issues (see experiment 1). the aforementioned metaphor detection methods are insufficient, because they only require the model to determine whether the target word is used metaphorically in context. additional information is needed , such as metaphor reason, seeing figure 1 . in metaphor-related interpretation researches, previous work has only replaced the target word used metaphorically in the context with a literal word. however, this method also do not address the issue of metaphor reason. ## implementation for the baseline in the experiments, the epoch of mlm is set to 25, the learning rate is initialized to 3e-5, and the metaphor weight is given to 5. the epoch of clm is set to 50. both mlm and clm are initialized using the weight parameter of huggingface library. the hidden layer unit of the classifier is set according to the size of the model, which is set to 768 for the base model and 1024 for the large model. the experiments are run on a cloud server with a single a100 80g gpu. ## experimental analysis experiment 1: the experimental results are shown in ## automatic evaluation experiment in comparison to the reason generated from the original dataset in section 3.2, we used automatic evaluation methods to assess the supervised method gpt2-large, as well as the llms methods chatgpt3.5 and llama3-70b. we employed three evaluation metrics: bleu, meteor, and rouge. all three metrics use n-gram matching mechanisms but differ slightly in the factors they consider. specifically, bleu and rouge emphasize precision and recall, respectively, while me-teor additionally takes into account synonyms and stems. the automatic evaluation results are shown in ## evaluation experiment of text-embedded fine-tuning model the fine-tuning model evaluation method first trains the roberta-large model on entailment datasets. then, it evaluates the results generated by gpt2-large, chatgpt3.5, and llama3. the advantage of this method is that, compared to direct vocabulary distribution calculation in automatic evaluation methods, the fine-tuned model often contains certain high-dimensional semantic information. in this experiment, we used two entailment datasets: semantic textual similarity benchmark (sts-b) and sentences involving compositional knowledge (sick). the sts-b is a similarity and paraphrase dataset, consisting of sentence pairs extracted from news headlines, video titles, image captions, and natural language inference data, each annotated by humans. the sick dataset is used for compositional distributional semantics. it includes a large number of sentence pairs that exhibit rich lexical, syntactic, and semantic phenomena. each pair of sentences is annotated with two dimensions: relatedness and entailment. we only consider relatedness, with scores ranging from 1 to 5. the experimental results are shown in ## chatgpt evaluation experiment the chatgpt evaluation method aims to use chatgpt to score the "similarity" between the original dataset output and the prediction model output. its advantage lies in the fact that chatgpt has been trained on a broader and higher-quality dataset, enabling it to understand more complex semantic information and relationships. for the i-th sample n @xmath0, target word w i , and context s i , the prompt for chatgpt is designed as follows: discuss the use of "w i " in "s i ". answer: a i prediction: a i,j rate the prediction based on the answer (1 to 5). ## manual evaluation experiment finally, we used a manual method for "similarity" evaluation. compared to the previous three methods, manual evaluation can comprehensively consider aspects such as semantics, fluency, and logic, thus providing more accurate results. to reduce subjectivity, we invited three volunteers to independently evaluate the correct answers and the modelpredicted answers, with a scoring range of 1-5. the results from the three volunteers were then averaged to provide the final evaluation score for each prediction. the experiment results are detailed in ## case study as shown in ## conclusion as an advanced approach, current textual metaphor detection tasks typically provide usage labels. however, this method falls short in understanding metaphors. metaphor research requires not only detection but also in-depth reasoning. to our knowledge, we are the first to utilize llms for metaphor reason to better understand metaphorical meanings. firstly, we expanded existing traditional metaphor datasets using the chatgpt and provided results for metaphor reason. secondly, we designed supervised experiments demonstrating that current metaphor detection models have poor generalization performance, while incorporating metaphorical reasons significantly improves the models' generalization ability. additionally, to evaluate llms performance in metaphor reason, we proposed four novel evaluation methods. overall, our experimental results indicate that llms can reduce resource consumption for data reasoning (compared to crowdsourcing) and show significant room for improvement.