Can C hat GPT ’s Performance be Improved on Verb Metaphor Detection Tasks? Bootstrapping and Combining Tacit Knowledge (2024)

Abstract: Metaphors detection, as an important task in the field of NLP, has been receiving sustained academic attention in recent years. Current researches focus supervised metaphors detection systems, which usually require large-scale, high-quality labeled data support. The emerge of large language models (e.g., ChatGPT) has made many NLP tasks (e.g., automatic summarization and dialogue systems) a qualitative leap. However, it is worth noting that the use of ChatGPT for unsupervised metaphors detection is often challenged with less-than-expected performance. Therefore, the aim of our work is to explore how to bootstrap and combine ChatGPT by detecting the most prevalent verb metaphors among metaphors. Our approach first utilizes ChatGPT to obtain literal collocations of target verbs and subject-object pairs of verbs in the text to be detected. Subsequently, these literal collocations and subject-object pairs are mapped to the same set of topics, and finally the verb metaphors are detected through the analysis of entailment relations. The experimental results show that our method achieves the best performance on the unsupervised verb metaphors detection task compared to existing unsupervised methods or direct prediction using ChatGPT. Our code is available at https://github.com/VILAN-Lab/Unsupervised-Metaphor-Detection.

## introduction metaphors are essentially mapping relationships between two different domains @xcite . according to the conceptual metaphor theory @xcite , linguistic metaphors derive from underlying conceptual metaphors that map a source concept to another, more abstract, target concept. metaphors detection aims at modeling nonliteral expressions (e.g., metaphors and metonymy) and generating corresponding metaphorical annotations. it is beneficial to many nlp tasks, e.g., information extraction @xcite , sentiment analysis @xcite , and machine translation @xcite . in past research, most metaphor detection methods primarily used supervised approaches @xcite . although these methods achieved excellent performance on test sets. however, they rely on well-labeled datasets. high-quality labeled data is both time-consuming and expensive, especially for metaphor samples. in addition, current supervised metaphor detection methods suffer from low generalization performance, impeding the efficacy of metaphor detection systems in practical applications. to cope with the above problems, researchers explored the unsupervised domain. @xcite constructed a topic list using latent derechter allocation (lda) @xcite . @xcite constructed a clustering map. @xcite and @xcite introduced lexical abstraction to study copular verbs metaphor and real verbs metaphor, respectively. however, these methods usually require complex hand-coding rules. to simplify the methods, @xcite and @xcite used cosine distance to determine whether subject-verb or verbobject pairs belong to the same conceptual domain. nevertheless, these methods still rely on partially manually labeled datasets. with the development of large language models (llms), and in particular chatgpt's excellent performance on zero-shot or few-shot nlp tasks @xcite , we are inspired to consider utilizing world knowledge of chatgpt to augment a metaphors detection system. given that verb metaphors occupy the broadest class of metaphors @xcite , many supervised @xcite and unsupervised methods @xcite focus on verbs, our work likewise concentrates on the verb part. we propose an unsupervised verb metaphors detection method based on chat-gpt. first, we build a verb list that records the literal meaning collocation of each verb. then, we introduce topical features that map the subject and object of the target verb to one or more topical categories. next, we analyze the subjects and objects of the verbs to be detected in the input text and map them to topical categories as well. finally, we detect verb metaphors through the analysis of entailment relations. we test our model on the vuaverb, moh-x, and trofi datasets, and the results show that by bootstrapping and integrating the implicit knowledge of chatgpt, it can effectively improve performance on the verb metaphors detection task. in summary, the main contributions of our work are summarized as follows: ## related work to minimize the reliance on labeled data, researchers have explored a lot on unsupervised methods. karov and edelman (1998) used a word sense disambiguation (wsd) algorithm to cluster sentences with target words, and then made metaphor predictions based on the principle of distance between literal meanings of words. @xcite also drew on the idea of clustering, and it used the gigaword corpus @xcite with noun-related of verb-noun combinations (grammatical features) to cluster the 2000 common nouns of the bnc. in this approach, the words to be detected acquire knowledge information at a certain layer in the clustering map, i.e., the nouns at that layer are non-metaphorically related to the words to be detected. @xcite presented an approximately unsupervised metaphors detection system. the system selects the best alternative to the target word by considering superlatives and synonyms in the context. when the cosine distance between the best alternative and the target word is greater than a specific threshold, it is detectd as a literal meaning. in addition, other studies @xcite have considered the cosine distance, although @xcite did not use a priori labeled data to set the threshold, instead it adopted a feature construction approach using clustering for metaphorical judgments. some studies @xcite explored the relationship between the abstraction degree of focus words and the expression of language metaphors. @xcite used the abstraction degrees of nouns, proper nouns, verbs and adverbs were first calculated, and then logistic regression to learn high-dimensional metaphoric features. @xcite used wordnet to generate n common collocations of the words to be detected and sorted these collocations according to the abstraction level. a metaphorical relationship word is detectd as a metaphor if it is not between the first k most concrete words. this idea is also reflected in the study of @xcite , which investigated three metaphorical relations, subject-be-object, verb-object and adjective-noun, and identified metaphors by determining whether the two focal words have a hyponymy relation. although the above methods have been effective to a certain extent, there are still problems such as complex parsing of metaphorical relationships, cumbersome construction of hand-coded knowledge, or reliance on manually labeled data. to overcome these challenges, we attempt to introduce generative language modeling into the metaphors detection task. the main function of generative language models is to generate natural language text, which can be used for conversing with humans or performing text generation tasks. in previous research, wachowiak and gromann (2023) used gpt-3 for supervised metaphor generation. the study first provided input text and target domain information, and then utilized gpt-3 to predict source domain information. this is a good attempt, but still relies on labeled data. the difference is that our study focuses on unsupervised method to acquire implicit knowledge of chatgpt through bootstrapping and integration. our approach achieves significant performance gains in the unsupervised metaphor detection task. ## method we divide the proposed method into three parts: definition of verb metaphors, topic mapping and verb list. ## results and discussion experiment 1. from the results in experiment 2. the results of comparing with the sample generation methods are shown in ## topic experiment we examined the impact of the three topic mappings introduced in §3.2 on model performance. for wordnet topics, we use the nltk library in python to extract the superlatives and synonyms of the central noun, and then combine all of them into the wordnet topics set corresponding to the target verb. for lda topics, we use wups @xcite to calculate the similarity between the central noun and the 60 lda topics words, and classify them into one or more lda topics based on the similarity. for oxford topics, we first access the oxford lexicon for pronoun disambiguation and named entity conversion, and then convert them into one or more topic categories corresponding to the oxford lexicon. specifically, we first parse the input text to extract the subject and object corresponding to the target verb. we select by default the first k nouns as the subject content to be converted (k is a hyperparameter). we consider the case of extracting 1 or 3 central nouns. specific topic types include wordnet_topic, wordnet_topic_k, lda_topic, lda_topic_k, oxford_topic, oxford_topic_k, where k means extracting the first k nouns as the center nouns. as shown in ## hyper-parameter experiment to balance the set size with the metaphors detection accuracy when introducing topic sets, we introduce two additional hyperparameters for control. specifically, k 1 represents the number of literal or non-metaphorical collocations selected from the verb list, while k 2 denotes the number of topics that may be covered by the subject and object corresponding to the target verb. larger values of k 1 imply that the model's predictions cover more literal-meaning collocations of verbs, while larger values of k 2 indicate that more meanings of the subject-or object-centered words are used in the metaphorical relations parsed in the text. in this regard, the hyper-parameter experiment aims to explore the effect of two hyper-parameters, k 1 and k 2 , on the model metaphor detection performance. considering the results of the previous topic experiment, we find that oxford_topic_k, which extracts 3 central nouns, performs better relative to oxford_topic_k, which extracts 1 central word. moreover, when only 1 central noun is extracted, there are relatively fewer topic types (which depends on the number of different meanings of that central noun). specifically, the hyperparameter experiment will fix the hyper-parameter of the center word as @xmath0, while setting the value range of k 1 and k 2 between 1 and 9. in addition, the experiments will be conducted on the moh-x. detailed results can be found in figure 1 . on the one hand, the model performance improves as the value of the hyperparameter k 1 increases. this can be attributed to the fact that increasing k 1 introduces more literal collocations from the verb list. as a result, the model is more capable of detecting the non-metaphorical content associated with a particular verb and reduces misclassification. on the other hand, the performance peaks when the hyperparameter k 2 is set to 3. however, when continuing to increase the value of k 2 , the model's performance in detecting metaphors decreases instead. this suggests that considering multiple meanings of the central word may introduce metaphorical information or redundant topics. thus, our experimental results emphasize the need to weigh the model performance and the impact of topic introduction when choosing the value of k 2 . ## conclusion we present a novel approach aimed at improving the performance of unsupervised verb metaphors detection task using chatgpt. this approach does not rely on hand-coded knowledge or manually labeled datasets. first, we construct a literal meaning collocation lookup list for each target verb. when parsing the input text, we pay special attention to the subjects and objects corresponding to the verbs to be detected. we introduced a variety of topics, including wordnet topics, lda topics, and oxford topics. by comparing the relationship between subject and object topics in the input text and the verb topics in the verb list, we determine whether the text contains metaphorical expressions. the results show that by delicately combining and directing the world knowledge, we are able to significantly improve the performance of chatgpt in the verb metaphors detection task.