Does GPT -3 Grasp Metaphors? Identifying Metaphor Mappings with Generative Language Models (2023)

Abstract: Conceptual metaphors present a powerful cognitive vehicle to transfer knowledge structures from a source to a target domain. Prior neural approaches focus on detecting whether natural language sequences are metaphoric or literal. We believe that to truly probe metaphoric knowledge in pre-trained language models, their capability to detect this transfer should be investigated. To this end, this paper proposes to probe the ability of GPT-3 to detect metaphoric language and predict the metaphor’s source domain without any pre-set domains. We experiment with different training sample configurations for fine-tuning and few-shot prompting on two distinct datasets. When provided 12 few-shot samples in the prompt, GPT-3 generates the correct source domain for a new sample with an accuracy of 65.15% in English and 34.65% in Spanish. GPT’s most common error is a hallucinated source domain for which no indicator is present in the sentence. Other common errors include identifying a sequence as literal even though a metaphor is present and predicting the wrong source domain based on specific words in the sequence that are not metaphorically related to the target domain.

## introduction metaphor processing with pre-trained language models (e.g. @xcite has been dominated by metaphor detection, that is, the classification of expressions into metaphoric or literal (e.g. @xcite . in metaphor interpretation, a common approach is to paraphrase metaphoric expressions into literal ones (e.g. @xcite . few approaches target metaphor identification, e.g. predicting the source domain of a metaphor in a linguistic sequence. for instance, @xcite relies on grammatical constructs and pre-defined labels. instead, in this paper, we test a generative language model's ability to predict the source domain given a target domain and sequence without grammatical assumptions or fixed source domain labels. conceptual metaphor theory (cmt) @xcite starts from the assumption that metaphors represent a powerful cognitive mechanism to transfer physical knowledge structures to abstract domains. in natural language, he was bombarded by insults or your words pierce my heart transfers the concrete domain of weapons to the abstract domain of words in the metaphor words are weapons. on the assumption that our cognitive organization relies on metaphors, automatically identifying metaphoric transfer holds the promise of contributing to more human-like computational models. from the overall success of pre-trained language models in metaphor detection, a certain degree of metaphoric knowledge in these models can be assumed @xcite . this paper aims to evaluate whether this inherent knowledge extends beyond contextual clues to predict the concrete domains in the metaphoric transfer. detecting a metaphor entails contrasting the physical with the abstract meaning of a sequence. however, the source domain is frequently a noncontextual attribute @xcite , while the target domain can be found directly using contextual clues. for instance, in the above example, pierce is more implicitly related to weapons than the explicit words is to words. to determine the accuracy of the predicted source domains from fine-tuning and few-shot prompting gpt-3 @xcite , we manually evaluate the results. to this end, we propose a classification of error types from too generic domains to relying on words in the sentence that are not connected to the metaphor, which we call trigger words. this provides further intuition on the nature and extent of metaphoric knowledge encoded in pre-trained language models. we compare methods to elicit metaphoric knowledge without any assumptions on grammar or source domains and test if it ex-tends across languages, i.e., spanish in addition to english. finally, we evaluate its generalization by testing on two distinct datasets and a set of nonmetaphoric sentences. ## preliminaries two major pillars that build the foundation for this approach are conceptual metaphors and generative language models, which we briefly introduce here. ## results this section describes the results from the experiments that determine the manually evaluated test set predictions, their accuracy, and types of errors. ## discussion we experimented with different gpt-3 variants and prompts containing varying numbers of few-shot samples to see whether gpt-3 can generate the source domain of a conceptual metaphor mapping given a context and a target domain. the best results were achieved with a long few-shot prompt containing 12 example completions. the largest model variant davinci-002 strongly outperformed the next biggest variant and a fine-tuned gpt-3. we also saw that fine-tuning the model can lead to a decrease in expressiveness, that is, fewer unique source domains being generated. in our case, this might be because the model fine-tuned on all data sees each source domain around three times per training. it might be possible to counteract the decrease in expressiveness by increasing the temperature parameter, thus, making less probable generations more likely. manually coding the errors made by the model, we saw that the model often fabricates source domains for which no related words are present in the sentence. other common errors included predicting a literal meaning although a metaphor was present, and generating wrong source domains based on trigger words that were not metaphorically related error code definition example % of all errors sentence wrong prediction wrong with trigger the model predicts a wrong source domain due to words in the sentence related to that domain the arms race competition is war 21.31 wrong without trigger the model predicts a wrong source domain without any noticeable triggers for that domain in the sentence sally gave the idea to sam ideas are chil-dren 27.32 too literal the model predicts a literal relationship instead of a metaphoric mapping i'm down to my bottom dollar money is in-vestment 7.10 should be nonmetaphoric the model predicted a metaphoric source domain instead of non-metaphoric they saw him advancing moving is com-ing 7.65 should be metaphoric the model wrongly predicted non-metaphoric under the cover of darkness darkness is non-metaphoric 25.14 too specific the predicted metaphor is more specific than what the sentence implies. he finally caught up to schedule schedule is people 2.73 too general the predicted source domain is too unspecific the idea slipped through my fingers mind is space 1.09 wrong subelement mapping the model predicts an aspect of the correct source domain, however, it is not the exact element let's strip away the unimportant details importance is clothing 7.65 to the target domain. discerning whether to predict a source domain for a given sentence or to label it as non-metaphoric seems to be quite challenging for the model as well. analyzing the errors of large language models as done here is essential to build appropriate trust or distrust in the model and allow for the use of error-correction methods in the future, for instance, the selection of better prompts or training samples. in the context of analyzing the model's misclassifications, we also experienced issues with the dataset, e.g. unintuitive metaphor mappings or lack of contextual clues for the provided target domain. the dataset's quality strongly affected the spanish test results and clearly indicated that more multilingual resources for metaphor identification are needed. the difference in the nature and quality of the datasets is also the main reason for the strong variation in accuracy results. the metaphor list dataset provides prototypical, general language examples, while the lcc dataset annotated real-world, domain-specific expert language. this affects the complexity as well as the length of sentences, both contributing to the difference in accuracy across datasets. application. using gpt-3 to analyze metaphors used in an unlabeled corpus comes with two problems: (1) we do not know what target domains are the right ones to provide to the model, (2) there will be an overwhelming amount of output given that most sentences contain at least subtle metaphoric language that will largely not even be relevant to the domain we are interested in. therefore, it would be useful to first filter sentences based on seed words whose usage interests us or that belong to a specific target domain we want to analyze @xcite . as such an approach already narrows down the candidate sentences to a pre-specified target domain, we can include that target domain in the prompt for the language model. lastly, it might help to restrict the context window around the words of interest so that the model is not distracted by other metaphors in the sentence. however, to confirm this, further research is needed. considering precise element mappings. as the capabilities of large neural language models continue to grow, it will be interesting to see if they can identify not only the correct source domains but also precise element-wise mappings between the concepts of the target and source domain. for example, the conceptual metaphor love is a jour-ney involves mapping lovers to travelers, difficulties to roadblocks, and progress to distance traveled forward. querying such an element-wise mapping could be facilitated through a set of the target domain's core elements being provided to the model. ## conclusion we analyzed how well gpt-3 can identify source domains of metaphors in natural language. across three different datasets in english and spanish, gpt-3 predicts the source domain with an accuracy of 60.22%. the best performance was achieved given 12 few-shot examples in the prompt, although the average performance was highest with 4 to 8 few-shot examples. however, the model still suffers from specific error types, such as hallucinating domains without any indicators being present. we believe future iterations of large language models like gpt-3 will become important tools in computational metaphor analysis, where one investigates conceptual metaphors in different domains, for instance, literature or political discourse. in the future, we want to experiment with using large language models to generate complete metaphors, i.e., generate both, source domain and target domain, given a sentence. we also plan to use the developed techniques in corpus analyses.