Testing the Ability of Language Models to Interpret Figurative Language (2022)

Abstract: Figurative and metaphorical language are commonplace in discourse, and figurative expressions play an important role in communication and cognition. However, figurative language has been a relatively under-studied area in NLP, and it remains an open question to what extent modern language models can interpret nonliteral phrases. To address this question, we introduce Fig-QA, a Winograd-style nonliteral language understanding task consisting of correctly interpreting paired figurative phrases with divergent meanings. We evaluate the performance of several state-of-the-art language models on this task, and find that although language models achieve performance significantly over chance, they still fall short of human performance, particularly in zero- or few-shot settings. This suggests that further work is needed to improve the nonliteral reasoning capabilities of language models.

## introduction all our words are but crumbs that fall down from the feast of the mind @xcite . when humans read such a metaphorical phrase, how do they interpret it? conceptual metaphors structure our everyday language and are used to map everyday physical experiences and emotions onto abstract concepts @xcite . they allow us to communicate complex ideas, to emphasize emotions, and to make humorous statements @xcite . however, despite relating words in a way that differs from their accepted definition, these phrases are readily interpreted by human listeners, and are common in discourse @xcite , occurring on average every three sentences @xcite the ability to interpret figurative language has been viewed as a bottleneck in natural language un-derstanding, but it has not been studied as widely as literal language @xcite . figurative language often relies on shared commonsense or cultural knowledge, and in some cases may be difficult to solve using language statistics. this presents a challenge to language models (lms), as strong lms trained only on text may not be able to make sense of the physical world, nor the social or cultural knowledge that language is grounded in @xcite . most previous work on figurative language focuses on metaphor detection, where a model is trained to identify the existence of metaphors in text @xcite @xcite , with datasets consisting mostly of conventionalized metaphors and idioms in wide use. however, identifying these common metaphors that already appear often in language may be an easy task for lms, and not fully test their ability to interpret figurative language. the little work that exists on metaphor interpretation frames it as a task linking metaphorical phrases to literal rewordings, either through paraphrase detection @xcite or paraphrase generation @xcite @xcite (details in § 7) another line of work probes for metaphorical understanding in lms, but this is similar to the metaphor detection task, in that the lm is not actually asked to choose an interpretation for the metaphor @xcite . while interesting, this work does not take into account the fact that metaphors are rich with different implications that may vary depending on the context. in this work, we ask whether or not lms can correctly make inferences regarding creative, relatively novel metaphors generated by humans. this task is harder for two reasons: (1) inference is harder than identification or paraphrasing, as it requires understanding the underlying semantics, and (2) the metaphors in our dataset are novel cre-ations, and many may not appear even once in the lms' training data. we propose a minimal task inspired by the winograd schema @xcite , where lms are tasked with choosing the entailed phrase from two opposite metaphorical phrases. an example of a paired sentence is "her commitment is as sturdy as (plywood/oak)". the correct answer would be either "she was (committed/uncommitted)". this can also be seen as an entailment task, where input x is the premise, and the output y is the hypothesis. 2 we crowdsource a benchmark fig-qa, consisting of 10,256 such metaphors and implications ( § 2), which can be used to evaluate the nonliteral reasoning abilities of lms or for more broad studies of figurative language in general (we provide preliminary analyses in § 3). through extensive experiments over strong pre-trained lms ( § 4), we find that although they can be fine-tuned to do reasonably well, their few-shot performance falls significantly short of human performance ( § 5). an in-depth analysis ( § 6) uncovers several insights: (1) lms do not make use of the metaphorical context well, instead relying on the predicted probability of interpretations alone, (2) the task of associating a metaphor with an interpretation is more difficult than the reverse, (3) even strong models such as gpt-3 make inexplicable errors that are not well-aligned with human ones, indicating that further work is needed to properly model nonliteral language. ## figurative language typologies in this sample, we perform an analysis of the collected data to demonstrate its trends and categorize examples for further error analysis. specifically, we examine (a) subjects, objects, and relations, and (b) types of common-sense knowledge needed to interpret the metaphor. ## performance and error analysis with these results in mind, we examine what kinds of errors models make, and what factors make the task difficult.. this is covered in § 6. we find that autoregressive models rely on the predicted probability of each answer by itself to predict the answer, and that this holds true for all models, before and after training. we find that models have difficulty in interpreting "sarcastic" metaphors, and sometimes inexplicably interpret very simple metaphors wrong. we also examine error typology according to the commonsense typology of § 3.2 and find that models improve significantly on object, visual and social commonsense when trained, but not on cultural commonsense. ## conclusion we present a winograd-like benchmark task to test the ability of lms to reason about figurative language, based on large-scale collection of creative metaphors written by humans. we find a large gap between lm zero-shot and human performance on this dataset, but show that models can be fine-tuned to perform well on this particular task. we hope that this work will encourage further study of nonliteral reasoning in lms, especially in few-shot settings. given that metaphorical reasoning may play a role in problem-solving and linguistic creativity, the development of models, training methods, or datasets that enable metaphorical reasoning may improve models' abilities to reason creatively and draw analogies between situations that may appear to be different on the surface. one avenue we hope to investigate is multimodal metaphors, as this dataset currently includes only text-based metaphors. nonliteral expressions also remain understudied cross-linguistically, but further work on identifying and interpreting metaphors in other languages may also improve the abilities of multilingual models.