Figurative and metaphorical language are commonplace in discourse , and figurative expressions play an important role in communication and cognition . However , figurative language has been a relatively under-studied area in NLP , and it remains an open question to what extent modern language models can interpret nonliteral phrases . To address this question , we introduce Fig-QA , a Winograd-style nonliteral language understanding task consisting of correctly interpreting paired figurative phrases with divergent meanings . We evaluate the performance of several state-of-the-art language models on this task , and find that although language models achieve performance significantly over chance , they still fall short of human performance , particularly in zero-or few-shot settings . This suggests that further work is needed to improve the nonliteral reasoning capabilities of language models . 1 All our words are but crumbs that fall down from the feast of the mind ( Gibran , 1926 ) . When humans read such a metaphorical phrase , how do they interpret it ? Conceptual metaphors structure our everyday language and are used to map everyday physical experiences and emotions onto abstract concepts ( Lakoff and Johnson , 1981 ) . They allow us to communicate complex ideas , to emphasize emotions , and to make humorous statements ( Fussell and Moss , 2008 ) . However , despite relating words in a way that differs from their accepted definition , these phrases are readily interpreted by human listeners , and are common in discourse ( Shutova , 2011 ) , occurring on average every three sentences ( Mio and Katz , 1996 ; Fussell and Moss , 2008 ) The ability to interpret figurative language has been viewed as a bottleneck in natural language un-derstanding , but it has not been studied as widely as literal language ( Shutova , 2011 ; Tong et al . , 2021 ) . Figurative language often relies on shared commonsense or cultural knowledge , and in some cases may be difficult to solve using language statistics . This presents a challenge to language models ( LMs ) , as strong LMs trained only on text may not be able to make sense of the physical world , nor the social or cultural knowledge that language is grounded in ( Bender and Koller , 2020 ; Bisk et al . , 2020 ) . Most previous work on figurative language focuses on metaphor detection , where a model is trained to identify the existence of metaphors in text ( Tsvetkov et al . , 2014 ; Stowe and Palmer , 2018 ; Leong et al . , 2020 ) , with datasets consisting mostly of conventionalized metaphors and idioms in wide use . However , identifying these common metaphors that already appear often in language may be an easy task for LMs , and not fully test their ability to interpret figurative language . The little work that exists on metaphor interpretation frames it as a task linking metaphorical phrases to literal rewordings , either through paraphrase detection ( Bizzoni and Lappin , 2018 ) or paraphrase generation ( Shutova , 2010 ; Su et al . , 2017 ; Mao et al . , 2018 ) ( details in § 7 ) Another line of work probes for metaphorical understanding in LMs , but this is similar to the metaphor detection task , in that the LM is not actually asked to choose an interpretation for the metaphor ( Pedinotti et al . , 2021 ; Aghazadeh et al . , 2022 ) . While interesting , this work does not take into account the fact that metaphors are rich with different implications that may vary depending on the context . In this work , we ask whether or not LMs can correctly make inferences regarding creative , relatively novel metaphors generated by humans . This task is harder for two reasons : ( 1 ) inference is harder than identification or paraphrasing , as it requires understanding the underlying semantics , and ( 2 ) the metaphors in our dataset are novel cre-ations , and many may not appear even once in the LMs ' training data . We propose a minimal task inspired by the Winograd schema ( Levesque et al . , 2012 ) , where LMs are tasked with choosing the entailed phrase from two opposite metaphorical phrases . An example of a paired sentence is " Her commitment is as sturdy as ( plywood / oak ) " . The correct answer would be either " She was ( committed / uncommitted ) " . This can also be seen as an entailment task , where input x is the premise , and the output y is the hypothesis . 2We crowdsource a benchmark Fig-QA , consisting of 10,256 such metaphors and implications ( § 2 ) , which can be used to evaluate the nonliteral reasoning abilities of LMs or for more broad studies of figurative language in general ( we provide preliminary analyses in § 3 ) . Through extensive experiments over strong pre-trained LMs ( § 4 ) , we find that although they can be fine-tuned to do reasonably well , their few-shot performance falls significantly short of human performance ( § 5 ) . An in-depth analysis ( § 6 ) uncovers several insights : ( 1 ) LMs do not make use of the metaphorical context well , instead relying on the predicted probability of interpretations alone , ( 2 ) the task of associating a metaphor with an interpretation is more difficult than the reverse , ( 3 ) even strong models such as GPT-3 make inexplicable errors that are not well-aligned with human ones , indicating that further work is needed to properly model nonliteral language . 2 Dataset Creation and Validation We present a Winograd-like benchmark task to test the ability of LMs to reason about figurative language , based on large-scale collection of creative metaphors written by humans . We find a large gap between LM zero-shot and human performance on this dataset , but show that models can be fine-tuned to perform well on this particular task . We hope that this work will encourage further study of nonliteral reasoning in LMs , especially in few-shot settings . Given that metaphorical reasoning may play a role in problem-solving and linguistic creativity , the development of models , training methods , or datasets that enable metaphorical reasoning may improve models ' abilities to reason creatively and draw analogies between situations that may appear to be different on the surface . One avenue we hope to investigate is multimodal metaphors , as this dataset currently includes only text-based metaphors . Nonliteral expressions also remain understudied cross-linguistically , but further work on identifying and interpreting metaphors in other languages may also improve the abilities of multilingual models .

Challenge: Figurative language has been under-studied and existing works on metaphorical understanding of language models do not take the richness of such expressions into account.
Approach: They introduce a Winogrand-style nonliteral language understanding task to test if language models can make correct inferences on human-generated metaphors.
Outcome: Evaluation of state-of-the-art models with the proposed dataset shows that they underperform humans in a zero-shot setting but can improve with fine-tuning.