Does the “Most Sinfully Decadent Cake Ever” Taste Good? Answering Yes/No Questions from Figurative Contexts (2023)

Abstract: Figurative language is commonplace in natural language, and while making communication memorable and creative, can be difficult to understand. In this work, we investigate the robustness of Question Answering (QA) models on figurative text. Yes/no questions, in particular, are a useful probe of figurative language understanding capabilities of large language models. We propose FigurativeQA, a set of 1000 yes/no questions with figurative and non-figurative contexts, extracted from the domains of restaurant and product reviews. We show that state-of-the-art BERT-based QA models exhibit an average performance drop of up to 15% points when answering questions from figurative contexts, as compared to non-figurative ones. While models like GPT-3 and ChatGPT are better at handling figurative texts, we show that further performance gains can be achieved by automatically simplifying the figurative contexts into their non-figurative (literal) counterparts. We find that the best overall model is ChatGPT with chain-of-thought prompting to generate non-figurative contexts. Our work provides a promising direction for building more robust QA models with figurative language understanding capabilities.

## related work figurative language has been a difficult problem for many natural language processing (nlp) applications. a number of computational approaches have been proposed to study their occurrence in text @xcite @xcite @xcite , including generation of figurative language @xcite . the idea of converting metaphors to their literal counterparts has been previously explored for machine translation by @xcite , where metaphors in english text are first identified and then converted to a literal version by using word embeddings and wordnet, before doing machine translation into chinese. in dialog systems, a similar approach was employed by @xcite , where idioms and metaphors in utterances are converted to literal versions using a dictionary lookup-based method. our work is closest to @xcite , except that we explore the robustness of qa systems in a machine comprehension setup, instead of dialog models, to figurative language, which, to the best of our knowledge, is a first. our automatic approach to creating rephrased non-figurative versions of figurative text is done using pre-trained language models, rather than rulebased methods which have been shown to be errorprone @xcite . in a concurrent work, split source example figurative amazon context: the album , like almost everything krush has released , slays . question: is the album good? answer: yes non-fig. version of the context (manual): the album is really good, like most of krush's work. non-fig. version of the context (from gpt-3): the album is really good, like almost everything krush has released. figurative yelp context: although, the menu items doesnt scream french cuisine. most foods looks like you can get at any american place. question: is the menu authentic french? answer: no non-fig. context (manual): the menu items aren't typical of french cuisine. rather, they are common at most american eateries. non-fig. context (from gpt-3): although, the menu items doesn't look very french. most foods look like you can get at any american place. non-figurative amazon context: nice ring, but the color is paler than the picture . question: is the ring brightly colored? answer: no non-figurative yelp context: the chicken is delicious and so are the ribs question: did the food taste good? answer: yes the figurative text fragments within the contexts are shown in bold and italics. @xcite have also done prompting on gpt-3 to create their figurative nli dataset, flute, as well as obtain an explanation of the nli labels in this dataset. to our knowledge, there are no qa datasets specifically designed for figurative language understanding, but some existing qa datasets do contain figurative language. the friendsqa dataset (yang and choi, 2019) is a dialog-based qa dataset constructed from dialogs from the tv series friends. while it does contain metaphors and sarcasm, the focus of the dataset is not figurative language, and it is not ideal for testing figurative language understanding as it is unclear how much of the dataset is figurative. the dialog nature of the dataset further contributes to making it challenging and complicates studying the effect of figurativeness. another dataset that requires figurative language understanding is the riddlesense dataset @xcite , which comprises of riddles, but unlike ours, it's modeled as an open-domain qa task rather than a machine comprehension task. @xcite show that questions about novel metaphors from literature are judged to be deeper than nonmetaphorical or non-conventional metaphors by humans, but their focus is on generating deep questions rather than testing the robustness of qa models. dankin et al. construct yes/no questions using templates to detect the presence of metaphors in a few-shot setting. ## figurativeqa dataset the contexts in figurativeqa comes from two sources: amazon product reviews @xcite , and yelp restaurant reviews @xcite . we extract both figurative and non-figurative contexts from each source. we manually construct yes/no questions and answers on top of these contexts. figure 2 shows examples from figurativeqa. the data statistics from each source (amazon and yelp) and each split (figurative and non-figurative) are summarized in for the amazon part of figurativeqa, we use @xcite 's dataset of figurative comparisons. of the 1260 comparisons in this dataset, we extract instances where all 3 annotators are in agreement about figurativeness (i.e., average figurativeness score of greater than 3). we then randomly pick 150 exam- split context fig. construct amazon the books are like potato chips -you can't eat just one . simile, idiom so when my laptop battery puffed up like a balloon , i dreaded paying the cost of replacement . simile, hyperbole really , this novel feels more like a footnote to the series whereas the gunslinger was a novel that stood extremely well on its own . simile, idiom these horrible recordings contain treasure more precious than gold. simile, sarcasm yelp i had the chicken fajitas , which came with a giant flour tortilla that was as hot as hades . for the yelp part of the dataset, the contexts are sourced from @xcite 's nlg dataset for the restaurant domain. since highly positive or highly negative reviews are more likely to contain figurative language, we extract these first, and then, similar to @xcite , use comparator expressions to get a set of reviews likely to be rich in figurative content. we then manually examine these reviews to annotate 350 examples of figurative contexts and non-figurative contexts, each. the figurative contexts from figurativeqa tend to contain more similes, since comparator patterns ("like", "as", or "than") were used to extract the text. however, we observe that many of these examples also contain other kinds of figurative constructs such as metaphor, idiom, hyperbole, sarcasm, etc. for each context in figurativeqa, we construct a yes/no question. for the figurative contexts, we make sure to pose a question such that answering it would require an understanding of the figurative text present in the context. for the non-figurative contexts, we construct questions similar to the ones for the figurative contexts. additionally, for the fig-urative contexts extracted from amazon and yelp, we manually create non-figurative counterparts that preserve the meaning and overall content. ## conclusion and future work to automate this approach, we propose a method of prompting gpt-3 to produce simplified, nonfigurative contexts, which yields significant performance gains over the baseline. chain-of-thought prompting using chatgpt has the best overall performance on figurativeqa. we hope that our method and dataset will spur more research into question answering with figurative language.