F igurative QA : A Test Benchmark for Figurativeness Comprehension for Question Answering (2022)

Abstract: Figurative language is widespread in human language (Lakoff and Johnson, 2008) posing potential challenges in NLP applications. In this paper, we investigate the effect of figurative language on the task of question answering (QA). We construct FigQA, a test set of 400 yes-no questions with figurative and non-figurative contexts, extracted from product reviews and restaurant reviews. We demonstrate that a state-of-the-art RoBERTa QA model has considerably lower performance in question answering when the contexts are figurative rather than literal, indicating a gap in current models. We propose a general method for improving the performance of QA models by converting the figurative contexts into non-figurative by prompting GPT-3, and demonstrate its effectiveness. Our results indicate a need for building QA models infused with figurative language understanding capabilities.

## introduction understanding figurative language can be a challenging task for humans, let alone for machines @xcite . although native speakers may effortlessly infer the meaning of similes and metaphors, it may be particularly difficult for nonnative speakers to understand. effects of the presence of figurative language has been studied for various downstream nlp tasks such as machine translation @xcite , recognizing textual entailment @xcite , and dialog models @xcite , inter-alia. to the best of our knowledge, there is no prior line of work investigating question answering (qa) on figurative text. figurative language has a limited presence in existing question answering (qa) datasets in popular use such as squad @xcite and natural questions @xcite , where the contexts are typically literal and factual, constructed from wikipedia passages. 1 we propose a new task of answering questions from text that is figurative, and consequently, more challenging to answer. for this task, we present a test dataset, figurativeqa, consisting of 400 questions and accompanying figurative contexts constructed from amazon product reviews @xcite and yelp restaurant reviews @xcite . we leverage existing resources for figurative contexts @xcite and manually construct questionanswer pairs from these contexts. further, we create non-figurative versions of this dataset, both automatically by prompting gpt-3 @xcite as well as manually. we show that it is harder to answer questions from figurative context for current state-of-the-art models. in fig. 1 , we show examples of figurative contexts from amazon product reviews and yelp restaurant reviews, a question answer pair for the contexts, along with automatically and manually constructed non-figurative versions of the context. the contributions of this work are the following: we found two examples of figurative questions out of 5,928 answerable questions in the squad dev set, one of them being "who is eligible to toss their name in the hat to be first minister?". figurative context: the album , like almost everything krush has released , slays . question: is the album good? answer: yes non-fig. version (manually created): the album is really good, like most of krush's work. non-fig. version (from gpt-3): the album is really good, like almost everything krush has released. figurative context: although, the menu items doesnt scream french cuisine. most foods looks like you can get at any american place. question: is the menu authentic french? answer: no non-fig. version (manually created): the menu items aren't typical of french cuisine. rather, they are common at most american eateries. non-fig. version (from gpt-3): although, the menu items doesn't look very french. most foods look like you can get at any american place. contexts for comparison. the outline of the paper is as follows: after reviewing related work ( §2), we introduce our new qa dataset for figurative language ( §3). we next introduce a general method for converting figurative language to non-figurative language by prompting gpt-3 ( §4), which we use to improve our baseline qa model. we then present our experimental results ( §5), and finally conclude ( §6). ## related work handling of figurative language is of significance in many downstream nlp tasks such as machine translation @xcite , recognizing textual entailment @xcite , sentiment analysis @xcite , among others. @xcite investigate the robustness of state-of-the-art entailment models on figurative examples on test sets for similes, metaphors, and irony. @xcite test figurative language understanding in pre-trained language models by evaluating continuation of text in narratives, while @xcite investigate non-literal reasoning capabilities of language models on a winograd-style non-literal language understanding task. the idea of converting metaphors to their literal counterparts has been previously explored for machine translation by @xcite , where metaphors in english text are first identified and then converted to a literal version, by making use of word embeddings and wordnet, before doing machine translation into chinese. in dialog systems, a similar approach was employed by @xcite , where idioms and metaphors in utterances are converted to literal versions using a dictionary lookup-based method. our work is closest to @xcite , except that we explore the robustness of qa systems in a machine comprehension setup, instead of dialog models, to figurative language, which, to the best of our knowledge, is a first. our automatic approach to creating rephrased non-figurative versions of figurative text is done using pre-trained language models, rather than rule-based methods which have been shown to be error-prone @xcite . related qa datasets include the friendsqa dataset @xcite , which is a dialogbased qa dataset constructed from dialogs from the tv series friends. while it does contain metaphors and sarcasm, it may not be ideal to test figurative language understanding as it is unclear how much of the dataset is actually figurative. the dialogic nature of the dataset further contributes to making it challenging. another dataset that requires figurative language understanding is the riddlesense dataset @xcite , which comprises of riddles, but unlike ours, it's modeled as an open domain qa task, rather than a machine comprehension task. @xcite show that questions about novel metaphors from literature are judged to be deeper than non-metaphorical or non-conventional metaphors by humans, but their focus is on generating deep questions, rather than testing the robustness of qa models. ## figurativeqa dataset the figurative data in figurativeqa comes from two sources: amazon product reviews @xcite , and yelp restaurant reviews @xcite . for comparison, we also extract non-figurative contexts from each of these sources to form the non-figurative split of figurativeqa. we construct a question answering dataset on top of these contexts. for simplicity, we only work with yes-no questions. fig 1 shows examples from the figurativeqa dataset. the data statistics from each source (amazon and yelp) and each split (figurative and non-figurative) are summarized in we select figurative texts for annotation with question-answer pairs from amazon product reviews using the following procedure. niculae and danescu-niculescu-mizil (2014) construct a dataset of figurative comparisons extracted using comparator patterns (such as "like", "as", or "than") from amazon product reviews, and then obtain 3 sets of figurativeness scores (on a scale of 1 to 4) on amazon mechanical turk (with scores of 1-2 binned as literal and 3-4 as figurative). of the 1260 comparisons in this dataset, we extract the sentences which have an average figurativeness score of greater than 3. this leaves us with 254 sentences, of which we manually pick 100 instances, and construct a yes-no question for each sentence. we select examples for annotating with questionanswer pairs from yelp reviews using a similar procedure. @xcite construct a dataset for nlg in the restaurant domain from yelp reviews, which comes labeled with sentiment information (1-2 rating for negative, 3 for neutral and 4-5 for positive). since positive or negative reviews are more likely to contain figurative language, from the set of positive and negative reviews, we extract instances using comparator patterns such as avg. ## amazon 52 48 16 for each context in figurativeqa, we construct a yes-no question. for the figurative contexts, we make sure to pose a question such that answering it would require understanding of the figurative text present in the context. for the non-figurative contexts, we construct questions that look similar to the ones for the figurative contexts. additionally, for the figurative contexts extracted from amazon and yelp, we manually create non-figurative counterparts that preserve the meaning and overall content. ## non-figurative contexts from gpt-3 we obtain non-figurative versions of the figurative contexts from figurativeqa by prompting gpt-3 (da-vinci-002). please refer to appendix a for model details and the prompts we used. as an intrinsic evaluation of the effectiveness of our prompting method, we manually evaluate the correctness of the non-figurative/literal contexts generated by prompting gpt-3. figurative context: not a storybook , by any means , this one is more like a visit to the zoo . to be correct. in fig. 2 , we show examples of nonfigurative text generated from gpt-3. ## experiments and results as a baseline, we run roberta-base @xcite finetuned on the training set of boolq @xcite . the performance on figurativeqa is summarized in tables 3 and 4 . we find that the roberta qa model performs poorly on the figurative contexts compared to the non-figurative contexts, and that manually changing the figurative language to non-figurative language improves performance. this indicates that automatic conversion of figurative language to non-figurative language may improve performance. ## conclusion and future work we show that current qa models do not perform so well when answering questions from figurative contexts as compared to non-figurative text. on manually created non-figurative versions of these contexts, we are able to show significant improvements. however, the manual annotation being an expensive step, we use an automatic method of prompting of gpt-3 and were still able to achieve performance gains. this highlights a need to build qa models that can handle figurative text. in the future, we would like to do a fine-grained analysis of qa performance on different kinds of figurative constructs, including similes, metaphors, irony, idioms, rhetorical questions, hyperbole, personification, etc.