FLUTE : Figurative Language Understanding through Textual Explanations (2022)

Abstract: Figurative language understanding has been recently framed as a recognizing textual entailment (RTE) task (a.k.a. natural language inference (NLI)). However, similar to classical RTE/NLI datasets they suffer from spurious correlations and annotation artifacts. To tackle this problem, work on NLI has built explanation-based datasets such as eSNLI, allowing us to probe whether language models are right for the right reasons. Yet no such data exists for figurative language, making it harder to assess genuine understanding of such expressions. To address this issue, we release FLUTE, a dataset of 9,000 figurative NLI instances with explanations, spanning four categories: Sarcasm, Simile, Metaphor, and Idioms. We collect the data through a Human-AI collaboration framework based on GPT-3, crowd workers, and expert annotators. We show how utilizing GPT-3 in conjunction with human annotators (novices and experts) can aid in scaling up the creation of datasets even for such complex linguistic phenomena as figurative language. The baseline performance of the T5 model fine-tuned on FLUTE shows that our dataset can bring us a step closer to developing models that understand figurative language through textual explanations.

## introduction figurative language such as metaphors, similes or sarcasm plays an important role in enriching human communication, allowing us to express complex ideas and emotions in an implicit way @xcite . however, understanding figurative language still remains a bottleneck for natural language processing @xcite . recently @xcite show that when faced with dialog contexts consisting of figurative language,some models show very large drops in performance compared to contexts without figurative language. despite the fact that transformer-based pre-trained language models (lms) get even larger @xcite , they are still unable to comprehend the physical world, cultural knowledge, or social context in which figurative language is embedded @xcite . in recent years, there have been several benchmarks dedicated to figurative language understanding, which generally frame "understanding" as a recognizing textual entailment (a.k.a natural language inference (nli)) task -deciding whether one sentence (premise) entails/contradicts another (hypothesis) @xcite @xcite . however, similar to general nli datasets, these benchmarks suffer from spurious correlations and annotation artifacts @xcite . these can allow large language models (llms) to achieve near human-level performance on in-domain test sets, yet turn brittle when evaluated against out-ofdomain or adversarial examples @xcite @xcite . to tackle these problems, research in nli has argued that it is not enough to correctly predict the entail/contradict labels, but also to explain the decision using natural language explanations that are comprehensible to an end-user assessing model's reliability @xcite @xcite , leading to novel datasets such as e-snli @xcite . yet, there is no such dataset for figurative language, hindering our ability to assess llms' genuine understanding of figurative language. in this paper, we make several contributions towards the goal of building models and assessing their ability to understand figurative language: type premise (literal) hypothesis (figurative*) label explanation paraphrase + sarcasm my next door neighbors are always arguing in our shared hallway. it's so annoying to have to hear my next door neighbors argue all the time in our shared hallway. ## results and discussion in terms of the rational quality ( ## conclusion we release flute, a dataset for figurative language understanding spanning across sarcasm, similes, metaphors, and idioms collected via a model-inthe-loop framework. to encourage genuine understanding of figurative language, our data also contains free-form textual explanations. upon conducting baseline experiments with state-of-the-art benchmark models (i.e., models trained on the the e-snli dataset), we notice those models perform poorly. in contrast, performance of the t5 model fine-tuned on flute shows that our dataset can bring us a step closer to developing models that understand figurative language through textual explanations. we hope our research on explanation generation for figurative language will be a fruitful future direction, and our dataset will be a challenging testbed for experimentation.