Just- DREAM -about-it: Figurative Language Understanding with DREAM - FLUTE (2022)

Abstract: Figurative language (e.g., “he flew like the wind”) is challenging to understand, as it is hard to tell what implicit information is being conveyed from the surface form alone. We hypothesize that to perform this task well, the reader needs to mentally elaborate the scene being described to identify a sensible meaning of the language. We present DREAM-FLUTE, a figurative language understanding system that does this, first forming a “mental model” of situations described in a premise and hypothesis before making an entailment/contradiction decision and generating an explanation. DREAM-FLUTE uses an existing scene elaboration model, DREAM, for constructing its “mental model.” In the FigLang2022 Shared Task evaluation, DREAM-FLUTE achieved (joint) first place (Acc@60=63.3%), and can perform even better with ensemble techniques, demonstrating the effectiveness of this approach. More generally, this work suggests that adding a reflective component to pretrained language models can improve their performance beyond standard fine-tuning (3.3% improvement in Acc@60).

## introduction understanding figurative language is a particularly challenging problem in nlp since the underlying meaning of the utterance is very different from the surface meaning of its constituent words @xcite . in this paper we focus on the task of recognizing and explaining textual entailment between a premise and hypothesis involving figurative language (figlang 2022 shared task in @xcite . we propose dream-flute, foot_1 a system that makes use of scene elaboration for building a "mental model" of the situations figure 1 : overview of dream-flute: it first uses dream @xcite to generate an elaboration of the situation in the premise and hypothesis (separately), then uses this additional context for entailment classification and explanation generation. dream-flute (consequence), using the "likely consequence" elaboration dimension as additional context, achieved top scores. such systems also form the building blocks of dream-flute (ensemble), our best system. presented in the premise and hypothesis to detect textual entailment between them (see figure 1 ). the design of dream-flute builds upon the scene elaboration model, dream, presented by @xcite . dream uses a t5-based @xcite sequence-to-sequence model to generate additional, pertinent details about each given situation in the input text, along key conceptual dimensions informed by cognitive science, story understanding and planning literature @xcite @xcite . using such scene elaboration as additional context has been shown to improve questionanswering (qa) performance on different models and across different downstream tasks such as ethics @xcite , codah @xcite and social iqa @xcite . to adapt it for the figurative language understanding shared task, we made three significant extensions to using dream that have not been previously explored. first, we incorporate dream for elaborating the premise and hypothesis in a natural language inference (nli) task involving figurative language understanding @xcite . we hypothesize that such additional, pertinent details could also improve a model's ability to judge whether there is an entailment or contradiction between the premise and hypothesis. this could be especially helpful for the instances that use figurative language, where the underlying meaning might be opaque to the model: further elaborating the context can make certain inferences more explicit. second, beyond improvements on label prediction accuracy (i.e. choosing from multiple-choice options) shown in @xcite , our work uncovers the use of such additional context for improving explanation quality. and lastly, we exploit the dimensions in dream to train different models for an ensemble system representing a cognitive continuum (figure 2 ), further improving accuracy and explanation quality. our approach is easily adaptable to other language models, and task-agnostic in format (e.g. qa or nli) and domain (e.g. ethical decisions or figurative language understanding). we demonstrate the effectiveness of our single model system in terms of achieving top scores in the task, as well as the flexibility of implementing an ensemble system that not only yields further improvements for this task but also allows customization to suit the requirements of different downstream applications. ## approach we first describe our single model systems in section 2.1. next, we present a two-step "classify then explain" pipeline in section 2.2. in section 2.3, we take advantage of all information learned by the different models and propose an ensemble approach inspired by cognitive science. ## experiment settings data this shared task has a two-phases timeline: the development phase then the test phase. during the development phase, ∼7500 samples are provided as the training set. we used a 80-20 split to create our own training (6027 samples) and validation (1507 samples) partitions on which we build our models. later at the test phase, separate 1500 test samples (without gold labels) are released on which all models are tested. note that our model is primarily developed during the training phase without having access to the test data. ## conclusion in this work we showed how dream-flute, a competitive system for the figurative language understanding nli task, can be built by utilizing scene elaborations from an existing model, dream. compared to a model without such scene elaborations, dream-flute makes use of scene elaboration for building a "mental model" of situations in the premise and hypothesis to make inferences more explicit, thus improving label prediction accuracy and explanation quality. dream-flute (ensemble) uses different elaborations to form building blocks for implementing a continuum with varying levels of intuition and analysis, modeling deriving answers and rationalizing by considering different positions on a cognitive continuum. this novel use of dream not only obtained the highest scores for the figurative language understanding shared task, but could also easily be applied to the situational qa tasks in @xcite , and beyond. our approach is easily adaptable to other language models, and task-agnostic in format (e.g. qa or nli) and domain (e.g. ethical decisions or figurative language understanding). more generally, our work demonstrates that adding a reflective component helps to improve answer accuracy and explanation quality in pretrained language models.