SBU Figures It Out: Models Explain Figurative Language (2022)

Abstract: Figurative language is ubiquitous in human communication. However, current NLP models are unable to demonstrate a significant understanding of instances of this phenomena. The EMNLP 2022 shared task on figurative language understanding posed the problem of predicting and explaining the relation between a premise and a hypothesis containing an instance of the use of figurative language. We experiment with different variations of using T5-large for this task and build a model that significantly outperforms the task baseline. Treating it as a new task for T5 and simply finetuning on the data achieves the best score on the defined evaluation. Furthermore, we find that hypothesis-only models are able to achieve most of the performance.

## introduction figurative language is an important component of discourse, ranging from daily interactions to books. it is used as a tool to convey complex and deeper emotions that are often difficult to express literally @xcite . despite the fact that transformer-based pretrained language models (lms) get even larger, they are still unable to comprehend the physical world, cultural knowledge, or social context in which figurative language is embedded. large-scale crowdsourced datasets often contain these phenomena inherently. to show true conceptual understanding of figurative language, the model should not only be able to correctly differentiate a figurative instance from its literal counterpart, but also explain its decision. these natural language explanations should be readily comprehensible by an end-user who needs to assert a model's reliability @xcite . this paper describes the experiments and submission of the lunr lab at stony brook univer- * first two authors have equal contribution sity, usa to the shared task on figurative language understanding @xcite organized at emnlp 2022. given a premise and a hypothesis, the shared task required predicting the relation between them as well as an explanation for the same. we use variations in input format, separator and sequential fine-tuning techniques to build our final model. since the task involves predicting the label as well as an explanation for it, in this paper we vary the order of generation of each target in our models. prior work @xcite highlighted the importance of separator tokens. it helps the model distinguish between different portions of the input. additionally, since this task is not a common one, variations in input format and keywords dictate how well a model performs. to that end, we experimented with different formats prescribed for t5 models as well as a simple one for an unseen, new task. finally, we also experimented with sequential fine-tuning on several related datasets to improve performance on the shared task. our final model is a simple t5-large model finetuned on the task data, trained to generate the explanation before the label. the input format does not contain any task-specific keys and does not resemble any of the ones described in @xcite . the model uses a "\n" separator, which is a prominent part of how unifiedqa @xcite was built over t5. it improves significantly over the task baseline. we observe that (1) treating this as a new task leads to best model performance, (2) the dataset contains artifacts that hypothesis-only models use to reach significant performance, and (3) knowing the type of phenomena being encapsulated does not help the model. ## data the shared task data @xcite ) contains 9,000 high-quality literal, figurative sentence pairs with entail/contradict labels and the associated explanations. the benchmark spans five types of figurative language: paraphrase, sarcasm, simile, metaphor, and idiom. the definition of each type is explained as follows: paraphrase is a rephrasing of something that is written. all sentences in this category belongs to the entailment category. sarcasm is using phrases which have the opposite meaning from what they are intended to convey. it can be used for creating contradiction labels. simile is using a figure of speech to compare something with something else. it can be used for both entailment and contradiction labels. metaphor is when a word or phrase used to describe something that it cannot literally describe. it can be used for both entailment and contradiction labels. it can be used for both entailment and contradiction labels. idiom is established by usage as having a meaning not derived from their individual meanings. it can be used for both entailment and contradiction labels. a noteworthy property of this data is that both the entailment/contradiction labels and the explanations are w.r.t the figurative language expression (i.e., metaphor, simile, idiom) rather than other parts of the sentence. the task is challenging because it inherently requires 1) relational reasoning using background commonsense knowledge, and 2) finegrained understanding of figurative language. we split 7,500 examples into a 80-20 train and dev set randomly. these sets are then used to build models for the overall shared task. ## experiment design we use the t5 @xcite family of models for our submission. particularly, we build over t5large. since this is a new task for t5, we experiment with various input and output formats. we build models where the label is placed before and after explanation on the target side. large language models have also been shown to be sensitive to the choice of separators. to this end, we build models that conform to different input/output formats as well as separators. prior work has shown that pretraining on large amounts of data similar to the task improves the downstream performance of models. to this end, we use e-snli @xcite to sequential fine-tuning our model before finetuning on downstream task data to obtain a final model. e-snli is an extension of the snli dataset @xcite with an additional layer of humanannotated natural language explanations of the entailment relations. similarly, sume @xcite is a biomedical mechanism explanation dataset which contains a set of supporting sentence about two main entities, the relation between the entities, and a sentence explaining the mechanism behind this relation. they explored the generation of explanation and target label at the same time given the supporting sentences, using different transformer based models. they use [explanation. label] as the output format while we explore all possible orders and separator tokens. we used the model pretrained on sume dataset and finetuned on this task. @xcite used hypothesis-only models showed that statistical irregularities may allow a model to perform natural language inference in some datasets beyond what should be achievable without access to the context. motivated by that, we also build hypothesis-only models to analyze whether models require contexts to perform this nli + explanation task. ## analysis we analyse the performance of the numerous models that we have built to understand the impact of various design decisions that we took -input format, sequential fine-tuning, and order of required predictions. further, we also want to understand the impact of artifacts present in the dataset itself on model performance. we use the evaluation described in subsection 5.1 on the dev set for analysis. the data input formats vary in two aspects -taskspecific keywords and the separator. specifically, the task-specific keywords can correspond to a new task for t5 (no keywords), rte and mnli (appendix d.2 and d.3 of @xcite respectively). we experiment with three possible separators between pieces of input text -' ' (whitespace), [sep] (the sep token), and "\n" (the newline character). both \n and [sep] are predefined to the tokenizer as one unique token before training. the effects of these design choices can be seen in ## conclusion figurative language is an important component of discourse, often used as a tool to convey complex emotions usually difficult to express literally. the shared task is designed to test whether models can predict the relation between a pair of sentences that contains figurative language as well as explain that phenomena. we experiment with building several models based on t5-large varying the input format, order of prediction and sequential fine-tuning. our final model is a simple t5-large model finetuned on the task data, trained to generate the explanation before the label. the input format does not contain any task-specific keys and does not resemble any of the ones described in @xcite but uses a "\n" separator. it improves significantly over the task baseline. we observe that (1) treating this as a new task leads to best model performance, (2) the dataset contains artifacts that hypothesis-only models use to reach significant performance, and (3) knowing the type of phenomena being encapsulated does not help the model.