IMPLI : Investigating NLI Models’ Performance on Figurative Language (2022)

Abstract: Natural language inference (NLI) has been widely used as a task to train and evaluate models for language understanding. However, the ability of NLI models to perform inferences requiring understanding of figurative language such as idioms and metaphors remains understudied. We introduce the IMPLI (Idiomatic and Metaphoric Paired Language Inference) dataset, an English dataset consisting of paired sentences spanning idioms and metaphors. We develop novel methods to generate 24k semiautomatic pairs as well as manually creating 1.8k gold pairs. We use IMPLI to evaluate NLI models based on RoBERTa fine-tuned on the widely used MNLI dataset. We then show that while they can reliably detect entailment relationship between figurative phrases with their literal counterparts, they perform poorly on similarly structured examples where pairs are designed to be non-entailing. This suggests the limits of current NLI models with regard to understanding figurative language and this dataset serves as a benchmark for future improvements in this direction.

## introduction understanding figurative language (i.e., that in which the intended meaning of the utterance differs from the literal compositional meaning) is a particularly difficult area in nlp @xcite , but is essential for proper natural language understanding. we consider here two types of figurative language: idioms and metaphors. idioms can be viewed as non-compositional multiword expressions @xcite , and have been historically difficult for nlp systems. for instance, sentiment systems struggle with multiword expressions in which individual words do not directly contribute to the sentiment @xcite . idioms jamie was pissed off this afternoon. → jamie was irritated this afternoon there's a marina down in the docks. there's a marina down under scrutiny. ## building a dataset our impli dataset is built from idiomatic and metaphoric sentences paired with entailing and nonentailing counterparts, from both silver pairs ( §3.1) and manually written sentences ( §3.4). for our purposes, we follow @xcite in conflating the neutral and contradiction categories into a nonentailment label. we then label every pair as either entailment (→) or non-entailment ( ). due to the difficult nature of the task and to avoid issues with crowdsourcing (bowman et al., 2020), we employed expert annotators. we used two fluent english speakers, both graduate students in linguistics with strong knowledge in figurative language, paid at a rate of $20/hr. for each method below, we ran pilot studies, incorporated annotator feedback and iteratively assessed the viability of identifying and generating appropriate expressions. as the annotators were working on generating new expressions, agreement was not calculated: we instead assessed the quality of the resulting expressions (see section 3.3). ## experiments / results using the impli dataset, we aim to answer a series of questions via nli pertaining to language models' ability to understand and represent figurative language accurately. these questions are: our dataset provides unique advantages in addressing these research questions that cover gaps in previous work: it contains a large number of both entailments and non-entailments and is large enough to be used for training the models. ## conclusions and future work in this work, we introduce the impli dataset, which we then use to evaluate nli models' capabilities on figurative language. we show that while widely used mnli models handle entailment admirably and metaphoric expressions are relatively easy, nonentailment idiomatic relationships are more difficult. additionally, adding idiom-specific training data fails to alleviate poor performance for nonentailing pairs. this highlights how currently language models are inherently limited in representing some figurative phenomena and can provide a target for future model improvements. for future work, we aim to expand our data collection processes to new data sources. our dataset creation procedure relies on annotated samples and definitions: as more idiomatic and metaphoric resources become available, this process is broadly extendable to create new figurative/literal pairs. additionally, we only explore this data for evaluating nli systems: this data could also be used for other parallel data tasks such as figurative language interpretation @xcite and figurative paraphrase generation. as natural language generation often relies on training or fine-tuning