TINA : Textual Inference with Negation Augmentation (2022)

Abstract: Transformer-based language models achieve state-of-the-art results on several natural language processing tasks. One of these is textual entailment, i.e., the task of determining whether a premise logically entails a hypothesis. However, the models perform poorly on this task when the examples contain negations. In this paper, we propose a new definition of textual entailment that captures also negation. This allows us to develop TINA (Textual Inference with Negation Augmentation), a principled technique for negated data augmentation that can be combined with the unlikelihood loss function.Our experiments with different transformer-based models show that our method can significantly improve the performance of the models on textual entailment datasets with negation – without sacrificing performance on datasets without negation.

## introduction textual entailment (te, also called natural language inference) is the task of recognizing whether one natural language sentence (the premise) semantically entails another one (the hypothesis). for example, the premise "i live in paris" entails the hypothesis "i live in france". te is at the heart of natural language understanding, as it is closely related to question answering and natural language reasoning @xcite . nowadays, the state of the art performance in te is achieved by transformer-based models such as bert @xcite . however, transformer-based models can get derailed easily by trap words or syntactic variations (see, e.g., @xcite for a survey). in particular, such models have difficulties with negation in textual entailment @xcite . here is an example from hossain et al. (2020)'s dataset: premise: green cards are not becoming more difficult to obtain. hypothesis: green card is now difficult to receive. ## experiments we conducted several experiments to investigate the robustness of models trained with our data augmentation technique, tina, for the task of textual entailment with negation. ## conclusion in this paper, we have studied the problem of negation in textual entailment in detail. we have argued that the previous formal definition of textual entailment is problematic, and we have proposed a new probabilistic definition. based on this definition, we have proposed tina, a principled negated data augmentation technique. tina can be combined with the unlikelihood loss to improve the robustness of language models to negation in textual entailment tasks. our experimental results across different negated textual entailment benchmarks show that our method can significantly increase the performance of different transformer-based models. future work can explore how different loss functions, such as contrastive loss, could be used with our augmented datasets. acknowledgements. this work was partially funded by anr-20-chia-0012-01 ("nordf").