I nfer ES : A Natural Language Inference Corpus for S panish Featuring Negation-Based Contrastive and Adversarial Examples (2022)

Abstract: In this paper we present InferES - an original corpus for Natural Language Inference (NLI) in European Spanish. We propose, implement, and analyze a variety of corpus-creating strategies utilizing expert linguists and crowd workers. The objectives behind InferES are to provide high-quality data, and at the same time to facilitate the systematic evaluation of automated systems. Specifically, we focus on measuring and improving the performance of machine learning systems on negation-based adversarial examples and their ability to generalize across out-of-distribution topics. We train two transformer models on InferES (8,055 gold examples) in a variety of scenarios. Our best model obtains 72.8% accuracy, leaving a lot of room for improvement. The “hypothesis-only” baseline performs only 2%-5% higher than majority, indicating much fewer annotation artifacts than prior work. We show that models trained on InferES generalize very well across topics (both in- and out-of-distribution) and perform moderately well on negation-based adversarial examples.

## introduction in the task of natural language inference (nli), an automated system has to determine the meaning relation that holds between two texts. the model has to make a three-way choice between entailment: a hypothesis (h) is true given a premise (p) (e.g. 1.); contradiction: a hypothesis (h) is false given a premise (p) (e.g. 2.); or neutral: the truth value of the hypothesis (h) cannot be determined solely based on the premise (p) (e.g.: 3.). 1. p) john goes to work every day with a car. ## related work the task of recognizing textual entailment (rte) was proposed in @xcite as a binary clas-1 at https://github.com/venelink/inferes inferes is also added as a huggingface dataset sification ("entailment" / "non-entailment"). the rte competition ran for seven editions @xcite @xcite @xcite . rte was later reformulated as a three-way decision and ultimately renamed natural language inference in the snli @xcite and the mnli @xcite corpora. both the rte and the nli tasks form part of the natural language understanding benchmarks glue @xcite and super-glue @xcite . the nlu benchmarks attracted a lot of attention from the community and by 2020 the state-of-the-art systems reported human level performance. @xcite proposed a "linguist-in-the-loop" corpus creation to improve the quality of the data. the "super-human" performance of nli systems has been questioned by a number of researchers. @xcite found that annotation artifacts in the datasets enable the models to predict the label by only looking at the hypothesis. @xcite and @xcite demonstrate that state-of-the-art nli systems often rely on heuristics and annotation artifacts. systematic approaches to evaluation propose different sets of stress-tests for nli and nlu systems @xcite @xcite @xcite . the attacks can be inspired by linguistic phenomena or empirical use cases. systematic evaluations show that nli and other nlu systems often underperform on complex linguistic phenomena such as conjunction @xcite , negation @xcite , and coreference @xcite . researchers also experimented with creating contrastive examples that differ only slightly from training examples, but have a different label @xcite @xcite . adversarially created datasets such as adversarial nli @xcite and dynabench nli @xcite demonstrate that there is a lot of room for improvement regarding nli datasets and models. most of the available resources for nli research are in english. @xcite present xnli, a multilingual dataset created by translating english nli examples into other languages. the interest in multilingual nli has resulted in the creation of some novel non-english resources such as the korean nli corpus @xcite , chinese nli corpus @xcite , persian nli corpus @xcite , indonesian nli corpus @xcite , and indigenous languages of the americas nli corpus @xcite . for spanish, the only available resources are the spanish portion of xnli and the sparte corpus for rte @xcite which was adapted from question answering data. ## objectives and corpus creation when creating inferes we experimented with different strategies for obtaining gold examples. to the best of our knowledge, this is the first time various annotation strategies are combined and compared in a single nli corpus. we adopt three different approaches, used in prior work: our generation strategy is similar to the original rte and nli corpus creation; our rewrite strategy is inspired from work in generating adversarial and contrastive examples; our annotation strategy scales well with data and allows us to compare expert-and crowdcrated datasets. our aim was to provide interesting and diverse examples that cover a large range of use cases and linguistic phenomena. we hope that inferes can be used not only to train automated systems, but also to better understand the nature of inference. we formulated three main objectives: o1 to create a native nli dataset for the spanish language. the existing resources are either an adaptation from a different task or a translation from english. o2 to promote better data quality and corpus creation practices. we aim to create a more challenging dataset and simultaneously reduce the number of annotation artifacts. o3 to facilitate the research on negation and coreference in the context of nli. more specifically, we focus on contrastive and adversarial examples. ## inferes we combined the examples from all corpus creation strategies to create inferes -a corpus of nli for spanish containing 8,055 text pairs. the full inferes has a vocabulary size of 12,877 unique types. on average, 22.6% of the tokens from the premise also appear in the hypothesis. 33.4% of the tokens from the hypothesis also appear in the premise. the two numbers differ since we count the number of non-unique tokens, including repetition, and we normalize them using a different denominator (length of premise/hypothesis). the overlap is comparable with prior work for english (20% and 38% for mnli and 18% and 34% for linguist-in-the-loop nli). ## machine learning experiments to demonstrate the utility of inferes , we carried out a set of machine learning experiments. the design of inferes allows us to test nli models under a variety of conditions: standard train/test split, hypothesis-only condition, performance on negation-based adversarial examples, and performance by topic in-and out-of-distribution. ## discussion in section 3, we formulated three main objectives behind inferes . in this section, we want to revisit those objectives and briefly discuss the importance of our work for the nlp and nli communities. our first objective was "to create a native nli dataset for the spanish language". to the best of our knowledge, inferes is the first native nli dataset for spanish which is not adapted or translated. we described the creation process and validated that it can be used to train different machine learning models. we have successfully contributed a new resource to the spanish nlp community and we hope that inferes can facilitate the further creation of tools and resources for that language. our second objective was "to promote better data quality and corpus creation practices.". we proposed, implemented, analyzed, and compared several different strategies for creating text pairs. the resulting dataset proves non-trivial to state-of-the-art nlp models with an overall accuracy in the low 70s. this leaves a lot of room for improvement and future research. the results using a "hypothesis-only" baseline indicate that inferes contains fewer annotation artifacts than prior work. at the same time, models trained on the dataset are able to generalize well across different topics, even in our "out-of-distribution" condition. overall, we can conclude that inferes is of high quality and achieves the objective of promoting better data by design. our third objective was "to facilitate the research on negation and coreference in the context of nli.". our rewrite strategy was focused on creating naturally occurring contrastive and adversarial examples based on negation and coreference. we followed prior work on evaluating systems' performance and demonstrated that those examples are non-trivial to solve. however, the two models are still able to outperform the majority baseline by over 10%. these findings indicate that the problem is not unsolvable and the models are learning something about complex negation from the data. inferes can facilitate the resaerch of negation in spanish both in the context of nli and in isolation. we leave quantifying the importance of coreference in the rewrite section for future work. overall, we have achieved all our objectives: 1) we created a novel dataset for spanish; 2) we used different generation and annotation strategies to obtain a challenging corpus with fewerr annotation artifacts; 3) we created a set of high-quality contrastive and adversarial examples based on negation and coreference. we believe that inferes is an important contribution to spanish nlp, and also to researchers interested in nli, negation, and coreference. we hope that this dataset can be used to train and evaluate more accurate automated sys-tems, but also to better understand the nature of those linguistic phenomena. ## conclusions we presented inferes -a new corpus of natural language inference for spanish. to the best of our knowledge, this is the first original spanish nli corpus that is not a translation or an adaptation of an existing dataset. we explored several different strategies for corpus creation and put the emphasis on creating diverse and non-trivial examples, that are also linguistically interesting. more specifically, we created contrastive and adversarial examples involving complex negation and coreference. we provided two baseline transformer-based systems finetuned on the dataset. we demonstrated that inferes is challenging and contains fewer anotation artifacts than prior work. we also evaluated the performance of automated systems on adversarial examples and the ability of the models to generalize across topics in-and out-of-distribution. the results validated the quality and the difficulty of the corpus. inferes leaves a room for analysis and improvement. our work opens several directions for future work: studying and improving the performance of nli models for spanish; expanding the research on negation in spanish, and specifically the complex and lexical negation; evaluating the importance of coreference in the context of nli. we believe that inferes will be useful both to spanish researchers and to the general nlp community.