Nature language inference (NLI) task is a predictive task of determining the inference relationship of a pair of natural language sentences. With the increasing popularity of NLI, many state-of-the-art predictive models have been proposed with impressive performances. However, several works have noticed the statistical irregularities in the collected NLI data set that may result in an over-estimated performance of these models and proposed remedies. In this paper, we further investigate the statistical irregularities, what we refer as confounding factors, of the NLI data sets. With the belief that some NLI labels should preserve under swapping operations, we propose a simple yet effective way (swapping the two text fragments) of evaluating the NLI predictive models that naturally mitigate the observed problems. Further, we continue to train the predictive models with our swapping manner and propose to use the deviation of the model's evaluation performances under different percentages of training text fragments to be swapped to describe the robustness of a predictive model. Our evaluation metrics leads to some interesting understandings of recent published NLI methods. Finally, we also apply the swapping operation on NLI models to see the effectiveness of this straightforward method in mitigating the confounding factor problems in training generic sentence embeddings for other NLP transfer tasks. Natural Language Inference (NLI) task is testing the ability of a computational model to understand the natural language by the evaluation of a three-way classification for two fragments of natural language texts (Bowman et al. 2015) . The two fragments, namely premise (P) and hypothesis (H), are labeled in three different categories (i.e. entailment (E), neutral (N), and contradiction (C)) based on their entailment relation. More specifically, we can have: • E: H is definitely true given P • C: H is definitely not true given P • N : H may or may not be true given P Formally, we could rewrite the natural language described linguistic entailment relation into propositional logic as following: To appear at Association for the Advancement of Artificial Intelligence (AAAI) 2019. where → stands for implication, ¬ stands for negation, and we use ⊥ to denote that there is no clear relation between P and H. The essential part of this paper lies in the fact that for any two propositions A and B, we have: In simpler words, swapping A and B will retain the → ¬ relation and ⊥ relation, but not → relation. Therefore, we can simply evaluate an NLI predictive model by swapping the premise and hypothesis in testing data set with the argument: If a model can truly predict inference relationship between pairs of text fragments, it should report comparable accuracy between the original test set and swapped test set for contradiction pairs and neutral pairs, and lower accuracy in swapped test set for entailment pairs. If we do not observe such an accuracy pattern, it is very likely that the evaluated NLI predictive model is questionable despite its impressive performance. One may argue that the label may not necessarily preserve for neutral pairs after swapping, we explain in the Discussion section later. This work is inspired by several recent works that have observed the statistical irregularities of constructed NLI data sets. For example, Gururangan et al. (2018) noticed that during the data construction phase, the workers create a heuristic to generate hypothesis with minimum risks of being wrong, which lead to the fact that there are different distributions of words for different labels, and a powerful model can easily bypass the semantic information of the texts and predict the label with reasonable accuracy. The similar phenomenon has been observed by Poliak et al. (2018) , who noticed that a model could infer the NLI label with only hypotheses, and proposed a hypothesis-only baseline. Recently, Naik et al. (2018) also noticed that the machine learning models might exploit the idiosyncrasies in the construction of the data set in predicting NLI labels and proposed a "stress test" to evaluate whether the models can make semantic-level inferences. All these previous work have discussed the inherent limitations of current NLI tasks and proposed solutions such as more comprehensive evaluation criteria (Naik et al. 2018), guidance in future constructions of data sets (Gururangan et al. 2018) , or more powerful methods such as utilizing the propositional logic relationship to regularize the neural network to ignore statistical irregularities of words and predict through semantic-level information (Minervini and Riedel 2018). While this paper agrees with the necessity of these carefully-designed remedies, we want to bring the community's attention to a straightforward, easily-implemented, yet meaningful solution: swapping the two text fragments. Our contribution of this paper can be summarized in three-fold: • By recognizing the spurious signals between sentences' semantic and words' distribution of hypothesis created through a confounding factor of word choices, we introduce a straightforward and effective way to break such dependency: swapping the two text fragments. We apply the swapping evaluation metric on several recent NLI models and notice that some methods seem to predict based on local statistical patterns, in contrast to semantic level understanding. • Inspired by the swapping evaluation metric, we propose to train the NLI models with a sequence of different training data sets that are defined by the percentages of sentence pairs swapped in the training set. We examine the "deviation" of performances of the model trained by these data sets evaluated by more powerful metrics. We propose the "deviation" to be a good measure of the robustness of the models to confounding signals. • We further investigate how the swapping training procedure help mitigate the confounding factor problem during NLI training by applying the trained sentence embeddings to other NLP transfer tasks. The remainder of the paper is organized as follows. We first introduce the background of this paper, where we explain why swapping the two text fragments is necessary, rather than just a random trick we adopt, and we also explain the concept of confounding factors in the context of NLI. Then we introduce the swap evaluation and the results, which further disclose some mechanism of the evaluated models. Further, we continue to test the robustness of the models to confounding signals. Afterward, we test to see if the swapping training in NLI will lead to more meaningful sentence embeddings that help in other NLP transfer tasks. Finally, with a brief discussion of the related work, we conclude the paper.
REF further investigate models' robustness to the bias pattern using swapping operations.
What If We Simply Swap the Two Text Fragments? A Straightforward yet Effective Way to Test the Robustness of Methods to Confounding Signals in Nature Language Inference Tasks
ArXiv
ArXiv
Computer Science