An Empirical Study on Model-agnostic Debiasing Strategies for Robust Natural Language Inference (2020)

Abstract: The prior work on natural language inference (NLI) debiasing mainly targets at one or few known biases while not necessarily making the models more robust. In this paper, we focus on the model-agnostic debiasing strategies and explore how to (or is it possible to) make the NLI models robust to multiple distinct adversarial attacks while keeping or even strengthening the models’ generalization power. We firstly benchmark prevailing neural NLI models including pretrained ones on various adversarial datasets. We then try to combat distinct known biases by modifying a mixture of experts (MoE) ensemble method and show that it’s nontrivial to mitigate multiple NLI biases at the same time, and that model-level ensemble method outperforms MoE ensemble method. We also perform data augmentation including text swap, word substitution and paraphrase and prove its efficiency in combating various (though not all) adversarial attacks at the same time. Finally, we investigate several methods to merge heterogeneous training data (1.35M) and perform model ensembling, which are straightforward but effective to strengthen NLI models.

## introduction natural language inference (nli) (also known as recognizing textual entailment) is a widely studied task which aims to infer the relationship (e.g., entailment, contradiction, neutral) between two fragments of text, known as premise and hypothesis @xcite . recent works have found that nli models are sensitive to the compositional features @xcite , syntactic heuristics @xcite , stress test @xcite and human artifacts in the data collection phase @xcite @xcite . ## benchmark datasets our benchmark datasets include the adversarial datasets 2 and some widely used general-purpose 2 some datasets listed in ## mixture of experts (moe) debiasing we utilize the moe ensemble model @xcite as the backbone to mitigate three known biases in nli. concretely, we implement the 'instance reweighting' and 'bias product' methods in @xcite . based on these methods, we perform several trials on combating several distinct nli biases at the same time. ## data augmentation in this section, we explore 3 automatic augmentation ways without collecting new data. for fair comparison, in all the following settings, we double the training size by automatically generating the same number of augmented instances as the original training sets as shown in ## dataset merging and model ensemble in this section we explore 1) to what extend larger dataset and ensemble would make the nli models more robust to distinct adverserial datasets. 2) what is the best way to combine the large-scale nli training sets in very different domains. ## conclusions we try to investigate how to build robust and generalized nli models by model-agnostic debiasing strategies, including mixture of experts ensemble (moe), data augmentation (da), dataset merging and model ensemble, and benchmark these methods on various adversarial and general purpose datasets. our findings suggest model-level moe ensemble, text swap da and performance based dataset merging would effectively combat multiple (though not all) distinct biases. although we haven't found a debiasing strategy that can guarantee the nli models to be more robust on every adversarial dataset used in this paper, we leave the question of whether such a debiasing method exists for future research.