Double Perturbation: On the Robustness of Robustness and Counterfactual Bias Evaluation (2021)

Abstract: Robustness and counterfactual bias are usually evaluated on a test dataset. However, are these evaluations robust? If the test dataset is perturbed slightly, will the evaluation results keep the same? In this paper, we propose a “double perturbation” framework to uncover model weaknesses beyond the test dataset. The framework first perturbs the test dataset to construct abundant natural sentences similar to the test data, and then diagnoses the prediction change regarding a single-word substitution. We apply this framework to study two perturbation-based approaches that are used to analyze models’ robustness and counterfactual bias in English. (1) For robustness, we focus on synonym substitutions and identify vulnerable examples where prediction can be altered. Our proposed attack attains high success rates (96.0%-99.8%) in finding vulnerable examples on both original and robustly trained CNNs and Transformers. (2) For counterfactual bias, we focus on substituting demographic tokens (e.g., gender, race) and measure the shift of the expected prediction among constructed sentences. Our method is able to reveal the hidden model biases not directly shown in the test dataset. Our code is available at https://github.com/chong-z/nlp-second-order-attack .

## introduction recent studies show that nlp models are vulnerable to adversarial perturbations. a seemingly "invariance transformation" (a.k.a. adversarial perturbation) such as synonym substitutions @xcite or syntax-guided paraphrasing @xcite can alter the prediction. to mitigate the model vulnerability, robust training methods have been proposed and shown effective @xcite @xcite . in most studies, model robustness is evaluated based on a given test dataset or synthetic sentences constructed from templates @xcite . specifically, the robustness of a model is often evaluated by the ratio of test examples where the model prediction cannot be altered by semantic-invariant perturbation. we refer to this type of evaluations as the first-order robustness evaluation. however, even if a model is first-order robust on an input sentence x 0 , it is possible that the model is not robust on a natural sentence x0 that is slightly modified from x 0 . in that case, adversarial examples still exist even if first-order attacks cannot find any of them from the given test dataset. throughout this paper, we call x0 a vulnerable example. the existence of such examples exposes weaknesses in models' understanding and presents challenges for model deployment. fig. 1 illustrates an example. in this paper, we propose the double perturbation framework for evaluating a stronger notion of second-order robustness. given a test dataset, we consider a model to be second-order robust if there is no vulnerable example that can be identified in the neighborhood of given test instances ( §2.2). in particular, our framework first perturbs the test set to construct the neighborhood, and then diagnoses the robustness regarding a single-word synonym substitution. taking fig. 2 as an example, the model is first-order robust on the input sentence x 0 (the prediction cannot be altered), but it is not second-order robust due to the existence of the vulnerable example x0 . our framework is designed to identify x0 . we apply the proposed framework and quantify second-order robustness through two second-order attacks ( §3). we experiment with english sentiment classification on the sst-2 dataset @xcite across various model architectures. surprisingly, although robustly trained cnn @xcite and transformer @xcite can achieve high robustness under strong attacks @xcite ) (23.0%-71.6% success rates), for around 96.0% of the test examples our attacks can find a vulnerable example by perturbing 1.3 words on average. this finding indicates that these robustly trained models, despite being first-order robust, are not second-order robust. furthermore, we extend the double perturbation framework to evaluate counterfactual biases @xcite ( §4) in english. when the test dataset is small, our framework can help improve the evaluation robustness by revealing the hidden biases not directly shown in the test dataset. intuitively, a fair model should make the same prediction for nearly identical examples referencing different groups @xcite with different protected attributes (e.g., gender, race). in our evaluation, we consider a model biased if substituting tokens associated with protected attributes changes the expected prediction, which is the average prediction among all examples within the neighborhood. for instance, a toxicity classifier is biased if it tends to increase the toxicity if we substitute @xmath0. in the experiments, we evaluate the expected sentiment predictions on pairs of protected tokens (e.g., (he, she), (gay, straight)), and demonstrate that our method is able to reveal the hidden model biases. our main contributions are: (1) we propose the double perturbation framework to diagnose the robustness of existing robustness and fairness evaluation methods. (2) we propose two second-order attacks to quantify the stronger notion of second- figure 2 : an illustration of the decision boundary. diamond area denotes invariance transformations. blue x 0 is a robust input example (the entire diamond is green). yellow x0 is a vulnerable example in the neighborhood of x 0 . red x 0 is an adversarial example to x0 . note: x 0 is not an adversarial example to x 0 since they have different meanings to human (outside the diamond). ## the double perturbation framework in this section, we describe the double perturbation framework which focuses on identifying vulnerable examples within a small neighborhood of the test dataset. the framework consists of a neighborhood perturbation and a word substitution. we start with defining word substitutions. ## evaluating counterfactual bias in addition to evaluating second-order robustness, we further extend the double perturbation framework ( §2) to evaluate counterfactual biases by setting p to pairs of protected tokens. we show that our method can reveal the hidden model bias. ## conclusion this work proposes the double perturbation framework to identify model weaknesses beyond the test dataset, and study a stronger notion of robustness and counterfactual bias. we hope that our work can stimulate the research on further improving the robustness and fairness of natural language models.