A Curious Case of Searching for the Correlation between Training Data and Adversarial Robustness of Transformer Textual Models (2024)

Abstract: Existing works have shown that fine-tuned textual transformer models achieve state-of-the-art prediction performances but are also vulnerable to adversarial text perturbations. Traditional adversarial evaluation is often done only after fine-tuning the models and ignoring the training data. In this paper, we want to prove that there is also a strong correlation between training data and model robustness. To this end, we extract 13 different features representing a wide range of input fine-tuning corpora properties and use them to predict the adversarial robustness of the fine-tuned models. Focusing mostly on encoder-only transformer models BERT and RoBERTa with additional results for BART, ELECTRA and GPT2, we provide diverse evidence to support our argument. First, empirical analyses show that (a) extracted features can be used with a lightweight classifier such as Random Forest to effectively predict the attack success rate and (b) features with the most influence on the model robustness have a clear correlation with the robustness. Second, our framework can be used as a fast and effective additional tool for robustness evaluation since it (a) saves 30x-193x runtime compared to the traditional technique, (b) is transferable across models, (c) can be used under adversarial training, and (d) robust to statistical randomness. Our code is publicly available at https://github.com/CaptainCuong/RobustText_ACL2024 .

## introduction pre-trained transformer models such as bert @xcite and roberta @xcite have recently demonstrated superior performance in various downstream nlp classification tasks. however, they are also vulnerable to adversarial text attacks @xcite , which aim to generate adversarial examples by applying imperceptible perturbations to input texts such that the resulting examples cause a target text classifier to make incorrect predictions @xcite . this makes the robustness of transformer models against adversarial attacks crucial, especially in high-stake domains such as banking, law, and content moderation @xcite @xcite where susceptibility to such attacks can result in detrimental consequences such as giving out high-risk loans, wrongful indictments and enabling hate speech and disinformation. thus, ml practitioners must ensure their models are robust against text perturbations before deploying them to the public. to achieve this, existing works have proposed several ways to benchmark and analyze the robustness of perturbations of transformer models. in general, they often take a model-first approach-i.e., assuming that the model itself, such as its architecture or loss function formulation, is mainly responsible for its adversarial vulnerability and aiming to understand what kinds of changes in a model would shift its adversarial robustness @xcite @xcite . particularly, this approach iteratively makes a controlled alternation in the model-e.g., changing the architecture type, experimenting with novel attention layers, adding noises to the embeddings, etc., and then fine-tune the new model on the same finetuning dataset, followed by generating adversarial examples and benchmarking the model using the generated examples. although this model-first approach has resulted in several useful insights in practice, it assumes that adversarial robustness can only be evaluated only after a model has already been fine-tuned and adversarial examples have been generated. this approach tends to isolate the effects of the fine-tuning dataset and hence does not provide many insights on how such training data affect a model's robustness, such as "how the distribution of fine-tuning texts' embeddings and labels affect a model's robustness?", "how do the unique vocabulary and lengths of fine-tuning texts correlate with a model's robustness?", etc. exploring the relationship between fine-tuning data and model robustness promises to open up data-centric research directions to improve model robustness, introduced in detail in sec. 8.5. therefore, in this paper, we propose a global interpretation framework, as shown in fig. 1 , to investigate whether there is a strong direct correlation between finetuning data and model robustness and interpret the features of fine-tuning data that have the greatest influences. to this end, we take a different approach from model-first analysis and propose to analyze the adversarial robustness from a data-first approach. to do this, we extract 13 different features that comprehensively capture several important properties of not individual training examples but of the fine-tuning dataset as a whole. then, via regression analysis, we attempt to correlate them with the adversarial robustness of the models to be fine-tuned on the dataset measured as the average attack success rates (asrs) of 4 representative text perturbation methods on an unseen test set. to demonstrate one application of such novel analysis, we also try utilizing our interpretation framework to estimate the adversarial robustness of transformer classifiers even before they are fine-tuned and without the need to generate adversarial examples, only by analyzing their fine-tuning dataset. this approach is 30-193 times faster than the traditional method, as shown in fig. 1 . contributions of our paper are as follows. ## problem formulation we propose to develop a function g f θ (d) parameterized by θ that can effectively approximate the adversarial robustness of a pre-trained transformer-based classification model f when it is fine-tuned by an input training dataset d. in other words, g f θ (d) estimates the difference between predictions on examples of a clean, unseen test set d * (d ∩ d * =∅) that is drawn from the same distribution with d and is sufficiently large and on their corresponding adversarial examples. let's denote r(f, d, d * ) such adversarial robustness, we have: feature engineering label distribution token-based statistics pearson median skewness misclassification rate # of classes embedding distribution class separation kurtosis fisher's discriminant ratio mean distance among classes # of unique tokens avg # of tokens # of clusters min # of tokens max # of tokens davies-bouldin index calinski-harabasz index clustering surrogate model's learnabitliy bustness to be estimated as r(f, d)≈r(f, d, d * ) with any arbitrary d * . to train g f θ , which is specific to the model type f such as bert or roberta, we can then formulate this as a regression prediction problem and minimize the l 2 loss for an arbitrary fine-tuning dataset d as follows. in this work, we want to evaluate g f θ in two prediction scenarios, namely interpolation and extrapolation. in (1) interpolation or in-distribution evaluation, we want to validate g f θ on a fine-tuning corpus that is similar to one of the corpus included in q q q that the model g f θ has been trained on. this is also the standard evaluation setting in a typical machine learning problem. in (2) extrapolation or out-of-distribution evaluation, we want to validate g f θ on a dataset that is very different from corpus included in q q q-e.g., training on sentiment classification datasets and evaluating on a non-sentiment dataset such as q&a or fakenews detection. ## method overview. our goal is to create a regression dataset that includes (1) the features of the several smaller datasets and (2) their adversarial robustness-i.e., attack success rate (asr) of a transformer-based model f fine-tuned on each of them. then, we can use regression ml algorithms to predict such adversarial robustness and then analyze the influence of those features on the robustness of the model. fig. ## related work adversarial attacks in nlp. the general framework for adversarial attacks on a sentence includes two steps: (1) choosing which words in the sentence a target text classifier is most vulnerable to and (2) replacing them with a candidate such that the prediction label crosses the original prediction. thus, most of the attack methods differ in how they come up with new replacements, with the majority of them using word-level perturbation strategies such as via word-substitution ( @xcite @xcite ) or character-level attack such as via swapping and deleting characters within an original word ( @xcite ). while one can choose a set of random words in a sentence to perturb, existing works also propose several optimization schemes such as greedy search or genetic algorithm to select the optimal words to perturb and also their replacements. although these mechanisms help maximize the changes in the target classifier's behaviors while still preserving the sentence's original semantic meaning, the fact that they work with discrete nlp domain induces a substantial additional computational cost due to the need to continuously ping the target model for fine-tuning their perturbations, often on one token at a time. interpreting the adversarial robustness of models. @xcite claimed that a lack of model robustness is caused by non-robust features. as a result, they improved text classification models by including a bottleneck layer in their architectures to eliminate the effects of low-quality features. moreover, @xcite attributed the non-robust transformer models to outliers, and presented a resilient framework called transformer-rkde by replacing the dot-product attention with attention deriving from robust kernel density estimators. in addition to these works that focus more on model architectures, works such as @xcite focused more on drawing the relationship between specific linguistic patterns and the adversarial robustness, but only on unseen test sentences during inference. distinguished from these works, we emphasize and analyze the role of the fine-tuning dataset during model training on adversarial robustness and isolate the effects of the model architecture and inference inputs. datasets. we include 9 diverse publicly available classification corpus in the set d d d, namely ag news @xcite , amazon reviews full, amazon reviews polarity @xcite , dbpedia @xcite , yahoo answers, yelp reviews full, yelp reviews polarity @xcite , banking77 @xcite , and tweet eval review @xcite target models. we focus on studying the adversarial robustness of encoder-only transformer language models (lm) bert @xcite and roberta @xcite , which are often the standard baseline for text classification tasks. moreover, we also report experiment results on decoder-only lm gpt2 (radford et al., 2019), encoder-decoder lm bart @xcite transformer models, and electra @xcite , which is an encoder-only model but trained with an additional discriminator. interpolation and extrapolation evaluation. for interpolation, we employ overlapped k-fold crossvalidation of 80%:20% split and with @xmath0. for extrapolation, data points are split based on their original dataset. for example, we have a list of datasets d l and split them into three sets d 1 , d 2 , and d 3 such that d @xmath1, d 2 , d 3 } and ∅ = i,@xmath2,2,3} and i̸ =j {d i , d j } for training, validation and testing purposes, and to be more specific, and |d 3 | = 2. the train, val, and test sets of the extrapolation prediction include the data points respectively sampled from datasets in d 1 , d 2 , and d 3 . with this strategy, the train, val, and test sets have different contexts and ranges which are useful for extrapolation testing purposes. evaluation metrics. we employ standard evaluation metrics of regression prediction problems including root mean square error (rmse), r squared (r 2 ), mean absolute error and percentage error (mae, mape), explained variance score (evs). 6 results, analyses, and discussions 2 finding 1: fine-tuning data have a strong correlation with model robustness. finding 2: embedding distribution and tokenbased statistics features are among the most influential indicators of adversarial robustness. finding 1 demonstrates that our engineered features are highly informative about the fine-tuned model's robustness. fig. 5 further summarizes the order of influence of each feature in the case of random forest, which is the best regression predictor we found in finding 1. we only show features that have an average influence score twice greater than their variance. overall, embedding distribution and token-based statistics are the two groups of most influential features. in interpolation, chi, fr, and # of unique tokens have a significant influence on the adversarial robustness of bert (fig. 5a ), whereas such feature set of roberta also includes md (fig. 5b ). we also observe a similar pattern in predicting the adversarial robustness of bert (fig. 5c ) and roberta (fig. 5d ) in extrapolation prediction. finding 3: chi, fr, # unique tokens and # classes have clear correlations with asr. fig. 6 provides the correlation between notable features discussed in finding 2 and how they influence the asr prediction on average. these results show that the distances among classes in the embedding space-i.e., class separation sub-group (fig. 3 ), are highly indicative of the adversarial robustness of the fine-tuned models. when the embedding among classes disperses in the space and is not concentrated, fr feature has a low value and chi feature has a high value, which correlates to a greater robustness against adversarial examples. the opposite also holds as well, as illustrated in fig. 4 . furthermore, token-based statistics of the dataset such as # of unique tokens and # of classes also contribute to the influence on adversarial robustness. as # of classes increases, the embedding space becomes denser and clusters among prediction labels show more overlaps, the less robustness observed in the fine-tuned models. moreover, a large # of unique tokens often informs a diverse fine-tuning dataset, which makes the pre-trained transformer-based models more generalizable and hence more difficult to attack. error analysis. 3 the top three error-inducing features in asr prediction are dbi, # of classes, and mr. unlike md, fdr, and chi, dbi lacks robustness and fails to accurately represent embedding concentration because it is based on the distance to the nearest cluster compared to the original cluster. the increasing # of classes makes the decision boundary more complicated and greatly affects asr, but when a saturation threshold is crossed, this phenomenon no longer occurs. this explains our observation that although there is a strong correlation between the # of classes and asr shown in finding 3, our predictor has poor performance when the # of classes is greater than 10. cnn calculates the misclassification rate (mr) of the surrogate model, leveraging its focus on local metric bert distil-bert roberta distil-roberta rmse↓ 0.070 0.100 0.061 0.072 r 2 ↑ 0.806 0.621 0.782 0.740 mae↓ 0.045 0.075 0.052 0.049 evs↑ 0.812 0.790 0.918 0.760 mape↓ 0.145 0.173 0.139 0.109 ## another tool for robustness analysis the proposed approach saves significant runtime in estimating adversarial robustness with reasonable accuracy. the advantage of our method lies in skipping the adversarial example generation of four attacking methods used for evaluating adversarial robustness, making our inference time 30x-193x faster than the traditional approach when evaluating adversarial robustness on 100 examples (fig. 7 ). for example, when inferring the robustness of a transformer model fine-tuned on 900 test samples shown in fig. 7 , our method takes 153.02s including feature extraction (152.48s) and robustness inference by random forest (0.18s). conversely, the traditional method takes 4807.83s including finetuning plm (130.51s) and adversarial generation (4677.32s). as a result, our proposed framework is 31.4 times faster. another possible confounding factor that affects both the training data and a model's adversarial robustness is the label and data curation process. for example, some malicious actors might intentionally poison the training data to affect the training features, labels, and a model's adversarial robustness. we assume that all of our training data is clean (we use official, published sources of all datasets) and we do not expect any confounding factors might affect our analysis. there might be other confounding factors that we might overlook. within the pioneering nature of our work on this research topic, we hope to see future works that explore those confounding factors, for example, from the perspective of causality. ## conclusion in this paper, we introduce an approach to correlate the adversarial robustness of transformer models fine-tuned on new downstream datasets. by learning a lightweight regression-based robustness predictor on a taxonomy of 13 features of a fine-tuning dataset, we empirically demonstrate that our framework can effectively predict the model robustness in both interpolation and extrapolation settings with a significant speedup in inference.