Sample Attackability in Natural Language Adversarial Attacks (2023)

Abstract: Adversarial attack research in natural language processing (NLP) has made significant progress in designing powerful attack methods and defence approaches. However, few efforts have sought to identify which source samples are the most attackable or robust, i.e. can we determine for an unseen target model, which samples are the most vulnerable to an adversarial attack. This work formally extends the definition of sample attackability/robustness for NLP attacks. Experiments on two popular NLP datasets, four state of the art models and four different NLP adversarial attack methods, demonstrate that sample uncertainty is insufficient for describing characteristics of attackable/robust samples and hence a deep learning based detector can perform much better at identifying the most attackable and robust samples for an unseen target model. Nevertheless, further analysis finds that there is little agreement in which samples are considered the most attackable/robust across different NLP attack methods, explaining a lack of portability of attackability detection methods across attack methods.

## introduction with the emergence of the transformer architecture @xcite , natural language processing (nlp) models have demonstrated impressive performance in many tasks, ranging from standard sentiment classification @xcite to summarisation @xcite and translation @xcite . however, @xcite demonstrated that deep learning models are susceptible to adversarial attacks, where carefully crafted small imperceptible changes applied to original, natural inputs can cause models to mis-classify. in response, extensive efforts have explored methods to combat the threat of adversarial attacks by training with adversarial examples @xcite or building separate detection systems @xcite . however, little or no work has sought to determine which input samples are the most susceptible to adversarial attacks. are certain input samples easier to adversarially attack and if so can we efficiently identify these attackable samples? the ability to identify the attackable and in converse the robust samples has applications in a range of sample-selection tasks. for example, in the field of active learning @xcite , the query system can be designed to select the most attackable samples. similarly, knowledge of sample attackability is useful for weighted adversarial training @xcite , where the aim is to augment the training set with only the most useful adversarial examples. in the image domain, @xcite formally define the notion of sample attackability as the minimum perturbation size required to change a sample's output prediction from the target model. running iterative adversarial attacks to determine this minimum perturbation size for a single sample is inefficient. @xcite use entropy (uncertainty) as a proxy function for sample attackability, but, raina and gales (2023) demonstrate that training a deep learning based classifier to predict the most attackable samples (and most robust samples) is the most effective method in the image domain. therefore, this works extends the use of a deep learning based system to identify the most attackable and robust samples in nlp tasks. as a measure of a sample's attackability, it is challenging to define a sample's perturbation size for natural language. following raina and gales (2023) in the image domain, this work uses the imperceptibility threshold in the definition of an adversarial attack as a measure of the perturbation size. to align with human perception, imperceptibility constraints for nlp aim to limit the seman-tic change in the text after an adversarial attack. these imperceptibility constraints can be grouped into two stages: 1) pre-transformation constraints (e.g. no stopword changes) that limit the set of acceptable adversarial examples; and 2) distance constraints that only allow for a subset of the acceptable adversarial examples, where the distance constraint explicitly restricts the distance moved by an adversarial example from the original example to satisfy a specified imperceptibility threshold. this distance can be measured for example using the universal sentence encoder @xcite . a sample subject to a specific nlp attack method (with defined pre-transformation constraints) will have an associated set of acceptable adversarial examples. the attackability of the sample can thus be given by the smallest distance constraint imperceptibility threshold that at least one acceptable adversarial example in the set satisfies. default imperceptibility thresholds for the distance constraints proposed for nlp attack methods can often lead to unnatural adversarial examples @xcite . hence, in this work, we use separate thresholds for defining attackable and robust samples. a sample's minimum perturbation size is required to be within a much stricter imperceptibility threshold to be termed attackable, whilst in converse a sample's minimum perturbation size has to be greater than a more generous imperceptibility threshold to be termed robust. the deep learning based attackability classifier proposed in @xcite is successfully used to identify the attackable and robust samples for unseen data and unseen target models. however, in contrast to the image domain, it is found in nlp that the trained attackability detector fails to determine the attackable samples for different unseen nlp attack methods. this work extensively analyzes this observation and offers an explanation rooted in the inconsistency of imperceptibility definitions for different nlp attack methods. ## related work in the image domain @xcite introduce the notion of sample attackability through the language of vulnerability of a sample to an adversarial attack. this vulnerability is abstractly defined as the distance of a sample to a model's decision boundary. raina and gales (2023) offer a more formal and extensive estimate of a sample's vulnerability/attackability by considering the smallest perturbation size, aligned with an adversarial attack's imperceptibility measure, to change a sample's class prediction. other research in the field of weighted adversarial training @xcite , has also implicitly considered the notion of sample attackability. the aim in weighted adversarial training is train with the more useful adversarial examples, which are arguably sourced from the more attackable original samples. for example @xcite use model entropy to estimate this attackability, whilst @xcite use model confidence and raina and gales (2023) are successful in using a deep-learning based estimator of attackability. in the field of nlp, little work has explored weighted adversarial training. @xcite propose a meta-learning algorithm to lean the importance of each adversarial example, but this has no direct relation to a source sample's attackability. finally, in the field of active learning @xcite there has also been implicit consideration of adversarial perturbation sizes as a measure of a sample's value. the aim in active learning is to select the most useful subset of samples in a dataset to train a model on. in the image domain, @xcite propose the use of the smallest adversarial perturbation size for each sample to measure the distance to the decision boundary. however, there is no explicit consideration of sample attackability or design of an efficient method to identify the attackable samples. ## adversarial attacks in both the image and nlp domain, an untargeted adversarial attack is able to fool a classification system, f(), by perturbing an input sample, x to generate an adversarial example x to cause a change in the output class, f(x) ̸ = f(x). it is necessary for adversarial attacks to be imperceptible, such that adversarial examples, x are not easily detectable/noticeable by humans. it is inefficient and expensive to rely on manual human measures of attack imperceptibility, so instead proxy measures are used to enforce imperceptibility of an adversarial attack. for images, the l p norm is considered a good proxy for human perception of imperceptibility. however, in nlp it is more challenging to ensure imperceptibility. despite earlier research introducing only visual constraints on the adversarial attacks @xcite @xcite @xcite , e.g. number of words changed as per the levenshtein distance, recent research considers more sophisticated measures seeking to measure the semantic change in text sequences @xcite @xcite @xcite @xcite . in general, modern nlp imperceptibility constraints can be separated into two stages: pretransformation constraints and distance constraints. pre-transformation constraints typically limit the attack mechanism to encourage little change in semantic content. for example, stop-word transformations will be prevented or any word substitutions will be restricted to appropriate synonyms. a collection of pre-transformation constraints, as specified by a particular attack method, limit the available set, a of possible adversarial examples that can be considered for a specific sample, x, such that @xmath0. the distance-based constraints are further constraints that explicitly aim to limit the distance between the original sample x and the adversarial example, x to ensure a small perceived semantic change. this distance can be measured via a proxy ## sample attackability definition sample attackability is concerned with how easy it is to adversarially attack a specific sample. the notion of sample attackability is formally introduced by raina and gales (2023), where a specific input sample, x n 's attackability for a specific model, f k is given by the theoretical minimum perturbation size, δ(k) n within which a sample can be successfully attacked. however, it is not simple to define the perturbation size for an adversarial attack in nlp. the simplest definition for the perturbation size, δ, for a specific attack method with a specific set of acceptable adversarial examples, a (equation 2), is to use the distance-based proxy function, g (equation 3 ), such that δ = g(x, x). then the minimum perturbation size, δ(k) n for sample n and model k is, we aim to use a sample's minimum perturbation size to classify it as attackable, robust or neither. default distance-based imperceptibility constraints defined using g for various nlp attack methods can lead to unnatural adversarial examples and so we use separate and stricter thresholds for classifying samples as attackable or robust. hence, as in raina and gales (2023), we define sample n as attackable for model k if the smallest adversarial perturbation is less than a strict threshold, ## attackability detector the definition of attackable and robust samples uses the minimum perturbation size (as per a distance-based constraint) for an nlp adversarial attack on a sample. when trying to determine which samples are attackable, it is slow and expensive to run an adversarial attack iteratively to find the minimum perturbation size. further, often one may not have access to an unseen target model, f t or even the target sample, n to perform an adversarial attack upon. hence, in this setting, it is necessary to have a simple and efficient process that can determine whether samples in an unseen dataset are attackable for an unseen target model. inspired by raina and gales (2023), this section describes a method to train a simple deep-learning attackability detector to identify the attackable and robust samples in an unseen dataset, for an unseen target model, f t . we give the deep-learning attackability detector access to a seen dataset, {x n , y n } n @xmath1, @xmath2, . . . , f |m| }, such that f t / ∈ m. each model can be represented as an encoder embedding stage, followed by a classification stage, consistent with raina and gales (2023), we use a simple, single hidden-layer fully connected network architecture for each attackability detector, d, such that, raina and gales (2023) demonstrated that this estimate in the image domain does not capture the samples that are attackable specifically for the target model, f t 's specific realisation. therefore, we seek instead to estimate the probability of a universally attackable sample (defined in equation 6 ), the attackability/robustness of samples can also be estimated using simple uncertainty based approaches, such as entropy @xcite or a sample's class margin measured by model confidence @xcite . these uncertainty measures can then also be compared to strict thresholds to classify samples as attackable or robust. experiments in section 6 compare the deep-learning based attackability detector to uncertainty-based attackability detectors. to assess which attackability detector performs the best in identifying attackable samples for the unseen target model, f t / ∈ m, we consider four variations on defining a sample, n as attackable @xcite . allthe sample is attackable for the unseen target model. given that the deep learning based attackability detectors are trained to identify universally attackable samples (equation 11 ), they are expected to perform best in the uni evaluation setting. the corpus-level performance of an attackability detector for an unseen dataset can be reported using precision and recall. a selected threshold, β, is used to class the output of detectors, e.g. p(a by sweeping the threshold β a full precision-recall curve can be generated and typically the threshold with the greatest f1-score is selected as an appropriate operating point. ## experiments 6.1 setup experiments in this section aim to understand how well a deep-learning based detector, described in section 5, performs in identifying attackable samples for an unseen dataset and an unseen target model, f t , where the detector only has access to a separate set of seen models, m during training. there are equivalent experiments looking to detect the most robust samples too. the performance of the deep learning based detector is compared to a baseline of uncertainty-based detectors (model confidence), inspired by @xcite , in which the samples with the most uncertain model predictions are identified as attackable and in converse the most certain samples are deemed to be robust. specifically, two forms of uncertainty-based detectors are considered: 1) conf-u, where there is no access to the confidence from the unseen target model and so a sample's uncertainty is measured by an average of the confidence of the seen models, m; and as a realistic reference we also consider 2) conf-s, where there is access to the target model output such that the target model's confidence is used directly as a measure of sample uncertainty. two popular natural language classification datasets are used in these experiments. first, the stanford sentiment treebank2 dataset (sst) @xcite is a movie review dataset with each review labelled as positive or negative. there are 6920 training samples, 872 validation samples and 1820 test samples. we also consider the twitter emotions dataset @xcite , which categorizes tweets into one of six emotions: sadness, love, anger, surprise, joy and fear. this dataset contains 16,000 training samples, 2000 validation samples and 2000 test samples. for training of the attackability detectors, access was provided to only the validation data and hence the test data was used as an unseen set of samples to assess the performance of attackable sample detection. these experiments work with four state of the art nlp transformer-based models: bert (bert) @xcite , xlnet (xlnet) @xcite , roberta (roberta) @xcite and electra (electra) @xcite . each model is of base-size (110m parameters). finetuning on sst and twitter used adamw optimizer, 3 epochs and a learning rate of 1e-5. the performance of the models is given in four adversarial attack types are considered in these experiments: textfooler (tf) @xcite , bert-based adversarial examples (bae) @xcite , improved genetic algorithm (iga) @xcite and probability weighted word saliency (pwws) @xcite . in the bae attack we consider specifically the bae-r attack mode from the paper, where the aim is to replace tokens. for nlp adversarial attacks section 3 discusses the nature of imperceptibility constraints, where constraints can either be pretransformation constraints (equation 2) or distancebased constraints (equation 3 ). ## conclusions little research has sought to determine the level of vulnerability of individual samples to an adversarial attack in natural language processing (nlp) tasks. this work formally extends the definitions of sample attackability to the field of nlp. it is demonstrated that uncertainty-based approaches are insufficient in characterising the most attackable and the most robust samples in a dataset. instead, a deeplearning based detector can be used to effectively to identify these attackable/robust samples for an unseen dataset and more powerfully for an unseen target model. however, it is also observed that different attack methods in natural language have a different set of imperceptibility constraints, leading to a lack of consistency in determining sample attackability across different attack methods. as a consequence, the success of a deep-learning based attackability detector is limited to the attack method it is trained with.