Residue-Based Natural Language Adversarial Attack Detection (2022)

Abstract: Deep learning based systems are susceptible to adversarial attacks, where a small, imperceptible change at the input alters the model prediction. However, to date the majority of the approaches to detect these attacks have been designed for image processing systems. Many popular image adversarial detection approaches are able to identify adversarial examples from embedding feature spaces, whilst in the NLP domain existing state of the art detection approaches solely focus on input text features, without consideration of model embedding spaces. This work examines what differences result when porting these image designed strategies to Natural Language Processing (NLP) tasks - these detectors are found to not port over well. This is expected as NLP systems have a very different form of input: discrete and sequential in nature, rather than the continuous and fixed size inputs for images. As an equivalent model-focused NLP detection approach, this work proposes a simple sentence-embedding “residue” based detector to identify adversarial examples. On many tasks, it out-performs ported image domain detectors and recent state of the art NLP specific detectors.

## introduction in the last decade deep learning based models have demonstrated success in a wide range of application areas, including natural language processing (nlp) @xcite and object recognition @xcite . these systems may be deployed in mission critical situations, where there is the requirement for a high level of robustness. however, @xcite demonstrated that deep models have an inherent weakness: small perturbations in the input can yield significant, undesired, changes in the output from the model. these input perturbations were termed adversarial examples and their generation adversarial attacks. adversarial attacks have been developed for systems operating in various domains: image systems @xcite @xcite and nlp systems @xcite @xcite . the characteristics of the input can be very different between these application domains. broadly, the nature of inputs can be described using two key attributes: static (fixed length) vs sequential and continuous vs discrete. under this categorisation, image inputs are continuous and static, whilst nlp inputs are discrete and sequential. this work argues that due to the fundamental differences in the input and resulting adversarial perturbations in the different domains, adversarial attack behaviour can vary significantly from one domain to another. hence, the extensive research on exploring and understanding adversarial perturbation behaviour in the continuous, static world of image systems does not necessarily transfer well to the nlp tasks. for adversarial attack generation, a number of specific nlp attacks have been proposed that are designed for nlp task inputs @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . however, there has been less research on developing defence schemes. these defence strategies can be split into two main groups: model modification, where the model or data is altered at training time (e.g. adversarial training @xcite ) and detection, where external systems or algorithms are applied to trained models to identify adversarial attacks. as model modification approaches demand re-training of models, detection approaches are usually considered easier for implementation on de-ployed systems and thus are often preferred. hence, this work investigates the portability of popular detection approaches designed for image systems to nlp systems. furthermore, this work introduces a specific nlp detection approach that exploits the discrete nature of the inputs for nlp systems. this approach out-performs standard schemes designed for image adversarial attack detection, as well as other nlp detection schemes. the proposed nlp specific detection approach will be referred to as residue detection, as it is shown that adversarial attacks in the discrete, word sequence space result in easily detectable residual components in the sentence embedding space. this residue can be easily detected using a simple linear classifier operating in the encoder embedding space. in addition, this work shows that even when an adversary has knowledge of the linear residual detector, they can only construct attacks at a fraction of the original strength. hence this work argues that realistic (word level, semantically similar) adversarial perturbations at the natural language input of nlp systems leave behind easily detectable residue in the sentence embedding. interestingly, the residue detection approach is shown to perform poorly when used to detect attacks in the image domain, supporting the hypothesis that the nature of the input has an important influence on the design of effective defence strategies. ## related work previous work in the image domain has analysed the output of specific layers in an attempt to identify adversarial examples or adversarial subspaces. first, @xcite proposed that adversarial subspaces have a lower probability density, motivating the use of the kernel density (kd) metric to detect the adversarial examples. nevertheless, @xcite found local intrinsic dimensionality (lid) was a better metric in defining the subspace for more complex data. in contrast to the local subspace focused approaches of kd and lid, @xcite showed that trajectories of hidden layer features can be used to train a lstm network to accurately discriminate between authentic and adversarial examples. out performing all previous methods, @xcite introduced an effective detection framework using mahalanobis distance analysis (mda), where the distance is calculated between a test sample and the closest class-conditional gaussian distribution in the space defined by the output of the final layer of the classifier (logit space). @xcite also explored using the output of convolutional layers for image classification systems to identify statistics that distinguish adversarial samples from original samples. they find that by performing a pca decomposition the statistical variation in the least principal directions is the most significant and can be used to separate original and adversarial samples. however, they argue this is ineffective as an adversary can easily suppress the tail distribution. hence, @xcite extract statistics from the convolutional layer output to train a cascade classifier to separate the original and adversarial samples. most recently, @xcite avoid the use of artificially designed metrics and combine the adversarial subspace identification stage and the detecting adversaries stage into a single framework, where a parametric model adaptively learns the deep features for detecting adversaries. in contrast to the embedding space detection approaches, @xcite shows that influence functions combined with nearest neighbour distances perform comparably or better than the above standard detection approaches. other detection approaches have explored the use of uncertainty: @xcite argues that adversarial examples are out of distribution and do not lie on the manifold of real data. hence, a discriminative bayesian model's epistemic (model) uncertainty should be high. therefore, calculations of the model uncertainty are thought to be useful in detecting adversarial examples, independent of the domain. however, bayesian approaches aren't always practical in implementation and thus many different approaches to approximate this uncertainty have been suggested in literature @xcite @xcite . there are a number of existing nlp specific detection approaches. for character level attacks, detection approaches have exploited the grammatical @xcite and spelling @xcite inconsistencies to identify and detect the adversarial samples. however, these character level attacks are unlikely to be employed in practice due to the simplicity with which they can be detected. therefore, detection approaches for the more difficult semantically similar attack samples are of greater interest, where the meaning of the textual input is maintained without compromising the spelling or grammatical in-tegrity. to tackle such word-level, semantically similar examples, @xcite designed a discriminator to classify each token representation as part of an adversarial perturbation or not, which is then used to 'correct' the perturbation. other detection approaches @xcite @xcite have shown some success in using perplexity to identify adversarial textual examples. most recently, @xcite achieved state of the art performance with the frequency guided word substitution (fgws) detector, where a change in model prediction after substituting out low frequency words is revealing of adversarial samples. ## adversarial attacks an adversarial attack is defined as an imperceptible change to the input that causes an undesired change in the output of a system. often, an attack is found for a specific data point, x. consider a classifier f θ, with parameters θ, that predicts a class label for an input data point, x, sampled from the input distribution x . a successful adversarial attack is where a perturbation δ at the input causes the system to miss-classify, when defining adversarial attacks, it is important consider the interpretation of an imperceptible change. adversarial perturbations are not considered effective if they are easy to detect. hence, the size of the perturbation must be constrained: for whitebox attacks in the image domain, the dominant attack approach has proven to be projected gradient descent (pgd) @xcite . the pgd approach, iteratively updates the adversarial perturbation, δ, initialised as δ 0 = 0. each iterative step moves the perturbation in the direction that maximises the loss function, l, used in the training of the model, this requires a change in the interpretation of the perturbation δ. it is not simple to define an appropriate function g() in equation 2 for word sequences. perturbations can be measured at a character or word level. alternatively, the perturbation could be measured in the vectorized embedding space (equation 5), using for example l p -norm based @xcite metrics or cosine similarity @xcite , which have been used in the image domain. however, constraints in the embedding space do not necessarily achieve imperceptibility in the original word sequence space. the simplest approach is to use a variant of an editbased measurement @xcite , l e (), which counts the number of changes between the original sequence, w 1:l and the adversarial sequence w ′ 1:l ′ , where a change is a swap/addition/deletion, and ensures it is smaller than a maximum number of changes, n , for the nlp adversarial attacks this work only examines word-level attacks, as these are considered more difficult to detect than character-level attacks. as an example, for an input sequence of l words, a n -word substitution adversarial attack, w ′ 1:n , applied at word positions n 1 , n 2 , . . . , n n gives the adversarial output, w ′ the challenge is to select which words to replace, and what to replace them with. a simple yet effective substitution attack approach that ensures a small change in the semantic content of a sentence is to use saliency to rank the word positions, and to use word synonyms for the substitutions @xcite . this attack is termed probability weight word saliency (pwws). the highest ranking word word can be swapped for a synonym from a preselected list of given synonyms. the next most highly ranked word is substituted in the same manner and the process is repeated till the required n words have been substituted. the above approach is limited to attacking specific word sequences and so cannot easily be generalised to universal attacks @xcite , where the same perturbation is used for all inputs. for this situation, a simple solution is concatenation @xcite , where for example, the same n -length sequence of words is appended to each input sequence of words, as described in raina et al. (2020). here, in both the substitution attack (equation 8 ) and the concatenation attack (equation 9 ), the size of the attack can be measured using the number of edits, l e (w 1:l , w ′ 1:l ′ ) = n . ## adversarial attack detection for a deployed system, the easiest approach to defend against adversarial attacks is to use a detection process to identify adversarial examples without having to modify the existing system. for the image domain section 2 discusses many of the standard detection approaches. in this work, we select two distinct approaches that have been generally successful: uncertainty @xcite , where adversarial samples are thought to result in greater epistemic uncertainty and mahalanobis distance @xcite , where the mahalanobis distance in the logit space is indicative of how out of distribution a sample is (adversarial samples are considered more out of distribution). in the nlp domain, when excluding trivial grammar and spelling based detectors, perplexity based detectors can be used @xcite . many other nlp specific detectors @xcite @xcite have been proposed, but mozes et al. (2020)'s fgws detector is considered the state of art and is thus selected for comparison. here low frequency words in an input are substituted for higher frequency words and the change in model prediction is measured -adversarial samples are found to generally have a greater change. this work introduces a further nlp specific detector: residue detection, described in detail in section 4.1. when considering any chosen detection measure f d , a threshold β can be selected to decide whether an input, w 1:l , is adversarial or not, where f d (w 1:l ) > β, implies that w 1:l is an adversarial sample. to assess the success of the adversarial attack detection processes, precision-recall curves are used. for the binary classification task of identifying an input as adversarially attacked or not, at a given threshold β, the precision and recall values can be computed as @xmath0, where tp, fp and fn are the standard true-positive, false-positive and falsenegative definitions. a single point summary of precision-recall curves is given with the f 1 score. ## conclusions in recent years, deep learning systems have been deployed for a large number of tasks, ranging from the image to the natural language domain. however, small, imperceptible adversarial perturbations at the input, have been found to easily fool these systems, compromising their validity in high-stakes applications. defence strategies for deep learning systems have been extensively researched, but this research has been predominantly carried out for systems operating in the image domain. as a result, the adversarial detection strategies developed, are inherently tuned to attacks on the continuous space of images. this work shows that these detection strategies do not necessarily transfer well to attacks on natural language processing systems. hence, an adversarial attack detection approach is proposed that specifically exploits the discrete nature of perturbations for attacks on discrete sequential inputs. the proposed approach, termed residue detection, demonstrates that imperceptible attack perturbations on natural language inputs tend to result in large perturbations in word embedding spaces, which result in distinctive residual components. these residual components can be identified using a simple linear classifier. this residue detection approach was found to out-perform both detection approaches ported from the image domain and other state of the art nlp specific detectors. the key finding in this work is that the nature of the data (e.g. discrete or continuous) strongly influences the success of detection systems and hence it is important to consider the domain when designing defence strategies.