Input-specific Attention Subnetworks for Adversarial Detection (2022)

Abstract: Self-attention heads are characteristic of Transformer models and have been well studied for interpretability and pruning. In this work, we demonstrate an altogether different utility of attention heads, namely for adversarial detection. Specifically, we propose a method to construct input-specific attention subnetworks (IAS) from which we extract three features to discriminate between authentic and adversarial inputs. The resultant detector significantly improves (by over 7.5%) the state-of-the-art adversarial detection accuracy for the BERT encoder on 10 NLU datasets with 11 different adversarial attack types. We also demonstrate that our method (a) is more accurate for larger models which are likely to have more spurious correlations and thus vulnerable to adversarial attack, and (b) performs well even with modest training sets of adversarial examples.

## introduction self-attention heads are characteristic of transformer models. individual attention heads are interpretable in different ways. one, for a token in an input sentence, we can visualize the attention paid by a head to all other tokens. such attention patterns are attractive linguistically and have come to define roles for attention heads @xcite . two, the output of attention heads from various layers can be probed for their ability to encode information related to the "nlp pipeline" @xcite @xcite . three, attention patterns of heads can represent knowledge learnt by a teacher model when distilling to a smaller student model @xcite . while individual attention heads are interpretable in the above ways, it is found that attention heads in models such as bert are over-provisioned and can be pruned. for instance, @xcite showed that a model with 16 attention heads per layer can be pruned to just one. @xcite and @xcite have shown similar results with different pruning techniques across tasks. in the above methods, while interpretation of attention heads is input-specific, pruning of heads is input-agnostic. can these two be combined, i.e., can we prune attention heads in an input-specific manner creating opportunities for interpretation? we explore this idea to identify an altogether different utility of attention heads -namely adversarial detection which is the task of differentiating between authentic and adversarial inputs. specifically, we propose a method to obtain an input-specific attention subnetwork (ias), which is a subnetwork where a subset of attention heads is masked without affecting the output of the model for that input. such subnetworks could vary across inputs representing how the model works for each input. this is particularly important for adversarial detection, as adversarial inputs do not reveal themselves in what the model outputs but may leave tell-tale signs in how the model computes this output. in this work, we present a technique to efficiently compute ias and demonstrate its utility in adversarial detection with significantly improved accuracy over all current methods. to this end, we propose three sets of features from ias. the first feature, f mask , is simply the attention mask that identifies if an attention head is retained or pruned in ias. the second feature, f flip , characterizes the output of a "mutated" ias obtained by toggling the mask used for attention heads in the middle layers of ias. the third feature, f lw , characterizes the outputs of ias as obtained layer-wise with a separately trained classification head for each layer. we train a classifier, called advnet, with these features as inputs to predict if an input is adversarial. we report results on 10 nlu tasks from the glue benchmark (sst2, mrpc, rte, snli, mnli, qqp, qnli) and elsewhere (yelp, ag news, imdb). for each of these tasks, we first create a benchmark of adversarial examples com-bining 11 attack methodologies like word order swap @xcite , embedding swap @xcite , word deletion @xcite , etc. in total, the benchmark contains 5,686 adversarial examples across tasks and attack types. to the best of our knowledge, this dataset is the most extensive benchmark available on the considered tasks. across all these tasks and attack types, we compare our adversarial detection technique against state-ofthe-art methods such as disp @xcite , nws @xcite , and fgws @xcite . our method establishes the best results in all tasks and attack types, with an average improvement of 7.45% over the best method for each task. our detector achieves an accuracy of 80-90% across tasks suggesting effective defense against adversarial attacks. having established the utility of attention heads for adversarial detection, we perform several ablation studies. first, we compare different combinations of the features demonstrating that they are mutually informative and thus combining them all works best. second, we show that cutmix data augmentation @xcite improves accuracy, demonstrating the first use of this method in adversarial detection in nlp tasks. third, we show that the detector is more accurate as the size of the language model scales. this is encouraging because larger language models are expected to have increased spurious correlations and thus are more vulnerable to adversarial attacks. fourth, we show that the detector performs well even for modest training sizes of adversarial examples, suggesting effective generalization. in summary, we propose a novel relation between attention heads and adversarial detection. the effectiveness of the resultant detector establishes that the mask of attention heads captures critical information about how a transformer model works for a given input. the rest of the paper is organized as follows. we detail our core method of computing ias in the next section. in section 3 we discuss the features from ias for adversarial detection. we detail the experimental setup along with the dataset creation process in section 4. we present our results in section 5 and conclude in section 6. ## input-specific attention subnetworks in this section, we describe input-specific attention subnetworks (ias) and the computational approach to identify ias for a given input. ## model for adversarial detection in this section, we explain how we extract features from the ias and the design of the classifier for adversarial detection. we use the term target class to refer to the class predicted by the complete finetuned network for an input. for authentic inputs, this translates to the true class while for adversarial inputs, this refers to the adversarial class that the model is fooled into predicting. ## results & discussion in this section, we first analyse the ias (section 5.1) and the constituent features of advnet (section 5.2). we then perform a comparative study with state-of-the-art adversarial detection methods (section 5.3). lastly, we perform ablation studies to understand the effect of task, model size, feature combinations and training set attacks on the performance of advnet (section 5.4). unless otherwise stated, the plots pertain to experiments on the sst-2 dataset with the bert-base model. ## conclusion and future work in this work, we present an altogether new utility of attention heads in transformer networks -to detect adversarial attacks. we defined input-specific attention subnetworks (ias) and proposed a method to compute them efficiently. we extracted 3 features from ias and showed their utility in distinguishing adversarial samples from authentic ones. we demonstrated that our approach significantly improves the state-of-the-art accuracy across datasets and attack types. our work suggests that input-specific model perturbations provide strong signals to interpret transformer-based models such as large language models. further, the sparse nature of the identified ias indicate opportunities for input-specific model optimization. in future work, we would like to extend this study to tasks beyond nlu, including vision and speech-related tasks.