“That Is a Suspicious Reaction!”: Interpreting Logits Variation to Detect NLP Adversarial Attacks (2022)

Abstract: Adversarial attacks are a major challenge faced by current machine learning research. These purposely crafted inputs fool even the most advanced models, precluding their deployment in safety-critical applications. Extensive research in computer vision has been carried to develop reliable defense strategies. However, the same issue remains less explored in natural language processing. Our work presents a model-agnostic detector of adversarial text examples. The approach identifies patterns in the logits of the target classifier when perturbing the input text. The proposed detector improves the current state-of-the-art performance in recognizing adversarial inputs and exhibits strong generalization capabilities across different NLP models, datasets, and word-level attacks.

## introduction despite recent advancements in natural language processing (nlp), adversarial text attacks continue to be highly effective at fooling models into making incorrect predictions @xcite @xcite . in particular, syntactically and grammatically consistent attacks are a major challenge for current research as they do not alter the semantical information and are not detectable via spell checkers @xcite . while some defense techniques addressing this issue can be found in the literature @xcite @xcite , results are still limited in performance and text attacks keep evolving. this naturally raises concerns around the safe and ethical deployment of nlp systems in real-world processes. previous research showed that analyzing the model's logits leads to promising results in discriminating manipulated inputs @xcite @xcite . however, logits-based adversarial detectors have been only studied on computer vision applications. our work transfers this type of methodology to the nlp domain and its contribution can be summarized as follows: ## methodology the defense approach proposed in this work belongs to the category of adversarial detection. it defends the target model from attacks generated via word-level perturbations belonging to the semantic similarity category. the intuition behind the method is that the model's reaction to originaland adversarial samples is going to differ even if the inputs are similar. hence, it relies on feature attribution explanations coupled with a machine learning model to learn such difference and thus identify artificially crafted inputs. figure 1 shows the overall pipeline of the approach. given a text classifier f trained on the task at hand, the pipeline's goal is to detect whether the currently fed input x is adversarial. in 3.1, we explain in greater detail how we measure the model f 's reaction to a given input x. this quantitylater indicated with wdr(x, f )-is then passed to the adversarial detector, whose training procedure is described in 3.2. finally, in 3.3, we provide detailed information about the setup of our experiments such as target models, datasets, and attacks. ## experimental results in this section, we report the experimental results of our work. in 4.1, we study various detector architectures to choose the best performing one for the remaining experiments. in 4.2, we measure our pipeline's performance in several configurations (target model, dataset, attack) and we compare it to the current state-of-the-art adversarial detectors. while doing so, we also assess transferability via observing the variation in performance when changing the dataset, the target model, and the attack source without retraining our detector. finally, in 4.3, we look at how different decision boundaries affect performance metrics. ## discussion and qualitative results section 4 discussed quantitative results and emphasized the competitive performance that the proposed approach achieves. here, instead, we focus on the qualitative aspects of our research findings. for instance, we try to understand why our pipeline works while also discussing challenges, limitations, ethical concerns, and future work. ## conclusion adversarial text attacks are a major obstacle to the safe deployment of nlp models in high-stakes applications. however, although manipulated and original samples appear indistinguishable, interpreting the model's reaction can uncover helpful signals for adversarial detection. our work utilizes logits of original and adversarial samples to train a simple machine learning detector. wdr scores are an intuitive measure of word relevance and are effective for detecting text components having a suspiciously high impact on the output. the detector does not make any assumption on the classifier targeted by the attacks and can be thus considered model-agnostic. the proposed approach achieves very promising results, considerably outperforming the previous state-of-the-art in word-level adversarial detection. experimental results also show the detector to possess remarkable generalization capabilities across different target models, datasets, and text attacks without needing to retrain. these include transformer architectures such as bert and wellestablished attacks such as pwws, genetic algorithms, and context-aware perturbations. we believe our work sets a strong baseline on which future research can build to develop better defense strategies and thus promoting the safe deployment of nlp models in practice. we release our code to the public to facilitate further research and development foot_2 .