Fooling the Textual Fooler via Randomizing Latent Representations (2024)

Abstract: Despite outstanding performance in a variety of Natural Language Processing (NLP) tasks, recent studies have revealed that NLP models are vulnerable to adversarial attacks that slightly perturb the input to cause the models to misbehave. Several attacks can even compromise the model without requiring access to the model architecture or model parameters (i.e., a blackbox setting), and thus are detrimental to existing NLP applications. To perform these attacks, the adversary queries the victim model many times to determine the most important parts in an input text and transform. In this work, we propose a lightweight and attack-agnostic defense whose main goal is to perplex the process of generating an adversarial example in these query-based black-box attacks; that is to fool the textual fooler. This defense, named AdvFooler, works by randomizing the latent representation of the input at inference time. Different from existing defenses, AdvFooler does not necessitate additional computational overhead during training nor does it rely on assumptions about the potential adversarial perturbation set while having a negligible impact on the model’s accuracy. Our theoretical and empirical analyses highlight the significance of robustness resulting from confusing the adversary via randomizing the latent space, as well as the impact of randomization on clean accuracy. Finally, we empirically demonstrate near state-of-the-art robustness of AdvFooler against representative adversarial attacks on two benchmark datasets.

## introduction in the last decade, deep neural networks have achieved impressive performance in the natural * the work was done prior to joining amazon. language processing (nlp) domain. several deep nlp models have reached state-of-the-art results in several nlp tasks @xcite using models such as recurrent neural networks, transformers, and pretrained language models (prlms). however, several works @xcite also reveal that deep nlp models can be fooled by the creation of adversarial examples @xcite . adversarial examples are synthetically perturbed inputs that are optimized to increase the errors between the predictions of the model and the true labels while being imperceptible to human evaluators @xcite . these developments have sparked concerns about the security and robustness of deep neural networks deployed in nlp applications. to generate adversarial examples, adversarial attack methods manipulate different aspects of the input sentence, from introducing character errors such as typos or visually similar characters @xcite and replacing words without significantly changing the original semantic in the perturbed inputs @xcite to recreating sentences with similar meanings using paraphrasing @xcite . among these methods, black-box adversarial attacks @xcite are some of the most widely studied and effective adversarial approaches. these methods query the victim model multiple times to find the corresponding adversarial perturbations of important words of an input text. in response to the adversarial threat and to enhance the robustness of nlp models, numerous defenses against textual adversarial attacks have been developed @xcite @xcite @xcite . the most popular approaches are adversarial training @xcite @xcite , which augments the training data with adversarial examples using an additional optimization step, and randomized smoothing @xcite , which replaces the model with its stochastic ensemble based on random perturbations of the input; both of these approaches require training modifications and additional non-trivial training computation (e.g., for adversarial training). due to the discrete nature of texts, to create adversarial inputs, these defenses also substitute input words with adversarial words sampled from predefined perturbation sets constructed based on the potential attacks instead of gradient-based optimization. this substitution process also aims to preserve the original input semantics. nonetheless, assuming knowledge of the potential attacks is often unrealistic and impractical. in this paper, we propose a lightweight, attackagnostic defensive method that can increase the robustness of nlp models against textual adversarial attacks. our defense, called advfooler, randomizes the latent representation of the input at test time to fool the adversary throughout their attack, which typically involves iteratively sampling of discrete perturbations to generate an adversarial sample. being a test-time defense with negligible computational overhead, advfooler also does not incur any training-time computation. furthermore, as advfooler operates within the latent space of the model, it remains agnostic of the perturbation sets associated with potential attacks. while there are some randomization defenses in nlp @xcite , they rely on randomized smoothing with necessary modifications to the model's training to reduce the variance in the model's outputs caused by randomizing the input or embedding space. the advantages of advfooler over the other defenses are shown in ## conclusion in this work, we proposed a lightweight, attackagnostic defense, advfooler, that can improve the robustness of nlp models against textual adversarial attacks. different from existing defenses, advfooler does not incur any training overhead nor relies on assumptions of the potential attacks. the main idea of advfooler is to mislead wordlevel perturbation-based adversary into selecting unimportant words by randomizing the latent space of the model, resulting in a significantly more challenging synonym-replacement process. we then provide both theoretical and empirical analyses to explain how advfooler fools the adversary, as well as its effect on the model's accuracy. our extensive experiments validate that advfooler improves the robustness of nlp models against multiple state-ofthe-art textual adversarial attacks on two datasets. finally, being a pluggable defense, advfooler can also be combined with existing defenses, such as adversarial training, to further protect the nlp models against these threats.