Textual Manifold-based Defense Against Natural Language Adversarial Examples (2022)

Abstract: Despite the recent success of large pretrained language models in NLP, they are susceptible to adversarial examples. Concurrently, several studies on adversarial images have observed an intriguing property: the adversarial images tend to leave the low-dimensional natural data manifold. In this study, we find a similar phenomenon occurs in the contextualized embedding space of natural sentences induced by pretrained language models in which textual adversarial examples tend to have their embeddings diverge off the manifold of natural sentence embeddings. Based on this finding, we propose Textual Manifold-based Defense (TMD), a defense mechanism that learns the embedding space manifold of the underlying language model and projects novel inputs back to the approximated structure before classification. Through extensive experiments, we find that our method consistently and significantly outperforms previous defenses under various attack settings while remaining unaffected to the clean accuracy. To the best of our knowledge, this is the first kind of manifold-based defense adapted to the NLP domain.

## introduction the field of nlp has achieved remarkable success in recent years, thanks to the development of large pretrained language models (plms). however, multiple studies have shown that these models are vulnerable to adversarial examples -carefully optimized inputs that cause erroneous predictions while remaining imperceptible to humans @xcite @xcite . this problem raises serious security concerns as plms are widely deployed in many modern nlp applications. various defenses have been proposed to counter adversarial attacks, which can be broadly categorized into empirical @xcite @xcite and certified @xcite @xcite defenses. adversarial training is currently the most successful empirical method @xcite @xcite . it operates by jointly training the victim model on clean and adversarial examples to improve the robustness. however, one major drawback of this approach is the prohibitively expensive computational cost. @xcite has theoretically shown that with just simple models, the sample complexity of adversarial training already grows substantially compared to standard training. alternatively, certified defenses aim to achieve a theoretical robustness guarantee for victim models under specific adversarial settings. however, most certified defenses for nlp are based on strong assumptions on the network architecture @xcite , and the synonym set used by attackers is often assumed to be accessible to the defenders @xcite . @xcite has shown that when using different synonym set during the attack, the effectiveness of these methods can drop significantly. concurrent with the streams of attack and defense research, numerous efforts have been made to understand the characteristics of adversarial examples @xcite @xcite @xcite . one rising hypothesis is the off-manifold conjecture, which states that adversarial examples leave the underlying low-dimensional manifold of natural data @xcite @xcite . this observation has inspired a new line of defenses that leverage the data manifold to defend against adversarial examples, namely manifold-based defenses @xcite @xcite . despite the early signs of success, such methods have only focused on images. it remains unclear if the off-manifold conjecture also generalizes to other data domains such as texts and how one can utilize this property to improve models' robustness. in this study, we empirically show that the offmanifold conjecture indeed holds in the contextualized embedding space of textual data. based on this finding, we develop textual manifold-based defense (tmd), a novel method that leverages the manifold of text embeddings to improve nlp robustness. our approach consists of two key steps: in summary, the key contributions in this paper are as follows: ## textual manifold-based defense in this section, we introduce our method in greater detail. we first introduce how we approximate the contextualized embedding manifold using deep generative models. then, we describe how to perform on-manifold projection with the trained model in the inference phase. the general overview of our method is visualized in figure 1 . ## conclusion in this paper, we show empirical evidence that the manifold assumption indeed holds for the textual domain. based on this observation, we propose a novel method that projects input embeddings onto the approximated natural embedding manifold before classification to defend against adversarial examples. extensive experiments show that our method consistently outperforms previous defenses by a large margin under various adversarial settings. future research on developing better manifold approximation and on-manifold projection methods are interesting directions to further improve the robustness of this type of defense. we hope that the findings in this work can enable broader impacts on improving nlp robustness.