Random Smooth-based Certified Defense against Text Adversarial Attack (2024)

Abstract: Certified defense methods have identified their effectiveness against textual adversarial examples, which train models on the worst-case text generated by substituting words in original texts with synonyms. However, due to the discrete word embedding representations, the large search space hinders the robust training efficiency, resulting in significant time consumption. To overcome this challenge, motivated by the observation that synonym embedding has a small distance, we propose to treat the word substitution as a continuous perturbation on the word embedding representation. The proposed method Text-RS applies random smooth techniques to approximate the word substitution operation, offering a computationally efficient solution that outperforms conventional discrete methods and improves the robustness in training. The evaluation results demonstrate its effectiveness in defending against multiple textual adversarial attacks.

## introduction language models are powerful tools for natural language processing; however, they have been found to be vulnerable to textual adversarial examples @xcite , which are carefully crafted through human-imperceptible changes. these textual adversarial examples pose a significant threat to real-world applications, such as text classification @xcite ), text translation @xcite , question answering @xcite , text-driven image generation @xcite , etc. textual adversarial attacks can be categorized into three types, namely character-level perturbation @xcite , word-level substitution @xcite @xcite , and sentencelevel rephrasing @xcite . among these, fig. 1 : by adding continuous permutations on word embeddings, our method maps one word to both real words and virtual words, which potentially broadens the optimized region and improves the training efficiency. various defense approaches have been proposed to mitigate the impact of word-level text perturbations, such as input transformations @xcite , adversarial training @xcite , and certified defense @xcite . for example, @xcite insert a synonym encoder before the input layer to eliminate adversarial substitutions by mapping various synonyms into the same tokens. adversarial training methods train models on adversarial examples to improve robustness @xcite @xcite . certified defense methods provide a provable defense radius that theoretically blocks all adversarial examples within that radius @xcite . among these defense methods, certified defense methods achieve a strong defense performance with a theoretical robustness guarantee. however, it is time-consuming because of the construction of a word substitutionbased candidate set for the worst-case optimization for training. the aforementioned defense methods, especially certified defense methods, mainly perform word substitution in the discrete token space, which has an enormous search space and usually results in low efficiency during optimization due to the enumeration and substitution operations for each word. however, for modern language models, input tokens are commonly projected into continuous word embeddings before being fed into subsequent neural networks. the l 2 distance between synonyms in the embedding space approximately follows a compact exponential distribution (sec. 2.2). this observation naturally motivates us to continuously treat text manipulation and design efficient adversarial defense techniques. in this work, we propose manipulating texts in the continuous embedding space to approximate the word substitution operation for certified defense. fig. 1 shows an intuitive example of our approach. for the word "adversarial", conventional methods that operate on the word level would map the "adversarial" to the real word "adverse" as an adversarial example, while our method can map the "adversarial" to both real and virtual words by adding permutations on embedding representations. besides, such a continuous assumption allows us to perturb multiple words in parallel, which significantly broadens the optimized region for compact text representation and improves the training efficiency for certified defense. on top of continuous perturbation, we further propose a random smooth-based certified adversarial defense framework text-rs. we integrate the continuous perturbation for word substitution into the certified defense, thus achieving smooth text representation for better model robustness against the text adversarial attack. extensive results of experiments on popular datasets using different models demonstrate the effectiveness of our method against advanced adversarial text attacks. ## conclusion in our work, motivated by the compact exponential distribution of word embedding space, we propose approximating the discrete word substitution operation as a continuous perturbation on the word embedding representation, thus achieving efficient certified defense training. numeric results demonstrate the effectiveness of our proposed method.