Robustness-Aware Word Embedding Improves Certified Robustness to Adversarial Word Substitutions (2023)

Abstract: Natural Language Processing (NLP) models have gained great success on clean texts, but they are known to be vulnerable to adversarial examples typically crafted by synonym substitutions. In this paper, we target to solve this problem and find that word embedding is important to the certified robustness of NLP models. Given the findings, we propose the Embedding Interval Bound Constraint (EIBC) triplet loss to train robustness-aware word embeddings for better certified robustness. We optimize the EIBC triplet loss to reduce distances between synonyms in the embedding space, which is theoretically proven to make the verification boundary tighter. Meanwhile, we enlarge distances among non-synonyms, maintaining the semantic representation of word embeddings. Our method is conceptually simple and componentized. It can be easily combined with IBP training and improves the certified robust accuracy from 76.73% to 84.78% on the IMDB dataset. Experiments demonstrate that our method outperforms various state-of-the-art certified defense baselines and generalizes well to unseen substitutions. The code is available at https://github.com/JHL-HUST/EIBC-IBP/ .

## introduction deep neural networks have achieved impressive performance on many nlp tasks @xcite . however, they are known to be brittle to adversarial examples: the model performance could dramatically drop when applying imperceptible crafted perturbations, especially synonym substitutions, into the input text. these phenomena have been observed in a wide range of practical applications @xcite @xcite @xcite @xcite . to mitigate the vulnerability of nlp models, many adversarial defense methods have been proposed to boost the model robustness from various perspectives, such as adversarial training @xcite @xcite , advanced training strategy @xcite , input transformation @xcite , and robust word embedding @xcite . however, these methods could only provide empirical robustness, i.e., the robust accuracy of these models varies depending on the heuristic search used in the attacks. in contrast, certified robustness guarantees that a model is robust to all adversarial perturbations of a given input, regardless of the attacks for evaluation. certified robustness provides a lower bound on the robust accuracy of a model in the face of various adversarial attacks. in this work, we aim to design better training methods for certified robustness. in particular, our algorithm is mainly based on interval bound propagation (ibp). ibp is initially designed for images @xcite and is also utilized to provide certified robustness in nlp models @xcite . in the first step, we compute the interval of embedding of all possible texts perturbed on the current input by word substitutions, where the embedding layer is fixed using the commonly used word embeddings, such as glove @xcite . then, in the second step, given the pre-computed interval, ibp is used to estimate the upper and lower bounds of the output layer by layer and minimize the worstcase performance to achieve certified robustness. however, previous works of ibp method @xcite use fixed word embeddings and we argue that may not be good enough for certified robustness. as shown in the experiments of @xcite , the embedding space significantly impacts the ibp bounds and the effectiveness of ibp training. though the close neighbor words in the embedding space are selected for the synonym set, the volume of the convex hull constructed by them is still large for ibp training, which will lead to loose bounds through propagating and a poor robustness guarantee. inspired by the above observation, in this work, we develop a new loss to train robustness-aware word embeddings for higher certified robustness. we first decompose certified robust accuracy into robustness and standard accuracy. we optimize for robustness from the perspective of embedding constraint and optimize for standard accuracy by training the model normally. it can be proved that the upper bound of certified robustness can be optimized by reducing the interval of the convex hull constructed by synonyms in the embedding space. therefore, we propose a new loss called embedding interval bound constraint (eibc) triplet loss. specifically, given a word, on each dimension in the embedding space, we aim to reduce the maximum distance between each word and its synonyms, which is actually to make a smaller interval of the convex hull formed by synonyms. then, we freeze the embedding layer after training the word embeddings by eibc triplet loss, and train the model by normal training or ibp training to achieve higher certified robust accuracy. extensive experiments on several benchmark datasets demonstrate that eibc could boost the certified robust accuracy of models. especially when eibc is combined with ibp training, we could achieve sota performance among advanced certified defense methods. for instance, on imdb dataset, eibc combined with ibp training achieves 84.78% certified robust accuracy, surpassing ibp by about 8%, which indicates that constraining the embedding interval bound will significantly boost the performance of ibp. our main contributions are as follows. ## related work there are many adversarial defense methods to boost the model's robustness to adversarial word substitutions. adversarial training (at), one of the most popular defense approaches, crafts adversarial examples during the training and injects them into the training set @xcite @xcite . a stream of work aims to improve the effectiveness and efficiency of textual adversarial training by adversary generation based on gradient optimization @xcite @xcite . to eliminate the differences between clean samples and adversarial examples, wang et al. (2021a) insert a synonym encoder before the input layer, and yang et al. (2022) propose fast triplet metric learning (ftml) to train robust word embeddings. liu et al. ( 2022 ) leverage the flooding training method @xcite to guide the model into a smooth parameter landscape that leads to better adversarial robustness. besides, adversarial detection methods detect the adversarial examples before feeding the input samples to models by training a classifier @xcite or randomized substitution @xcite . however, these methods can only provide empirical robustness, which is unstable for attacks based on different heuristic searches. certified robustness is proposed to guarantee that a model is robust to all adversarial perturbations of any given input. interval bound propagation (ibp) calculates the input interval involving all possible word substitutions and propagates the upper and lower bounds through the network, then minimizes the worst-case loss that any combination of the word substitutions may cause @xcite . randomized smoothing methods, such as safer @xcite and ranmask @xcite , mask a random portion of the words in the input text to construct an ensemble and utilize the statistical properties of the ensemble to predict the output. @xcite propose causal intervention by semantic smoothing (ciss), which associates causal intervention with randomized smoothing in latent semantic space to make provably robust predictions. most previous works do not attach importance to word embeddings concerning certified robustness. our work introduces eibc triplet loss to achieve certified robustness through constraining word embeddings and incorporates it into ibp to boost certified robustness. in the field of adversarial images, @xcite improve the ibp training method by mitigating the issues of exploded bounds at initialization and the imbalance in relu activation states. it is worth noting that our work differs from @xcite . we particularly focus on reducing the difference between the upper and lower bounds of initial inputs by fine-tuning the embeddings. the reduction of bounds interval provably causes the tightening of bounds in following propagation. ## preliminaries for the text classification task, a model f : @xmath0, where @xmath1, x 2 , • • • , x n ⟩ is a sequence consisting of n words, and the output space @xmath2, y 2 , • • • , y c } contains c classes. in this paper, we focus on an adversarial scenario in which any word in the textual input can be arbitrarily replaced by its synonyms so as to change the model's prediction. formally, we use s(x i ) to denote the synonym set of the i th word x i of input x. then, we formulate the set consisting of all the adversarial examples with allowed perturbations of x: we can easily decompose certified robust accuracy into robustness and standard accuracy. robustness cares about whether the model prediction is consistent under perturbations. clearly, achieving robustness is a necessary condition for obtaining models with high certified robust accuracy. we then illustrate the conditions to be satisfied for robustness in terms of interval bound. for a k-layer neural network, assuming we can calculate the interval bound of the output logits z k : z @xmath3, the model with robustness satisfies that the lower bound of the model's largest logit z k ymax is greater than the upper bound of other logits, i.e., to evaluate the model's certified robust accuracy, we just need to replace the model's largest logit z k ymax with the logit of the true class z k ytrue in eq. ( 3 ). interval bound propagation ibp provides the solution to estimate the interval bound layer by layer. we could represent a k-layer neural network model as a series of transformations f k (e.g., linear transformation, relu activation function): to calculate the interval bound of the output logits, we need to construct the interval bound of the input vector and propagate it through the network. let φ(x i ) ∈ r d denote the embedding word vector of word x i with d dimensions. the word vector input is we obtain the interval bounds of the word vector input z 0 by constructing the convex hull of s(x i ) in the embedding space: ## methodology in this section, we first theoretically demonstrate the influence of word embedding on the model robustness and then introduce the proposed eibc triplet loss to optimize the word embedding. finally, we describe how to incorporate eibc into the training process. ## experiments this section evaluates the proposed method with three advanced certified defense methods on three benchmark datasets. in addition, we further study eibc on the generalization to unseen word substitutions, the empirical robustness, the trade-off between clean and robust accuracy, the training procedure, and the robustness with different distance metrics. ## conclusion in this work, we attach importance to word embeddings and prove that the certified robustness can be improved by reducing the interval of the convex hull constructed by synonyms in the embedding space. we introduce a novel loss termed the embedding interval bound constraint (eibc) triplet loss to constrain the convex hull. since eibc merely provides word embeddings with certified robustness, which is componentized, we could incorporate eibc into the normal training or ibp training to boost the certified robust accuracy. experiments on three benchmark datasets show that eibc combined with ibp training achieves much higher certified robust accuracy than various stateof-the-art defense methods. eibc also exhibits good generalization to unseen word substitutions. we will further study how to incorporate eibc with other certified defense methods in future work. moreover, we will apply the proposed method in transformer-based models and extend the research to defend against character-level or sentence-level perturbations. an essential difference between image and text data is that text data is discrete and needs to be transformed into continuous word vectors by word embeddings. tightened bounds of word embeddings benefiting from eibc could boost the certified robustness of ibp, which is a typical example to indicate that word embeddings are vital to the robustness of nlp models. we hope our work could inspire more studies on the robustness of nlp models enhanced by word embeddings.