Similarizing the Influence of Words with Contrastive Learning to Defend Word-level Adversarial Text Attack (2023)

Abstract: Neural language models are vulnerable to word-level adversarial text attacks, which generate adversarial examples by directly substituting discrete input words. Previous search methods for word-level attacks assume that the information in the important words is more influential on prediction than unimportant words. In this paper, motivated by this assumption, we propose a self-supervised regularization method for Similarizing the Influence of Words with Contrastive Learning (SIWCon) that encourages the model to learn sentence representations in which words of varying importance have a more uniform influence on prediction. Experiments show that SIWCon is compatible with various training methods and effectively improves model robustness against various unforeseen adversarial attacks. The effectiveness of SIWCon is also intuitively shown through qualitative analysis and visualization of the loss landscape, sentence representation, and changes in model confidence.

## introduction neural language models have achieved impressive performance in various natural language processing (nlp) tasks, but they are also proven vulnerable to adversarial examples, which induce incorrect model output by adding small perturbations to natural inputs @xcite . unlike attacks on images, which are performed by directly adding imperceptible continuous noise to the input, adversarial text attacks are commonly performed by substituting input text due to the discrete and non-differentiable nature of text @xcite @xcite @xcite . among the various granularities of adversarial text attacks, word-level attacks have been more focused on by recent works for their effectiveness in maintaining semantic similarity and grammatical * corresponding author. the normally trained model considers the information in important words to have a significant impact on prediction, and as a result, search methods that prioritize substituting important words are more likely to find adversarial examples. siwcon, on the other hand, considers the information in both important and unimportant words to have a similar degree of influence on prediction, thus making it less possible for search methods that focus on important words to find optimal substitutions. previous search methods for word-level attacks are based on the assumption: different words in a sentence contribute differently to model prediction, with the information in important words being more influential than the information in unimportant words. therefore, following the word importance scores obtained through attribution methods, the attack can be seen as a process of iteratively substituting words in a sentence, with important words substituted first, followed by unimportant words. for example, the search methods word importance ranking (wir) @xcite @xcite and pwws @xcite obtain word importance using occlusion @xcite , then wir performs substitution in descending order of word importance and pwws formulates token scores that use word importance as weights to guide the attack. following this assumption, the success of wordlevel attacks can be explained. the words in a sentence can be classified as important words, which contain more influential information for prediction, or unimportant words, which contain less influential information. search methods that substitute important words first can perturb more influential information in each attack step, making the model more likely to be deceived. therefore, it is natural to wonder: will the model be more robust when the information in both important and unimportant words has a similar degree of influence on prediction? motivated by this question, we propose a selfsupervised regularization method for similarizing the influence of words with contrastive learning (siwcon) that improves the model robustness against word-level attacks. the motivation of our method is illustrated in figure 1 . we summarize our main contributions as follows: while some works attempt to detect or transform potential adversarial examples before the model @xcite , this does not actually improve the model's robustness. other methods, such as performing certifiably robust training through interval bound propagation (ibp), can be computationally costly and difficult to scale to large models like bert @xcite . additionally, it has been reported that while ibp improves adversarial accuracy, it comes at the huge cost of reduced clean accuracy @xcite . some works try to perform adversarial training by incorporating adversarial examples in the training set @xcite , while this method can only improve the robustness against the adversarial perturbations that the model has seen. moreover, generating adversarial examples is time-consuming, thus adversarial training is difficult to scale to a large dataset. in this paper, based on the ignored assumption, we discuss the model robustness from new perspectives, focusing on attribution and sentence representation. contrastive learning. contrastive learning is first proposed in computer vision tasks to help models learn better image representation @xcite @xcite . this self-supervised learning method alleviates the dependence on the costly labeled data. recently, encouraged by the superior performance, various contrastive learning methods have been proposed for nlp tasks. following the discrete nature of text, some previous works construct the pair examples by augmenting the input sentence @xcite @xcite @xcite , e.g., by word deleting, reordering, substituting, and back-translating, or by augmenting the word embedding @xcite , e.g., by shuffling, cutting off, dropping out the embedding matrix. unlike the previous works that aim to improve the downstream performance, we focus on improving the model robustness. ## conclusion this paper presents siwcon, a self-supervised regularization method based on contrastive learning. siwcon improves the robustness of language models by encouraging the words of different importance to have more similar influence on prediction. experiments show that siwcon effectively improves model robustness without depending on adversarial perturbation. we hope the insights provided in this paper will inspire further research.