M ocking BERT : A Method for Retroactively Adding Resilience to NLP Models (2022)

Abstract: Protecting NLP models against misspellings whether accidental or adversarial has been the object of research interest for the past few years. Existing remediations have typically either compromised accuracy or required full model re-training with each new class of attacks. We propose a novel method of retroactively adding resilience to misspellings to transformer-based NLP models. This robustness can be achieved without the need for re-training of the original NLP model and with only a minimal loss of language understanding performance on inputs without misspellings. Additionally we propose a new efficient approximate method of generating adversarial misspellings, which significantly reduces the cost needed to evaluate a model’s resilience to adversarial attacks.

## introduction while artificial neural networks have been able to achieve human level performance on many realworld tasks, they sometimes fail in surprising ways. @xcite showed that state of the art computer vision models can be fooled into misclassifying objects with only limited perturbations imperceptible to human viewers. along similar lines, it was shown in @xcite ) that very constrained attacks can successfully trick classification algorithms into making incorrect predictions. in fact such attacks have been used for a long time, chiefly for evading spam classifiers while remaining legible to human readers. protecting against such misspellings, whether accidental or intentional, has been a focus of research in the nlp field for many years @xcite . recently, defenses suggested by (pruthi * mockingbert is a reference to mockingbirds, a group of birds known for mimicking the sounds of other animals. the code for reproducing our results as well as instructions for obtaining the trained models are available at https:// github.com/akash13singh/resilient_nlp/ . @xcite and @xcite can partially remediate adversarial attacks by adding a pre-processing step, at the cost of a drop in classification performance. @xcite proposed replacing a fixed word embeddings with trained character-based ones and observed improved resilience to adversarial attacks. in existing systems a tension exists between modularity and accuracy. @xcite and @xcite propose fully modular systems that are completely oblivious of the downstream language understanding model. this provides explainability (by providing a verbatim sequence of corrected tokens) but comes at a cost of reduced accuracy on unperturbed inputs. additionally there is an added drawback of not being able to preserve potential ambiguity present in the input, making these systems 'destructive'. conversely, @xcite is able to represent ambiguous inputs, however at the cost of losing modularity. our central hypothesis is that original accuracy can be preserved while at the same time ensuring modularity. in particular we show that existing classifiers based on bert and roberta, two widely used pre-trained models, can be retroactively made resilient to perturbations even if only unperturbed data was used during the initial finetuning. this can be done by replacing the heuristic subword tokenizer and token embedding with a machine learned replacement which we call mockingbert. the mockingbert tokenizer and embedder learns to mimic a transformer model's tokenization and layer 0 embedding mechanism while providing resilience to input perturbations. we evaluate the performance of such models when trained on both unperturbed and perturbed training sets to understand their suitability for data augmentation. we perform a comparative analysis with the methods proposed in @xcite , as well as with a regular finetuned bert model trained with data augmentation. we also propose and evaluate wordscoreat-tack, an efficient and effective method for generating adversarial samples without the need for exhaustively considering all possible perturbations. wordscoreattack works by carefully choosing information bearing words in the input text. this provides a much faster alternative to the exhaustive method proposed by @xcite at the cost of perturbing a larger number of characters in the sentence. crucially this method requires significantly fewer calls to the underlying nlp model, which more closely approximates real world scenarios that are likely to involve rate limiting and/or a limited query budget. we manually verify that the outputs of this perturbation can still be classified correctly by a human reader with a high probability. our central findings are that the mockingbert tokenizer and embedder model paired with a finetuned bert or roberta classifier achieves a high level of resilience to character-level adversarial perturbations when pre-trained on perturbed data. this results in a higher accuracy on perturbed inputs on multiple well known datasets than state of the art methods for combating adversarial misspellings such as the one described in @xcite . crucially, the impact on accuracy for unperturbed inputs, while measurable, is noticeably lower than for comparable methods for protecting against adversarial attacks. additionally, wordscoreattack significantly reduces the number of model queries required to find adversarial inputs. this dramatically speeds up evaluation, while also making the attack more practical in real scenarios. ## related work previous research has explored both adversarial attacks against nlp systems as well as possible defenses. this section gives a brief overview of related work and concepts. @xcite proposes an attack that exhaustively searches for a modification of up to two characters with the intent of causing a wrong prediction. the allowed modifications are from a narrow set: adding or deleting a single internal character, swapping two neighboring internal characters or replacing an internal character with one of its neighbors in the qwerty keyboard layout. this choice of perturbations was based on linguistic re-search which suggests that modifications to internal characters in a word do not significantly hinder legibility for a human reader @xcite . ## data for training the mockingbert tokenizer and embedder we use the unlabeled bookcorpus dataset @xcite , with minimal pre-processing to reverse the existing tokenization present in that dataset. the goal is to utilize textual data that is as close as possible to unprocessed text. for evaluation purposes we use the large movie review (imdb) dataset @xcite , the stanford sentiment treebank (sst) dataset @xcite and the large yelp review dataset @xcite . for the latter two datasets we use both the 2-class (binary) and 5class variants. we subsequently refer to these datasets as imdb, sst-2, sst-5, yelp-2 and yelp-5. for each task we evaluate each model's accuracy on a randomly chosen subset of 500 sentences from each dataset's test set. the reason for the limited size of the test set is that adversarial attacks require potentially hundreds of model inferences for each sentence to find a successful perturbation. ## models we propose a model, mockingbert, that can be used in place of a transformer model's tokenizer and word embedding. the model consists of a character embedding layer that transforms each input character into a numeric vector of size 768. characters are converted to lower case before embedding, but no other preprocessing is done (e.g. no special handling for whitespace or punctuation). the layer is followed by three stacked bidirectional lstm layers, with a hidden size of 768. the final lstm layer is connected to two parallel dense layers: the subword boundary detection layer and the subword embedding layer (figure 1 ). the subword boundary detection layer uses a sigmoid activation function and has an output dimension of 1. the subword embedding layer uses hyperbolic tangent as its activation function and has an output dimension of 768, matching the subword embedding size of both bert base and roberta base. the model is trained with two objectives. the first objective is a character-level sequence classification task. the subword boundary detection layer is used as a classifier. for every input character it predicts whether the character is a subword boundary, i.e. the last character of a subword. the second objective is a regression task where the subword embedding layer outputs the embedding for each subword. the embeddings are only used for input characters deemed to be subword boundaries. for non-boundary characters the embeddings are discarded by using a mask. the tokenizer and embedder model is trained on sentences from bookcorpus @xcite . during training an unperturbed sentence is processed by the transformer model's regular tokenizer and its context free word embedding (trained layer 0 embeddings without the position and segmentation embeddings). this gives us the subword boundaries and embeddings which are used as labels to train the model. the sentence may subsequently be perturbed by adding or deleting a character, swapping two neighboring characters, inserting whitespace in the middle of long words, or removing whitespace between two words. the subword boundaries are updated accordingly. in the case of character deletion, special handling is present for characters that are subword boundaries. in this case the immediately preceding character is marked as a subword boundary. in the uncommon case when the preceding character is already a subword boundary, the embedding for the deleted character is simply discarded. in our opinion this situation is sufficiently rare that it does not warrant adding more complexity to the models. when training, we can choose which contextfree subword embedding mockingbert will approximate: in our experiments we evaluate both of these approaches to understand the tradeoffs. the models are trained on 64000 randomly selected sentences from bookcorpus for 5 epochs. the loss function is a combination of the mean squared error (mse) losses for the subword boundary detection task and for the embedding prediction task. mse was chosen since it can be applied both for subword boundary detection (classification) and embedding prediction (regression). the loss values for the two components are scaled to have the same magnitude. this is to prevent one of the subtasks from dominating the other one during training. when training on perturbed data and targeting generic embeddings for bert and roberta, the models achieved accuracies for the subword boundary detection task of 99.30% and 99.25% respectively on a held-out evaluation set. during evaluation and inference, the input text is converted to lower case and the characters are embedded as a 768-dimensional vector representation. subsequently the mockingbert model is executed to obtain subword embeddings for the input sequence. the embeddings are then bookended by the fixed representations of the transformer's special tokens, [cls] and @xcite in the case of bert @xmath0. finally, the sequences of embeddings are passed to the finetuned transformer model. the transformer models are based on the pretrained bert base and roberta base models and finetuned using the huggingface transformers package @xcite . for finetuning we attach a linear layer on top of the [cls] (for bert) @xmath1, and train the entire model using cross-entropy loss. we restrict the sequence length to 128 tokens, and the model is trained with a batchsize of 32, learning rate of 2e-05, for up to 5 epochs. we used an early stopping patience of 10, evaluated every 100 training steps. ## proposed attacks we propose wordscoreattack , a cost effective way to generate adversarial attacks which can occur in an real-world setting. typically trained models are hosted and exposed through an api and the users including adversaries can only query the models by sending input and receiving output. thus we only focus on black-box attacks, where one does not have access to the trained model. wordscoreattack intelligently selects input words to perturb in order to maximize the chances of finding an adversarial example with the minimum number of perturbations. this is achieved by computing corpus specific word scores which are based on per-word conditional class probabilities for the corpus. for a binary classification task we compute the log likelihood of each word as shown in equations 1 to 3, where f req pos , f req neg are the frequencies of the word in positive and negative classes respectively. n pos , n neg are the total words in the positive and negative corpus and v is the total vocabulary size. we remove stop words and low frequency words before computing word scores. in the case of multi-class classification, for each class a separate score is computed by considering that class as positive and combining all other classes into the negative class. given an input text and the original predicted class, wordscoreattack targets words in the input text which have the highest scores for the given class and perturbs them in the decreasing order of scores, until the model prediction is flipped. our hypothesis here is that the words with the highest scores are critical to the model's predictions and perturbing them can fool the model to make the wrong classification. we impose a query budget setting expressed by two parameters: (max_words_to_perturb, max_tries_per_word). the first one denotes the maximum number of words that can be perturbed for each input text. the second parameter denotes the number of perturbation attempts allowed per word. when the max_tries_per_word for a word are exhausted and do not yield a successful attack, the attack greedily preserves the perturbation that decreases the model confidence the most (i.e. the one with the lowest score for the original predicted class) and moves to the next word in the order of word scores. the maximum number of model queries per input text are max_words_to_perturb * max_tries_per_word. in order to mimic real world misspellings, we only allow one perturbation per input word. the perturbations considered are adding or deleting a character, swapping of adjacent characters, as well as splitting of a word (adding whitespace) and merging of adjacent words (deleting whitespace). furthermore for the non-whitespace perturbations, only the internal letters of a word are perturbed, and the first and last letters remain unmodified. this ensures the perturbed text can be comprehended by humans @xcite . though we allow one perturbation per word, multiple words per input text can be perturbed until a successful attack is found. this is in contrast to @xcite , where the authors do an exhaustive search to find a single character perturbation that flips the model prediction. the design goal for wordscoreattack is to find an adversarial perturbation with a significantly lower query budget compared to an exhaustive search. this is done with the objective of constructing a practical framework for both simulating adversarial attacks to analyse model resilience as well as for constructing adversarial samples for data augmentation and adversarial training. ## experiments we evaluated each model's performance on an unperturbed version of the test sets, as well as on wordscoreattack. we also evaluated the finetuned bert model on the imdb task using the exhaustive adversarial attack as described in @xcite . textattack @xcite , a python framework for adversarial attacks, was used to execute the exhaustive attacks. due to the prohibitive computational cost of this attack we were not able to evaluate it on the other models. we evaluated variations both including and excluding whitespace modifications. this is because the remediations proposed in @xcite have not been specifically designed for combating such modifications. to establish baselines, we evaluated finetuned bert and roberta models with their default tokenizers, both with and without data augmentation. we also evaluated the conncomp and aggclust approaches proposed in @xcite . accuracy is used as the primary evaluation metric. a second stream of experiments is focused on evaluating the efficacy and efficiency of wordscoreattack. we attack the bert model finetuned for the imdb task and vary the max_tokens_to_perturb from 1 to 40 and max_tries_per_token from 1 to 4. for each setting, we calculate the model accuracy on the 500 reviews in the test set. the results are shown in figure 2 . the original accuracy on the test set is 88%. which is reduced to 26.6% in the most adversarial setting of @xcite . in order to compare wordscoreattack to the exhaustive adversarial attack of @xcite , we evaluate a forgetful mode for word-scoreattack, where an unsuccessful perturbation is reset when the attack moves to a new token. in this mode, the attack tries to flip the model's prediction by making only one perturbation to the input text. the results are shown in ## analysis pruthi wsa-forgetful wsa orig. accuracy 89.6% 88% 88% attack success % 25.45% 18% 70% final accuracy 66.8% 72.6% 26.6% avg. queries 1574 133 83 crucially, our model's accuracy on unperturbed data is typically only slightly lower than when using the standard bert or roberta tokenization/embedding procedure, with the accuracy scores ranging from being 0.2% better to being 4.6% worse than the standard model. this is in contrast with roben, where the accuracy on unperturbed data is lower by between 4.8% and 17.4%. interestingly the version of the tokenizer that targets default pre-trained transformer embeddings performed slighly better overall than the version targeting embeddings finetuned specifically for individual tasks. this suggests that a universal tokenizer and embedder model can be used for a variety of tasks, with no need to adapt it specifically for each task. this hypothesis requires more research on a wider variety of tasks. through our analysis of the wordscoreat-tack, we demonstrated that it is a cost effective way of constructing adversarial examples. it provides a flexible framework for evaluating models under attack. as shown in figure 2 model accuracy drops steadily as the max_tokens_to_perturb is increased. accuracy declines faster with higher values of max_tries_per_token as more random perturbations can be tried per token. the cost effectiveness of the attack is borne by the fact that the most adversarial budget setting of (40,4) requires an average of 83 model queries and reduces the accuracy to 26.6%. on the other the setting of (20,1) will require at most 20 queries on average but still reduces the model accuracy to 60%. we wanted to construct adversarial samples such that humans can infer the original intent without much difficulty. thus we considered a limited set of perturbations with constraints such as perturbing only one internal character per token, in addition to adding/deleting whitespace. two examples of original inputs and their adversarial counterparts are given in ## future work in our opinion it would be interesting to see if an approach similar to mockingbert would work with other practical transformer-based models such as t5, xlnet or electra. due to differences in how their tokenizers work some adaptations might be necessary. nevertheless we see no fundamental issue that would prevent this approach from being applicable for these other model architectures. another potentially interesting direction would be to evaluate whether character-based transformer models such as canine @xcite , char-former @xcite or byt5 @xcite are better suited to data augmentation with character-level perturbations than subword based transformer models. intuitively subword based models are not well equipped for handling misspellings due to the fact that misspelled words might end up being mapped to unrelated tokens and that the number of possible misspellings for each token is very large. it should be noted that mocking-bert has a very clean separation between the tokenization/embedding procedure (which provides resilience to misspellings) and the main transformerbased language understanding layers. this means that it is easy to swap out the tokenizer and embedder if a new attack is devised, which may not be trivial for purely character-based transformer models. ## conclusion we have demonstrated that our proposed mocking-bert embedder is able to successfully mimic the operation of a traditional tokenizer and embedder as used in the bert and roberta models, with only a modest decrease in performance on classification tasks. in the presence of input perturbations, mockingbert outperforms both a data augmented bert model and the state of the art roben procedure. furthermore we have provided evidence that a universal embedder can achieve similar results to one that is specifically trained for a particular finetuned embedding, suggesting that embeddings might not need to be trained with specific tasks in mind. we have also proposed an efficient and effective method for constructing adversarial attacks, wordscoreattack, which allows constructing such attacks at a fraction of the cost of an exhaustive search, at the expense of possibly perturbing more words within a sentence in order to achieve a similar attack success rate. training of the mockingbert models described in this article took slightly over 20 hours of compute time on nvidia tesla k80 and nvidia geforce gtx 1070ti gpus. evaluation of the models on wordscoreattack took 25 hours, with an additional 50 hours of gpu time spent during development.