Semi-supervised Adversarial Text Generation based on S eq2 S eq models (2022)

Abstract: To improve deep learning models’ robustness, adversarial training has been frequently used in computer vision with satisfying results. However, adversarial perturbation on text have turned out to be more challenging due to the discrete nature of text. The generated adversarial text might not sound natural or does not preserve semantics, which is the key for real world applications where text classification is based on semantic meaning. In this paper, we describe a new way for generating adversarial samples by using pseudo-labeled in-domain text data to train a seq2seq model for adversarial generation and combine it with paraphrase detection. We showcase the benefit of our approach for a real-world Natural Language Understanding (NLU) task, which maps a user’s request to an intent. Furthermore, we experiment with gradient-based training for the NLU task and try using token importance scores to guide the adversarial text generation. We show that our approach can generate realistic and relevant adversarial samples compared to other state-of-the-art adversarial training methods. Applying adversarial training using these generated samples helps the NLU model to recover up to 70% of these types of errors and makes the model more robust, especially in the tail distribution in a large scale real world application.

## introduction over the past years, neural machine learning models have become more popular in a wide range of real world natural language applications. while these models have become very accurate, they are susceptible to errors due to small changes in the input, e.g. adding a stop word. this makes it difficult for a natural language understanding system to recognize all utterances correctly since there are many ways to formulate one request. in fact, while common user commands can be understood very well by the system, a system can react differently due to minor input changes, e.g., article variations, paraphrasing, or adding functional words. natural language understanding (nlu) is at the core of voice assistants and maps a user's request (also referred to as utterance later) to a specific intent within a certain domain, i.e. playmusic intent within the music domain. in this paper, we identify the weaknesses of an nlu text classification model by finding the types of inputs that have high probability of causing prediction errors and show how to mitigate them. inspired by recent work in adversarial training and adversarial sample generation @xcite , we describe how we employ adversarial text perturbation to identify and fix such samples to ensure a stable behavior of the model and thereby significantly boost the tail accuracy in a large scale real world application through adversarial training. we show that while common text perturbation methods relying on character manipulation @xcite , word swap @xcite could generate adversarial samples on the original text inputs to trick the model, most of them often generate many irrelevant samples that either do not make sense (e.g., grammatically incorrect, unnatural, out of domain) or change the meaning of the original text utterance. therefore, applying these methods in a real world application without any domain adaptation and additional constraints is not only unsuitable but could even harm the model. in this work, we describe how we use pseudo-labeled in-domain text data for task adaptation and thereby create adversarial perturbations that are more natural and relevant to the nlu task. our approach uses in-domain data to train a seq2seq model that generates an adversarial version of the input utterance. we introduce multiple constraints including a paraphrase detector model to make sure that the generated adversarial text samples are of high quality. finally, we show that including these generated adversarial samples during training helps to improve the model's robustness. we compare the results with adversarial training based on the fast gradient sign method (fgsm) @xcite often used in computer vision. ## related work adversarial sample generation in the text domain can be categorized by the level of the attemptstowards a target sentence. at the lowest level, hotflip @xcite and textbugger @xcite perform character level perturbation by character manipulations (swap, insert or delete). other methods by @xcite , @xcite and @xcite create attempts at word level using synonym swaps or masked language model. the most broad type of attempt is sentence level attempt where multiple words within a sentence are changed, such as paws @xcite , scpn @xcite and sears @xcite . while these methods can sometimes produce very high attempt success rates, they are not straightforward to apply out-of-the-box to real world applications, especially when the domains are specific and small changes in the original inputs can lead to a valid change in the labels (i.e., invalid attempts). in this work, we mainly deal with the problem of generating in-domain valid ad-versarial attempts by using a seq2seq model that learns from live traffic/un-annotated data in real world applications. ## seq2seq-based adversarial text perturbation figure 1 describes our semi-supervised seq2seqbased adversarial text perturbation (ssat) framework which consists of three components: (1) adversarial candidate extraction from unannotated data based on pseudo-labels, (2) a paraphrase detector model that keeps only candidates that are actual paraphrases, and (3) a seq2seq (t5) model that is trained on the data generated in the first two steps to generate adversarial samples for a given input text. ## results and discussion in this section, we present our results on adversarial sample generation and adversarial training on top of the generated samples for the target models. ## conclusions in this paper, we proposed a framework that uses pseudo-labeled data for learning and training an target model class std test set adv test set music 0% 19% bert-base-german-cased books -1% -3% fgsm-adv. trained calendar -1% 6% commercial dataset shopping -1% 15% notification 7% -1% overall -1% 10% music -0% 74% bert-base-german-cased books -1.11% 61% ssat adversarially trained calendar -2.13% 63% commercial dataset shopping -1.04% 73% notification 7.6% 80% overall -1.03% 70% bert-base-multilingual-cased fgsm-adv. trained overall 0% 0% massive dataset bert-base-multilingual-cased ssat adversarially trained overall -1.01% 25.23% massive dataset