Consistency Training with Virtual Adversarial Discrete Perturbation (2022)

Abstract: Consistency training regularizes a model by enforcing predictions of original and perturbed inputs to be similar. Previous studies have proposed various augmentation methods for the perturbation but are limited in that they are agnostic to the training model. Thus, the perturbed samples may not aid in regularization due to their ease of classification from the model. In this context, we propose an augmentation method of adding a discrete noise that would incur the highest divergence between predictions. This virtual adversarial discrete noise obtained by replacing a small portion of tokens while keeping original semantics as much as possible efficiently pushes a training model’s decision boundary. Experimental results show that our proposed method outperforms other consistency training baselines with text editing, paraphrasing, or a continuous noise on semi-supervised text classification tasks and a robustness benchmark.

## introduction building a natural language processing (nlp) system often requires an expensive process to collect a massive amount of labeled text data. semisupervised learning (ssl) @xcite mitigates the requirement for such labeled data by exploiting the structure of unlabeled data. among the ssl methods, the consistency training framework @xcite enforces a model to produce similar predictions of original and perturbed inputs. this method has several advantages over other training algorithms such as naively adding augmented samples into the training set @xcite in that it provides a richer training signal than a one-hot label, and also applies to both labeled and unlabeled data @xcite . for perturbing a text while preserving its semantics, some approaches inject continuous noise to embedding vectors @xcite , and others modify text itself in discrete fashion by edit operations @xcite or paraphrasing with back-translation @xcite @xcite . however, adding continuous noise might not strongly regularize the training model, compared to diverse discrete noise-based augmentation methods @xcite . also, the augmentations with discrete noise are mostly black-box approaches based on simple rules or fixed models without access to the training model's internal states, having no control over output augmentations that would aid in the regularization of the training model. as seen in fig. 1 (d ), the 5646 augmented samples with similar semantics but that are outside the training model's decision boundary (i.e. adversarial) are the ones that would effectively regularize the model to fit into the complex real data distribution. to this end, we explore virtual adversarial training with discrete token replacements (vat-d). our framework (1) first perturbs a given input text by replacing a small subset of tokens to maximize the divergence between the original and the perturbed samples' model predictions (i.e., virtual adversarial) while filtering tokens to replace for constraining the semantic similarity, and (2) train a model to minimize the divergence of the predictions of original and perturbed inputs. vat-d shares the advantages of virtual adversarial training (vat) with continuous noise @xcite in that the perturbation is modeldependent, changing over the training time to approximate the augmented samples that would effectively push the decision boundary. on the other hand, vat-d differs from vat in that the search space is discrete rather than continuous, thus not constrained by the pre-defined norm on the embedding space. our method relies on the training model's predictions which do not require label information, hence being the first work to successfully apply the adversarial training with perturbation on discrete space to the ssl framework. our proposed method empirically outperforms previous state-of-the-art methods on topic classification datasets @xcite @xcite under various ssl scenarios. we additionally conduct experiments on anli robustness benchmark dataset @xcite for testing the robustness when only labeled samples are given where the method improves over the roberta-large @xcite ) by 8 points. ## background we explain the concept of consistency training and vat that our framework relies on. ## method we aim to generate a perturbed sample by adding discrete noise that incurs the highest divergence of the model's prediction logits from the original one without significant changes in its semantics. our augmentation is made on-the-fly depending on the current model to push the decision boundary during training effectively.