Multi-Layer Random Perturbation Training for improving Model Generalization Efficiently (2021)

Abstract: We propose a simple yet effective Multi-Layer RAndom Perturbation Training algorithm (RAPT) to enhance model robustness and generalization. The key idea is to apply randomly sampled noise to each input to generate label-preserving artificial input points. To encourage the model to generate more diverse examples, the noise is added to a combination of the model layers. Then, our model regularizes the posterior difference between clean and noisy inputs. We apply RAPT towards robust and efficient BERT training, and conduct comprehensive fine-tuning experiments on GLUE tasks. Our results show that RAPT outperforms the standard fine-tuning approach, and adversarial training method, yet with 22% less training time.

## introduction although deep learning models have been very successful in various kinds of nlp problems, they are known to be sensitive towards input data distribution change which are pervasive across language tasks. motivated by this, a recent line of work investigate the adversarial training technique to enhance the model robustness. adversarial training has proven effective in improving model generalization and robustness in computer vision @xcite and natural language processing (nlp) @xcite @xcite @xcite @xcite . it works by augmenting the input with a small perturbation to steer the current model prediction away from the correct label, thus forcing subsequent training to make the model more robust and generalizable. in nlp, the cutting-edge research in adversarial training tends to perform an inner search for the most adversarial direction using gradient steps @xcite @xcite ; pereira et @xcite @xcite . this causes a significant overhead in training time. moreover, such methods tend to add the adversarial perturbation only to the embedding layer, which might not be optimal. by contrast, in this paper, we investigate a simpler direction by using only randomly sampled noise to generate label-preserving artificial input points. we thus propose a simple yet effective random perturbation training algorithm (rapt) for enhancing model robustness and generalization. for each instance, instead of using gradient steps to generate adversarial examples, rapt adds randomly sampled noise to the hidden representaions of a randomly chosen layer, among multiple intermediate transformer layers (i.e. bert layers). we hypothesize this might encourage the model to generate more diverse examples, and improve model generalization capability. our model then regularizes the model posterior difference between clean and noisy inputs. on the overall glue benchmark, rapt outperforms the standard fine-tuning approach, and matches or improves the performance of strong adversarial training methods such as smart @xcite , yet with a significantly reduced training time. figure 1 shows the accuracy gain and training time drop of rapt compared to smart on the mnli (matched) development dataset. ## rapt in this paper, we focus on fine-tuning bert models @xcite , as this approach has proven very effective for a wide range of nlp tasks. the standard training algorithm seeks to learn a function f (x; θ) : @xmath0, where c is the class label set. given a training dataset d of input-output pairs (x, y) and the loss function l(., .) (e.g. cross entropy), the standard training objective would minimize the empirical risk: by contrast, in adversarial training, as pioneered in computer vision @xcite @xcite , the input would be augmented with a small perturbation that maximize the adversarial loss: recently, adversarial training has been successfully applied to nlp as well @xcite @xcite . in particular, smart @xcite regularizes the standard training objective using virtual adversarial training @xcite , by performing an inner loop to search for the most adversarial direction: effectively, the adversarial term encourages smoothness in the input neighborhood, and α is a hyperparameter that controls the trade-off between standard errors and adversarial errors. current adversarial methods for nlp are slower than standard training, due to the inner maximization. smart, for instance, requires an additional k projected gradient steps to find the perturbation that maximizes the adversarial loss (violation algorithm 1 rapt input: n : the number of training epochs, d, {(x1, y1), ..., (xn, yn)}: the dataset, x : the minibatch of the dataset, f (x; θ): the machine learning model parametrized by θ, σ 2 : the variance of the random noise δ, η: the number to controll the size of the noise, l: the number of transformer based model's layers, f layer : the function that computes the hidden representations of a given layer, h: the hidden representations of a layer of the model, δr: the noise added to the hidden states of layer r , τ : the global learning rate, α: the hyperparameter for balancing the standard loss and the regularization term, max_layer: the number of the maximum layer where the noise can be added during training. 1: for @xmath1, 2, .., n do 2: generate a random integer @xmath2, ...,max_layer} 4: rapt does not require extra backward computations and empirically works as well as or better than smart. we consider the posterior regularization using the kl-divergence. for all tasks in this work, an input text sequence is divided into subword units w t , @xmath3, . . . , t . the tokenized input sequence is then transformed into embeddings, x 1 , . . . , x @xmath4, through a token encoder, which combines a token embedding, a (token) position embedding and a segment embedding (i.e. which text span the token belongs to) by elementwise summation. the embedding layer is used as the input to multiple transformer layers @xcite to generate the contextual representations, h layer 1 , . . . , h layer @xmath5, which are the hidden states of an intermediate layer of the bert model. in rapt, we sample noise vectors δ 1 , . . . , δ t from n (0, σ 2 i), with mean 0 and variation of σ 2 . we first set a maximum layer (among all bert intermediate layers) where the noise vector can be added. in each epoch, for each mini-batch selected, a layer among the first layer and the maximum layer previously set is randomly chosen. the noise input is then constructed by adding the noise vector δ r (equation 2 ) to the hidden state vector of the ramdomly chosen layer (h layer ). specifically, the model first performs a forward pass up to the chosen layer, then the noise vector is added to its hidden states, i.e. h layer the model is then updated according to the taskspecific objective for the task. to preserve the semantics, we constrain the noise to be small, and assume the model's prediction should not change after adding the perturbation. the algorithm of rapt is shown in algorithm 1. ## analysis here, we take a closer look at the embeddings and hidden representations of the intermediate layers of bert using standard fine-tuning (standard), smart, and rapt. after training, we extract the embeddings (for standard and smart) and the intermediate layer representations (where noise has been added) of the second intermediate layer (for rapt) of a sentence. then, we compute the top most similar words for each word in the sentence. we use the cosine similarity for computing the similarity between the vectors. as shown in although having more diversity, the hidden representations are kept meaningful. to further illustrate the diversity rapt introduces to the model's hidden representations, regarding which layer combination setting is best for adding the noise, we found that adding the noise to the layers 1 to 3 worked best in our experiments, as shown in figure 2 and figure 3 , for the mnli and mrpc datasets. figure 2 : performance on the mnli development set as we change the layer combination to add the noise. on the x-axis, @xmath6. all the other values denote that, for each mini-batch, a layer among the layer 1 up to this layer value is randomly chosen. the model is then updated according to the task-specific objective for the task. figure 3 : performance on the mrpc development set as we change the layer combination to add the noise. on the x-axis, @xmath7. all the other values denote that, for each mini-batch, a layer among the layer 1 up to this layer value is randomly chosen. the model is then updated according to the task-specific objective for the task. ## conclusion we proposed rapt, a simple and efficient random perturbation training algorithm for fine-tuning large scale pre-trained language models. our experiments demonstrated that it achieves competitive results on glue tasks, without relying on any additional resource other than the target task dataset. moreover, our model can significanlty reduce the training time compared to adversarial training. rapt is model-agnostic, and can also be generalized to solve other downstream tasks as well, and we will explore these directions as future work.