Rethinking Perturbations in Encoder-Decoders for Fast Training (2021)

Abstract: We often use perturbations to regularize neural models. For neural encoder-decoders, previous studies applied the scheduled sampling (Bengio et al., 2015) and adversarial perturbations (Sato et al., 2019) as perturbations but these methods require considerable computational time. Thus, this study addresses the question of whether these approaches are efficient enough for training time. We compare several perturbations in sequence-to-sequence problems with respect to computational time. Experimental results show that the simple techniques such as word dropout (Gal and Ghahramani, 2016) and random replacement of input tokens achieve comparable (or better) scores to the recently proposed perturbations, even though these simple methods are faster.

## introduction recent advances in neural encoder-decoders have driven tremendous success for sequenceto-sequence problems including machine translation @xcite , summarization @xcite , and grammatical error correction (gec) @xcite . since neural models can be too powerful, previous studies have proposed various regularization methods to avoid over-fitting. to regularize neural models, we often apply a perturbation @xcite , which is a small difference from a correct input. during the training process, we force the model to output the correct labels for both perturbed inputs and unmodified inputs. in sequenceto-sequence problems, existing studies regard the following as perturbed inputs: (1) sequences containing tokens replaced from correct ones @xcite , (2) embeddings injected small differences @xcite . for example, @xcite proposed the scheduled sampling that samples a token from the output probability distribution of a decoder and uses it as a perturbed input for the decoder. @xcite applied an adversarial perturbation, which significantly increases the loss value of a model, to the embedding spaces of neural encoder-decoders. those studies reported that their methods are effective to construct robust encoder-decoders. however, their methods are much slower than the training without using such perturbations because they require at least one forward computation to obtain the perturbation. in fact, we need to run the decoder the same times as the required number of perturbations in the scheduled sampling @xcite . for adversarial perturbations @xcite , we have to compute the backpropagation in addition to forward computation because we use gradients to obtain perturbations. those properties seriously affect the training budget. for example, it costs approximately 1,800 usd for each run when we train transformer (big) with adversarial perturbations @xcite on the widely used wmt english-german training set in aws ec2 foot_0 . most studies conduct multiple runs for the hyper-parameter search and/or model ensemble to achieve better performance @xcite , which incurs a tremendous amount of training budget for using such perturbations. @xcite and @xcite indicated that recent neural approaches increase computational costs substantially, and they encouraged exploring a cost-efficient method. for instance, @xcite explored a training strategy to obtain the best model in a given training time. however, previous studies have paid little attention to the costs of computing perturbations. thus, we rethink a time efficient perturbation method. in other words, we address the question whether perturbations proposed by recent studies as effective methods are time efficient. we compare several perturbation methods for neural encoderdecoders in terms of computational time. we introduce light computation methods such as word dropout @xcite and using randomly sampled tokens as perturbed inputs. these methods are sometimes regarded as baseline methods @xcite , but experiments on translation datasets indicate that these simple methods surprisingly achieve comparable scores to those of previous effective perturbations @xcite in a shorter training time. moreover, we indicate that these simple methods are also effective for other sequence-to-sequence problems: gec and summarization. ## definition of encoder-decoder in this paper, we address sequence-to-sequence problems such as machine translation with neural encoder-decoders, and herein we provide a definition of encoder-decoders. in sequence-to-sequence problems, neural encoder-decoders generate a sequence corresponding to an input sequence. let x 1:i and y 1:j be input and output token sequences whose lengths are i and j, respectively: x 1:@xmath0, ..., x i and y 1:@xmath1, ..., y j . neural encoder-decoders compute the following conditional probability: in the training phase, we optimize the parameters θ to minimize the negative log-likelihood in the training data. let d be the training data consisting of a set of pairs of x n and this section briefly describes perturbations used in this study. this study focuses on three types of perturbations: word replacement, word dropout, and adversarial perturbations. figure 1 shows perturbations used in this study. as shown in this figure, we can use all types of perturbations in the figure 1 : overview of perturbations used in this study. we can combine perturbations as shown in this figure because each type of perturbation is orthogonal. ## experiments on machine translation to obtain findings on sequence-to-sequence problems, we conduct experiments on various situations: different numbers of training data and multiple tasks. we mainly focus on translation datasets because machine translation is a typical sequenceto-sequence problem. we regard the widely used wmt english-german dataset as a standard setting. in addition, we vary the number of training data in machine translation: high resource in section 4.2 and low resource in section 4.3. on other sequence-to-sequence problems: grammatical error correction (gec) in section 5 and summarization in appendix a to confirm whether the findings from machine translation are applicable to other tasks. ## conclusion we compared perturbations for neural encoderdecoders in view of computational time. experimental results show that simple techniques such as word dropout @xcite and random replacement of input tokens achieved comparable scores to sophisticated perturbations: the scheduled sampling @xcite and adversarial perturbations @xcite , even though those simple methods are faster. in the low resource setting in machine translation, adversarial perturbations achieved high bleu score but those simple methods also achieved comparable scores to the adversarial perturbations when we spent almost the same time for training. for the robustness of trained models, rep(sim) is superior to others. this study indicates that simple methods are sufficiently effective, and thus, we encourage using such simple perturbations as a first step. in addition, we hope for researchers of perturbations to use the simple perturbations as baselines to make the usefulness of their proposed method clear.