mix S eq: A Simple Data Augmentation Methodfor Neural Machine Translation (2021)

Abstract: Data augmentation, which refers to manipulating the inputs (e.g., adding random noise,masking specific parts) to enlarge the dataset,has been widely adopted in machine learning. Most data augmentation techniques operate on a single input, which limits the diversity of the training corpus. In this paper, we propose a simple yet effective data augmentation technique for neural machine translation, mixSeq, which operates on multiple inputs and their corresponding targets. Specifically, we randomly select two input sequences,concatenate them together as a longer input aswell as their corresponding target sequencesas an enlarged target, and train models on theaugmented dataset. Experiments on nine machine translation tasks demonstrate that such asimple method boosts the baselines by a non-trivial margin. Our method can be further combined with single input based data augmentation methods to obtain further improvements.

## introduction data augmentation, which enlarges the training corpus by manipulating the inputs through given rules, has been widely used in machine learning tasks. for image classification, there are various data augmentation methods, including cropping, flipping, rotating,cut-out (devries and taylor, 2017), etc. for natural language processing (briefly, nlp), similar data augmentation methods also exist, like randomly swapping words @xcite , dropping words @xcite , and masking specific words @xcite . with data augmentation, the main content of the input is not affected but the noise is introduced so as to increase the diversity of the training set. the effectiveness of above data augmentation methods has been verified by their strong performance improvements in both image processing and nlp tasks. for example, with the combination of data augmentation and meta-learning, state-of-the-art result of image classification is achieved @xcite . most existing data augmentation methods take one sample from the training set as input, which might limit the scope and diversity of the training corpus. mixup @xcite is a recently proposed data augmentation method, where two samples from the training corpus are leveraged to build a synthetic sample. specifically, let x 1 , x 2 denote two images from the training set, and y 1 , y 2 denote their corresponding labels. the synthetic data (λx 1 + (1 -λ)x 2 , λy 1 + (1 -λ)y 2 ) is introduced to the augmented dataset, where λ is randomly generated. such a strategy is further enhanced in follow-up works @xcite . pair sampling @xcite is another data augmentation method where the synthetic sample is built as (0.5x 1 + 0.5x 2 , y 1 ). in comparison, according to our knowledge, such ideas are not leveraged in nlp tasks (e.g., machine translation). therefore, in this work, we explore along this direction to see whether augmenting data through mixing multiple sentences is helpful. in sequence learning tasks, two inputs x 1 and x 2 might contain different numbers of units (e.g., words or subwords). besides, for sequence generation tasks, their labels y 1 and y 2 are of different lengths. therefore, it is not practical to sum them up directly. instead, we choose to concatenate two inputs and the two labels to get the synthetic data. we find that it is important to use a special token to separate the two sentences in a synthetic data. we name our proposed method as mixseq. normally, mixseq randomly samples the two concatenated sequences. however, if the two concatenated sequences are contextually related, we can enhance our mixseq to a context-aware version: ctxmixseq, which will result in better performance (see ## experiments we conduct experiments on the following machine translation tasks to evaluate our method: iwslt'14 german↔english and spanish↔english; flores english↔nepali and english↔sinhala; and wmt'14 @xmath0. we abbreviate english, german, spanish, nepali and sinhala as en, de, es, ne and si. ## related work most existing data augmentation methods in nmt operate on one single input. @xcite replaced common words with rare words under the guidance of language models to improve the translation of rare words. in unsupervised learning, @xcite proposed to randomly drop, swap, or mask words. @xcite verifies the effectiveness of such methods in supervised nmt. raml @xcite randomly inserted, deleted or substituted words in the target sequence with probability exponentially decreasing with the edit distance. switchout @xcite extended raml by both manipulating on the source side and the target side. @xcite proposed to "softly replace" words by replacing the one-hot representation of words with a distribution on the vocabulary. a concurrent work similar to ours is @xcite , @xmath1. in other fields, data augmentation methods operating on multiple samples have been proposed. mixup @xcite generated a synthetic sample by averaging two inputs and the two labels. it is further applied to semi-supervised learning to enlarge the dataset @xcite . pair sampling @xcite only averaged the two inputs but not the labels. ## conclusion and future work in this work, we proposed a simple yet effective data augmentation method for nmt, which randomly concatenates two training samples to enlarge the datasets. experiments on nine machine translation tasks demonstrate the effectiveness of our method. for future work, there are a few directions to explore. first, we will apply our method to more nlp tasks. second, we will theoretically analyze when and why it works. third, we will study and design more effective data augmentation methods.