Sentence Concatenation Approach to Data Augmentation for Neural Machine Translation (2021)

Abstract: Recently, neural machine translation is widely used for its high translation accuracy, but it is also known to show poor performance at long sentence translation. Besides, this tendency appears prominently for low resource languages. We assume that these problems are caused by long sentences being few in the train data. Therefore, we propose a data augmentation method for handling long sentences. Our method is simple; we only use given parallel corpora as train data and generate long sentences by concatenating two sentences. Based on our experiments, we confirm improvements in long sentence translation by proposed data augmentation despite the simplicity. Moreover, the proposed method improves translation quality more when combined with back-translation.

## introduction neural machine translation (nmt) can be used to achieve high translation quality. however, it has certain drawbacks, such as the degradation in the translation quality for long sentences. @xcite reported that the translation quality of nmt is superior to that of statistical machine translation (smt) for input sentences within a certain length. however, they also stated that when the sentence length exceeds a particular value, the quality of nmt becomes inferior to that of smt, and the greater the sentence length, the lower the translation quality. additionally, they presented the correlation between the size of the training data and the translation quality @xcite . in other words, the less training data we have, the lower will be the accuracy of the translation. this issue is prevalent in low-resource languages. there-fore, various data augmentation methods for lowresource parallel corpora have been studied. for instance, the generation of pseudo data was proposed by back-translating the monolingual corpora or paraphrasing the parallel corpora as additional training data @xcite @xcite . hence, this study proposes a data augmentation method that can be effective in long sentence translations. the proposed method is illustrated in figure 1 . long sentences were obtained by concatenating two sentences at random and adding them to the original data. the translation quality is expected to be improved by this method because the low quality of translation of long sentences was caused by insufficient number of long sentences in the training data, which reduces this concern in the proposed method. this study presents an improved bleu score and higher quality in long sentence translations on english-japanese corpus. moreover, the bleu score further increases by incorporating backtranslation. in addition, human evaluation shows that fluency is increased more than adequacy. in summary, the main contributions of this paper are as follows: ## related works nmt exhibits a significant decrease in the translation quality for very long sentences. @xcite analyzed the correlation between the translation quality and the sentence length by comparing nmt with smt. they showed that the overall quality of nmt is better than that of smt but that smt outperforms nmt on sentences of 60 words and longer. they stated that this degradation in quality was caused by the short length of the translations. additionally, @xcite propose to use the relative position information instead of the absolute position information to mitigate the performance drop of nmt models for long sentences. they conducted an analysis of the translation quality and sentence length on lengthcontrolled english-to-japanese parallel data and showed that the absolute positional information sharply drops the bleu score of the transformer model @xcite in translating sentences that are longer than those in the training data. several data augmentation methods have been proposed for nmt, such as back-translation, which involves translating the target-side monolingual data to create a pseudo dataset @xcite . in their method, the back-translation model is first learned by using parallel corpora from the target-side to the source-side. once converged, this model generates pseudo data by translating the target-side monolingual corpora to the source-side language. a translation model is then trained using both the pseudo-parallel and original-parallel data. @xcite analyzed multiple data augmentation methods. in their experiments, they applied self-training and back-translation. in selftraining, they fixed the source-side and used a forward translation model to generate the target-side, and in back-translation, they fixed the target-side and used a backward translation model to generate the source-side. it was observed that these methods can effectively improve the translation accuracy for infrequent tokens. these methods can be used with the sentence concatenation method proposed in this study. in multi-source neural machine translation, @xcite proposed concatenating source sentences in different languages corresponding to a target sentence in training. however, they did not aim to improve the translation accuracy of long sentences. our method concatenates two source sentences in the same language at random. ## data augmentation by sentence concatenation the proposed method augments the parallel data by back-translation and concatenation. a schematic overview of the proposed method is shown in figure 1. first, we back-translate the target-side of the parallel corpus @xcite to create pseudo data as additional training data. note that we do not use external data in backtranslation, and the diversity of target sentences does not change. then, we randomly select two sentences exclusively in the original or pseudo data and concatenate them to create another training data. technically, we concatenate two source sentences and insert a special token, "<sep>," between them. corresponding target sentences are concatenated in the same way. afterwards, we remove the sentences consisting of less than 25 words from the pseudo data. finally, we obtain an augmented training data comprising original, pseudo, and concatenated sentences, which has the quadruple data size of the original training data. we train our models on both single and concatenated sentences first because models can learn to translate single sentences. we also expect models to acquire a better absolute position encoding to translate long sentences in the better quality without generating a special token (i.e., <@xmath0. during the testing process, a single sentence is fed as the input, even though the training data contains concatenated sentences. foot_0 ## conclusion this study proposes a data augmentation method to improve the translation quality of long sentences. the experimental results confirmed that the data augmentation method is straightforward but useful, especially for the translation of very long sentences. however, the quality of the translation of short sentences is reduced. in the future, we would like to develop a method that works well when there is a large amount of available parallel data. moreover, since the adequacy of the translation of short sentences is considerably low in the proposed method, we would like to compensate for this weakness by considering the reconstruction loss @xcite . also, it would be interesting to explore the use of interpolation of hidden space for data augmentation considering long sentences @xcite .