Data Augmentation by Concatenation for Low-Resource Translation: A Mystery and a Solution (2021)

Abstract: In this paper, we investigate the driving factors behind concatenation, a simple but effective data augmentation method for low-resource neural machine translation. Our experiments suggest that discourse context is unlikely the cause for concatenation improving BLEU by about +1 across four language pairs. Instead, we demonstrate that the improvement comes from three other factors unrelated to discourse: context diversity, length diversity, and (to a lesser extent) position shifting.

## introduction many attempts have been made to augment neural machine translation (mt) systems to use discourse context @xcite @xcite @xcite @xcite @xcite . one particularly simple method is to concatenate consecutive pairs of sentencepairs during training, but not during translation @xcite @xcite . 1 in this paper, we confirm that this simple method helps, by roughly +1 bleu across four lowresource language pairs. but we demonstrate that the reason it helps is not discourse context, because concatenating random pairs of sentence-pairs yields the same improvement. instead, we view concatenation as a kind of data augmentation or noising method (one which pleasantly requires no alteration to the text, unlike data augmentation methods that disturb word order 1 as this paper was being finalized, @xcite published independent work also presenting random concatenation as data augmentation for nmt. they find that concatenation helps the model translate long sentences better, while the focus of the present paper is to explain thoroughly why it helps. @xcite or replace words with automatically-selected words @xcite @xcite ). concatenating random sentences is easier than concatenating consecutive sentences, because many parallel corpora discard document boundaries, drop sentence-pairs, or even reorder sentence-pairs, so it can be difficult to know which sentence-pairs are truly consecutive. but the fact that random concatenation helps so much creates a mystery, which is the focus of the paper. if the reason is not discourse context, what is the reason? we consider three new hypotheses: through a careful ablation study, we demonstrate that all three of these factors more or less contribute to the improvement, and together completely explain the improvement. ## concatenation we first present the concatenation methods and confirm that they improve low-resource translation. ## analysis why does a method as simple as concatenation help so much? we reject the initial hypothesis that the model is assisted by discourse context ( §3.1) and consider three new hypotheses related to data augmentation ( §3.2- §3.4). ## conclusion random concatenation is a simple and surprisingly effective data augmentation method for lowresource nmt. although the improvement of +1 bleu it yields seems mysterious at first, we have shown that it can be explained by the fact that concatenation increases positions, context, and length diversity. of these three factors, context diversity seems to be the most important.