Target-Side Augmentation for Document-Level Machine Translation (2023)

Abstract: Document-level machine translation faces the challenge of data sparsity due to its long input length and a small amount of training data, increasing the risk of learning spurious patterns. To address this challenge, we propose a target-side augmentation method, introducing a data augmentation (DA) model to generate many potential translations for each source document. Learning on these wider range translations, an MT model can learn a smoothed distribution, thereby reducing the risk of data sparsity. We demonstrate that the DA model, which estimates the posterior distribution, largely improves the MT performance, outperforming the previous best system by 2.30 s-BLEU on News and achieving new state-of-the-art on News and Europarl benchmarks.

## introduction document-level machine translation @xcite @xcite @xcite has received increasing research attention. it addresses the limitations of sentence-level mt by considering cross-sentence co-references and discourse information, and therefore can be more useful in the practical setting. document-level mt presents several unique technical challenges, including significantly longer inputs @xcite and relatively smaller training data compared to sentence-level mt (junczys-dowmunt, 2019; @xcite . the combination of these challenges leads to increased data sparsity @xcite @xcite , which raises the risk of learning spurious patterns in the training data @xcite and hinders generalization @xcite . to address these issues, we propose a targetside data augmentation method that aims to reduce sparsity by automatically smoothing the training distribution. the main idea is to train the document mt model with many plausible potential translations, rather than forcing it to fit a single human translation for each source document. this allows the model to learn more robust and generalizable patterns, rather than being overly reliant on features of particular training samples. specifically, we introduce a data augmentation (da) model to generate possible translations to guide mt model training. as shown in figure 1 , the da model is trained to understand the relationship between the source and possible translations based on one observed translation (step 1), and then used to sample a set of potentially plausible translations @xcite . these translations are fed to the mt model for training, smoothing the distribution of target translations @xcite . we use standard document-level mt models including transformer @xcite and g-transformer @xcite for both our da and mt models. for the da model, in order to effectively capture a posterior target distribution given a reference target, we concatenate each source sentence with a latent token sequence as the new input, where the latent tokens are sampled from the observed translation. a challenge to the da model is that having the reference translation in the input can potentially decrease diversity. to address this issue, we introduce the intermediate latent variable on the encoder side by using rules to generate ngram samples, so that posterior sampling (wang and park, 2020) can be leveraged to yield diverse translations. results on three document-level mt benchmarks demonstrate that our method significantly outperforms transformer and g-transformer baselines, achieving an improvement of 1. @xcite and 1.75 s-bleu on average, respectively, and the state-die meisten freien gesellschaften halten diese einschränkungen für sinnvoll , aber die gesetze wurden in letzter zeit verschärft . die meisten freien gesellschaften halten diese einschränkungen für sinnvoll , aber die gesetze wurden in letzter zeit verschärft . die meisten freien gesellschaften halten diese einschränkungen für sinnvoll , aber die gesetze wurden in letzter zeit verschärft . step 1. da model training: step 2. target-side data augmentation: da model 𝑃 𝑑𝑎 (𝑦|𝑥 𝑖 , 𝑦 𝑖 ) 𝑦 𝑖 𝑥 𝑖 da model 𝑃 𝑑𝑎 (𝑦|𝑥 𝑖 , 𝑦 𝑖 ) ො 𝑦 𝑗 𝑥 𝑖 mt model 𝑃 𝑚𝑡 (𝑦|𝑥 𝑖 ) ො 𝑦 𝑗 𝑥 𝑖 𝑥 𝑖 ~𝑃𝑑𝑎𝑡𝑎 (𝑥), 𝑦 𝑖 ~𝑃𝑑𝑎𝑡𝑎 (𝑦|𝑥 𝑖 ) most free societies accept such limits as reasonable , but the law has recently become more restrictive . samples from data distribution for training: 𝑦 𝑖 𝑦 𝑖 while most free societies consider these restrictions useful , the law has recently been tightened . most free societies regard such restrictions as reasonable , but the law has been strengthened lately . ... ො 𝑦 1 : ො 𝑦 2 : ො 𝑦 3 : sample from da model: ## target-side augmentation the overall framework is shown in figure 1 . formally, denote a set of training data as @xmath0, y i )} n @xmath1, where (x i , y i ) is the i-th sourcetarget pair and n is the number of pairs. we train a data augmentation (da) model (section 3.1) to generate samples with new target translations (section 3.2), which are used to train an mt model (section 3.3). ## experiments baselines. we apply target-side augmentation to two baselines, including sentence-level transformer @xcite and document-level g-transformer @xcite . we further combine back-translation and target-side augmentation, and apply it to the two baselines. training settings. for both transformer and g-transformer, we generate m new translations (9 for ted and news, and 3 for europarl) for each sentence and augment the data to its m + 1 times. for back-translation baselines, where the training data have already been doubled, we further augment the data 4 times for ted and news, and 1 for europarl, so that the total times are still 10 for ted and news, and 4 for europarl. we obtain the translations by sampling latent z with an observed ratio from a beta distribution beta(2, 3) and running a beam search with a beam size of 5. we run each main experiment three times and report the median. more details are described in appendix b.3. ## conclusion we investigated a target-side data augmentation method, which introduces a da model to generate many possible translations and trains an mt model on these smoothed targets. experiments show our target-side augmentation method reduces the effect of data sparsity issues, achieving strong improvement upon the baselines and new state-ofthe-art results on news and europarl. analysis suggests that a balance between high diversity and low deviation is the key to the improvements. to our knowledge, we are the first to do target-side augmentation in the context of document-level mt.