Importance-Aware Data Augmentation for Document-Level Neural Machine Translation (2024)

Abstract: Document-level neural machine translation (DocNMT) aims to generate translations that are both coherent and cohesive, in contrast to its sentence-level counterpart. However, due to its longer input length and limited availability of training data, DocNMT often faces the challenge of data sparsity. To overcome this issue, we propose a novel Importance-Aware Data Augmentation (IADA) algorithm for DocNMT that augments the training data based on token importance information estimated by the norm of hidden states and training gradients. We conduct comprehensive experiments on three widely-used DocNMT benchmarks. Our empirical results show that our proposed IADA outperforms strong DocNMT baselines as well as several data augmentation approaches, with statistical significance on both sentence-level and document-level BLEU.

## introduction document-level neural machine translation (docnmt) has achieved significant progress in recent years, as evidenced by notable studies @xcite @xcite @xcite @xcite @xcite . by effectively incorporating contextual information, doc-nmt aims to enhance the coherence and cohesion between the translated sentences, compared with its sentence-level counterpart (sentnmt). however, training docnmt models requires documentlevel parallel corpora, which are more difficult and expensive to obtain than sentnmt. this data sparsity issue can cause docnmt models to learn spurious patterns in the training data, leading to poor generalization @xcite . to overcome this issue, the data augmentation (da) technology @xcite offers a promising solution. these da methods for sentnmt typically generate synthetic data by randomly perturbing tokens in the training instances @xcite @xcite . on top of this, in this paper, we propose a novel important-aware data augmentation (iada) method, which provides explicit signals for training the docnmt models to proactively utilize document contextual information. specifically, as shown in figure 1 , iada first perturbs the important tokens (i.e., she and attack) in the current sentence to be translated, which enforces the docnmt models to recover those information using the document context. iada further perturbs the less important tokens in the context (i.e., because and have), highlighting the useful information in the document context. to determine token importance, we propose two novel measures derived from the docnmt model: the topmost hidden states of the encoder/decoder (tnorm), which leverages context-dependent information, and training gradients (gnorm), which takes source-target alignment information into account. finally, as iada perturbs the important information in current sentences and could increase learning difficulty. we combat this issue by adding an agreement loss between the original and perturbed instances. in this work, we combine iada with two pop-ular data augmentation methods, word dropout @xcite (i.e., iada drop ) and word replacement @xcite (i.e., iada repl ). we evaluate these versions on three widely-used docnmt benchmarks: ted, news, and europarl. our experiments consistently demonstrate that both iada drop and iada repl outperform various strong doc2doc models with statistical significance. we perform ablation studies to validate the effectiveness of our design choices. through our analyses, we show that iada enhances contextual awareness and robustness in the docnmt model. additionally, we demonstrate that iada can be combined with back/forwardtranslation techniques and is particularly beneficial in low-resource settings. lastly, our linguistic study confirms iada's ability to effectively identify important tokens in the text. data augmentation data augmentation (da) approaches for nmt are commonly categorized into two classes, word replacement and back/forward translation. @xcite and @xcite introduce word dropout (worddrop), where word embeddings are zeroed out at random positions in the input sequence. @xcite incorporate a dropout-like mechanism into the bpe segmentation process @xcite , generating multiple segments for the same sequence. @xcite utilize language models and phrasal alignment with causal modeling to augment sentence pairs. takase and kiyono (2021) demonstrate that word dropout (worddrop) and word replacement (wordrepl) can achieve strong performance with improved computational efficiency. @xcite expand the training corpus by enciphering the text with deterministic rules. back-translation (bt) translates the monolingual corpus from the target language back to the source language, resulting in significant performance improvements @xcite . @xcite perform iterative bt and observe substantial performance gains. another approach, known as forward-translation (ft) or self-training, translates the monolingual source corpus into the target language @xcite . recent works perform bt with a docnmt model, known as docbt @xcite . ours our novel important-aware data augmentation (iada) method effectively encourages the docnmt model to leverage the contextual information. our empirical results conform that iada is compatible with the classical da approaches, such as docbt and docft. ## method in this section, we introduce the task of docnmt in section 3.1, our proposed iada framework in section 3.2, our token importance measures in section 3.3, and our training objective in section 3.4. ## analysis we analyze iada from various aspects in this section, including contextual awareness, robustness, compatibility with docbt/docft, simulated low-resource scenario, and linguistic analysis. ## conclusion in this paper, we present iada, a new method for generating high-quality syntactic data for doc-nmt. by leveraging token importance, iada augments existing training data by perturbing important tokens in the current sentences while keeping those less important ones unchanged. this encourages docnmt models to effectively utilize contextual information. we propose tnorm and gnorm to measure token importance. we also introduce the agreement loss to prevent the training samples from being overly hard to learn after perturbation. results demonstrate that iada outperforms competitive docnmt approaches as well as several data augmentation methods. our analysis reveals that iada enhances doc-nmt models' contextual awareness, robustness, and is compatible with docbt and docft techniques. iada also shows significant benefits in low-resourced settings. linguistic analysis validates the effectiveness of iada in identifying important tokens. overall, our findings highlight the efficacy of iada in improving syntactic data generation for docnmt.