Beyond Noise: Mitigating the Impact of Fine-grained Semantic Divergences on Neural Machine Translation (2021)

Abstract: While it has been shown that Neural Machine Translation (NMT) is highly sensitive to noisy parallel training samples, prior work treats all types of mismatches between source and target as noise. As a result, it remains unclear how samples that are mostly equivalent but contain a small number of semantically divergent tokens impact NMT training. To close this gap, we analyze the impact of different types of fine-grained semantic divergences on Transformer models. We show that models trained on synthetic divergences output degenerated text more frequently and are less confident in their predictions. Based on these findings, we introduce a divergent-aware NMT framework that uses factors to help NMT recover from the degradation caused by naturally occurring divergences, improving both translation quality and model calibration on EN-FR tasks.

## introduction while parallel texts are essential to neural machine translation (nmt), the degree of parallelism varies widely across samples in practice, for reasons ranging from noise in the extraction process @xcite to nonliteral translations @xcite . for instance (figure 1 ), a french source could be paired with an exact translation into english (eq), with a mostly equivalent translation where only a few tokens convey divergent meaning (fine-div), or with a semantically unrelated, noisy reference (coarse-div). yet, prior work treats parallel samples in a binary fashion: coarse-grained divergences are viewed as noise to be excluded from training @xcite , whilst others are typically regarded as gold-standard equivalent translations. as a result, the impact of fine-grained divergences on nmt remains unclear. this paper aims to understand and mitigate the impact of fine-grained semantic divergences in nmt. we first contribute an analysis of how finegrained divergences in training data affect nmt quality and confidence. starting from a set of equivalent english-french wikimatrix sentence pairs, we simulate divergences by gradually "corrupting" them with synthetic fine-grained divergences. following @xcite -who, in contrast, study the impact of noise on mt-we control for different types of fine-grained semantic divergences and different ratios of equivalent vs. divergent data. our findings indicate that these imperfect training references: hurt translation quality (as measured by bleu and meteor) once they overwhelm equivalents; output degenerated text more frequently; and increase the uncertainty of models' predictions. based on these findings, we introduce a divergent-aware nmt framework that incorporates information about which tokens are indicative of semantic divergences between the source and target side of a training sample. source-side divergence tags are integrated as feature factors @xcite @xcite , while target-side divergence tags form an additional output sequence generated in a multi-task fashion @xcite . results on en↔fr translation show that our approach is a successful mitigation strategy: it helps nmt recover from the negative impact of fine-grained divergences on translation quality, with fewer degenerated hypotheses, and more confident and better calibrated predictions. we make our code publicly available: https://github.com/elbria/xling-semdiv-nmt . ## background & motivation cross-lingual semantic divergences we use this term to refer to meaning differences in aligned bilingual text @xcite . divergences in manual translation might arise due to the translation process @xcite and result in non-literal translations @xcite . divergences might also arise in parallel text extracted from multilingual comparable resources. for instance, in wikipedia, documents aligned across languages might contain parallel segments that share important content, yet they are not perfect translations of each other, yielding fine-grained semantic divergences @xcite . finally coarse-grained divergences might result from the process of automatically mining and aligning corpora from monolingual data @xcite , or web-scale parallel text @xcite @xcite . noise vs. semantic divergences in the context of mt, noise often refers to mismatches in webcrawled parallel corpora that are collected without guarantees about their quality. @xcite define five frequent types of noise found in the german-english paracrawl corpus: misaligned sentences, disfluent text, wrong language, short segments, and untranslated sentences. they examine the impact of noise on translation quality and find that untranslated training instances cause nmt models to copy the input sentence at inference time. their findings motivated a shared task dedicated to filtering noisy samples from webcrawled data at wmt, since 2018 @xcite @xcite . this work moves beyond such coarse divergences and focuses instead on finegrained divergences that affect a small number of tokens within mostly equivalent pairs and that can be found even in high-quality parallel corpora. training assumptions nmt models are typically trained to maximize the log-likelihood of the training data, d ≡ {(x (n) , y (n) )} n @xmath0, where (x (n) , y (n) ) is the n-th sentence pair consisting of sentences that are assumed to be translations of each other. under this assumption, model parameters are updated to maximize the token-level crossentropy loss: in figure 1 , we illustrate how semantic divergences interact with nmt training. in the case of coarse divergences, both the prefixes y (n) @xmath1, yield a noisy training signal at each time step t, which motivates excluding them from the training pool entirely. in the case of fine-grained divergences, the assumption of semantic equivalence is only partially broken. depending on the time step t, we might thus condition the prediction of the next token on partially corrupted prefixes, encourage the model to make a wrong prediction, or do a combination of the above. this suggests that fine-grained divergent samples provide a noisy yet potentially useful training signal depending on the time step. meanwhile, fine-grained divergences increase uncertainty in the training data, and as a result might impact models' confidence in their predictions, as noisy untranslated samples do @xcite . this work seeks to clarify and mitigate their impact on nmt, accounting for both translation quality and model confidence. ## mitigating the impact of fine-grained divergences we now turn to naturally occurring divergences in wikimatrix. we will see that their impact on model quality and uncertainty is consistent with that of synthetic divergences ( § 4.3). we propose a divergent-aware framework for nmt ( § 4.1) that successfully mitigates their impact ( § 4.3). ## related work we discuss work related to cross-lingual semantic divergences and noise effects in section 2 and now turn to the literature that connects with the methods used in this paper. factored models factored models are introduced to inject word-level linguistic annotations (e.g., part-of-speech tags, lemmas) in translation. source-side factors have been used in statistical mt @xcite and in nmt @xcite . target-side factors are used by @xcite as an extension to the traditional nmt framework that outputs multiple sequences. although their main motivation is to enable models to handle larger vocabularies, @xcite propose a list of novel applications of target-side factors beyond their initial purpose, such as wordcase prediction and subword segmentation. our approach draws inspiration from all the aforementioned works, yet it is unique in its use of both source and target factors to incorporate semantics in nmt. calibration @xcite find that nmt models are miscalibrated, even when conditioned on gold-standard prefixes. they attribute this behavior to the poor calibration of the eos token and the uncertainty of attention and design a recalibration model to improve calibration. @xcite argue that miscalibration can be attributed to the "extrinsic" uncertainty of the noisy, untranslated references found in the training data. @xcite investigate the effect of label smoothing on calibration. on a similar spirit, @xcite propose graduated label smoothing to improve calibration at inference time. they also link miscalibration to linguistic properties of the data (e.g., frequency, position, syntactic roles). our work, in contrast, focuses on the semantic properties of the training data that affect calibration. ## conclusion this work investigates the impact of semantic mismatches beyond noise in parallel text on nmt quality and confidence. our experiments on en↔fr tasks show that fine-grained semantic divergences hurt translation quality when they overwhelm the training data. models exposed to fine-grained divergences at training time are less confident in their predictions, which hurts beam search and produces degenerated text (repetitive loops) more frequently. furthermore, we also show that, unlike noisy samples, fine-grained divergences can still provide a useful training signal for nmt when they are modeled via factors. evaluated on en↔fr translation tasks, our divergent-aware nmt framework mitigates the negative impact of divergent references on translation quality, improves the confidence and calibration of predictions, and produces degenerated text less frequently. more broadly, this work illustrates how understanding the properties of training data can help build better nmt models. in future work, we will extend our analysis to other properties of parallel text and to other language pairs, focusing on low-resource conditions where divergences are expected to be even more prevalent.