Rare but Severe Neural Machine Translation Errors Induced by Minimal Deletion: An Empirical Study on C hinese and E nglish (2022)

Abstract: We examine the inducement of rare but severe errors in English-Chinese and Chinese-English in-domain neural machine translation by minimal deletion of source text with character-based models. By deleting a single character, we can induce severe translation errors. We categorize these errors and compare the results of deleting single characters and single words. We also examine the effect of training data size on the number and types of pathological cases induced by these minimal perturbations, finding significant variation. We find that deleting a word hurts overall translation score more than deleting a character, but certain errors are more likely to occur when deleting characters, with language direction also influencing the effect.

## introduction pathological machine translation (mt) errors have been a problem since the field's inception, and they have been analyzed and categorized in the context of both statistical (smt) and neural machine translation (nmt). recent work examines pathologies in nlp models on classification problems: cases in which the models make wildly inaccurate predictions, often confidently, when input tokens are removed @xcite . identifying these enriches our understanding of neural models and their points of failure. mt pathologies take the form of severe translation errors, the worst being hallucinations @xcite . these rare errors are difficult to study precisely because they are rare. in this paper, we examine severe errors induced by minimal deletions by automatically extracting translations with severe errors and manually categorizing them. previous work taxonomizes smt errors @xcite and analyzes their effects on translation quality @xcite . more recently, @xcite propose a taxonomy of mt pathologies, of which hallucinations are a category. 1 they note the shortcomings of current automatic detection methods, e.g., those based on quality estimation and heuristics, and look for critical errors in naturalistic settings. they also propose dehallucinator, which flags problematic translations and replaces them with re-ranking. other work on chinese-english (zh-en) smt examines tense errors caused by incorrectly translating 了 (le) @xcite and syntactic failures caused by 的 (de). more recent work uses input perturbation to argue that nmt models, including those based on transformers @xcite , are brittle: @xcite examine the effect on nmt systems of several kinds of randomized perturbations by adding tokens, and @xcite study subword regularization to increase robustness to randomized perturbations. @xcite argue that memorized training examples are more likely to hallucinate, and @xcite examine the contribution of source and target tokens to errors. also related, @xcite suggest that bert is less robust to misspellings than other kinds of noise, which can occur naturalistically or through other errors (e.g., encoding). while we expect targeted adversarial examplesthose explicitly designed to cause a system to fail @xcite to cause serious errors, we focus on the ostensibly more benign case of in-domain en↔zh nmt with minimal deletions. adding valid words introduces distractors with which the mt system must cope, while deleting words more often removes information without explicitly introducing lexical distractors. both are noise, but the latter is more naturally framed as requiring recovery from missing information, while the former introduces irrelevant and misleading information. at the character level, this distinction is less clear, since both adding and re-moving characters requires that the model translate despite unseen input substrings-minimally corrupted inputs. are minimal word or character corruptions more harmful to a purely character-based nmt model? the answer is not obvious. while most prior work examines western european languages, we examine translation between chinese and english, building upon work identifying errors by observing change in bleu @xcite after perturbation @xcite . but in contrast this prior work, which adds tokens, we focus exclusively on single deletions to examine minimal conditions-i.e., a missing character or word, as in a typo or corruption-under which severe errors are newly induced. for our purposes, a severe error leads to a translation in which the original meaning is unrecoverable, but there are others, as well @xcite . for our purposes, we use word changing to cover these cases. ## finding candidates we now describe the training of our nmt model, method for extracting severe error candidates (enumerations), and the results of this extraction. for our extraction experiments, we begin by examining character deletion before repeating the same experiments with word deletion. all experiments are done in both directions and for two different training data sizes (1m and 10m sentences), allowing us to observe the effect of training data size, translation direction, and deletion type. ## experiments and results we now discuss our experiments and the results of our enumeration extraction and the errors contained therein. all results are summarized in ## conclusion and future work we examine the effect of minimal deletions on rare but severe mt errors on chinese and english, using outlier changes in bleu after deletion to find candidates. we find that the error rate for the model with a larger dataset is always lower, suggesting more data can improve models' performance against severe errors. removing single words is more likely to cause severe errors but less likely to cause missing parts in our models, despite character deletion introducing invalid words. on @xmath0, we observe none when removing words. with the important caveat that these errors are already rare, limiting the conclusions we can make, this may suggest that zh↔en models are better able to recover when characters are missing, even if the substrings themselves have never been observed, despite not having been trained with such noise. this is not obvious for a character-based model. nor is it obvious that @xmath1, but this is what we find, especially for words, perhaps because english words are simply longer. furthermore, that ∆bleu is not predictive of significantly more severe errors suggests that these errors are a different phenomenon from typical mt shortcomings. further research is needed to determine the effect various variables on robustness with targeted probes; future work can also determine how findings generalize across more language pairs (potentially typologies), tokenization schemes, and architectures. training models with missing source words may increase robustness. for detection, unusually large disparities in length between source and target could signal inability or missing parts errors, and vocabulary or semantic distance checks could flag bad translations (e.g., @xcite . it would also be instructive to examine the extent to which nmt robustness to noise mirrors that of humans.