Addressing the Vulnerability of NMT in Input Perturbations (2021)

Abstract: Neural Machine Translation (NMT) has achieved significant breakthrough in performance but is known to suffer vulnerability to input perturbations. As real input noise is difficult to predict during training, robustness is a big issue for system deployment. In this paper, we improve the robustness of NMT models by reducing the effect of noisy words through a Context-Enhanced Reconstruction (CER) approach. CER trains the model to resist noise in two steps: (1) perturbation step that breaks the naturalness of input sequence with made-up words; (2) reconstruction step that defends the noise propagation by generating better and more robust contextual representation. Experimental results on Chinese-English (ZH-EN) and French-English (FR-EN) translation tasks demonstrate robustness improvement on both news and social media text. Further fine-tuning experiments on social media text show our approach can converge at a higher position and provide a better adaptation.

## introduction recent techniques @xcite @xcite in nmt have gained remarkable improvement in translation quality. however, robust nmt that is immune to real input noise remains a big challenge for nmt researchers. real input noises can exhibit in many forms such as spelling and grammatical errors, homophones replacement, internet slang, new words or even a valid word used in an unfamiliar or a new context. unlike humans who can easily comprehend and translate such texts, most nmt models are not robust to generate appropriate and meaningful translations in the presence of such noises, challenging the deployment of nmt system in real scenarios. noisy words have long been discussed in previous work. @xcite proposed the normalization approach to reduce the noise before translation. @xcite addressed the character-level noise directly in the nmt model. though these approaches addressed the effect of noisy words to some extent, they are limited to spelling errors, inflectional variations, and other noises definable during training. in addition, strong external supervision like a parallel corpus of noisy text translation or dictionary containing the translation of those noisy words are hard and expensive to obtain; they are also not practical in handling real noises as noisy words can exhibit in random forms and cannot be fully anticipated during training. @xcite pointed out nmt models are sensitive to small input perturbations and if this issue is not addressed, it will continue to bottleneck the translation quality. in such cases, not only the word embeddings of perturbations may cause irregularities with the local context, the contextual representation of other words may also get affected by such perturbations @xcite . this phenomenon applies to valid words in unfamiliar context as well, which will also cause the translation to fail as illustrated in in this paper, we define "noisy word" as a valid or invalid word that is uncommonly used in the context or not observed frequently enough in the training data. when encoding a sentence with such a noisy word, the contextual representation of other words in the sentence are affected by the "less jointly trained" noisy word embeddings. we refer this process as "noise propagation". noise propagation can extend to the decoder and finally distort the overall translation. the main intuition of our proposed method is to minimize this noise propagation and reduce the irregularities in contextual representation due to these words via a context-enhanced reconstruction (cer) approach. to reduce the sensitivity of contextual towards noisy words in the encoder, we inject made-up words randomly to the source side of the training data to break the text naturalness. we then use a noise adaptation layer (nal) to enable a more stable contextual representation by minimizing the reconstruction loss. in the decoder, we add perturbations with a semantic constraint and apply the same reconstruction loss. unlike adversarial examples which are crafted to cause the target model to fail, our perturbation process does not have such constraint and does not rely on a target model. our input perturbations are randomly generated, representing any types of noises that can be observed in real-world usage. this makes the perturbation process generic, easy and fast. following @xcite , we generate semantically related perturbations in the decoder to increase the diversity of the translations. together with nal, our model shows its ability to resist noises in the input and produce more robust translations. results on zh-en and fr-en translation significantly improve over the baseline by +1.24 (mt03) and +1.4 (n15) bleu on news domain, and +1.63 (social), +1.3 (mtnt18) on social media domain respectively. further fine-tuning experiments on fr-en social media text even witness an average improvement of +1.25 bleu over the best approach. robust neural machine translation: methods have been proposed to make nmt models resilient not only to adequacy errors @xcite but also to both natural and synthetic noise. incorporating monolingual data into nmt has the capacity to improve the robustness @xcite @xcite . some non data-driven approaches that specifically designed to address the robustness problem of nmt @xcite @xcite @xcite explored effective ways to synthesize adversarial examples into the training data. @xcite showed a structure-invariant word representation capable of addressing multiple typo noise. @xcite used adversarial stability training strategy to make nmt resilient to arbitrary noise. @xcite added an additional phonetic embedding to overcome homophone noise. meanwhile, michel and neubig (2018) released a dataset for evaluating nmt on social media text. this dataset was used as a benchmark for wmt 19 robustness shared task @xcite to improve the robustness of nmt models on noisy text. we show our approach also benefits the fine-tuning process using additional social media data. ## approaches we propose a context-enhanced reconstruction (cer) approach to learn robust contextual representation in the presence of noisy words through a perturbation step and a reconstruction step in both encoder and decoder during model training. figure 1 shows the architecture. the perturbation step automatically inserts made-up words in the input sequence x to generate a noisy example x . the noisy example mimics input where text naturalness is broken due to the noisy words. similarly, we perturb the output sequence y to y using a semantic constraint to generate noisy examples for the decoder to have more diversity in the translations. the reconstruction step in the model aims to restore the contextual representation c x of x to be similar to its corresponding original contextual representation c x in the encoder. specifically, under the transformer architecture (figure 1 ), the reconstruction step aims to stabilize and minimize the disruption of attention distribution for a word over the whole input in the presence of inserted noise. the stabilization is needed for both clean and noisy words as both of their contextual representations are affected. for a noisy word, reconstruction reduces the attention to itself and encourages the construction of the contextual representation to leverage more on its clean neighbors. for clean words, reconstruction works as a denoise module to mitigate the interference of noisy words. for c y in the decoder, the aim is to generate more examples with similar context as c y . the reconstruction helps to normalize the contextual representation of semantically similar words. ## experiment settings experiments are conducted on zh-en and fr-en translation tasks for both news and social media domains. we also use social media text to fine-tune the nmt systems on fr-en. ## conclusions in this work, we propose an approach to reduce the vulnerability of nmt models to input perturbations. our input perturbation is easy, fast and not specific to a target victim model. experimental results show our proposed approach improves the robustness on both news and social media text and helped to improve the translation of real input.