Evaluating Robustness to Input Perturbations for Neural Machine Translation (2020)

Abstract: Neural Machine Translation (NMT) models are sensitive to small perturbations in the input. Robustness to such perturbations is typically measured using translation quality metrics such as BLEU on the noisy input. This paper proposes additional metrics which measure the relative degradation and changes in translation when small perturbations are added to the input. We focus on a class of models employing subword regularization to address robustness and perform extensive evaluations of these models using the robustness measures proposed. Results show that our proposed metrics reveal a clear trend of improved robustness to perturbations when subword regularization methods are used.

## introduction recent work has pointed out the challenges in building robust neural network models @xcite . for neural machine translation (nmt) in particular, it has been shown that nmt models are brittle to small perturbations in the input, both when these perturbations are synthetically created or generated to mimic real data noise @xcite . consider the example in improving robustness in nmt has received a lot of attention lately with data augmentation @xcite @xcite @xcite and adversarial training methods @xcite @xcite as some of the more popular approaches used to increase robustness in neural network models. in this paper, we focus on one class of methods, subword regularization, which addresses nmt original input se kyllä tuntuu sangen luultavalta. translation it certainly seems very likely. perturbed input se kyllä tumtuu sangen luultavalta. translation it will probably darken quite probably. reference it certainly seems probable. previous work has used translation quality measures such as bleu on noisy input as an indicator of robustness. absolute model performance on noisy input is important, and we believe this is an appropriate measure for noisy domain evaluation @xcite @xcite . however, it does not disentangle model quality from the relative degradation under added noise. for this reason, we propose two additional measures for robustness which quantify the changes in translation when perturbations are added to the input. the first one measures relative changes in translation quality while the second one focuses on consistency in translation output irrespective of reference translations. unlike the use of bleu scores alone, the metrics introduced show clearer trends across all languages tested: nmt models are more robust to perturbations when subword regularization is employed. we also show that for the models used, changes in output strongly correlate with decreased quality and the consistency measure alone can be used as a robustness proxy in the absence of reference data. ## evaluation metrics robustness is usually measured with respect to translation quality. suppose an nmt model m translates input x to y and translates its perturbed version x δ to y δ , the translation quality (tq) on these datasets is measured against reference translations y: tq(y , y) and tq(y δ , y). tq can be implemented as any quality measurement metric, such as bleu @xcite or 1 minus ter @xcite . previous work has used tq on perturbed or noisy input as an indicator of robustness. however, we argue that assessing models' performance relative to that of the original dataset is important as well in order to capture models' sensitivity to perturbations. consider the following hypothetical example: selecting m 1 to translate noisy data alone is preferable, since m 1 outperforms m 2 (38 > 37). however, m 1 's quality degradation (40 → 38) reflects that it is in fact more sensitive to perturbation δ comparing with m 2 . to this end, we use the ratio between tq(y , y) and tq(y δ , y) to quantify an nmt model m 's invariance to specific data and perturbation, and define it as robustness: robust(m |x, y, δ) = tq(y δ , y) tq(y , y) . when evaluating on the dataset (x, y), robust(m |x, y, δ) < 1 means the translation quality of m is degraded under perturbation δ; robust(m |x, y, δ) = 1 indicates that m is robust to perturbation δ. it is worth noting that: (1) robust can be viewed as the normalized ∆@xmath0, y) -tq(y δ , y) because ∆tq/tq(y , y) = 1-robust. we opt for the ratio definition because it is on a [0, 1] scale, and it is easier to interpret than ∆tq since the latter needs to be interpreted in the context of the tq score. (2) high robustness can only be expected under low levels of noise, as it is not realistic for a model to recover from extreme perturbations. evaluation without references reference translations are not readily available in some cases, such as when evaluating on a new domain. inspired by unsupervised consistency training @xcite , we test if translation consistency can be used to estimate robustness against noise perturbations. specifically, a model is consistent under a perturbation δ if the two translations, y δ and y are similar to each other. note that consistency is sufficient but not necessary for robustness: a good translation can be expressed in diverse ways, which leads to high robustness but low consistency. sim can be any symmetric measure of similarity, and in this paper we opt for sim(y δ , y ) to be the harmonic mean of tq(y δ , y ) and tq(y , y δ ), where tq is bleu between two outputs. ## experimental set-up we run several experiments across different language families with varying difficulties, across different training data conditions (i.e. with different training data sizes) and evaluate how different subword segmentation strategies performs across noisy domains and noise types. ## experimental results as shown in however, both our proposed metrics (i.e., robustness and consistency) show clear trends of models' robustness to input perturbations across all languages we tested: bpe-@xmath1. this suggests that although we did not observe a significant impact of subword regularization on generic translation quality, the robustness of the models is indeed improved drastically. unfortunately, it is unclear if subword regularization can help translating real-world noisy input, as shown in robustness versus consistency variation in output is not necessarily in itself a marker of reduced translation quality, but empirically, consistency and robustness nearly always provide same model rankings in as illustrated in figure 1 , consistency strongly correlates with robustness (sample pearson's @xmath2.91 to 0.98) within each language pair. this suggests that for this class of models, low consistency signals a drop in translation quality and the consistency score can be used as a robustness proxy when the reference translation is unavailable. robustness versus noise level in this paper, robustness is defined by giving a fixed perturbation function and its noise level. we observe consistent model rankings across language pairs, but is it still true if we vary the noise level? to test this, we plot the robustness data points from the last section against the noise level. focusing on the misspelling perturbation for @xmath3, figure 2 shows that varying the word misspelling probability does not change the ranking of the models, and the gap in the robustness measurement only increases with larger amount of noise. this observation applies to all perturbations and language pairs we investigated. ## conclusion we proposed two additional measures for nmt robustness which can be applied when both original and noisy inputs are available. these measure robustness as relative degradation in quality as well as consistency which quantifies variation in translation output irrespective of reference translations. we also tested two popular subword regularization techniques and their effect on overall performance and robustness. our robustness metrics reveal a clear trend of subword regularization being much more robust to input perturbations than standard bpe. furthermore, we identify a strong correlation between robustness and consistency in these models indicating that consistency can be used to estimate robustness on data sets or domains lacking reference translations.