Machine Translation Robustness to Natural Asemantic Variation (2022)

Abstract: Current Machine Translation (MT) models still struggle with more challenging input, such as noisy data and tail-end words and phrases. Several works have addressed this robustness issue by identifying specific categories of noise and variation then tuning models to perform better on them. An important yet under-studied category involves minor variations in nuance (non-typos) that preserve meaning w.r.t. the target language. We introduce and formalize this category as Natural Asemantic Variation (NAV) and investigate it in the context of MT robustness. We find that existing MT models fail when presented with NAV data, but we demonstrate strategies to improve performance on NAV by fine-tuning them with human-generated variations. We also show that NAV robustness can be transferred across languages and find that synthetic perturbations can achieve some but not all of the benefits of organic NAV data.

## introduction while current machine translation (mt) models often perform decently in regular conditions, robustness to slight modifications on input remains a challenge. several works have investigated the devastating effects that minor noise can cause in mt @xcite . addressing this is critical as robustness to naturally occurring variation is important in deployed systems. some classes of perturbations to which humans exhibit robustness but studies have shown mt models struggle with include speling errors or rtypos, casing @xcite , v15sua11y-51m1ll4r ch4r4ct3r5 @xcite , and synonym replacement substitution @xcite . however, we focus on a less-known class we call natural asemantic variation (nav). ## nav robustness in robustness of nlp, it is difficult to determine improvements without considering both what types of input models are meant to be robust to and what robustness looks like. in this section, we will first discuss nav with a formal definition and explanatory examples. then we will define nav robustness in terms of desired behaviors and present metrics as proxies for measuring nav robustness. ## analysis setup in this work, we seek to provide answers to several analysis questions related to nav perturbations and mt models' robustness to them. using our quality and consistency metrics we investigate the following: ## experiments and results for our experiments and results section, we will progress chronologically as we address each of our analysis questions as defined in section 3. ## discussion overall our work raises several questions about nav phenomena and how to address them in mt. we are able to provide some answers to a few research questions and open pathways for future work. nav describes a subtle yet substantial mark we find that nav-robust models can perform worse in non-nav settings, which does not differ from several other findings in robustness work in which a more robust model may not perform better on original, non-noised input. we also address scaling issues by synthetically generating nav and non-nav examples. the synthetic nav examples seem to help more than the non-nav examples, but none of them are able to obtain the same improvements as human-created nav data. the major results from all of our experiments are displayed in mt robustness several works investigate robustness in mt by considering different classes of perturbations and developing strategies to improve performance at test time. @xcite perform misspelling-and casing-related perturbations on mt test input and evaluate different models on their robustness to these. @xcite address several classes of perturbations by replacing a standard unicode-based encoder with a visual encoder, which demonstrates higher robustness to perturbations associated with visual appearance of language, such as 1337speak. @xcite generate additional clean and noisy data using backtranslation and different datasets to improve performance on noisy text. we add to this line of work by addressing a specific kind of robustness (nav). ## conclusion we present the reader with an under-studied subject in mt and contribute more formalized definitions and simple evaluation metrics for the phenomenon of natural asemantic variation. we also perform experiments, showing how nav perturbations can be used during fine-tuning to improve robustness of mt models, even when evaluating on a different language-pair. several questions remain to be explored to further formalize nav, investigate its role in nlp modeling and improve techniques to increase nuanced understanding.