Sometimes We Want Ungrammatical Translations (2021)

Abstract: Rapid progress in Neural Machine Translation (NMT) systems over the last few years has focused primarily on improving translation quality, and as a secondary focus, improving robustness to perturbations (e.g. spelling). While performance and robustness are important objectives, by over-focusing on these, we risk overlooking other important properties. In this paper, we draw attention to the fact that for some applications, faithfulness to the original (input) text is important to preserve, even if it means introducing unusual language patterns in the (output) translation. We propose a simple, novel way to quantify whether an NMT system exhibits robustness or faithfulness, by focusing on the case of word-order perturbations. We explore a suite of functions to perturb the word order of source sentences without deleting or injecting tokens, and measure their effects on the target side. Across several experimental conditions, we observe a strong tendency towards robustness rather than faithfulness. These results allow us to better understand the trade-off between faithfulness and robustness in NMT, and opens up the possibility of developing systems where users have more autonomy and control in selecting which property is best suited for their use case.

## introduction recent advances in neural machine translation (nmt) have resulted in systems that are able to effectively translate across many languages @xcite , and we have already seen many commercial deployments of nmt technology. yet some studies have also reported that nmt systems can be surprisingly brittle when presented with out-of-domain data @xcite , or when trained with noisy input data containing small orthographic @xcite @xcite or lexical perturbations @xcite . uncovering these sorts of errors has lead the research community to develop new nmt models that are more robust to noisy inputs, using techniques such as targeted data augmentation @xcite and adversarial approaches @xcite . unfortunately an approach that (over-)emphasized robustness can lead to "hallucinations"-translating source input to an output that is not faithful to the source, and sometimes is even factually incorrect @xcite @xcite @xcite @xcite @xcite . moreover, such an approach hinges on the key assumption that orthographic, lexical or grammatical variants in the input are mistakes, to be corrected by the translation system. this ignores the wealth of applications where it may be preferable for a system to offer more faithfulness to the original text. it is worthwhile to consider the diversity of applications where having a faithful translation (opting literal translation over paraphrasing) is desirable. first, consider an automatic language tutoring system: a (human) second-language learner will often produce language that has grammatical mistakes of various types. this learner can be empowered by having a (ai-produced) faithful translation, so that s/he can see what mistakes were made vs. what would be the more common phrasing. second, recall that many languages, including english, use word order to encode argument structure information (cf. @xcite : while "the dog bit the man" might be more frequent compared to "the man bit the dog", the latter has a very clear meaning that we may wish to preserve in some (albeit rarer) cases. third, consider poetry: it is often the case that unusual word order is used to influence rhythm and rhyme. it would be a shame if all our state-of-the-art nmt systems lost such poetic beauty in translation. figure 1 : machine translation systems (a-d) tend to favor robust or faithful translations as measured by computing the difference between faithfulness and robustness scores across seven languages (aggregated across all perturbations). although models with different sizes were analysed, we did not find a strong correlation between the robustness or faithfulness to the model sizes. but, m2m-100-1.2b showed a higher tendency to be robust when compared with m2m-418m or mbart (smaller that m2m-1.2b model). preferentially output "normative" language (regardless of whether the nonstandard languages affects spelling, word order, or choice of vocabulary). @xcite note that word order is an important problem in distant-language translation. when we increase model robustness (at least with the solutions proposed to date), we generally enforce even stronger tendencies towards the norm, at the expense of diversity of language, of thought, and, perhaps, of our very culture. although bisazza et al. (2021)'s observation on word order flexibil-ity only minimally affect the performance of nmt systems is encouraging towards building robust systems, the trade-off on preserving diversity in expression is seldom understood. we believe it will be necessary in future to propose solutions that can explicitly enable a better trade-off between robustness and faithfulness, and can give the user autonomy and control in specifying their preference. it is therefore our goal with this work to draw attention to this important compromise, and to provide tools to detect, quantify, and compare such aspects of nmt systems. more specifically, this paper is not only the first to deeply analyze the effects of particular perturbations on existing nmt systems, but is the first to investigate their effects in the sphere of generation. we investigate 16 unique perturbations that fall into three categories-dependency tree based, pos-tag based and random shuffles. we introduce two novel metrics for evaluating machine translation models' preference for robustness or faithfulness. taking english as the common source, we run a case study with three widely used transformer-based machine translation mod-els -helsinki/opus machine translation model @xcite , the multilingual bart model @xcite , and the many-to-many multilingual translation model @xcite (in two sizes)-into 7 target languages from several families (german, french, spanish, italian, russian, chinese, and japanese). across several experimental conditions, we observe a strong tendency towards robustness rather than faithfulness (figure 1 ) that varies somewhat depending on the particular perturbation (figure 2 ). more specifically, we observe that (1) state-of-the art nmt systems tend to produce translations that are unaffected by the noisy source (more robust), ( 2 ) accuracy (bleu score) correlates with model robustness, (3) certain perturbations involving partof-speech-based word reordering tend to further encourage robustness, and ( 4 ) results vary by somewhat by target language, with the models producing translations of japanese that are more faithful than for the other languages (except for helsinki-opus). overall, our analysis suggests that overemphasizing accuracy and robustness may limit richer development and broader usefulness of nmt systems. ## related work the idea to randomly shuffle linguistic elements to evaluate nlp model performance goes back fairly far @xcite , and has even been used to determine which tasks are "syntax-light" in human sentence processing @xcite . recent work on classification tasks, such those on the glue benchmark @xcite , has shown that pre-trained transformer-based models trained with a masked language modeling objective are shockingly insensitive to word order permutations. @xcite @xcite @xcite . given these recent findings, we might expect insensitivity to word order permutation in the sphere of generation as well, leading to robust machine translations. the mismatching of default word orders between target and source has long been an important consideration for multilingual tasks including automatic machine translation. @xcite find that word order agnostic models (recurrent neural networks) trained to dependency parse can transfer better than word order sensitive ones (selfattention) to distantly related languages. also in the context of transfer, @xcite propose for reference-free mt that the delta between originally ordered and permuted sentences be used as an evaluation technique. even when considering multilingual sequence labeling tasks in general, @xcite ; @xcite find that limiting word order information in the multilingual setting can enable models to achieve better zeroshot cross-lingual performance. taken together, these works also suggest that our models tend to overfit on source word order to the detriment of that of the target, which might lead one to predict that our models will be more robust than they are faithful in our case as well. however, nmt systems have use cases in diverse applications that require the preservation of word order, local syntax and other linguistic components @xcite . translation systems that are contingent on preserving syntax and semantics are used as interpretors to decode the interaction between components of a neural network @xcite . further, in practical applications like translating a sentence that is a mixture of two different languages requires the mt systems to strike some balance between preserving l1 syntax and/or word-order and correctly adhering to the grammatical rules of l2 @xcite . in nlp tasks, where the end-user could be a human, benchmarking the robustness of nlp systems is done by evaluating a model's performance on willfully perturbed examples that could potentially expose fragility of the systems @xcite . towards averting such scenarios, efforts along the lines of building robust models with adversarial training have been a common topic of study in natural language processing @xcite . our word order perturbations also share some points of synergy with work across nlp that aims to devise supplementary heuristics to explicate the inner workings of our machine learning systems. for specific nlp tasks, probe tasks are engineered to measure specific kinds of linguistic knowledge encoded in the systems @xcite @xcite @xcite . swapping the arguments of verbs is a classic way to measure the effects of word order both in humans @xcite and in models, largely because changing the order of verbal arguments maintains high word overlap between related examples @xcite @xcite ; however, although limited word order permutation is applied in this case, it is generally restricted to licit, grammatical sequences of words. when perturbation has been used to evaluate model performance, the utilized perturbation functions have been predominantly fairly simple, including reverse and word shuffle, and usually target only single sentences @xcite @xcite . for tasks like dialogue prediction that requires multiple input sentences, perturbation functions like reordering the conversation history have been adopted @xcite . to the best of our knowledge, the set of perturbation functions we propose is the most detailed set explored thus far, perturbing not only tokens, but pos and dependency structure. changing the order of words in the context of nmt also has its roots in classical, syntactically sophisticated models that used parses (of various kinds) to pre-order abstract syntactic representations as an early step in a multi-step translation pipeline from source to target @xcite @xcite @xcite ; miceli-barone and attardi 2013 i.a.). our approach differs from these approaches in that our main aim is not to incorporate word order changes into the translation pipeline itself, but, instead use them to better understand the behavior of nmt models. ## metrics let g x be a sentence where x takes one of two values: e if it is a sentence in the source language (english) or o if it is a gold target sentence. let φ @xmath0 we define three metrics β 1 , β 2 , and α. β 1 is our measure of robustness to perturbation by quantifying the similarity according to κ between the translation of a perturbed sentence in source into the target, and the gold sentence in target: the difficulty of the perturbation function is measured with α, which scores the similarity between perturbed sentence and the unperturbed sentence in the source language: α e ← 1 n n @xmath1, g - e ) i . β measures the standard translation performance metric on any given source-target sentence pair: β ← 1 d d @xmath2, t o ) i , where d is the size of the dataset. ## perturbations we propose 16 different functions to perturb the structure of an input sentence. the perturbations can be broadly classified in three categories-random shuffles, pos-tag based and dependency tree based-comprised of 4, 8, and 4 perturbation functions respectively. the functions vary in complexity and linguistic sophistication so that we can score whether a model translates faithfully or stays robust to the perturbed inputs. we applied all perturbations in seven languages-de, fr, ja, ru, zh, it, and es-and describe each perturbation in turn below. see figure 3 for a selection of examples. some of the perturbations we explore are "possible", in the sense that applying them will result (in most cases) in a grammatical sentence treemirrorpost: to live a decent place he could n't find tom said . treemirrorpre: said find place live to a decent he could n't tom . treemirrorin: live to place a decent find he could n't said tom . rotatearoundroot: live find said tom he could n't a decent place to . wordshuffle: place to could live said decent a tom n't find he . reversed: live to place decent a find n't could he said tom . verbswaps: tom live he find n't said a decent place to could . nounswaps: tom said a decent place could n't find he to live . nounverbswaps: said tom could he n't a decent place find to live . nounverbmismatched: live a decent place find could n't he said to tom . shufflefirst: he tom find could said n't a decent place to live . shufflelast: tom said he could n't find a decent live place to . figure 3 : effect of the different perturbation functions on the sentence -tom said he could n't find a decent place to live. the perturbation functions do not inject new tokens or delete a token to perturb the sentence. ## experiments we experiment with some of the state of the art translation models -opus translation models @xcite , mbart @xcite ), facebook's m2m @xcite (both 418m and 1.2b models). we construct the perturbed dataset using the eval set of opus corpus @xcite in 7 different languages paired with english as source -french (fr), german (de), russian (ru), japanese (ja), chinese (zh), spanish (es), and italian (it). our experiments 3 have a twofold objective: (1) compute the robustness (β 1 ) and faithfulness (β 2 ) of the translations in different languages when the input is perturbed, and (2) analyse the β 1 and β 2 scores with different levels of perturbations.