Fine-Tuning MT systems for Robustness to Second-Language Speaker Variations (2020)

Abstract: The performance of neural machine translation (NMT) systems only trained on a single language variant degrades when confronted with even slightly different language variations. With this work, we build upon previous work to explore how to mitigate this issue. We show that fine-tuning using naturally occurring noise along with pseudo-references (i.e. “corrected” non-native inputs translated using the baseline NMT system) is a promising solution towards systems robust to such type of input variations. We focus on four translation pairs, from English to Spanish, Italian, French, and Portuguese, with our system achieving improvements of up to 3.1 BLEU points compared to the baselines, establishing a new state-of-the-art on the JFLEG-ES dataset. All datasets and code are publicly available here: https://github.com/mahfuzibnalam/finetuning_for_robustness .

## introduction neural machine translation (nmt) approaches have aided the machine translation field in achieving great advances in the recent years, starting with encoder-decoder models with attention @xcite , to transformers using self-attention @xcite , to massively multilingual models that yield large improvements even in low-resource settings @xcite . despite these very encouraging developments, the list of shortcomings of nmt is also quite vast @xcite , and one of the most crucial shortcomings is the lack of robustness to source-side noise. 2 when confronted with inputs that are even slightly different from the inputs that the models were trained on, the quality of the outputs significantly degrades. this observation has been confirmed for noise due to typos or character scrambling @xcite , due to faulty speech recognition @xcite , or due to naturally-occurring errors by second-language non-native speakers @xcite . however, this issue can particularly degrade the user experience for millions of potential users. for example, the number of non-native english speakers is three times larger than the number of native english speakers (c.f. around 1 billion for the former and about 300 million for the latter). had one had access to large amounts of data for all different language varieties, it would be straightforward to train variety-specific mt models. such data, though, are of course scarce. in this paper we work on addressing this particular shortcoming, in an attempt to make nmt systems more robust to source-side variations that nonnative speakers produce. since english is the language with the largest amount of second-language learners and non-native speakers, we only focus on mt systems translating out of english, but we point out that such work is urgently needed for other colonial languages (i.e. french, spanish) or majority languages (such as russian, mandarin, or hindi) that are taking over minority ones. 3 the main difference of our approach compared to previous work is that we do not attempt to synthesize different types of noise, but rather use naturally-occurring texts, as produced by nonnative speakers. we utilize grammar error correction corpora and produce pseudo-references, which we then use to fine-tune a nmt system with a goal of increasing its robustness to such sourceside noise. in our view, our approach has two main advantages and a single disadvantage over previous 3 that is also not to say that robustness is not necessary for low-resource languages; to the contrary! we just focus on high-resource settings first as they are the ones that have the potential to affect a larger number of downstream users. ## related work our work is inspired by and combines two lines of research: (1) robustness studies in nmt and (2) data augmentation. robust nmt making robust models for nmt has recently gained popularity, with shared tasks organized in the conference of machine translation @xcite and several solutions put forth @xcite @xcite @xcite . liu et al. (2018); karpukhin et al. (2019) focus on creating black-box methods for making synthetic or natural noises. ebrahimi et al. (2018) use white-box methods and creates adversarial examples for character-level nmt. anastasopoulos et al. (2019) show that including noisy synthetic data in the training data can increase the model's robustness without sacrificing performance on clean data, an approach that tan et al. (2020) extend to more nlp tasks. while these approaches are indeed meritorious and indeed improve a model's robustness, we argue that one needs to use natural noise instead, on account of two phenomena. the first is language change: the different variations that the models will have to contend with are not static, but rather constantly changing at an ever-increasing pace. second, and perhaps a partial direct consequence of the first point, one cannot rely on synthetic examples to properly capture the wide variety of naturallyoccurring variations. besides, if one could properly model noise creation, they could also similarly model the inverse problem adequately, namely remove said noise, in which case a noise-removing preprocessing step would be most likely suffiecient to tackle the issue. on working with real-world noise, the approach of @xcite is the most similar to ours. they collected "noisy" english, french, and japanese sentences from reddit, created translations, and split their dataset (mtnt) into train, development, and test, ranging from 5 to 36 thousand training examples. to build robust nmt systems, they first train a model on standard clean data and then fine-tune it on the training portion of mtnt using techniques from domain adaptation. the main difference between this worthy effort and our approach is two-fold. first, our approach does not require gold translations of the noisy inputs, which can be expensive and hard to collect, but we instead rely on the abundance of corrected second-language learner data (which we use to create pseudo-references, see §3). second, we attest that reddit language translation is much closer to a domain adaptation scenario, and includes additional noise types that are not pertinent to non-native language translation such as emoji, reddit jargon such us "upvote" or "gild", and internet slang such as "tbh" and "smh". 4 on working with pseudo-references, the approach of @xcite is the most similar to ours. they use asr corpora to create synthetic asr-induced noise and try to make nmt system more robust to this type of noise. as speech-to-transcription-to-translation datasets are very costly to produce, they use standard speechto-transcription datasets instead. they translate the gold transcription data set to get translation pseudoreferences. then they jointly train the model on noisy source transcription using the pseudoreference translations as the target. data augmentation data augmentation techniques have become increasingly popular for mt and other nlp tasks, from back-translation of monolingual data @xcite to counterfactual augmentation to address gender bias issues @xcite . for our purposes, we will focus on data augmentation techniques aimed at increasing nmt robustness. simple perturbations typically used include the infusion of character-level noise (e.g. character scrambling @xcite or typos (belinkov and bisk, 2018)) or word order scrambling @xcite . @xcite propose a gradient based method to attack the translation model with adversarial source examples, but there's not guarantee that the adversarial attack results in realistic noise @xcite . @xcite add specific types of errors (such as subject-verb agreement or determiner errors) on the source-side of parallel data, while @xcite specifically perturb the inflectional morphology of words to create adversarial examples and show that adversarial finetuning them for a single epoch significantly improves robustness. our work is highly motivated from these last two works, but instead of creating synthetic perturbed adversarial examples we use real noisy examples. ## fine-tuning for robustness our goal is to achieve robustness to source-side variations that are similar to the mistakes that nonnative english speakers make. to do so, we will utilize state-of-the-art pretrained systems and finetune them using pseudo-references over corpora that include real-world noise. the general outline of our approach is straightforward: notation throughout this work, we use the notation of @xcite to denote different types of data: for the sake of readability, we use the terms grammatical errors, noise, or edits interchangeably. in the context of this work, they will all denote the annotated grammatical errors in the source sentences ( ## conclusion in this work, we studied the effect of fine-tuning a nmt model using real source-side noise paired with pseudo-references obtained by translating grammar error correction corpora. we confirmed previous works on the utility of training with source-side noise, as it leads to models more robust to non-native english inputs, but also showed that instead of using synthetically-induced noise, we can (a) use real-user data with pseudo-references and (b) fine-tune a pre-trained system, rather than training from scratch. we will release all pseudoreferences and our code upon acceptance. our approach of fine-tuning a pre-trained system with pseudo-references approach has particular appealing advantages (less training time, no need for costly translation references) and it improves the robustness of mt systems significantly on all language pairs we tested. for future work, we will explore ways to integrate strategies for avoiding catastrophic forgetting, in order to achieve multilingual robustness without needing to fine-tune a multilingual model on all interested languages, as well as incorporating robustness rewards through reinforcement learning in the fine-tuning process. in addition, we will investigate how the quality of the pseudo-references affects the downstream results, and we also plan to explore the trade-off between language-specific and multilingual fine-tuning.