Revisiting Robust Neural Machine Translation: A Transformer Case Study (2021)

Abstract: Transformers have brought a remarkable improvement in the performance of neural machine translation (NMT) systems but they could be surprisingly vulnerable to noise. In this work, we try to investigate how noise breaks Transformers and if there exist solutions to deal with such issues. There is a large body of work in the NMT literature on analyzing the behavior of conventional models for the problem of noise but Transformers are relatively understudied in this context. Motivated by this, we introduce a novel data-driven technique called Target Augmented Fine-tuning (TAFT) to incorporate noise during training. This idea is comparable to the well-known fine-tuning strategy. Moreover, we propose two other novel extensions to the original Transformer: Controlled Denoising (CD) and Dual-Channel Decoding (DCD), that modify the neural architecture as well as the training process to handle noise. One important characteristic of our techniques is that they only impact the training phase and do not impose any overhead at inference time. We evaluated our techniques to translate the English–German pair in both directions and observed that our models have a higher tolerance to noise. More specifically, they perform with no deterioration where up to 10% of entire test words are infected by noise.

## introduction nmt is the task of transforming a source sequence into a new form in a particular target language using deep neural networks. such networks commonly have an encoder-decoder architecture @xcite , in which an encoder maps a given input sequence to an intermediate representation and a decoder then uses this representation to generate candidate translations. both encoder and decoder are neural networks that are trained jointly. due to the sequential nature of the nmt task, early models usually relied on recurrent architectures @xcite , or benefited from the sliding feature of convolutional kernels to encode/decode variable-length sequences @xcite . recently, transformers @xcite ) have shown promising results for nmt and become the new standard in the field. they follow the same concept of encoding and decoding but in a relatively different fashion. a transformer is fundamentally a feed-forward model with its unique neural components (self-attention, layer norm, etc) that altered the traditional translation pipeline accordingly. it is expected if such a new architecture would behave differently than its recurrent or convolutional counterparts, and our goal in this research is to study this aspect in the presence of noise. nmt engines trained on clean samples provide high-quality results when tested on similarly clean texts, but they break easily if noise appears in the input @xcite . they are not designed to handle noise by default and transformers are no exception. many previous works have focused on this issue and studied different architectures @xcite . however, in this work, we particularly focus on transformers foot_0 as they are relatively new and to some extent understudied. a common approach to make nmt models immune to noise is fine-tuning (ft), where a noisy version of input tokens is intentionally introduced during training and the decoder is forced to generate correct translations despite deformed inputs. ft is quite useful for almost all situations but it needs to be run with an optimal setting to be effective. in our experiments, we propose a slightly different learning-rate scheduler to improve ft. we also define a new extension that not only modifies input words but also adds complementary tokens to the target side. we refer to this extension as target augmented fine-tuning (taft), which is the first contribution of this paper. in our study, we realized that data augmentation techniques (ft and taft) might not be sufficient enough and we need a compatible training process and neural architecture to deal with noise. therefore, we propose controlled denoising (cd) whereby noise is added to source sequences during training and the encoder is supposed to fix noisy words before feeding the decoder. this approach is implemented via an auxiliary loss function and is similar to adversarial training. cd is our second contribution. cd only takes care of noise on the encoder side. we also propose a dual-channel decoding (dcd) strategy to study what happens if the decoder is also informed about the input noise. dcd supports multi-tasking through a 2-channel decoder that samples target tokens and corrects noisy input words simultaneously. this form of fusing translation knowledge with noise-related information has led to interesting results in our experiments. dcd is the third and last contribution of this work. the remainder of the paper is organised as follows: we first review previously reported solutions for the problem of noise in nmt in section 2. then, we present details of our methods and the intuition behind them in section 3. to validate our methods, we report experimental results in section 4. finally, we conclude the paper and discuss possible future directions in section 5. ## related work fine-tuning (ft) is one of the most straightforward and reliable techniques to protect nmt systems from noise. berard et al. (2019), dabre and sumita (2019), and helcl et al. (2019) studied its impact and showed how it needs to be utilized to boost nmt quality. adversarial training is another common solution to build noise-robust models. @xcite proposed a gradient-based method to construct adversarial examples for both source and target samples. source-side inputs are supposed to attack the model while adversarial target inputs help defend the translation model. in their model, a candidate word is replaced with its semantically-close peer to introduce noise. this way, the neural engine visits different forms of the same sample, which extends its generalization. in other words, the network is trained to deliver the same, consistent functionality even though it is fed with different forms of a sample. although this strategy showed promising results, in our setting we replace input words with real noisy candidates instead of synonyms or semantically-related peers. we find this way of adding noise more realistic and closer to real-world scenarios. @xcite experimented another idea by generating adversarial examples using synthetic noise. their proposed architecture relies on transformers but the encoder is equipped with a character-based convolutional model @xcite . this work is one of the few attempts that studied transformers' behaviour in the presence of noise. however, their results are based on relatively small datasets. we know that nmt models' performance could change proportionally with a change in the size of training sets. therefore, we used larger datasets in our experiments. the application of adversarial training is not limited to the aforementioned examples. cheng et al. (2018) defined additional loss functions which force the encoder and decoder to ignore perturbations and generate clean outputs. this idea is similar to our cd approach, but the underlying architecture is different. @xcite only reported results on recurrent nmt models. providing better representations is as important as designing tailored training strategies for noiserobust models. a group of researchers focused on how different segmentation schemes and encoding techniques can play a role. @xcite and @xcite showed that subwords are better alternatives than surface forms (words) to handle perturbations and outof-vocabulary words. @xcite comprehensively studied this by using different character-and subword-based representations in different architectures. @xcite also carried out a similar investigation where they proposed a new encoding that is invariant to the order of characters. besides these approaches, translating noisy inputs can be viewed as a two-pass process performed via two connected neural networks. the first one acts as a monolingual translator to correct noisy inputs and the second one is an engine that consumes denoised sequences to generate clean translations @xcite . this idea can be implemented as an end-to-end, differentiable solution or as a pipeline, but it should be noted that such a mechanism could be hard to deploy or slow(er) to run in practice. ## methodology this section covers details of our proposed methods. ft is a well-known technique so we skip its details and only focus on taft, our own extension of it (section 3.1). besides ft and taft that leverage data, we introduce cd (section 3.2) and dcd (section 3.3), which modify the training procedure as well as the neural architecture of transformers. ## experimental study datasets to evaluate our models, we trained engines to translate the english-german (en-de) pair in both directions. in the interest of fair comparisons, we used the same datasets as the original transformer @xcite , so our training set is the wmt-14 dataset foot_1 with 4.5m parallel sentences and for development and test sets we used newstest-13 and newstest-14, respectively. nowadays, almost all state-of-the-art nmt models rely on subwords. we also followed the same tradition and preprocessed the target side of our datasets with bpe @xcite . for the source side, we used different segmentation schemes with different granularities as we are studying the impact of noisy inputs. our source tokens can appear in surface forms (words) or can be segmented into bpe tokens or oven characters. we refer to these settings as word2bpe, bpe2bpe, and char2bpe, respectively. the size of the lexicon generated by our bpe model also matches the setting proposed for the original transformer model @xcite . for the word2bpe setting, we keep the top 48k frequent words for each english and german sides and ignore the rest by substituting with a special unk token. this configuration is learned through an empirical study to maximize translation quality. ## conclusion and future work in this paper, we studied the problem of noise in the context of nmt and particularly focused on transformers. we proposed three novel techniques to augment data and change the training procedure as well as the neural architecture. experimental results show that our techniques can protect nmt engines from noise. our models only affect the training phase and do not add any overhead in terms of space and/or time complexities at inference time. the findings of our research can be summarized as follows: in this research, we ran an extensive number of experiments in order to find the best configuration of each model and optimize hyper-parameters, but there still exist some unexplored topics/areas. in our future work, we are planning to experiment with other language pairs with different morphological and grammatical structures.we are also interested in studying other noise classes. we could only afford to work with one class and we selected natural noise as we find it more realistic among others, but this work can be extended to other noise classes. finally, our models are not unique to transformer and nmt. we aim to evaluate them in other language processing/understanding tasks with other architectures.