Counterfactual Data Augmentation for Neural Machine Translation (2021)

Abstract: We propose a data augmentation method for neural machine translation. It works by interpreting language models and phrasal alignment causally. Specifically, it creates augmented parallel translation corpora by generating (path-specific) counterfactual aligned phrases. We generate these by sampling new source phrases from a masked language model, then sampling an aligned counterfactual target phrase by noting that a translation language model can be interpreted as a Gumbel-Max Structural Causal Model (Oberst and Sontag, 2019). Compared to previous work, our method takes both context and alignment into account to maintain the symmetry between source and target sequences. Experiments on IWSLT’15 English → Vietnamese, WMT’17 English → German, WMT’18 English → Turkish, and WMT’19 robust English → French show that the method can improve the performance of translation, backtranslation and translation robustness.

## introduction neural machine translation (nmt) models @xcite @xcite have reached state-of-the-art performance on various benchmarks. however, these models frequently rely on large-scale parallel corpora for training, exhibiting degraded performance on low-resource languages @xcite . further, modern nmt systems are often brittle, as noises (e.g. grammatical errors) can cause significant mistranslations @xcite . data augmentation is a promising direction to overcome these issues. it works by enlarging the number of data points for training without manually collecting new data. it is widely used to improve diversity and robustness and to avoid overfitting on small datasets. even though data augmentation (e.g. image flipping, cropping and blurring) has s 4 r 2 a e 0 s u y m n + 3 q y s t m t i q s a n 6 h h + b w p 8 g d u h x n f / g b t t i s b o u b g x p p u z d 7 5 3 g r o 1 l z 9 r d r w l l d w 9 8 o b 1 a 2 t n d 2 9 8 z 9 g 7 y m y 0 g g r u i w i q 6 h j t d k o a w o y t c n b o d a y 9 d x j t e z 3 3 k e i w n i 7 9 u 0 a j f a i 0 5 9 s r d s 0 s a 8 6 g d y j t 0 / u u k h i z m + 5 c x b l o m m a 7 n q 1 + w c 1 j j x c l j f b z o d 8 6 c / d e k c a f e e y s l 7 j h 0 p n 8 f c u c i g r f r j c r e m e z y c n q y c b y d d j p 9 d a p 1 q z w j 5 o d c p k y t x / 0 4 k o j b y g n i 6 m z t r l n q z + j / x i 5 v / 5 s a u r 7 e c t m a l / j h z k r s y q k w h f u a u m 2 q c i a d 6 v o u m s c b e 6 d j m t k i d c w h g w t l o y g 7 l p f 2 v o e e 1 + t 1 f t w e x g z x r m t p b z 8 h b l 6 i b b l e t t r b b t + g f v a i 3 4 9 l 4 n z 6 m z 1 l r y s h m d t e c j k 9 f e y 2 h u @xmath0, it is non-trivial to apply in machine translation since even a slight modification to a sequence can result in drastic changes in its syntax and semantics. indeed there is relatively little work in this direction due to these difficulties @xcite @xcite @xcite . further, work based on word replacement either ignores the contexts of replaced words or breaks the alignment between source and target sequences, both detrimental for generating high-quality data. in this paper we observe that a translation language model can be interpreted as a causal model, as described in figure 1 . doing so allows us to ask counterfactual questions of the form: given source and target sequences, if a phrase in the source sequence is changed, how would the target sequence change? we propose a data augmentation method for machine translation that generates counterfactual parallel translation data. to ensure these counterfactuals are close to the original data we sample a new source phrase from a masked language model. we then consider the (path-specific) counterfactual target phrase that is aligned to that source phrase (given by an unsupervised phrasal alignment method). the idea is that this augmentation procedure exposes inductive biases in existing language models that enables new translation models to learn more efficiently and exhibit more robust generalisation. specifically, our augmentation procedure performs the following three steps: different from prior work, our approach takes advantage of both source/target context and alignment for data augmentation. experiments on iwslt'15 @xmath1, wmt'17 @xmath2, and wmt'18 @xmath3. we additionally demonstrate that our method complements existing approaches such as backtranslation @xcite . finally, we demonstrate that our method improves translation robustness (we evaluate this on the wmt'19 @xmath4. ## background in this section we describe background on neural machine translation (nmt), phrasal alignment, and causal modelling. ## method our goal is to take an input sequence pair (x , y) and create augmented data from it. we aim to do so by removing phrases, resampling them in the source sequence, and computing the counterfactual effect of doing so in the target sequence. we argue that for any such augmentation method for nmt, it is crucial to leverage both contextual and alignment information, for the following reasons. (1) context: as contextual information is widely used to disambiguate words @xcite and generate realistic-looking sequences @xcite , it is critical to utilize contextual information to obtain grammatically-correct and semanticallysound sequences. (2) alignment: phrasal alignment plays a critical role in statistical machine translation @xcite . as phrasal alignment provides information about which phrase in the source sequence produces a phrase in the target sequence, a data augmentation algorithm which disregards alignment risks breaking the symmetry between source and target sequences. to this end, in section 3.1, we introduce a technique called translation-counterfactual word replacement (tcwr) for leveraging both context and alignment to replace phrases in source and target sequences. in section 3.2, we propose a new data augmentation algorithm based on this replacement technique. in section 3.3, we describe the architectures used to parameterize the models. ## experiments we now describe the improvements with the data augmentation based on tcwr.