Semantically Consistent Data Augmentation for Neural Machine Translation via Conditional Masked Language Model (2022)

Abstract: This paper introduces a new data augmentation method for neural machine translation that can enforce stronger semantic consistency both within and across languages. Our method is based on Conditional Masked Language Model (CMLM) which is bi-directional and can be conditional on both left and right context, as well as the label. We demonstrate that CMLM is a good technique for generating context-dependent word distributions. In particular, we show that CMLM is capable of enforcing semantic consistency by conditioning on both source and target during substitution. In addition, to enhance diversity, we incorporate the idea of soft word substitution for data augmentation which replaces a word with a probabilistic distribution over the vocabulary. Experiments on four translation datasets of different scales show that the overall solution results in more realistic data augmentation and better translation quality. Our approach consistently achieves the best performance in comparison with strong and recent works and yields improvements of up to 1.90 BLEU points over the baseline.

## introduction neural network models have achieved remarkable results in many fields such as computer vision, natural language processing, and speech. in order to obtain adequate expressivity, the models usually come with a large number of parameters. however, such models are prone to overfitting if trained with an insufficient amount of training data. data augmentation (da) is an effective technique that has been used in many areas to augment existing labeled data and boost the performance of machine learning models. for example, in computer vision, training data is often augmented by ways such as horizontal flipping, random cropping, tilting, and color shifting @xcite . while da has become a standard technique to train deep networks for image processing, it is relatively under-explored in natural language processing (nlp). the exact mechanisms and theoretical foundations of data augmentation are still under investigation. most studies show empirically that data augmentation is effective and provide some intuitive explanations. a recent work in the field of vision @xcite demonstrates that affinity (the distributional shift caused by da) and diversity (the complexity of the augmentation) can predict the performance of data augmentation methods. however, neither metric can be measured without completing the entire da process. therefore, it is still challenging to evaluate the goodness of a da technique without full-fledged experimentation and it is not clear how the result can be used to guide the design of data augmentation schemes. generally speaking, da can be classified into two categories. the first tries to produce realistic samples that resemble the inherent semantics of naturally generated data. in areas such as computer vision, this is often achieved via heuristics that mimic the intrinsic processes that could have actually happened in the physical world, such as photometric noise, flipping, and scaling, etc. the second perturbs the data in a stochastic fashion, resulting in unrealistic samples. some (e.g., @xcite ) interpret this as a type of regularization that boosts model performance by reducing overfitting. both are being exploited in nlp. this paper focuses on lexical replacement methods that augment the training data by altering existing sentences in the parallel corpus of a neural machine translation (nmt) system. we have observed frequently in practice, as well as in literature @xcite @xcite @xcite , that augmented data samples that preserve the semantics of the real labeled data increase the effective training size and are beneficial for model performance. we call this property semantic consistency. in the case of nmt, the training data comes in the form of a collection @xmath0, @xmath1. semantic consistency requires that (1) both source and target are fluent and grammatically correct in their respective languages; and (2) target is a high quality translation of source. ## related work from a technical perspective, previous work on data augmentation for nlp can be classified as either context-independent or context-dependent. context-independent approaches often apply predetermined, easy-to-compute transformations that depend solely on the word or sentence to be altered. not surprisingly, most of them are not semantically consistent. @xcite improves performance on many text classification tasks through a set of word level random perturbation operations, including random insertion, deletion, and swapping. similar ideas have been applied to nmt, but the methods differ in how and what to alter. swap @xcite randomly swaps words in nearby positions within a window size k and drop @xcite randomly drops word tokens. blank @xcite replaces the candidate word with a placeholder token and smooth @xcite replaces it with a word sampled from the frequency distribution of the vocabulary, showing that data noising is an effective regularizer for nmt. switchout, introduced in @xcite , formulates the design of a da algorithm as an optimization problem that maximizes an objective that encourages two desired properties: smoothness and diversity. switchout independently replaces words in both source and target by other words uniformly sampled from their respective vocabularies. others try to preserve a certain level of semantic consistency by replacing words with their synonyms selected from a handcrafted ontology such as wordnet @xcite or words based on similarity calculation @xcite . these works do not make use of important context and label information and, in practice, usually cause a very small or even negative impact on performance. context-dependent approaches, on the other hand, modify words, phrases, or the whole sentence based on their contextual information that is usually modeled using neural networks. we summarize a few representative ones below. @xcite propose a simple but effective approach to augment the training data of nmt for low-resource language pairs. their work uses shallow lstm language models (lm) trained on large amounts of monolingual data to first substitute a word in source, and then put the corresponding translation in target, using automatic word alignments and the traditional statistical mt practice. lm sample @xcite proposes contextual augmentation for text classification by offering a wide range of substitute words, which are predicted by a label-conditional bidirectional language model. @xcite retrofit bert to conditional bert that allows it to augment sentences without breaking the label-compatibility. the bert-based solution brings two benefits. first, bert's transformer core provides a more structured memory for handling long-term dependencies in text. second, bert, as a deep bidirectional model, is strictly more powerful than the shallow concatenation of left-to-right and right-to-left models. a recent work @xcite treats a translation language model as a causal model and performs data augmentation by counterfactual-based causal inference. their da replaces source phrases according to a masked language model and the aligned target phrase by a cross-lingual language model (xlm) @xcite conditional on the changed source phrase. different from their work, we use two separate cmlms to augment source and target respectively, which means that, instead of model prediction, the condition is always true information for both cmlms. we show its superiority in section 5.1. the way we incorporate augmented data into the nmt training is drawn from the idea of "soft" word introduced by sca @xcite . basically, the embedding of a chosen word in a sentence is replaced by its probabilistic distribution predicted by a language model. this brings in more diversity to the da process. however, @xcite as a da solution is based on a uni-directional language model and is not label-conditional. as we show in section 4.3 that this is less optimal. ## approach in this section, we present our method in detail. we first introduce conditional mlm, then we show how to apply cmlm to data augmentation in neural machine translation tasks. let (x, y ) be a pair of source and target sentences where @xmath2, x 2 , . . . , x m ) and @xmath3, y 2 , . . . , y n ) are two sequences of tokens in source and target languages, with lengths m and n , respectively. a neural machine translation system learns the conditional probability p(y 1 , y 2 , . . . , y n |x 1 , x 2 , . . . , x m ). ## experiments in this section, we demonstrate the effectiveness of our method on four datasets with diverse language variation. they include three relatively small-scale datasets, {german, spanish, hebrew} to english ({de, es, he}-> en) from the well-known iwslt 2014, and one large-scale english to german (en->de) dataset from wmt14 . ## analysis our method consists of multiple modules, and we design several groups of comparative experiments to analyze their effects. ## conclusion in this paper, we advocate performing semantically consistent data augmentation for neural machine translation and propose a scheme based on conditional masked language model and soft word substitution. we show that a deep, bi-directional cmlm is capable of enforcing semantic consistency by conditioning on both source and target during data augmentation. experiments demonstrate that the overall solution results in more realistic data augmentation and better translation quality. our approach consistently achieves the best performance in comparison with strong and recent works and yields improvements of up to 1.90 bleu points over baseline.