Deterministic Reversible Data Augmentation for Neural Machine Translation (2024)

Abstract: Data augmentation is an effective way to diversify corpora in machine translation, but previous methods may introduce semantic inconsistency between original and augmented data because of irreversible operations and random subword sampling procedures. To generate both symbolically diverse and semantically consistent augmentation data, we propose Deterministic Reversible Data Augmentation (DRDA), a simple but effective data augmentation method for neural machine translation. DRDA adopts deterministic segmentations and reversible operations to generate multi-granularity subword representations and pulls them closer together with multi-view techniques. With no extra corpora or model changes required, DRDA outperforms strong baselines on several translation tasks with a clear margin (up to 4.3 BLEU gain over Transformer) and exhibits good robustness in noisy, low-resource, and cross-domain datasets.

## introduction recent neural machine translation (nmt) models have led to dramatic improvements in translation quality. however, the powerful learning and memorizing ability of these models also leads to poor generalization and vulnerability to small perturbations like misspelling and paraphrasing @xcite . a common solution to perturbation vulnerability is data augmentation @xcite , which is to create massive virtual training data with diverse symbolic representations under the premise of ensuring semantic consistency @xcite . symbolic diversity emphasizes that original and augmented data should differ significantly in token sequences, and semantic consistency requires that the two should be semantically similar. previous data augmentation methods employ irreversible substitutions, like direct dropping or replacing discrete tokens to generate diverse data (figure 1 a). despite being able to improve data diversity, these augmentation operations are not reversible, and will inevitably introduce semantic loss to original texts, thus compromising the semantic consistency between original and augmented data. yet another way to generate diverse augmentation data without employing irreversible operations is subword regularization @xcite . subword regularization adopts random segmentations to sample subwords probabilistically thus generating diverse data. these methods are reversible because of the inherent reversibility of segmentations. however, due to the random sampling procedure of segmentation, they may adopt inappropriate subword segmentations (e.g., "sup erm ark et" in figure 1 b ). these sub-optimal segmentations may result in semantic perturbations and do damage to semantic consistency. to summarize, previous methods have difficulty in completely retaining the semantics from corruption when diversifying the texts because of irreversible augmentation operations and probabilistic subword sampling. to generate symbolically diverse and semantically consistent data, we propose deterministic reversible data augmentation (drda), a simple but effective augmentation approach. drda augments source sentences with their token representations in different granularities as shown in figure 1 c. these representations are symbolically diverse, but also syntactically correct and semantically complete thanks to the reversible and deterministic segmentations in the multi-granularity segmentation process. to make full use of the semantic identity among all multi-granularity representations of one sentence, we also leverage the multi-view techniques in training to pull these representations closer together. we conduct extensive experiments of different languages and scales and find that drda gains consistent improvements over strong baselines with clear margins. to further understand the factors that make drda work, we conduct insightful analyses of the effects drda imposed on semantic consistency, subword frequency, and subword semantic composition. we combine the empirical and theoretical verification of the consistency and offer a subword-level explanation of the mechanism of multi-granularity segmentations and multi-view techniques. our contributions are summarized as follows: ## related work augmentation methods besides continuous approaches @xcite , data augmentation can be categorized into back-translation like methods @xcite @xcite and token substitution methods. drda is an instance of the latter category. several substitution methods uniformly select a word or token in a sentence and perform deletion or substitution @xcite @xcite @xcite . @xcite constrained the substitution of a word in a small subset of synonyms, thus improving the semantics consistency. kambhatla et al. (2022b ) viewed the original corpus as plain text and applies a rotation encryption as data augmentation. unlike previous methods, introducing multi-granularity takes advantage of the reversible nature of segmentation and causes no semantic loss. ## background: subword segmentation subword segmentation models the probability of token sequence @xmath0, x 2 , ..., x m given a source sentence s. previous deterministic subword segmentations choose the most probable sample: for example, byte pair encoding (bpe) assigns p (x|s; p) = 1 when x is obtained from the greedy merge process @xcite . to generate different segmentations for one word, subword regularization methods draw a segmentation from the segmentation distribution probabilistically: for example, @xcite makes use of a unigram language model to sample segmentations on, and provilkov et al. (2020) randomly interrupts the bpe merging process to generate multiple segmentations. ## deterministic reversible data augmentation previous data augmentation and subword regularization approaches take irreversible operation (like discrete token substitution) and probabilistic segmentation sampling, which may introduce semantic loss or inappropriate subwords, thus affecting the semantic consistency. our objective is to ensure the semantic consistency between original and augmented data when generating diverse data. we propose drda to generate augmentation data without introducing semantic perturbations. drda augments original data with multigranularity segmentations, and pulls representations of one sentence closer with multi-view learning. furthermore, we propose a dynamic selection technique to automatically choose an appropriate granularity in inference. ## experiments we evaluate drda with translation tasks in different language pairs and translation directions to show its universal property regardless of language features. we also conduct experiments on extremely low resources and noisy scenarios to show the robustness of drda. significance is computed via bootstrapping @xcite using compare-mt @xcite . ## analysis in this section, we conduct analysis experiments to answer the following research questions (rqs) respectively: ## conclusion in this paper, we identify the semantic inconsistency caused by irreversible operations or probabilistic segmentations, and propose a deterministic reversible data augmentation consisting of multigranularity segmentation and multi-view learning to ensure the consistency when generating diverse data. experiments demonstrate the superiority of our proposed drda over previous data augmentation and subword regularization in terms of translation accuracy and robustness. we also offer a combination of empirical and theoretical verification of semantic consistency, and insightful analyses about multi-granularity and multi-view techniques.