Learning to Generalize to More: Continuous Semantic Augmentation for Neural Machine Translation (2022)

Abstract: The principal task in supervised neural machine translation (NMT) is to learn to generate target sentences conditioned on the source inputs from a set of parallel sentence pairs, and thus produce a model capable of generalizing to unseen instances. However, it is commonly observed that the generalization performance of the model is highly influenced by the amount of parallel data used in training. Although data augmentation is widely used to enrich the training data, conventional methods with discrete manipulations fail to generate diverse and faithful training samples. In this paper, we present a novel data augmentation paradigm termed Continuous Semantic Augmentation (CsaNMT), which augments each training instance with an adjacency semantic region that could cover adequate variants of literal expression under the same meaning. We conduct extensive experiments on both rich-resource and low-resource settings involving various language pairs, including WMT14 English \rightarrow {German,French}, NIST Chinese \rightarrow English and multiple low-resource IWSLT translation tasks. The provided empirical evidences show that CsaNMT sets a new level of performance among existing augmentation techniques, improving on the state-of-the-art by a large margin. The core codes are contained in Appendix E.

## introduction neural machine translation (nmt) is one of the core topics in natural language processing, which aims to generate sequences of words in the target language conditioned on the source inputs @xcite @xcite . in the common supervised setting, the training objective is to learn a transformation from the source space to the target space @xmath0: f (y|x; θ) with the usage of parallel data. in this way, nmt models are expected to 1 the core codes are contained in appendix e. to address this problem, various methods have been developed to leverage abundant unlabeled data for augmenting limited labeled data @xcite @xcite @xcite @xcite . for example, backtranslation (bt) @xcite makes use of the monolingual data on the target side to synthesize large scale pseudo parallel data, which is further combined with the real parallel corpus in machine translation task. another line of research is to introduce adversarial inputs to improve the generalization of nmt models towards small perturbations @xcite @xcite @xcite . while these methods lead to significant boosts in translation quality, we argue that augmenting the observed training data in the discrete space inherently has two major limitations. first, augmented training instances in discrete space are lack diversity. we still take bt as an example, it typically uses beam search @xcite or greedy search @xcite to generate synthetic source sentences for each target monolingual sentence. the above two search strategies are approximate algorithms to identify the maximum a-posteriori (map) output @xcite , and thus favor the most frequent one in case of ambiguity. @xcite proposed a sampling strategy from the output distribution to alleviate this issue, but this method typically yields synthesized data with low quality. while some extensions @xcite @xcite augment each training instance with multiple literal forms, they still fail to cover adequate variants under the same meaning. second, it is difficult for augmented texts in dis-crete space to preserve their original meanings. in the context of natural language processing, discrete manipulations such as adds, drops, reorders, and/or replaces words in the original sentences often result in significant changes in semantics. to address this issue, @xcite and @xcite instead replace words with other words that are predicted using language model under the same context, by interpolating their embeddings. although being effective, these techniques are limited to word-level manipulation and are unable to perform the whole sentence transformation, such as producing another sentence by rephrasing the original one so that they have the same meaning. in this paper, we propose continuous semantic augmentation (csanmt), a novel data augmentation paradigm for nmt, to alleviate both limitations mentioned above. the principle of csanmt is to produce diverse training data from a semantically-preserved continuous space. specifically, (1) we first train a semantic encoder via a tangential contrast, which encourages each training instance to support an adjacency semantic region in continuous space and treats the tangent points of the region as the critical states of semantic equivalence. this is motivated by the intriguing observation made by recent work showing that the vectors in continuous space can easily cover adequate variants under the same meaning @xcite . we evaluate our framework on a variety of machine translation tasks, including wmt14 english-german/french, nist chinese-english and multiple iwslt tasks. specifically, csanmt sets the new state of the art among existing augmentation techniques on the wmt14 english-german task with 30.94 bleu score. in addition, our approach could achieve comparable performance with the baseline model with the usage of only 25% of training data. this reveals that csanmt has great potential to achieve good results with very few data. furthermore, csanmt demonstrates consistent improvements over strong baselines in low resource scenarios, such as iwslt14 english-german and iwslt17 english-french. ## framework problem definition supposing x and y are two data spaces that cover all possible sequences of words in source and target languages, respectively. we denote (x, y) ∈ (x , y) as a pair of two sentences with the same meaning, where @xmath1, x 2 , ..., x t } is the source sentence with t tokens, and @xmath2, y 2 , ..., y t ′ } is the target sentence with t ′ tokens. a sequence-tosequence model is usually applied to neural machine translation, which aims to learn a transformation from the source space to the target space @xmath3: f (y|x; θ) with the usage of parallel data. formally, given a set of observed sentence pairs the log-probability is typically decomposed as: log p (y|x; θ) = t ′ @xmath4, x; θ), where θ is a set of trainable parameters and @xmath5. however, there is a major problem in the common supervised setting for neural machine translation, that is the number of training instances is very limited because of the cost in acquiring parallel data. this makes it difficult to learn an nmt model generalized well to unseen instances. traditional data augmentation methods generate more training samples by applying discrete manipulations to unlabeled (or labeled) data, such as back-translation or randomly replacing a word with another one, which usually suffer from the problems of semantic deviation and the lack of diversity. ## experiments we first apply csanmt to nist chinese-english (@xmath6, wmt14 english-german (@xmath7, and conduct extensive analyses for better understanding the proposed method. and then we generalize the capability of our method to low-resource iwslt tasks. ## conclusion we propose a novel data augmentation paradigm csanmt, which involves with an adjacency semantic region as the vicinity manifold for each training instance. this method is expected to make more unseen instances under generalization with very limited training data. the main components of csanmt consists of the tangential contrastive learning and the mixed gaussian recurrent chain (mgrc) sampling. experiments on both rich-and low-resource machine translation tasks demonstrate the effectiveness of our method. in the future work, we would like to further study the vicinal risk minimization with the combination of multi-lingual aligned scenarios and large-scale monolingual data, and development it as a pure data augmentator merged into the vanilla transformer. marc'aurelio ranzato, sumit chopra, michael auli, and wojciech zaremba. 2016. sequence level training with recurrent neural networks. in 4th international conference on learning representations, iclr 2016, san juan, puerto rico, may 2-4, 2016, conference track proceedings. nils reimers and iryna gurevych. 2019. sentence-bert: sentence embeddings using siamese bertnetworks. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 3982-3992. association for computational linguistics. holger schwenk and matthijs douze. 2017. learning joint multilingual sentence representations with neural machine translation. in proceedings of the 2nd workshop on representation learning for nlp, pages 157-167, vancouver, canada. association for computational linguistics. rico sennrich, barry haddow, and alexandra birch. 2016a. improving neural machine translation models with monolingual data. in proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers), pages 86-96. association for computational linguistics. rico sennrich, barry haddow, and alexandra birch. 2016b. neural machine translation of rare words with subword units. in proceedings of the 54th annual meeting of the association for computational linguistics, pages 1715-1725, berlin, germany. association for computational linguistics. karen simonyan and andrew zisserman. 2014. very deep convolutional networks for large-scale image recognition. arxiv preprint arxiv:1409.1556. ziyi yang, yinfei yang, daniel cer, jax law, and eric darve. 2021. universal sentence representation learning with conditional masked language model. in proceedings of the 2021 conference on empirical methods in natural language processing, pages 6216-6228. association for computational linguistics. biao zhang, deyi xiong, jinsong su, hong duan, and min zhang. 2016. variational neural machine translation. in proceedings of the 2016 conference on empirical methods in natural language processing, pages 521-530. association for computational linguistics. hongyi zhang, moustapha cisse, yann n. dauphin, and david lopez-paz. 2018. mixup: beyond empirical risk minimization. in international conference on learning representations. wenzhao zheng, zhaodong chen, jiwen lu, and jie zhou. 2019. hardness-aware deep metric learning. in proceedings of the ieee/cvf conference on computer vision and pattern recognition (cvpr). jinhua zhu, yingce xia, lijun wu, di he, tao qin, wengang zhou, houqiang li, and tieyan liu. 2020. incorporating bert into neural machine translation. in international conference on learning representations. a details of rich-resource datasets for the @xmath8, the ldc corpus foot_0 is taken into consideration, which consists of 1.25m sentence pairs with 27.9m chinese words and 34.5m english words, respectively. the nist 2006 dataset is used as the validation set for selecting the best model, and nist 2002 (mt02), 2003 (mt03), 2004 (mt04), 2005 (mt05), 2008 (mt08) are used as the test sets. we created shared bpe (bytepair-encoding (sennrich et al., 2016b)) codes with 60k merge operations to build two vocabularies comprising 47k chinese sub-words and 30k english sub-words. for the @xmath9, we employ the popular wmt14 dataset, which consists of approximately 4.5m sentence pairs for training. we select newstest2013 as the validation set and newstest2014 as the test set. all sentences had been jointly byte-pair-encoded with 32k merge operations, which results in a shared source-target vocabulary of about 37k tokens. for the @xmath10, we use the significantly larger wmt14 dataset consisting of 36m sentence pairs. the combination of {newstest2012, 2013} was used for model selection and the experimental results were reported on newstest2014.