Improving Non-autoregressive Neural Machine Translation with Monolingual Data (2020)

Abstract: Non-autoregressive (NAR) neural machine translation is usually done via knowledge distillation from an autoregressive (AR) model. Under this framework, we leverage large monolingual corpora to improve the NAR model’s performance, with the goal of transferring the AR model’s generalization ability while preventing overfitting. On top of a strong NAR baseline, our experimental results on the WMT14 En-De and WMT16 En-Ro news translation tasks confirm that monolingual data augmentation consistently improves the performance of the NAR model to approach the teacher AR model’s performance, yields comparable or better results than the best non-iterative NAR methods in the literature and helps reduce overfitting in the training process.

## introduction neural machine translation (nmt) @xcite has achieved impressive performance in recent years, but the autoregressive decoding process limits the translation speed and restricts low-latency applications. to mitigate this issue, many non-autoregressive (nar) translation methods have been proposed, including latent space models @xcite @xcite , iterative refinement methods @xcite , and alternative loss functions @xcite @xcite @xcite . the decoding speedup for nar models is typically 2-15× depending on the specific setup (e.g., the number of length candidates, number of latent samples, etc.), and nar models can be tuned to achieve different trade-offs between time complexity and decoding quality @xcite @xcite . although different in various aspects, all of these methods are based on transformer modules @xcite , and depend on a well-trained ar model to obtain its output translations to create targets for nar model training. this training setup is well-suited to leverage external monolingual data, since the target side of the nar training corpus is always generated by an ar model. techniques like backtranslation @xcite are known to improve mt performance using monolingual data alone. however, to the best of our knowledge, monolingual data augmentation for nar-mt has not been reported in the literature. in typical nar-mt model training, an ar teacher provides a consistent supervision signal for the nar model; the source text that was used to train the teacher is decoded by the teacher to create synthetic target text. in this work, we use a large amount of source text from monolingual corpora to generate additional teacher outputs for nar-mt training. we use a transformer model with minor structural changes to perform nar generation in a noniterative way, which establishes stronger baselines than most of the previous methods. we demonstrate that generating additional training data with monolingual corpora consistently improves the translation quality of our baseline nar system on the wmt14 en-de and wmt16 en-ro translation tasks. furthermore, our experiments show that nar models trained with increasing amount of extra monolingual data are less prone to overfitting and generalize better on longer sentences. in addition, we have obtained @xmath0, just by using more monolingual data. ## nar-mt with monolingual data augmenting the nar training corpus with monolingual data provides some potential benefits. firstly, we allow more data to be translated by the ar teacher, so the nar model can see more of the ar translation outputs than in the original training data, which helps the nar model generalize better. secondly, there is much more monolingual data than parallel data, especially for low-resource languages. incorporating monolingual data for nar-mt is straightforward in our setup. given an ar model that we want to approximate, we obtain the sourceside monolingual text and use the ar model to generate the targets that we can train our nar model on. ## discussion we found that monolingual data augmentation reduces overfitting and improves the translation quality of nar-mt models. we note that the monolingual corpora are derived from domains which may be different from those of the parallel training data or evaluation sets, and a mismatch can affect nar translation performance. other work in nmt has examined this issue in the context of backtranslation (e.g., @xcite ), and we expect the conclusions to be similar in the nar-mt case. there are several open questions to investigate: are the benefits of monolingual data orthogonal to other techniques like iterative refinement? can the nar model perfectly recover the ar model's performance with much larger monolingual datasets? are the observed improvements language-dependent? we will consider these research directions in future work.