Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation (2022)

Abstract: Knowledge distillation (KD) is the preliminary step for training non-autoregressive translation (NAT) models, which eases the training of NAT models at the cost of losing important information for translating low-frequency words. In this work, we provide an appealing alternative for NAT – monolingual KD , which trains NAT student on external monolingual data with AT teacher trained on the original bilingual data. Monolingual KD is able to transfer both the knowledge of the original bilingual data (implicitly encoded in the trained AT teacher model) and that of the new monolingual data to the NAT student model. Extensive experiments on eight WMT benchmarks over two advanced NAT models show that monolingual KD consistently outperforms the standard KD by improving low-frequency word translation, without introducing any computational cost. Monolingual KD enjoys desirable expandability, which can be further enhanced (when given more computational budget) by combining with the standard KD, a reverse monolingual KD, or enlarging the scale of monolingual data. Extensive analyses demonstrate that these techniques can be used together profitably to further recall the useful information lost in the standard KD. Encouragingly, combining with standard KD, our approach achieves 30.4 and 34.1 BLEU points on the WMT14 English-German and German-English datasets, respectively. Our code and trained models are freely available at https://github.com/alphadl/RLFW-NAT.mono .

## introduction non-autoregressive translation (nat, @xcite has been proposed to improve the decoding efficiency by predicting all tokens independently and simultaneously. however, the independence assumption prevents a model from properly capturing the highly multimodal distribution of target translations. in response to this problem, a sequencelevel knowledge distillation (kd, @xcite becomes the preliminary step for training nat models, which produces more deterministic knowledge by reducing the translation modes of the bilingual data @xcite . although the standard kd on original bilingual data eases the training of nat models, distillation may lose some important information in the raw training data, leading to more errors on predicting low-frequency words @xcite . to remedy this problem, @xcite augmented nat models the ability to learn lost knowledge from the raw bilingual data with an additional objective, and @xcite first pre-trained nat models on the raw training data and then finetuned them on the distilled training data. while previous studies mainly focus on recalling the lost information during the distillation of the original bilingual data, in this work we propose to improve the prediction of low-frequency words by redistributing them in the external monolingual data, which has the great potential to complement the original bilingual data on the word distribution. specifically, we leverage the monolingual data to perform kd (monolingual kd, §2.2), and train the nat student model on the distilled monolingual data (figure 1b ). monolingual kd provides appealing benefits. firstly, the monolingual data and bilingual data in machine translation are generally complementary to each other @xcite @xcite @xcite . accordingly, monolingual kd is able to transfer both the knowledge of the bilingual data (implicitly encoded in the trained teacher model) and that of the monolingual data to the nat student, without introducing additional computational cost. secondly, the amount of available monolingual data is several orders of magnitude larger than that of bilingual data, which offers monolingual kd the potential to further improve translation performance by exploiting more monolingual data. furthermore, we analyze the bilingual links in the bilingual and monolingual distilled data from two alignment directions (i.e. source-to-target and target-to-source). we found that the monolingual kd makes low-frequency source words aligned with targets more deterministically compared to bilingual kd, but both of them fail to align lowfrequency words from target to source due to information loss. starting from this finding, we propose reverse monolingual kd to recall more alignments for low-frequency target words. we then concatenate two kinds of monolingual distilled data (bidirectional monolingual kd, §2.3) to maintain advantages of deterministic knowledge and lowfrequency information. we validated our approach on several translation benchmarks across scales (wmt14 en↔de, wmt16 ro↔en, wmt17 zh↔en, and wmt19 en↔de) over two advanced nat models: mask predict @xcite and levenshtein @xcite . experiments demonstrate the effectiveness and universality of our approach. specifically, we have the following findings: the paper is an early step in exploring monolingual kd for nat, which can narrow the performance gap between nat models and the sota at models. we hope the promising effect of monolingual kd on nat can draw more interest and can make nat a common translation framework. ## related work to bridge the performance gap, a number of recent efforts have explored, including model architectures @xcite @xcite , training objectives and methods @xcite @xcite . another thread of work focus on understanding and improving distillation training for nat @xcite @xcite . sequence-level kd @xcite is a preliminary step for training nat models to reduce the intrinsic uncertainty and learning difficulty @xcite . recent studies have revealed that kd reduces the modes (i.e. multiple lexical choices for a source word) in the original data by re-weighting the training examples @xcite , at the cost of losing some important information, leading to more errors on predicting low-frequency words @xcite . in response to this problem, @xcite proposed to rejuvenate low-frequency words by pretraining nat models on the raw bilingual data. in this study, we attempt to solve this problem from a different perspectiverediscovering low-frequency words from external monolingual data, which can simultaneously exploit the knowledge of bilingual data (implicitly encoded in the parameters of at teacher). closely related to our work, zhou and keung (2020) improved nat models by augmenting source-side monolingual data. their work can be regarded as a special case of our approach (i.e. "mono. kd + standard kd" in section 3.3), and our work has several more contributions. firstly, we demonstrated the effectiveness of using only monolingual kd for nat models, which can achieve better performance than the standard kd without introducing any computational cost. secondly, we proposed a novel bidirectional monolingual kd to exploit both the source-side and target-side monolingual data. finally, we provide insights into how monolingual kd complements the standard kd. ## conclusion in this work, we propose a simple, effective and scalable approach -monolingual kd to redistribute the low-frequency words in the bilingual data using external monolingual data. monolingual kd consistently outperforms the standard kd with more translation accuracy of low-frequency words, which attribute to its strength of exploiting both the knowledge of the original bilingual data (implicitly encoded in the parameters of at teacher) and that of the new monolingual data. monolingual kd enjoys appealing expandability, and can be further enhanced by (1) combining with a reverse monolingual kd to recall more alignments for low-frequency target words; (2) combining with the standard kd to explicitly combine both types of complementary knowledge; (3) enlarging the scale of monolingual data that is cheap to acquire. our study empirically indicates the potential to make nat a practical translation system. future directions include designing advanced monolingual kd techniques and validating on larger-capacity nat models (e.g. big setting) to strengthen the power of monolingual kd, and fully nat models @xcite to show the universality of monolingual kd. besides, it will be interesting to follow @xcite and @xcite to investigate the complementarity between our monolingual kd and pretrained language models to further enhance the nat models.