Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation (2021)

Abstract: Knowledge distillation (KD) is commonly used to construct synthetic data for training non-autoregressive translation (NAT) models. However, there exists a discrepancy on low-frequency words between the distilled and the original data, leading to more errors on predicting low-frequency words. To alleviate the problem, we directly expose the raw data into NAT by leveraging pretraining. By analyzing directed alignments, we found that KD makes low-frequency source words aligned with targets more deterministically but fails to align sufficient low-frequency words from target to source. Accordingly, we propose reverse KD to rejuvenate more alignments for low-frequency target words. To make the most of authentic and synthetic data, we combine these complementary approaches as a new training strategy for further boosting NAT performance. We conduct experiments on five translation benchmarks over two advanced architectures. Results demonstrate that the proposed approach can significantly and universally improve translation quality by reducing translation errors on low-frequency words. Encouragingly, our approach achieves 28.2 and 33.9 BLEU points on the WMT14 English-German and WMT16 Romanian-English datasets, respectively. Our code, data, and trained models are available at https://github.com/longyuewangdcu/RLFW-NAT .

## introduction recent years have seen a surge of interest in nonautoregressive translation (nat, @xcite , which can improve the decoding efficiency by predicting all tokens independently and simultaneously. the non-autoregressive factorization breaks conditional dependencies among output tokens, which prevents a model from properly capturing the highly multimodal distribution of target translations. as a result, the translation quality of nat models often lags behind that of autoregressive translation (at, @xcite models. to balance the trade-off between decoding speed and translation quality, knowledge distillation (kd) is widely used to construct a new training data for nat models @xcite . specifically, target sentences in the distilled training data are generated by an at teacher, which makes nat easily acquire more deterministic knowledge and achieve significant improvement @xcite . previous studies have shown that distillation may lose some important information in the original training data, leading to more errors on predicting low-frequency words. to alleviate this problem, @xcite proposed to augment nat models the ability to learn lost knowledge from the original data. however, their approach relies on external resources (e.g. word alignment) and human-crafted priors, which limits the applicability of the method to a broader range of tasks and languages. accordingly, we turn to directly expose the raw data into nat by leveraging pretraining without intensive modification to model architectures ( §2.2). furthermore, we analyze bilingual links in the distilled data from two alignment directions (i.e. source-to-target and target-to-source). we found that kd makes low-frequency source words aligned with targets more deterministically but fails to align low-frequency words from target to source due to information loss. inspired by this finding, we propose reverse kd to recall more alignments for low-frequency target words ( §2.3). we then concatenate two kinds of distilled data to maintain advantages of deterministic knowledge and low-frequency information. to make the most of authentic and synthetic data, we combine three complementary approaches (i.e. raw pretraining, bidirectional distillation training and kd finetuning) as a new training strategy for further boosting nat performance ( §2.4). we validated our approach on five translation benchmarks (wmt14 en-de, wmt16 ro-en, wmt17 zh-en, wat17 ja-en and wmt19 en-de) over two advanced architectures (mask predict, @xcite . experimental results show that the proposed method consistently improve translation performance over the standard nat models across languages and advanced nat architectures. extensive analyses confirm that the performance improvement indeed comes from the better lexical translation accuracy especially on low-frequency tokens. contributions our main contributions are: ## conclusion in this study, we propose simple and effective training strategies to rejuvenate the low-frequency information in the raw data. experiments show that our approach consistently and significantly improves translation performance across language pairs and model architectures. notably, domain shift is an extreme scenario to diagnose low-frequency translation, and our method significant improves them. extensive analyses reveal that our method improves the accuracy of lexical choices for low-frequency source words, recalling more low-frequency words in translations as well, which confirms our claim.