Revisiting Non-Autoregressive Translation at Scale (2023)

Abstract: In real-world systems, scaling has been critical for improving the translation quality in autoregressive translation (AT), which however has not been well studied for non-autoregressive translation (NAT). In this work, we bridge the gap by systematically studying the impact of scaling on NAT behaviors. Extensive experiments on six WMT benchmarks over two advanced NAT models show that scaling can alleviate the commonly-cited weaknesses of NAT models, resulting in better translation performance. To reduce the side-effect of scaling on decoding speed, we empirically investigate the impact of NAT encoder and decoder on the translation performance. Experimental results on the large-scale WMT20 En-De show that the asymmetric architecture (e.g. bigger encoder and smaller decoder) can achieve comparable performance with the scaling model, while maintaining the superiority of decoding speed with standard NAT models. To this end, we establish a new benchmark by validating scaled NAT models on the scaled dataset, which can be regarded as a strong baseline for future works. We release code and system outputs at https://github.com/DeepLearnXMU/Scaling4NAT .

## introduction recent years have seen a surge of interest in non-autoregressive translation (nat) @xcite , which can improve the decoding efficiency by predicting all tokens independently and simultaneously. the majority studies on nat focus on the base models trained on medium-scale datasets (e.g., mask-predict: 69m; wmt14 en-de: 4.5m) @xcite , while scaled models and datasets become de facto standard for autoregressive translation (at) models (e.g., transformer-big: 226m; wmt20 en-de: 45.1m) @xcite . the model-and datalevel gaps make the progress of nat lag behind that of at, which limits the applicability of nat models to practical scenarios. this general tendency motivates us to boost nat models from the scaling perspective, including the amounts of training data and the model size. in this paper, we aim to provide empirical answers to the following research questions: to answer these research questions, we investigate the effects of different scaling methods on two advanced nat models. experimental results show that scaling works well with knowledge distillation to alleviate commonly-cited weaknesses of nat. the scaled nat models achieve better translation quality at the expense of decreasing decoding speed. to balance effectiveness and efficiency, we com-pare various component-scaled nat models and find that scaling architecture in nat is more asymmetric than that in at. accordingly, we introduce a cone architecture for nat with a deeper and wider encoder and a shallower and narrower decoder, which boosts translation performance and maintain the decoding speed. specifically, our main contributions are as follows: ## scaling behaviors of nat models we investigate effects of scaling from three perspectives: translation quality, commonly-cited weaknesses and decoding efficiency. the settings are: • model scaling: based on traditional nat-base configurations ((6,6)×512), we conduct 1) width scaling, where the size of feed-forward dimensions are enlarged to 1024 (nat-big: (6,6)×1024); 2) depth scaling, where the number of stacked encoder-decoder layers is increased to to 24-24 (nat-deep: (24,24)×512)). we mainly investigate behaviors of nat-big as it has similar performance with nat-deep, while the training of nat-big is more stable. ## new nat benchmark most nat models are implemented upon the encoder-decoder framework, where the encoder summarizes the source sentence and the decoder learns to generate target words. we ask: how to scale this framework? in this section, we empirically search for a better nat architecture by considering both effectiveness and efficiency. ## conclusion and future work in this study we target bridging the gap of model and data scale between nat and at models by investigating the scaling behaviors of nat models. we find that simply scaling nat models (nat-big) can significantly improve translation performance, especially on large-scale training data. cally study the contributions of scaling encoder and scaling decoder, and find that scaling nat is more asymmetric than at. based on the observations, we design a new scaling architecture with deeper and wider encoder and shallower and narrower decoder (nat-cone), which achieves comparable performance with nat-big without scarifying decoding speed. our study empirically indicates the potential to make nat a practical translation system as its at counterpart. however, the sota nat models (including scaling nat) still rely on the distillation by an at teacher. future work will investigate better techniques to train scaled nat models from scratch (i.e. without distillation). we additionally experiment larger nat models in appendix §a.2, which can be regarded as a preliminary experiments for this. we will also explore scaling nag models in other nlp tasks, such as keyphrase generation @xcite ) and text-to-table generation @xcite . the advent of large language models (llms) like gpt-4 has ushered in a new era in mt @xcite @xcite . this innovation is causing us to reconsider conventional paradigms, especially with regards to nat models.