Exploring the Benefits and Limitations of Multilinguality for Non-autoregressive Machine Translation (2022)

Abstract: Non-autoregressive (NAR) machine translation has recently received significant developments and now achieves comparable quality with autoregressive (AR) models on some benchmarks while providing an efficient alternative to AR inference. However, while AR translation is often used to implement multilingual models that benefit from transfer between languages and from improved serving efficiency, multilingual NAR models remain relatively unexplored. Taking Connectionist Temporal Classification as an example NAR model and IMPUTER as a semi-NAR model, we present a comprehensive empirical study of multilingual NAR. We test its capabilities with respect to positive transfer between related languages and negative transfer under capacity constraints. As NAR models require distilled training sets, we carefully study the impact of bilingual versus multilingual teachers. Finally, we fit a scaling law for multilingual NAR to determine capacity bottlenecks, which quantifies its performance relative to the AR model as the model scale increases.

## introduction non-autoregressive (nar) models generate output tokens in parallel instead of sequentially, reducing potentially expensive inference dependencies. they rely on sequence-level knowledge distillation to reach the quality of autoregressive (ar) models @xcite . as the notion of nar has expanded to include semi-nar models that generate their outputs in multiple steps, with each step generating several tokens non-autoregressively @xcite , we have begun to see nar matching the quality of ar @xcite . prior works have benchmarked nar models for machine translation (mt) on a number of language pairs, but with very few exceptions, the nar models under test have been bilingual as opposed to multilingual. multilingual mt models @xcite @xcite , translating between multiple languages, have two major advantages. first, they offer better parameter efficiency than bilingual models via multi-tasking. second, they are able to transfer knowledge from highresource languages to low-resource ones. therefore they have become an attractive solution for expanding the language coverage of ar mt @xcite @xcite . the capability of multilingual modeling is a major feature of the ar regime, and it is one that we should seek to maintain in nar models. however, it is unclear to what extent the benefits of multilingual ar models transfer to nar modeling @xcite . do related languages help each other as easily (positive transfer)? do unrelated languages interfere with one another more (negative transfer)? furthermore, nar modeling raises a new issue of multilingual distillation. to retain the training-time efficiency of multilingual modeling, it is crucial that nar works well with multilingual teachers; otherwise, the prospect of training many bilingual teachers would greatly increase the effective training cost. it may actually be the case that multilingual teachers are better suited than bilingual ones, as the effective capacity reduction may result in less complex @xcite and less multi-modal outputs @xcite . we present an empirical study of multilingual nar modeling. taking ctc @xcite as our canonical nar method, and imputer @xcite as our canonical semi-nar model, we study how they respond to multilinguality through a series of "stress-tests", first in a six-language scenario designed to emphasize negative transfer ( §4), and then in twolanguage scenarios designed to emphasize positive transfer under data resource constraints ( §5). lastly, we fit a scaling law for our six-language sce-nario to measure the potential of increasing model sizes ( §6). the main findings can be summarized as follows: our extensive analysis on outputs from the nar models suggest that they still struggle to generate "valid" tokens with desired output length. furthermore, our results indicate that scaling up the nar models is not going to close the gap to multilingual ar, but our analysis points to promising directions for future work throughout the paper. ## non-autoregressive multilingual nmt let, d @xmath0, y) ∈ x × y denote the bilingual corpus of a language pair, l. given an input sequence x of length t ′ , an ar model @xcite predicts the target y with length t sequentially based on the conditional distribution p(y t @xmath1, x 1∶t ′ ; θ). nar models assume conditional independence in the output token space; that is, they model p(y t x 1∶t ′ ; φ). due to this conditional independence assumption, training nar models directly on the true target distribution leads to degraded performance @xcite . hence, nar models are typically trained with sequencelevel knowledge distillation @xcite to reduce the modeling difficulty. ## experimental setup data we perform our main experiments on six language pairs, translating from english into wmt-14 german (de) @xcite , wmt-15 french (fr) @xcite , wmt-19 russian (ru) @xcite , wmt-20 japanese (ja), wmt-20 polish (pl) @xcite and samanantar hindi (hi) @xcite . the lower-resourced wmt-19 english-kazakh (kk) @xcite is used for an additional transfer experiment in section 5. the properties of the datasets are listed in evaluation metrics translation quality is evaluated with bleu @xcite as calculated by sacrebleu @xcite with default tokenization ("13a") except for en-ja, where we use character-level tokenization. 3 architecture we train the imputer model using the same setup as described in @xcite : we follow their base model with d @xmath2, d @xmath3, n @xmath4, n @xmath5, and p @xmath6.1. ar models follow transformerbase @xcite and have similar parameter counts. we train both models using adam with learning rate of 0.0001. we train ctc models with a batch size of 2048 and 8192 sentences for 300k steps for the bilingual and multilingual models respectively. we train the imputer using ctc loss using a bernoulli masking policy for next 300k steps with a batch size of 1024 and 2048 sentences for the bilingual and multilingual models respectively. we upsample the source sequence by a factor of 2 for all our experiments. 4 we pick the best checkpoint based on validation bleu for bilingual models, and the last checkpoint for multilingual models, following @xcite . ## negative transfer scenario our main experiment compares bilingual, multilingual, ar and nar models for the six highresource languages from ## positive transfer scenario in this section we present two experimental setups designed to emphasize positive transfer, where languages are related and training data is limited. @xmath7, french} to isolate the effect of transfer via multilingual modelling, we relax the capacity bottleneck and competition for parameters: we combine the two most related languages (de, fr) @xcite , figure 2 ) and give them smaller, balanced training sets (1m sentences). we compare bilingual and multilingual ar and nar models trained on this reduced data. ## impact of model scale we hypothesized in section 4 that ctc might require more capacity than ar models. if we increase the parameters for nar models sufficiently, could we reach ar quality? scaling laws can characterize the relationship between mt quality, the cross-entropy loss and the number of parameters used for training the model @xcite . we derive the relationship between bleu and the number of parameters for our ar and ctc models directly from the scaling laws proposed by ## related work multiple approaches with varying architectures @xcite @xcite , custom loss functions @xcite and training strategies @xcite 8 size for experiments in section 4. have been used to enable parallel generation of output tokens for mt with sequence-level knowledge distillation as one of the key ingredient in the training of nar models. both supervised, and unsupervised @xcite mt have benefitted from training with multiple languages, especially those that have tiny @xcite to no training data @xcite . however, multilingual modeling has not yet received any attention in the nar literature, which we explore in this work. one limitation of our study is that we choose one representative system for nar and semi-nar modeling over the full breadth of nar options. ## conclusion multilingual translation is a valuable feature of ar models, therefore, we have tested nar models for that same capability. we focus on challenging scenarios to discover potential weaknesses and to identify areas for future work. in a relaxed setting with little interference between languages and balanced data, multilingual nar models nicely exhibit positive transfer, practically closing the gap to ar models with a few decoding iterations. however, we do not see the same positive transfer in a true low-resource scenario. experiments in a sixlanguage scenario reveal that multilingual nar models suffer proportionally more from negative interference than ar models. our derived scaling laws show that scaling up ctc model parameters is not a sufficient remedy. our analysis identified two issues that hurt translation quality and worsen with multilinguality, namely output length control and the generation of invalid words. we have also shown beneficial properties of using multilingual teachers for distillation. we hope that this work will serve as a call for increased focus on multilingual modeling in nar research.