CUNI Non-Autoregressive System for the WMT 22 Efficient Translation Shared Task (2022)

Abstract: We present a non-autoregressive system submission to the WMT 22 Efficient Translation Shared Task. Our system was used by Helcl et al. (2022) in an attempt to provide fair comparison between non-autoregressive and autoregressive models. This submission is an effort to establish solid baselines along with sound evaluation methodology, particularly in terms of measuring the decoding speed. The model itself is a 12-layer Transformer model trained with connectionist temporal classification on knowledge-distilled dataset by a strong autoregressive teacher model.

## introduction in the past few years, non-autoregressive (nar) models for neural machine translation (nmt) attracted interest from the research community @xcite . given the conditional independence between the output states, the decoding process can be parallelized across time steps. in theory, this leads to higher decoding speeds. since efficient decoding is claimed to be the main motivation of non-autoregressive models, the efficient translation shared task seems to be the appropriate venue to provide fair comparison between these models and their autoregressive counterparts. however, all submissions to this task were autoregressive so far @xcite @xcite . recently, @xcite pointed out common flaws in the evaluation methodology of nar models. we found that optimized autoregressive models still achieve superior performance over nar models. the only scenario where nar models showed some potential is gpu decoding with batch size of 1 (latency). nevertheless, optimized autoregressive models were still both faster and better in terms of translation quality. the main purpose of this submission is to provide a reasonable baseline to future non-autoregressive submissions. ## model in our experiments, we use the non-autoregressive model proposed by @xcite based on connectionist temporal classification (ctc; @xcite . we submit models that have been trained as a part of @xcite . architecture. the architecture is a 6-layer transformer encoder, followed by a state-splitting layer and another stack of 6 transformer layers. the state-splitting layer takes the encoder states, project them into k-times wider states using an affine transformation, and then split the states into k-times longer sequence while retaining the original model dimension. in the submitted model, we set @xmath0. the latter 6 layers cross-attend to the states immediately after state-splitting. we use transformer model dimension of 1,024, 16 attention heads and a dimension of 4,096 in the feed-forward sublayer. the defining property of non-autoregressive models is that the decoding process treats output states as conditionally independent. in this architecture, we set the output sequence length to k × t x where t x is the length of the source sentence. to allow for shorter output sequences, the any output state can produce an empty token. the training loss is then computed using a dynamic programming algorithm as a sum of losses of all possible empty token alignments which lead to the same output sentence. the schema of the architecture is shown in figure 1 . training. we train our model on the knowledgedistilled data generated by the provided teacher @xcite . we use learning rate of 0.0001 in a inverse square-root decay scheme with 8,000 warm-up and decay steps. implementation. we implement and train our model in the marian toolkit @xcite . for the ctc implementation, we use the warp-ctc library 1 . we release our code at https://github.com/jindrahelcl/ marian-dev . the trained model (and a number of different variants including models in opposite translation direction) can be downloaded at https://data.statmt.org/nar . ## results we refer the reader to the original paper for more details about the evaluation and its results. the model we submitted is denoted in the paper as "large". a summary of the results follows. translation quality. to summarize the main findings, the model achieves a competitive bleu score @xcite on the wmt 14 news test set @xcite , which serves as a comparison to other non-autoregressive models that use this test set as the de facto standard benchmark. when evaluated on the wmt 19 news test set, our model obtains bleu of 47.8, and a comet score @xcite of 0.1485. compared to an similarly-sized autoregressive teacher model with 50.5 bleu and comet of 0.4110, we see a somewhat surprising gap between the comet scores while bleu scores are relatively close. we hypothesize that the errors that the non-autoregressive model makes are out of the training domain of the comet models, which makes them more sensitive towards this kind of errors. we used a single nvidia a100 gpu for gpu decoding. in the latency setup, the translation took 7,020 seconds, and the batched decoding (@xmath1. when compared with other submissions to this task, we find that the smallest difference is indeed found in the gpu decoding latency setting. however, the optimized models submitted to last year's round still achieved significantly better decoding times. ## conclusions we submit a non-autoregressive system to the efficient translation shared task to the wmt 22. the model is trained with connectionist temporal classification, which allows the generation of empty tokens and thus making generation of sentences of various length possible while retaining the conditional independence among output tokens without explicit length estimation. the main motivation of this submission is to provide a reasonable baseline system for future research. we believe that the sub-field of nonautoregressive nmt cannot progress without controlled decoding speed evaluation, which is exactly what the shared task organizers provide.