Diformer: Directional Transformer for Neural Machine Translation (2022)

Abstract: Autoregressive (AR) and Non-autoregressive (NAR) models have their own superiority on the performance and latency, combining them into one model may take advantage of both. Current combination frameworks focus more on the integration of multiple decoding paradigms with a unified generative model, e.g. Masked Language Model. However, the generalization can be harmful on the performance due to the gap between training objective and inference. In this paper, we aim to close the gap by preserving the original objective of AR and NAR under a unified framework. Specifically, we propose the Directional Transformer (Diformer) by jointly modelling AR and NAR into three generation directions (left-to-right, right-to-left and straight) with a newly introduced direction variable, which works by controlling the prediction of each token to have specific dependencies under that direction. The unification achieved by direction successfully preserves the original dependency assumption used in AR and NAR, retaining both generalization and performance. Experiments on 4 WMT benchmarks demonstrate that Diformer outperforms current united-modelling works with more than 1.5 BLEU points for both AR and NAR decoding, and is also competitive to the state-of-the-art independent AR and NAR models.

## introduction machine translation can be considered as a conditional generation task, which has been dominated by neural networks, especially after transformer @xcite . conventional autoregressive (ar) nmt models obtain the impressive performance, but it's time-consuming to decode token one by one sequentially @xcite . aiming at fast inference, non-autoregressive (nar) nmt models enhance the parallelizability by reducing or removing the sequential dependency on the translation prefix inside the decoder, but suffering from performance degradation owing to the multi-modality problem, which is still an open-question @xcite @xcite @xcite @xcite @xcite . it's always non-trivial to balance high performance and low latency in a single model perfectly. therefore, another branch focuses on the unifiedmodeling of multiple decoding paradigms so that decoding with ar or nar in different scenarios (ar for quality-first and nar for speed-first) with one model can be achieved @xcite @xcite , making the performance and speed can be pursued more practically. whereas, challenges still exist. for example, a generalized conditional language model is often required to support the generation with customized orders or positions @xcite , which actually prevents the model from being fully trained on specific decoding method, leading to the declines in overall performance. in addition, in some works, ar and nar decoding may needs to be trained separately in the stage of pretraining or fine-tuning @xcite , making the training more expensive. to ameliorate these issues, we propose directional transformer (diformer) which resolve the unification of ar and nar in a more practical way. first of all, we abandon the compatible of multiple flexible decoding strategies, but focusing on the modeling of some commonly used strategies that have good performance. for the ar decoding, it has been proved that monotonic linear generation is still considered as the best strategy @xcite , so we choose to only model the left-to-right (l2r) and right-to-left (r2l) generation. for the nar decoding, we choose to follow the stream of masked-language model, like mask-predict in cmlm @xcite or parallel easy-first in disco @xcite , since they are simpler than insertion-based method but still being effective. to this end, we unify two decoding paradigms into three generation directions -l2r, r2l and straight, and formulate it through a new objective named as directional language model (dlm), making the prediction of tokens conditioned on contexts controlled by a newly introduced direction variable. it ties ar and nar into a unified generation framework while still preserving the original dependency assumptions of ar and nar, retaining both generalization and performance. meanwhile, all directions can be trained simultaneously with the time spent equally to the training of an independent nar model, which greatly reduces the training cost compared to twostages methods. experimental results on the wmt14 en↔de and wmt16 en↔ro datasets for all three directions indicate that diformer performs better than previous unification-based works by more than 1.5 bleu points. comparing to other state-of-theart independent ar and nar models, diformer is also competitive when decoding in the same mode. we summarize contributions of our work as: both of them achieves the compatible to customized decoding through position selection and applying the selected positions/orders on a generalized generative model, which leads to the gap between training and inference. in contrast to the position selection, we directly model the decoding process with three generation directions in a taskspecific manner, thereby without introducing additional complexity to the task and close the gap between training objective and inference strategy. we consider it is worthwhile to obtain performance improvements by abandon some flexibility. the likelihood of an ar model is a factorization following the product rule, assuming each token is conditioned on all previous generated context. taking the l2r and r2l ar model as examples: in the iterative-refinement based nar model like cmlm @xcite , the conditional dependency is loosed, assuming the prediction of target token can be independent with each other, but conditioned on the output tokens (context) from last iteration: instead of using the global context, in disco @xcite , the target token at each position is predicted with different context, namely, the disentangled context. in such case, all tokens can be used for training and updated at each iteration during inference: during the training of cmlm and disco, a subset of tokens are selected as the context, cmlm updates parameters only with the loss on masked tokens while disco uses all tokens for updating. in the diformer, we aim to unify the two exclusive dependency assumptions @xcite of ar and nar essentially by proposing a new training objective and model architecture that can make them trained jointly. ## conclusion in this paper, we present directional transformer which is able to model the autoregressive and nonautoregressive generation with a unified framework named directional language model which essentially links two types of conditional language model with three generation directions. compared to previous works, diformer exchanges the generalization on decoding strategies for better performance and thereby only support 4 decoding strategies. experimental results on wmt14 en↔de and wmt16 en↔ro demonstrate that the unification of ar and nar can be achieved by diformer without losing any performance. the bidirectional property of diformer allows it to perform self-reranking which is especially useful for nar decoding to improve performance with no additional computational cost. except from machine translation, diformer can be easily extended to other tasks like language modeling by removing the dependency on x. it has the potential to unify the representation learning and generation with a single model, which is actually our ongoing work.