Assessing Non-autoregressive Alignment in Neural Machine Translation via Word Reordering (2022)

Abstract: Recent work on non-autoregressive neural machine translation (NAT) that leverages alignment information to explicitly reduce the modality of target distribution has reported comparable performance with counterparts that tackle multi-modality problem by implicitly modeling dependencies. Effectiveness in handling alignment is vital for models that follow this approach, where a token reordering mechanism is typically involved and plays a vital role. We review the reordering capability of the respective mechanisms in recent NAT models, and our experimental results show that their performance is sub-optimal. We propose to learn a non-autoregressive language model (NALM) based on transformer which can be combined with Viterbi decoding to achieve better reordering performance. We evaluate the proposed NALM using the PTB dataset where sentences with words permuted in different ways are expected to have their ordering recovered. Our empirical results show that the proposed method can outperform the state-of-the-art reordering mechanisms under different word permutation settings, with a 2-27 BLEU improvement, suggesting high potential for word alignment in NAT.

## introduction non-autoregressive neural machine translation (nat) @xcite takes advantage of the parallel architecture of transformer @xcite to alleviate the translation latency issue in neural machine translation (nmt), achieving significant speed-up. yet it suffers from the multimodality problem, where a target token could be a result of different possible translations. word order errors are often resulted as compared to the autoregressive counterparts @xcite , arising from the lack of dependency amongst target tokens in nat models. some recently proposed nat models can achieve comparable performance to autoregres-sive models. this can be attributed to various approaches that reduce the dependency in handling word order errors via word alignment mechanisms @xcite . in particular, latent variables and alignments have been adopted for implicitly modelling the dependencies among the target tokens @xcite . while the latent alignment approach assumes monotonic alignment between the source and target language pair when handling token shifts in the output space @xcite , explicit modality reduction methods @xcite @xcite on the other hand sought to directly align the source and target language pair. despite some previous work being sub-optimal, recent work in this direction achieves state-of-the-art (sota) results rivaling that of implicit dependency modeling methods. establishing explicit alignment between tokens in parallel sentences of source and target languages typically involves fertility prediction and token reordering prediction. in this paper, we focus on the latter and argue that improving the reordering performance can contribute greatly towards the performance of nat models. with the sole exception of @xcite , architectural design of the aforementioned nat models includes a reordering sub-module as a key component. we therefore set forward to review in detail the capabilities of the various reordering mechanisms proposed in the nat models. we then propose a novel way to achieve the reordering prediction by learning a nonautoregressive language model (nalm) based on transformer with viterbi decoding @xcite combined. we evaluated the reordering sub-modules extracted from the various nat models and variants of our proposed nalm using the ptb dataset @xcite where sentences with words permuted in different ways are expected to have their ordering recovered. in particular, we adopt different degrees of permutation to mimic various levels of monotonicity (or reordering difficulty) between the source and target sentences. our experimental results show that the proposed nalm achieves significant and consistent improvement compared to the reordering sub-modules extracted from explicit modality reductionist nat models in all word permutation settings. our experiment also advances the sota performance of the word reordering task in low beam setting and achieves comparable performance with autoregressive models even in high beam setting (@xmath0. ## non-autoregressive language modelling in this section, we will first provide the formulation of the word reordering task and then present our proposed solution by taking a non-autoregressive language modelling approach. ## experimental setup we use the english penn treebank data @xcite in our evaluation, preprocessed (in various ways) as described in the section. ## conclusion in this paper, we review reordering mechanisms of nat models that directly model alignment using various settings of word permutation. we propose a non-autoregressive language model which outperforms in low-beam and competes with in higher-beam setting sota autoregressive models. our extended model further achieves significant improvement over all adapted nat reordering mechanisms in datasets of varying difficulty that reasonably resemble the reordering task encountered in nat alignment. performance of existing reordering mechanisms in nat models vary according to our experiment results, implying that more effort would be required in this area.