ngram- OAXE : Phrase-Based Order-Agnostic Cross Entropy for Non-Autoregressive Machine Translation (2022)

Abstract: Recently, a new training oaxe loss has proven effective to ameliorate the effect of multimodality for non-autoregressive translation (NAT), which removes the penalty of word order errors in the standard cross-entropy loss. Starting from the intuition that reordering generally occurs between phrases, we extend oaxe by only allowing reordering between ngram phrases and still requiring a strict match of word order within the phrases. Extensive experiments on NAT benchmarks across language pairs and data scales demonstrate the effectiveness and universality of our approach. Further analyses show that ngram noaxe indeed improves the translation of ngram phrases, and produces more fluent translation with a better modeling of sentence structure.

## introduction fully non-autoregressive translation (nat) has received increasing attention for its efficient decoding by predicting every target token in parallel @xcite . however, such advantage comes at the cost of sacrificing translation quality due to the multimodality problem: there exist many possible translations of the same sentence, while vanilla nat models may consider them at the same time due to the independent predictions, which leads to multi-modal outputs in the form of token repetitions @xcite . recent works have incorporated approaches to improving the standard cross-entropy (xe) loss to ameliorate the effect of multimodality. the motivation for these works is that modeling word order is difficult for nat, since the model cannot condition on its previous predictions like its autoregressive counterpart. starting from this intuition, a thread of research relaxes the word order restriction based on the monotonic alignment assumption @xcite @xcite . @xcite take a further step by removing the penalty of word order errors with a novel order-agnostic cross entropy (oaxe) loss, which enables nat models to handle word reordering -a common source of multimodality problem. accordingly, oaxe achieves the best performance among these model variants. however, oaxe allows reordering between every two words, which is not always valid in practice. for example, the reordering of the two words "this afternoon" is not correct in grammar. the reordering generally occurs between ngram phrases, such as "i ate pizza" and "this afternoon". starting from this intuition, we extend oaxe by constraining the reordering between ngrams and requiring a strict match of word order within each ngram (i.e., ngram-oaxe). to this end, we first build the probability distributions of ngrams in the target sentence using the word probabilities produced by nat models. then we find the best ordering of target ngrams to minimize the cross entropy loss. we implement the ngram-oaxe loss in an efficient way, which only adds one more line of code on top of the source code of oaxe. accordingly, ngram-oaxe only marginally increases training time (e.g., 3% more time) over oaxe. experimental results on widely-used nat benchmarks show that ngram-oaxe improves translation performance over oaxe in all cases. encouragingly, ngram-oaxe outperforms oaxe by up to +3.8 bleu points on raw data (without knowledge distillation) for wmt14 en-de translation ( order-agnostic cross entropy (oaxe) @xcite remove the word order restriction of xe, and assign loss based on the best alignment between target tokens and model predictions. they define the ordering space @xmath0, . . . , o j } for y , where o j is an ordering of the set of target tokens (y 1 , . . . , y i ). the oaxe objective is defined as finding the best ordering o j to minimize the cross entropy loss: ## analysis in this section, we provide some insights where ngram-oaxe improves over cmlm (i.e., xe) and oaxe from different perspectives. otherwise stated, we report results on the wmt14 en-de test set with multiple references (i.e., the column "w14 en-de" in ## conclusion in this work, we extend oaxe by modeling ordering at the ngram phrase granularity, which can better ameliorate the effect of multimodality for nat models. benefiting from modeling translation at a larger granularity, the proposed ngram-oaxe loss performs better at translating phrases and long sentences, and improves the fluency of generated translations. extensive experiments on representative nat benchmarks show that ngram-oaxe consistently improves translation performance over oaxe, and is especially effective on raw data without distillation.