Non-Autoregressive Document-Level Machine Translation (2023)

Abstract: Non-autoregressive translation (NAT) models achieve comparable performance and superior speed compared to auto-regressive translation (AT) models in the context of sentence-level machine translation (MT). However, their abilities are unexplored in document-level MT, hindering their usage in real scenarios. In this paper, we conduct a comprehensive examination of typical NAT models in the context of document-level MT and further propose a simple but effective design of sentence alignment between source and target. Experiments show that NAT models achieve high acceleration on documents, and sentence alignment significantly enhances their performance. However, current NAT models still have a significant performance gap compared to their AT counterparts. Further investigation reveals that NAT models suffer more from the multi-modality and misalignment issues in the context of document-level MT, and current NAT models struggle with exploiting document context and handling discourse phenomena. We delve into these challenges and provide our code at https://github.com/baoguangsheng/nat-on-doc .

## introduction non-autoregressive neural machine translation (nat) models achieve significant acceleration on the inference, with even better translation performance, when compared to auto-regressive translation (at) models on sentence-level mt @xcite @xcite @xcite @xcite . however, real applications (e.g., google translation and chatgpt ) typically need to understand and respond in discourse, which requires a quick process of document-level content. different * contributed equally. † corresponding author. from sentence-level mt, document-level mt considers a much larger inter-sentential context and models discourse dependence, such as anaphora, ellipsis, and lexical cohesion @xcite . the sentence-by-sentence translation done by nat models cannot possibly produce contextual coherent translations, hindering the usage of nat models in real scenarios. document-level context enhances the translation quality of at models significantly @xcite @xcite @xcite . however, no precedent research explores the possibility of applying nat models to document-level mt, which leaves open research questions including: 1) can nat models leverage document context to improve translation quality? 2) can nat models generate translations with cross-sentential cohesion and coherence? and 3) can nat models achieve the same performance as at models? to address these questions, we investigate the challenges and opportunities of nat models on document-level mt. nat models primarily face two challenges: multi-modality @xcite , which causes failure when one source has multiple possible translations, and misalignment @xcite , which causes repetition and disorder in translations. previous studies reduce modalities using knowledge distillation @xcite and improve alignment using new loss @xcite @xcite . these methods have proven effective on the sentence level, while their efficacy is uncertain on the document level. first, document-level mt necessitates the modeling of cross-sentential coherence and discourse structure. second, the extended length of input and output sequences intensifies the potential modalities. last, the alignment between the source and target becomes harder because the enlarged input and output sequences increase the alignment space exponentially. in this paper, we first assess recent nat models on document-level mt tasks to understand their abilities and limitations, where we observe model failures and unstable performance. according to these observations, we further introduce sentence alignment to nat models, which equips the encoder and decoder of the nat models with the group-attention @xcite , restricting the target-to-source attention inside each target and source sentence pairs. since each target sentence is aligned with a source sentence, we do not need to consider the possibility of aligning the target sentence with other source sentences. therefore, the alignment space between the input and output sequences is significantly reduced. experiments on three benchmark datasets ted, news, and europarl show that nat models achieve a translation speed of more than 30 times faster than their at counterparts on document-level mt, but still lag behind on the translation performance. sentence alignment significantly enhances the translation performance for nat models, closing the gap from 2.59 to 1.46 d-bleu (document-level bleu score) compared to the best at results. to the best of our knowledge, we are the first to discuss the challenges and opportunities of nat models in the context of document-level mt. we release our code and data to stimulate future research. nat models face two fundamental challenges: the multi-modality issue and the misalignment issue. the multi-modality issue happens when one source has multiple possible translations, breaking the conditional independence assumption. for example, "thank you ." in english can be translated into "vielen dank ." or "danke ." in german. the second output token could either be "dank" or ".", which cannot be determined without a condition on the first token. the multi-modality issue causes vanilla nat models to fail on complex datasets. previous research leverages knowledge distillation to reduce modalities of a dataset, so that the conditional independence assumption is more likely satisfied @xcite . they generate a translation for each training sample using an at model and then train nat models on this generated training set. in this paper, we compare nat models trained on both raw and knowledge-distilled data. the misalignment issue elicits various nat techniques. vanilla nat @xcite uses an implicit token-alignment approach, enforcing a monotonic alignment between source and target by copy encoder outputs to decoder inputs. nat+ctc @xcite introduces connectionist temporal classification (ctc) @xcite to model the monotonic alignment explicitly. others introduce non-monotonic alignment between source and target, such as n-gram matching @xcite , aligned crossentropy (axe) @xcite , and order-agnostic cross-entropy (oaxe) @xcite . in this paper, we investigate nat models using the first two techniques. ## the challenge of long sequence the long input and output in document-level mt bring unexplored challenges to nat models. in section 4, we learn that nat models have a significant performance gap with their at counterparts, and various nat models even fail in some settings. we investigate these challenges in this section, leaving other discussions in appendix d. ## conclusion and future work we investigated nat models on document-level mt, revealing more severe affections of multimodality and misalignment issues on documents than on sentences. we proposed nat models with sentence alignment, reducing the possible alignment space, and achieving the best results across three benchmarks. our experiments show that nat models significantly accelerate text generation in documents while their performance still lags behind their at counterparts. further analysis shows that fully nat models underutilize document context, leading to loose discourse relations. as the first research of nat models on document-level mt, we hope this work could stimulate future research on reducing the modalities, exploiting document contexts, and modeling discourse dependence.