Document Flattening: Beyond Concatenating Context for Document-Level Neural Machine Translation (2023)

Abstract: Existing work in document-level neural machine translation commonly concatenates several consecutive sentences as a pseudo-document, and then learns inter-sentential dependencies. This strategy limits the model’s ability to leverage information from distant context. We overcome this limitation with a novel Document Flattening (DocFlat) technique that integrates Flat-Batch Attention (FBA) and Neural Context Gate (NCG) into Transformer model to utilizes information beyond the pseudo-document boundaries. FBA allows the model to attend to all the positions in the batch and model the relationships between positions explicitly and NCG identifies the useful information from the distant context. We conduct comprehensive experiments and analyses on three benchmark datasets for English-German translation, and validate the effectiveness of two variants of DocFlat. Empirical results show that our approach outperforms strong baselines with statistical significance on BLEU, COMET and accuracy on the contrastive test set. The analyses highlight that DocFlat is highly effective in capturing the long-range information.

## introduction remarkable progress has been made in neural machine translation (nmt) @xcite @xcite , yet human translation still clearly outperforms nmt at the document level @xcite , because current sentence-level nmt systems ignore the inter-sentential relationships. to narrow this gap, numerous document-level nmt (docnmt) approaches have been proposed in recent years to improve the context awareness by incorporating the contextual information during the translation @xcite @xcite . existing docnmt systems commonly concatenate several consecutive sentences to form a pseudo-document, instead of processing the entire document @xcite @xcite . one typical pseudo-document contains the current sentence to be translated and the surrronding context. intuitively, larger context should result in better performance. in our preliminary study, the model performance does not always grow as the context size increases as shown in figure 1 . @xcite and @xcite also observe that transformer's performance declines with longer inputs. we refer to this phenomenon as the quality saturation problem @xcite . therefore, such formation of pseudo-document limits the docnmt systems to leverage the information from a relatively small context. consequently, once the entire original document is segmented into several pseudo-documents for reducing the sequence length, the information out of the pseudodocument's scope is no longer accessible to the current sentence. therefore, a natural research question to ask is that, is there a more effective way to model the parallel documents in docnmt? in this work, we seek docnmt approaches that could better expand the context scope and improve the corresponding translation performance. instead of directly training docnmt system on the entire document, we propose to store the document as multiple pseudo-documents in a single batch and optimize the docnmt models by leveraging the inter-pseudo-document relationships at the batch level. inspired by @xcite , we propose a document flattening (docflat) technique that integrates flat-batch attention (fba) and neural context gate (ncg) into the transformer model @xcite . fba flattens all the current sentences in the batch with the original order into a sequence along the temporal dimension. it then applies the attention mechanism to the flattened sequence. the goal of this design is to preserve the linguistic structure of documents and expand the scope of context by explicitly learning the pseudo-document relationships. as there is both supportive and noisy information in the longer context, we introduce ncg, a simple feed-forward network, to identify the usefulness of contextual information and filter out the noise. with the combination of fba and ncg, docflat effectively captures the information in the distant context. to the best of our knowledge, @xcite propose mini-batch embedding (mbe), which is the only close work to ours. they compute the average representation for all the source tokens in the batch and prepend it to the source and target pseudo-documents. the compressed representation ignores the linguistic structure of documents, providing limited contextual information. our contributions are summarized as follows. firstly, we propose a novel approach docflat that allows the model to attend the content beyond the pseudo-document boundaries using fba and ncg. secondly, we demonstrate that docflat outperforms strong baselines with statistical significance, in terms of bleu, comet and accuracy on the contrastive test set, on three doc-nmt benchmark datasets, including ted, news commentary and europarl. thirdly, we conduct comprehensive analyses to understand the effectiveness of docflat. the analyses highlight that docflat is highly effective in capturing the distant context. ## document flattening in this section, we firstly describe the overview of docflat (section 3.1). we then introduce docflat's core components, flat-batch at-tention (fba; section 3.2) and neural con-text gate (ncg; section 3.3). finally, we discuss the practical considerations (section 3.4 and section 3.5) of docflat along with a concrete example. ## analysis in this section, we investigate the effectiveness of docflat on the contextual awareness and the quality saturation problem. we also demonstrate how the inference batch size affects the model predictions. a visualization of fba attention map is presented in appendix d. ## limitations as suggested in figure 6 , the performance of docflat is positively correlated to the inference batch size. this is because large inference batch size could help docflat to better utilize distant context within the same inference batch. however, this property of docflat could become an issue when there are only limited inference computational resources available.