Rethinking Document-level Neural Machine Translation (2022)

Abstract: This paper does not aim at introducing a novel model for document-level neural machine translation. Instead, we head back to the original Transformer model and hope to answer the following question: Is the capacity of current models strong enough for document-level translation? Interestingly, we observe that the original Transformer with appropriate training techniques can achieve strong results for document translation, even with a length of 2000 words. We evaluate this model and several recent approaches on nine document-level datasets and two sentence-level datasets across six languages. Experiments show that document-level Transformer models outperforms sentence-level ones and many previous methods in a comprehensive set of metrics, including BLEU, four lexical indices, three newly proposed assistant linguistic indicators, and human evaluation.

## introduction neural machine translation @xcite @xcite has achieved great progress and reached near human-level performance. however, most current sequence-tosequence nmt models translate sentences individually. in such cases, discourse phenomena, such as pronominal anaphora, lexical consistency, and document coherence that depend on long-range context going further than a few previous sentences, are neglected @xcite . as a result, @xcite find human raters still show a markedly stronger preference for human translations when evaluating at the level of documents. many methods have been proposed to improve document-level neural machine translation (dnmt). among them, the mainstream studies * * work was done while at bytedance focus on the model architecture modification, including hierarchical attention @xcite @xcite , additional context extraction encoders or query layers @xcite @xcite @xcite @xcite @xcite @xcite , and cache-like memory network @xcite @xcite . these studies come up with different structures in order to include discourse information, namely, introducing adjacent sentences into the encoder or decoder as document contexts. experimental results show effective improvements on universal translation metrics like bleu @xcite and document-level linguistic indices @xcite @xcite @xcite . unlike previous work, this paper does not aim at introducing a novel model. instead, we hope to answer the following question: is the basic sequenceto-sequence model strong enough to directly handle document-level translation? to this end, we head back to the original transformer @xcite and conduct literal document-to-document (doc2doc) training. though many studies report negative results of naive doc2doc translation @xcite , we successfully activate it with multi-resolutional training, which involves multiple levels of sequences. it turns out that end-to-end document translation is not only feasible but also stronger than sentence-level models and previous studies. furthermore, if assisted by extra sentencelevel corpus, which can be much more easily obtained, the model can significantly improve the performance and achieve state-of-the-art results. it is worth noting that our method does not change the model architecture and need no extra parameters. our experiments are conducted on nine document-level datasets, including ted (zh-en, en-de), news (en-de, es-en, fr-en, ru-en), europarl (en-de), subtitles (en-ru), and a newly constructed news dataset (zh-en). additionally, two sentence-level datasets are adopted in further experiments, including wikipedia (en-de) and wmt @xcite . experiment results show that our strategy outperforms previous methods in a comprehensive set of metrics, including bleu, four lexical indices, three newly proposed assistant linguistic indicators, and human evaluation. in addition to serving as improvement evidence, our newly proposed document-level datasets and metrics can also be a boosting contribution to the community. ## re-examining recent dnmt studies for dnmt, though many improvements have been reported, a couple of studies have proposed challenges against these results @xcite @xcite . and we also find some of previous gains should be attributed to overfitting to some extent. the most used datasets of previous work are news commentary and ted talks, which contain only 200 thousand sentences. the small scale of the datasets gives rise to the frequent occurrence of overfitting, let alone that the distribution of the test set is highly similar to the training set. and some work even conduct an unfair comparison with @xmath0.1 for sentence-level models and @xmath1.2 for document-level models @xcite @xcite . as a result, regularization and overfitting on small datasets make the improvements not solid enough. to verify our assumption, we perform different training by switching hyperparameters on sentencelevel experiments. we follow the datasets provided by @xcite and @xcite , including ted (zh-en/en-de), news (en-de), and europarl (en-de), as well as all the model architecture settings they adopt, including a fourlayer transformer base version. as is shown in our results show that preceding experiments lack the comparison with a strong baseline. an important proportion of the improvements may come from the regularization of the models since they bring in extra parameters for context encoders or hierarchical attention weights. however, the regularization can be also achieved in sentence-level models and is not targeted at improving document coherence. essentially, the small scale of related datasets and identically distributed test sets make the improvements questionable. @xcite draw the same conclusion that well-regularized or pre-trained sentence-level models can beat document-level models in the same settings. they find that most improvements are not from coreference or lexical choice but "not interpretable". similarly, @xcite adopt a wide evaluation and find that the existing context-aware models do not improve discourserelated translations consistently across languages and phenomena. also, @xcite find that the extra context encoders act more like a noise generator and the bleu improvements mainly come from the robust training instead of the leverage of contextual information. all these three studies appeal for stronger baselines for a fair comparison. we suggest that the current research tendency in dnmt should be reviewed since it is hard to tell whether the improvements are targeted at document coherence or just normal regularization, let alone complicated modules are introduced. therefore, as a simpler alternative, we head back to the original but concise style, using end-to-end training framework to cope with document translation. ## doc2doc: end-to-end dnmt in this section, we attempt to analyze the different training patterns for dnmt. firstly, let us formulate the problem. let d @xmath4, x (2) , • • • , x (m ) } be a source-language document containing m source sentences. the goal of the document-level nmt is to translate the document d x in language x to a document d y in language y. d @xmath5, y (2) , • • • , y (n ) }. we use l (i) y to denote the sentence length of y (i) . previous work translate a document sentenceby-sentence, regarding dnmt as a step-by-step sentence generating problem (doc2sent) as: (i) is the context in the source side, depending on the model architecture and is comprised of only two or three sentences in many work. most current work focus on s (i) , by utilizing hierarchical attention or extra encoders. and t (i) is the context in the target side, which is involved by only a couple of work. they usually make use of a topic model or word cache to form t (i) . different from doc2sent, we propose to resolve document translation with the end-to-end, namely document-to-document (doc2doc) pattern as: ## further evidence with newly proposed datasets and metrics to further verify our conclusions and push the development of this field, we also contribute a new dataset along with new metrics. specifically, we propose a package of a large and diverse parallel document corpus, three deliberately designed metrics, and correspondingly constructed test sets foot_1 . on the one hand, they make our conclusions more solid. on the other hand, they may benefit future researches to expand the comparison scenes. ## limitation though multi-resolutional doc2doc achieves direct document translation and obtains better results, there still exists a big challenge: efficiency. the computation cost of self-attention in transformer rises with the square of the sequence length. as we feed the entire document into the model, the memory usage will be a bottleneck for larger model deployment. and the inference speed may be affected if no parallel operation is conducted. recently, many studies focus on the efficiency enhancement on long-range sequence processing @xcite @xcite @xcite . we leave reducing the computation cost to the future work. ## related work document-level neural machine translation is an important task and has been abundantly studied with multiple datasets as well as methods. the mainstream research in this field is the model architecture improvement. specifically, several recent attempts extend the sent2sent approach to the doc2sent-like one. wang et al. (2017) 2020); yun et al. (2020); xu et al. (2020 introduce additional encoders or query layers with attention model and feed the history contexts into decoders. @xcite ; @xcite ; @xcite propose to augment nmt models with a cache-like memory network, which generates the translation depending on the decoder history retrieved from the memory. besides, some works intend to resolve this problem in other ways. @xcite propose a regularization term for encouraging to focus more on the additional context using a multi-level pairwise ranking loss. yu et al. (2020 ) utilize a noisy channel reranker with bayes' rule. garcia et al. (2019) extends the beam search decoding process with fusing an attentional rnn with an sslm by modifying the computation of the final score. saunders et al. (2020) present an approach for structured loss training with document-level objective functions. liu et al. (2020); ma et al. (2020) combine large-scale pre-train model with dnmt. unanue et al. (2020); kang et al. (2020) adopt reinforcement learning methods. there are also some works sharing similar ideas with us. @xcite ; @xcite explore concatenating two consecutive sentences and generate two sentences directly. obviously, we leverage greatly longer information and capture the full context. junczys-dowmunt (2019) cut documents into long segments and feed them into training like bert @xcite . there are at least three main differences. firstly, they need to add specific boundary tokens between sentences while we directly translate the original documents without any additional processing. secondly, we propose a novel multi-resolutional training paradigm that shows consistent improvements compared with regular training. thirdly, for extremely long documents, they restrict the segment length to 1000 tokens or make a truncation while we preserve entire documents and achieve literal document-to-document training and inference. finally, our work is also related to a series of studies in long sequence generation like gpt (radford, 2018), gpt-2 @xcite , and transformer-xl @xcite . we all suggest that the deep neural generation models have the potential to well process long-range sequences. ## conclusion in this paper, we try to answer the question of whether document-to-document translation works. it seems naive doc2doc can fail in multiple scenes. however, with the multi-resolutional training proposed in this paper, it can be successfully activated. different from traditional methods of modifying the model architectures, our approach introduces no extra parameters. a comprehensive set of experiments on various metrics show the advantage of mr doc2doc. in addition, we contribute a new document-level dataset as well as three new metrics to the community. finally, we obtain 1.39 million parallel sentences within almost 60 thousand cleaned parallel documents. the dataset contains diverse domains including politics, finance, health, culture, etc.