Multilingual Document-Level Translation Enables Zero-Shot Transfer From Sentences to Documents (2022)

Abstract: Document-level neural machine translation (DocNMT) achieves coherent translations by incorporating cross-sentence context. However, for most language pairs there’s a shortage of parallel documents, although parallel sentences are readily available. In this paper, we study whether and how contextual modeling in DocNMT is transferable via multilingual modeling. We focus on the scenario of zero-shot transfer from teacher languages with document level data to student languages with no documents but sentence level data, and for the first time treat document-level translation as a transfer learning problem. Using simple concatenation-based DocNMT, we explore the effect of 3 factors on the transfer: the number of teacher languages with document level data, the balance between document and sentence level data at training, and the data condition of parallel documents (genuine vs. back-translated). Our experiments on Europarl-7 and IWSLT-10 show the feasibility of multilingual transfer for DocNMT, particularly on document-specific metrics. We observe that more teacher languages and adequate data balance both contribute to better transfer quality. Surprisingly, the transfer is less sensitive to the data condition, where multilingual DocNMT delivers decent performance with either back-translated or genuine document pairs.

## introduction recent years have witnessed a trend moving from sentence-level neural machine translation (sen-nmt) to its document-level counterpart (doc-nmt). sennmt inevitably suffers from translation errors related with document phenomena @xcite and delivers obviously inferior performance when compared against human translations and evaluated at a document level @xcite 2018). most efforts on docnmt aim at improving contextual modeling via dedicated model architectures and/or decoding algorithms @xcite @xcite and heavily rely on large-scale parallel document resources. nevertheless, document resources are unevenly distributed across language pairs, with most pairs having little to no such resources. 1 one promising way to accommodate languages with varied training data is multilingual modeling, as demonstrated in multilingual sennmt @xcite . by sharing parameters across languages, multilingual modeling encourages cross-lingual knowledge transfer, enabling performance improvement and even zeroshot transfer @xcite @xcite . in the context of translation, however, most studies on multilingual transfer center around sennmt, seldom going beyond sentence-level translation. so far, the question of whether and how document-level contextual modeling can be learned cross-lingually in multilingual docnmt is still unanswered. in this paper, we study zero-shot generalization for docnmt -the ability to attain plausible doc-nmt quality for some focused (student) language pair(s), with only parallel sentences for the student but parallel documents for other (teacher) languages in the multilingual mix. the high-level research question we seek to answer is illustrated in figure 1 . we resort to transfer learning via multilinguality to leverage document resources in teacher languages to help the student languages. we perform our analysis using a simple concatenation based docnmt, where consecutive sentences are chained into one sequence for translation. we investigate three dimensions extensively to understand the transfer in multilingual docnmt: 1) the number of languages with document level data (teacher languages), where we simplify our transfer setup to contain either only one teacher language (with n students) or n teachers (with one student); 2) the data balance for parallel documents, i.e. manipulating the ratio of document-level data to sentencelevel data during training; and 3) the data condition of parallel documents, where we adopt backtranslated parallel documents when only monolingual documents are given in teacher languages or use genuine parallel documents crawled natively. we conduct experiments on two publicly available datasets, namely europarl-7 and iwslt-10, covering 6 and 9 languages from/to english respectively. we analyze one-to-many (@xmath0. following recent work @xcite , we adopt document-specific metrics for evaluation apart from bleu and support our findings with human evaluations. we also propose a pronoun f1 metric (targeted at gendered pronouns: he/she) for @xmath1, and employ accuracy on contrastive test sets @xcite for @xmath2. our main findings are summarized below: but it has also motivated the field to revisit the common protocols resorted for evaluation @xcite . despite the hard to measure success, all the above mentioned methods implicitly assume an abundance of document resources and overlook the data scarcity problem. in this study, we adopt the simple concatenation model as our experimental protocol, and leave the exploration of various input formatting options and modelling to future work. considering the fast changing landscape of the (contextual) mt evaluation, we also provide multiple evaluation metrics including human evaluations, to give a full picture of the phenomena under investigation, while acknowledging the current imperfections of and disagreements on the right way of evaluating mt systems @xcite . zero-shot transfer via multilinguality multilingual modeling often clusters sentences of similar meaning from different languages within a shared semantic space @xcite . such representation space is hypothesized to enable zero-shot transfer, delivering improved performance in many cross-lingual tasks @xcite @xcite , especially based on large-scale pretrained multilingual transformers @xcite @xcite . when it comes to transla-tion, multilingual sennmt successfully achieves zero-shot translation, transferring sentence-level generation knowledge to language pairs unseen during training @xcite @xcite even in massively multilingual settings @xcite @xcite . our study extends multilingual sennmt to multilingual docnmt and aims at document-level knowledge transfer from languages that have document level data to languages that only have sentence level data. to the best of our knowledge, our study is the first demonstrating the emergence of document-level zero-shot transfer across languages for multilingual machine translation. ## zero-shot transfer in multilingual docnmt we first formulate the zero-shot generalization framework explored in this paper. given n+1 language pairs, we assume that all of them have parallel sentences for training, but only some of them have parallel documents (teachers). through multilingual training, we study to what degree contextual modeling in document-supervised docnmt can be transferred to those document-poor (student) languages as in figure 1 . any form of parallel document for student languages is disallowed at training, ensuring that the transfer is measured zero-shot. ## experimental settings datasets we conduct experiments on two public datasets: europarl-7 and iwslt-10. europarl-7 is extracted from european parliament (v10) and has translations between english and @xmath3, including czech, german, finnish, french, lithuanian and polish @xcite . this dataset offers sentence-aligned parallel documents (0.9k∼3.7k documents, 190k∼1.9m sentences) and also monolingual documents (9.7k∼11k documents, 0.65m∼2.28m sentences) for training. for evaluation, we use the wmt dev and test sets (barrault et al., 2020) available for each language pair (from 2013 to 2020). in contrast, iwslt-10 is collected from ted talks and covers translations between english and @xmath4, including arabic, german, french, italian, japanese, korean, dutch, romanian and chinese @xcite . unlike europarl-7, the distribution of training data over languages in iwslt-10 is much smoother (uniform). there are ∼1.9k sentencealigned parallel documents with ∼240k sentences for each language pair. we further collected about 1k ted talks for each language pair (crawled from feb 2018 to jan 2021) as monolingual documents. we use iwslt17 dev and test sets for evaluation. detailed statistics are given in appendix a. we preprocess all texts with the byte pair encoding (bpe) algorithm @xcite implemented in the sentencepiece toolkit @xcite , and set the vocabulary size to 32k and 64k for iwslt-10 and europarl-7, respectively. ## discussion apart from automatic evaluation, we also offer human evaluation on en-de. we choose en-de as its wmt20 test set is intentionally constructed for docnmt evaluation. we surprisingly find that docnmt with seninfer yields very competitive performance, although no contextual information is used for decoding. we also observe that such decoding tends to produce longer translations than sennmt despite using the same decoding hyperparameters. this behaviour should be shaped by the fact that docnmt is biased towards long concatenated target references. this partially agrees with the recent argument that context improves docnmt with some sort of regularization rather than teaching the model to deal with context @xcite . on the other hand, this challenges how to properly evaluate docnmt. another observation is that applying docinfer to sennmt delivers a significant accuracy improvement on en-fr contrastive test set (+8.5%, ## conclusion and future work this paper studies the variables playing role in achieving zero-shot document-level translation capability for languages that only have sentence level data (students), through multilingual transfer from languages that have access to document level data (teachers). we make the first step in this direction by extensively exploring properties of transfer by investigating three different variables. our experiments on europarl-7 and iwslt-10 confirm the feasibility, where we discover that increasing document-supervised teacher languages thereby increasing the document training data size, adequately balancing between document and sentence data at training, and leveraging monolingual documents via back-translation all benefit zero-shot transfer in varying degrees. the transferability of contextual modeling in docnmt demonstrates the potential of delivering multilingual docnmt with limited document resources. along with the success of document-level zeroshot transfer, problems with accurately estimating the document-level translation become challenging. bleu often fails to capture document phenomena, while contrastive test sets only cover few document-level aspects. neither perfectly corre-lates with human evaluation. besides, whether the gains really come from contextual modeling is still unclear. our human evaluation shows some preference to docnmt with seninfer where context is not used for decoding at all. designing better evaluation protocols (either automatic or human) is again confirmed to be critical. besides, performing analysis beyond 12n and n21 transfer deserves more effort and it is an interesting and plausible future direction to analyze how language similarity affects the transfer. zaixiang zheng, xiang yue, shujian huang, jiajun chen, and alexandra birch. 2020. towards making the most of context in neural machine translation.