Divide and Rule: Effective Pre-Training for Context-Aware Multi-Encoder Translation Models (2022)

Abstract: Multi-encoder models are a broad family of context-aware neural machine translation systems that aim to improve translation quality by encoding document-level contextual information alongside the current sentence. The context encoding is undertaken by contextual parameters, trained on document-level data. In this work, we discuss the difficulty of training these parameters effectively, due to the sparsity of the words in need of context (i.e., the training signal), and their relevant context. We propose to pre-train the contextual parameters over split sentence pairs, which makes an efficient use of the available data for two reasons. Firstly, it increases the contextual training signal by breaking intra-sentential syntactic relations, and thus pushing the model to search the context for disambiguating clues more frequently. Secondly, it eases the retrieval of relevant context, since context segments become shorter. We propose four different splitting methods, and evaluate our approach with BLEU and contrastive test sets. Results show that it consistently improves learning of contextual parameters, both in low and high resource settings.

## introduction neural machine translation (nmt) has seen substantial improvements in recent years, fostered by the advent of the transformer model @xcite . a remaining challenge for modern machine translation (mt) is the ability to contextualize translation of the current sentence with other sentences in the document @xcite . for this reason, contextual nmt has recently triggered a lot of attention and many approaches have been proposed in the literature. a common taxonomy @xcite divides them in two broad categories: single-encoder (concatenation) approaches @xcite @xcite and multi-encoder approaches @xcite @xcite @xcite @xcite . multi-encoder models are more flexible and can be more efficient than concatenation approaches, but they have been criticized as being mere regularization methods @xcite . in some cases, they have even been shown to perform worse than sentence-level systems on discourse-aware targeted test suites @xcite . in this work, we address this criticism by showing that training multi-encoder models is difficult because of two reasons: (i) the sparsity of contextual training signal, i.e. the signal that pushes systems to translate in a context-aware fashion, which comes from the words that need context to be correctly translated; (ii) the sparsity of relevant context words, the ones needed to disambiguate translation. a trivial way to improve context-aware learning is by increasing the amount of documentlevel training data. large document-level parallel corpora are not always available, but some works have proposed data augmentation techniques to remedy scarcity @xcite @xcite . however, as we will show in our experimental section, this solution is not efficient and often sub-optimal. we therefore introduce a novel pre-training strategy, divide and rule (d&r), that is based on a simple and yet powerful technique to augment the contextual training signal and to ease learning efficiently: splitting parallel sentences in segments (see figure 1 ). simply put, feeding a context-aware model with a sequence of incomplete, shorter, consecutive segments, forces it to look for context (i.e., surrounding segments) more frequently, and makes it easier to retrieve relevant context because segments are shorter. this results in faster and improved learning. we pre-train multi-encoder models on split datasets and evaluate them in two ways: bleu score, and s i,1 he said that it was a project of peace s i,2 and unity and that it brought people together . t i,1 il disait que c' était un projet de paix t i,2 et d' unité et qu' il réunissait les gens . s j,1 i think single-cell organisms are s j,2 possible within two years . t j,1 je pense que les organismes unicellulaires t j,2 sont possibles dans 2 ans . figure 1 : example of sentence pairs from @xmath0, after being tokenized and split in the middle. after the splitting, some syntactic relations span across two segments (underlined). also, some source-side words are not parallel with their reference (in bold). our main contributions are the following: (i) we show that context-aware multi-encoder models need to be trained carefully, because the contextual training signal is sparse, as well as the context elements useful for contextualization; @xcite we propose the d&r pre-training strategy, which fosters training of contextual parameters by splitting sentences into segments, with four splitting variants; (iii) we support this strategy with an analysis of the impact of splitting on the distribution of discourse phenomena; (iv) we demonstrate that this strategy is both effective and efficient, as it allows multi-encoder models to learn better and faster than by simply increasing the training data. ## the double challenge of sparsity some works criticized multi-encoder methods @xcite , arguing that they do not improve sentence-level baselines in terms of bleu when the baseline is well regularized. when there are improvements, it is argued that the context-encoder simply works as a noise-generator that makes training more robust, and the improvements are not due to better contextmodeling. along this path, @xcite showed that multi-encoder architectures struggle to model contextual information, and even deteriorate the performance of a sentence-level baseline on contrastive test sets. in fact, many proponents of multi-encoder models only show bleu improvements, without providing any kind of targeted evaluation. this doesn't allow a direct evaluation of their context-modeling capability. we posit that training the contextual parameters of multi-encoder models is non-trivial because of two challenges: (i) the sparsity of the training signal, which comes from the words that need context to be correctly translated (most of the words of a sentence can be translated without context); (ii) the sparsity of context words that are useful for contextualization (most of the context is useless). as such, missing the right experimental setting can lead to unsuccessful training and unconvincing results. more data? a trivial way to offset sparsity is to increase the volume of training data. in fact, existing works that report strong results with targeted evaluation train their contextual parameters with millions of document-level sentence pairs @xcite @xcite @xcite . in contrast, many works in the literature train models with the ted talks' subtitles released by the iwslt shared tasks @xcite , which only consist of a couple of hundred thousand parallel sentences. in the experimental section, we will show that iwslt's subtitles are not sufficient to effectively train multi-encoder models. it follows that one can not make fair comparisons between alternative architectures in such experimental settings. on the other hand, we will give an empirical confirmation to the intuition that increasing the volume of training data helps learning contextual parameters. however, increasing the amount of training data is an inefficient solution, and one that is not always feasible: large documentlevel training sets may not be available in many languages. in the following section, we propose a pretraining solution that makes an efficient use of the available data for learning contextual-parameters effectively. ## proposed approach one way to simulate document-level data is to split sentences in two or more segments @xcite . in this way intra-sentential syntactic relations are broken, and a word previously disam- algorithm 1: split parallel corpus 1: input: parallel corpus c, minimum source length l min , function wheresplit() 2: for @xmath1, . . . , |c| do 3: if len(s i ) ≥ l min then 4: m s , m @xmath2, t i , ...) 5: s i, @xmath3,2 = s @xmath4: t i,1 = t @xmath5,2 = t @xmath6: ## conclusions multi-encoder models are a broad family of context-aware nmt models. in this work we have discussed the difficulty of training contextual parameters due to the sparsity of the words in need of context, and the sparsity of their relevant context. we have proposed a pre-training approach called divide and rule, based on splitting the training sentences, with four variants. after having analysed the implications of splitting on the distribution of discourse phenomena within the training data, we have shown that d&r allows to learn contextual parameters better and faster than by simply increasing training data. we have also shown that the simplest and language independent splitting variant, middlesplit, is a strong baseline that can be easily applied for pre-training any multi-encoder nmt model.