Diverse Pretrained Context Encodings Improve Document Translation (2021)

Abstract: We propose a new architecture for adapting a sentence-level sequence-to-sequence transformer by incorporating multiple pre-trained document context signals and assess the impact on translation performance of (1) different pretraining approaches for generating these signals, (2) the quantity of parallel data for which document context is available, and (3) conditioning on source, target, or source and target contexts. Experiments on the NIST Chinese-English, and IWSLT and WMT English-German tasks support four general conclusions: that using pre-trained context representations markedly improves sample efficiency, that adequate parallel data resources are crucial for learning to use document context, that jointly conditioning on multiple context representations outperforms any single representation, and that source context is more valuable for translation performance than target side context. Our best multi-context model consistently outperforms the best existing context-aware transformers.

## introduction generating an adequate translation for a sentence often requires understanding the context in which the sentence occurs (and in which its translation will occur). although single-sentence translation models demonstrate remarkable performance @xcite @xcite , extra-sentential information can be necessary to make correct decisions about lexical choice, tense, pronominal usage, and stylistic features, and therefore designing models capable of using this information is a necessary step towards fully automatic high-quality translation. a series of papers have developed architectures that permit the broader translation model to condition on extra-sentential context @xcite , operating jointly on multiple sentences at once @xcite , or indirectly conditioning on target side document context using bayes' rule @xcite . while noteworthy progress has been made at modeling monolingual documents @xcite , progress on document translation has been less remarkable, and continues to be hampered by the limited quantities of parallel document data relative to the massive quantities of monolingual document data. one recurring strategy for dealing with this data scarcity-and which is the basis for this work-is to adapt a sentence-level sequence-to-sequence model by making additional document context available in a second stage of training @xcite @xcite . this two-stage training approach provides an inductive bias that encourages the learner to explain translation decisions preferentially in terms of the current sentence being translated, but these can be modulated at the margins by using document context. however, a weakness of this approach is that the conditional dependence of a translation on its surrounding context given the source sentence is weak, and learning good context representations purely on the basis of scarce parallel document data is challenging. a recent strategy for making better use of document context in translation is to use pretrained bert representations of the context, rather than learning them from scratch @xcite . our key architectural innovation in this paper is an architecture for two-staged training that enables jointly conditioning on multiple context types, including both the source and target language context. practically, we can construct a weak context representation from a variety of different contextual signals, and these are merged with the source sentence encoder's representation at each layer in the transformer. to examine the potential of this architec-ture, we explore two high-level research questions. first, using source language context, we explore the relative impact of different kinds of pretraining objectives on the performance obtained (bert and pegasus), the amount of parallel document training data required, and the size of surrounding context. second, recognizing that maintaining consistency in translation would seem to benefit from larger contexts in the target language, we compare the impact of source language context, target language context, and context containing both. our main findings are (1) that multiple kinds of source language context improves performance of document translation over existing contextual representations, especially those that do not use pretrained context representations; (2) that although fine-tuning using pretrained contextual representations improves performance, large performance is strongly determined by the availability of contextual parallel data; and (3) that while both source and target language context provide benefit, source language context is more valuable, unless the quality of the target language context translations is extremely high. ## model description our architecture is designed to incorporate multiple sources of external embeddings into a pretrained sequence-to-sequence transformer model. we execute this by creating a new attention block for each embedding we wish to incorporate and stack them. we then insert this attention stack as a branching path in each layer of the encoder and decoder. the outputs of the new and original paths are averaged before being passed to the feed forward block at the end of the layer. details are discussed below ( §2.4), and the architecture is shown in figure 1 . the model design follows the adapter pattern @xcite . the interface between the external model and translation model takes the form of an attention block which learns to perform the adaptation. the independence between the models means that different input data can be provided to each, which enables extra information during the translation process. in this work, we leverage this technique to: (1) enhance a sentence-level model with additional source embeddings; (2) convert a sentence-level model to a document-level model by providing contextual embeddings. like bertfused @xcite , we use pretrained masked language models to generate the external embed-dings. ## results in this section, we present our main results and explore the importance of each component in the multi-context model. additionally, we investigate the performance impact of document-level parallel data scarcity, the value of source-side versus targetside context, and the importance of target context quality. ## conclusion we have introduced an architecture and training regimen that enables incorporating representations from multiple pretrained masked language models into a transformer model. we show that this technique can be used to create a substantially stronger sentence-level model and, with sufficient document data, further upgraded to a document-level model that conditions on contextual information. through ablations and other experiments, we establish document augmentation and multi-stage training as effective strategies for training a document-level model when faced with data scarcity. and that source side context is sufficient for these models, with target context adding little additional value.