Better Document-Level Machine Translation with B ayes’ Rule (2020)

Abstract: We show that Bayes’ rule provides an effective mechanism for creating document translation models that can be learned from only parallel sentences and monolingual documents a compelling benefit because parallel documents are not always available. In our formulation, the posterior probability of a candidate translation is the product of the unconditional (prior) probability of the candidate output document and the “reverse translation probability” of translating the candidate output back into the source language. Our proposed model uses a powerful autoregressive language model as the prior on target language documents, but it assumes that each sentence is translated independently from the target to the source language. Crucially, at test time, when a source document is observed, the document language model prior induces dependencies between the translations of the source sentences in the posterior. The model’s independence assumption not only enables efficient use of available data, but it additionally admits a practical left-to-right beam-search algorithm for carrying out inference. Experiments show that our model benefits from using cross-sentence context in the language model, and it outperforms existing document translation approaches.

## introduction there have been many recent demonstrations that neural language models based on transformers @xcite are capable of learning to generate remarkably coherent documents with few @xcite or no @xcite conditioning variables. despite this apparent generation ability, in practical applications, unconditional language models are most often used to provide representations for natural language understanding applications @xcite @xcite , and how to use them for conditional generation applications remains an open question. our hypothesis in this work is that bayes' rule provides an effective way to leverage powerful unconditional document language models to improve a conditional task: machine translation. the application of bayes' rule to transform the translation modeling problem p(y | x), where y is the target language, and x is the source language, has a long tradition and was the dominant paradigm in speech and language processing for many years @xcite , where it is often called a ''noisy channel'' decomposition, by analogy to an information theoretic conception of bayes' rule. whereas several recent papers have demonstrated that the noisy channel decomposition has benefits when translating sentences one-by-one @xcite @xcite , in this paper we show that this decomposition is particularly suited to tackling the problem of translating complete documents. although using crosssentence context and maintaining cross-document consistency has long been recognized as essential to the translation problem @xcite , operationalizing this in models has been challenging for several reasons. most prosaically, parallel documents are not generally available (whereas parallel sentences are much more numerous), making direct estimation of document translation probabilities challenging. more subtly, documents are considerably more diverse than sentences, and models must be carefully biased so as not to pick up spurious correlations. our bayes' rule decomposition ( §2) permits several innovations that enable us to solve these problems. rather than directly modeling the conditional distribution, we rewrite it as p(y | x) ∝ p(y) × p(x | y). this changes the learning problem from estimating a single complex conditional distribution to learning two different distributions: a language model p(y), which provides unconditional estimates of the output (in this paper, documents); and p(x | y), which provides the probability of translating a candidate output y into the (observed) source document x. as we will discuss subsequently, although the problems of estimating p(y | x) and p(x | y) are formally similar, independence assumptions made in p(x | y) are less statistically costly than they might otherwise be since, at test time, we will be conditioning on x and reasoning about a posterior distribution over y, which will be jointly dependent on all (conditionally independent) parts of x. this statistical fact-which is the same trick that gives naïve bayes classifiers their expressiveness and ease of estimation-permits us to assume independence between sentence translations in the reverse translation model, and therefore to use parallel sentences (rather than parallel documents) to train it. in the posterior, we thus have an implicit estimate of a document-level translation system, even though we made no use of parallel documents when estimating the prior or likelihood models. this is particularly useful because parallel sentences are much more readily available than parallel documents. a second benefit of our approach is that the unconditional language model can be estimated from nonparallel data, which exists in vast quantities. although the noisy channel model is ideal for exploiting the data resources that naturally exist in the world (large corpora of parallel but independent sentences, and large corpora of monolingual documents), we are faced with a much harder decoding problem ( §3). to address this problem, we propose a new beam-search algorithm, exploiting the fact that our document language model operates left-to-right, and our reverse translation model treats sentences independently. the search is guided by a proposal distribution that provides candidate continuations of a document prefix, and these are reranked according to the posterior distribution. in particular, we compare two proposal models: one based on estimates of independent sentence translations @xcite and one that conditions on the source document context @xcite . although closely related, our algorithm is much simpler and faster than that proposed in @xcite . rather than using a specially designed channel model @xcite which is limited in process-ing long sequences like documents, our conditional sentence independence assumptions allow us to use any sequence-to-sequence model as the channel model, making it a better option for document-level translation. to explore the performance of our proposed model, we focus on chinese-english translation, following a series of papers on document translation @xcite @xcite . although in general it is unreasonable to expect that independent translations of sentences would lead to coherent translations of documents, the task of translating chinese into english poses some particularly acute challenges. as chinese makes fewer inflectional distinctions than english does, and the relevant clues for predicting, for example, what tense an english verb should be in, or whether an english noun should have singular or plural morphology, may be spread throughout a document, it is crucial that extra-sentential context is used. our experiments ( §4) explore: (1) different approaches to reranking, (2) different independence assumptions when modeling documents (i.e., whether sentences are generated independently or not), (3) different amounts of language modeling data, and (4) different proposal models. briefly summarized, we find that documentcontext language models significantly improve the translation quality obtained with our system, both in terms of bleu scores, and in terms of a human evaluation. targeted error analysis demonstrates the document prior is capable of enforcing consistency of tense and number and lexical choice across documents. ## model description we define @xmath0, x 2 , . . . , x i ) as the source document with i sentences, and similarly, @xmath1, y 2 , . . . , y j ) as the target document with j sentences. during the (human) translation process, translators may split or recombine sentences, but we will assume that @xmath2. 1 let m ) represent the ith sentence in the document, consisting of m words; likewise y @xmath3, y i 2 , . . . , y i n ) denote the ith sentence in the target document, containing n words. the translation of a document x is determined by finding the document ŷ, where p(ŷ | x) is optimal. instead of modeling the probability p(y | x) directly, we factorize it using bayes' rule: we further assume that sentences are independently translated, and that the sentences are generated by a left-to-right factorization according to the chain rule. therefore, we have ## decoding because of the global dependencies in the posterior distribution, decoding in our model is computationally complex. on one hand, similar to the decoding problem faced in standard sequence-tosequence models, we must search over the space of all possible outputs with a model that makes no markov assumptions. on the other hand, unlike traditional models, we have to have a complete y i before we can compute p(x i | y i ), making greedy and near-greedy algorithms ineffective. to address this issue, we use an auxiliary proposal model q(y | x), that approximates the posterior distribution using a direct model, to focus our search on promising parts of the output space. because of the autoregressive factorization of the language model (p lm ), and the independent sentence translation assumption in the channel model (p tm ), we can carry out the reranking process using a left-to-right beam search strategy with the aid of our proposal model (q). figure 2 illustrates the decoding process. for an input document of ℓ sentences, we let the proposal model propose k candidate translations for each sentence in the document. 2 we then search for the best document path through this lattice-or confusion network @xcite -of candidate sentence translations. to do so, we maintain a beam of the b active hypotheses (i.e., when considering the ith sentence, the prefix consists of i -1 sentences), and we consider the proposal's k one-sentence extensions (which we write y i ). we retain b partial translations from the k × b candidates according to the following linear objective, ## experiments we evaluate our model on two translation tasks, the nist open mt chinese-english task foot_3 and the wmt19 chinese-english news translation task. 5 on both tasks, we use the standard parallel training data, and compare our model with a strong transformer baseline, as well as related models from prior work. ## analysis in this section, we present the quantitative and qualitative analysis of our models. the analysis is performed on the experiments of the nist dataset. ## conclusion we have presented a noisy channel reranker and empirically validated it on chinese-english document-level translation tasks. the noisy channel formulation requires only parallel sentences (rather than documents) but we can use abundant monolingual documents to train the language model component. experiments show that our proposed model considerably improves translation quality-it achieves approximately 2.5 bleu higher than transformer baselines. subjective evaluation further confirms that the language model helps enforce consistency of tense, number, and lexical choice across documents.