Log-Linear Reformulation of the Noisy Channel Model for Document-Level Neural Machine Translation (2020)

Abstract: We seek to maximally use various data sources, such as parallel and monolingual data, to build an effective and efficient document-level translation system. In particular, we start by considering a noisy channel approach (CITATION) that combines a target-to-source translation model and a language model. By applying Bayes’ rule strategically, we reformulate this approach as a log-linear combination of translation, sentence-level and document-level language model probabilities. In addition to using static coefficients for each term, this formulation alternatively allows for the learning of dynamic per-token weights to more finely control the impact of the language models. Using both static or dynamic coefficients leads to improvements over a context-agnostic baseline and a context-aware concatenation model.

## introduction neural machine translation (nmt) @xcite has been reported to reach near human-level performance on sentence-by-sentence translation @xcite . going beyond sentence-level, documentlevel nmt aims to translate sentences by taking into account neighboring source or target sentences in order to produce a more cohesive output @xcite @xcite . these approaches often train new models from scratch using parallel data. in this paper, in a similar spirit to @xcite ; @xcite , we seek a document-level approach that maximally uses various available corpora, such as parallel and monolingual data, leveraging models trained at the sentence and document levels, while also striving for computational efficiency. we start from the noisy channel model @xcite which combines a target-to-source translation model and a document-level language model. by applying bayes' rule, we reformulate this approach into a log-linear model. it consists of a translation model, as well as sentence and document-level language models. this reformulation admits an auto-regressive expression of tokenby-token target document probabilities, facilitating the use of existing inference algorithms such as beam search. in this log-linear model, there are coefficients modulating the impact of the language models. we first consider static coefficients and, for more fine-grained control, we train a merging module that dynamically adjusts the lm weights. with either static or dynamic coefficients, we observe improvements over a context-agnostic baseline, as well as a context-aware concatenation model @xcite . similarly to the noisy channel model, our approach reuses off-the-shelf models and benefits from future translation or language modelling improvements. ## log-linear reformulation of the noisy channel model given the availability of various heterogeneous data sources that could be used for document-level translation, we seek a strategy to maximally use them. these sources include parallel data, at either the sentence or document level, as well as more broadly available monolingual data. as the starting point, we consider the noisy channel approach proposed by @xcite . given a source document (x (1) , . . . , x as such, the conditional probability of the target document given the source is expressed by we therefore generate context-aware translations by combining a translation model (tm) <n) ) language models (lm). to calibrate the generation process, we introduce coefficients α ∈ r and β ∈ r to control the contribution of each language model, which are tuned on a validation set: similarly to the noisy channel approach @xcite , we use off-the-shelf translation and language models. as such, future improvements to either translation or language modelling can easily be leveraged. our reformulation however admits a more efficient search procedure, unlike that by @xcite . ## dynamic merging as extra-sentential information is not uniformly useful for translation, we propose dynamic coefficients for the different models by generalizing eq. 1: ## conclusion in this paper, we set to use heterogeneous data sources in an effective and efficient manner for document-level nmt. we reformulated the noisy channel approach @xcite and end up with a left-to-right log-linear model combining a baseline machine translation model with sentence-level and document-level language models. to modulate the impact of the language models, we dynamically adapt their coefficients at each time step with a merging module taking into account the translation and language models. we observe improvements over a context-agnostic baseline and using dynamic coefficients helps capture documentlevel linguistic phenomena better. future directions include combining our approach with mt models trained on back-translated documents, exploring its applicability to other modalities such as vision and speech, and considering deeper fusion of the models.