Language Models not just for Pre-training: Fast Online Neural Noisy Channel Modeling (2020)

Abstract: Pre-training models on vast quantities of unlabeled data has emerged as an effective approach to improving accuracy on many NLP tasks. On the other hand, traditional machine translation has a long history of leveraging unlabeled data through noisy channel modeling. The same idea has recently been shown to achieve strong improvements for neural machine translation. Unfortunately, na ̈ıve noisy channel modeling with modern sequence to sequence models is up to an order of magnitude slower than alternatives. We address this issue by introducing efficient approximations to make inference with the noisy channel approach as fast as strong ensembles while increasing accuracy. We also show that the noisy channel approach can outperform strong pre-training results by achieving a new state of the art on WMT Romanian-English translation.

## introduction unlabeled data has been leveraged in many ways in natural language processing including backtranslation @xcite @xcite , self-training @xcite , or language model pre-training which led to improvements in many natural language tasks @xcite . while pre-training has achieved impressive results on tasks where labeled data is limited, improvements in settings with abundant labeled data are modest @xcite with controlled studies showing a clear trend of diminishing returns as the amount of training data increases @xcite . in this paper, we focus on noisy channel modeling for text generation tasks, a classical technique from the statistical machine translation literature which had been the workhorse of text gen-eration tasks for decades before the arrival of neural sequence to sequence models @xcite . unlike pre-training approaches, this approach is very effective irrespective of the amount of labeled data: since a recent revival @xcite , it has been an important part in the winning entries of several high resource language pairs at wmt 2019 @xcite , improving over strong ensembles that used 500m back-translated sentences. at the low resource wat 2019 machine translation competition, noisy channel modeling was also a key factor for the winning entry @xcite . noisy channel modeling turns text generation on the head: instead of modeling an output sequence given an input, bayes' rule is applied to model the input given the output, via a backward sequence to sequence model which is combined with the prior probability of the output, typically a language model. this enables the effective use of strong language models trained on large amounts of unlabeled data. the role of the backward model, or the channel model, is to validate outputs preferred by the language model with respect to the input. a straightforward way to use language models is to pair them with standard sequence to sequence models @xcite . however, this does not address explaining away effects under which modern neural sequence models still suffer @xcite . as a consequence, models are susceptible to producing fluent outputs that are unrelated to the input @xcite . the noisy channel approach explicitly addresses this via the channel model. however, a major obstacle to efficient noisy channel modeling is that generating outputs is much slower than decoding from a standard sequence to sequence model. we address this issue by introducing several simple yet highly ef-fective approximations which increase the speed of noisy channel modeling by an order of magnitude to make it practical. this includes smaller channel models as well as scoring only a subset of the channel model vocabulary. experiments on wmt english-romanian translation show that noisy channel modeling can outperform recent pretraining results. moreover, we show that noisy channel modeling benefits much more from larger beam sizes than strong pre-training methods. ## the noisy channel approach we assume a sequence to sequence task that takes the input x to predict the output y. a standard sequence to sequence model directly estimates the probability p(y|x), referred to as a direct model. on the other hand, the noisy channel approach applies bayes' rule to model p(y|x) = p(x|y)p(y)/p(x) where p(x|y) predicts the source x given the target y and is referred to as the channel model, p(y) is a language model over the target y, and p(x) is generally not modeled since it is constant for all y. @xcite use transformer models to parameterize the direct model, the channel model and the language model. similar to @xcite , they use the following linear combination of the channel model, the language model as well as the direct model for decoding: despite this approximation, noisy channel decoding is still significantly slower than decoding with the direct model alone as shown in figure 1 . the reason for this is that the channel model repeatedly scores the entire input sequence at each time-step and this is done k 2 times for each beam. specifically, both the direct model and the language model compute k 1 × v scores at each time-step in order to make a decoding decision for each target token during beam search, where v denotes the vocabulary size which we assume to be similar between the input and output. in contrast, the channel model computes k 1 × k 2 × s × v scores for each target token, where s is the maximum source sequence length. this adds substantial compute and memory overhead, to the extent that the batch size at decoding often needs to be substantially reduced. this leads to slower inference on gpus since less computation can be parallelized. ## fast noisy channel modeling naïve online noisy channel modeling is significantly slower than standard direct models. in this section, we present approximations to make noisy channel modeling substantially faster. ## conclusion we introduced a number of approximations which greatly speed up noisy channel modeling for neural sequence to sequence models. this includes using channel models which are a fraction of the size of commonly used sequence to sequence models, pruning most of the channel model output vocabulary, and reducing the number of beam candidates scored by the channel model. our approximations are simple, yet, highly effective and enable comparable inference speed to ensembles of direct models while delivering higher accuracy. our experiments show that noisy channel modeling can outperform pre-training approaches by being able to better exploit wider beams. moreover, this is achieved while using a smaller amount of monolingual data.