Improving Language Model Integration for Neural Machine Translation (2023)

Abstract: The integration of language models for neural machine translation has been extensively studied in the past. It has been shown that an external language model, trained on additional target-side monolingual data, can help improve translation quality. However, there has always been the assumption that the translation model also learns an implicit target-side language model during training, which interferes with the external language model at decoding time. Recently, some works on automatic speech recognition have demonstrated that, if the implicit language model is neutralized in decoding, further improvements can be gained when integrating an external language model. In this work, we transfer this concept to the task of machine translation and compare with the most prominent way of including additional monolingual data - namely back-translation. We find that accounting for the implicit language model significantly boosts the performance of language model fusion, although this approach is still outperformed by back-translation.

## introduction machine translation (mt) is the task of automatically translating text from one language to another. nowadays, the dominant approach is neural machine translation (nmt), where a neural network is used to predict the probability of a sentence in the target language, given a sentence in the source language @xcite . for this approach to be effective, a large number of bilingual training samples -consisting of sentences and their corresponding translationsis needed. this poses a challenge, especially when we want to build a system for a specific domain, where zero or only limited amounts of in-domain bilingual data are available. in these situations, people turn towards monolingual text data, which is simply text in the source or target language and of which plenty exists for most languages and domains. before nmt became feasible, the preferred way of incorporating additional monolingual data in the mt system was the usage of an external target-side language model (lm), which is trained on monolingual data to predict the probability of a sentence @xcite @xcite . however, with the rise of nmt, it was found that a technique called back-translation outperforms the lm incorporation by a large margin @xcite . back-translation is a two step process, where we first create synthetic parallel data by automatically translating target side monolingual data into the source language. then, the final nmt system is trained on the combination of the real and synthetic parallel data. it was argued that the backtranslation approach better suits the nmt framework because the nmt system implicitly learns an internal language model (ilm) as part of the training, which might interfere with an additional external lm @xcite . more recently, for automatic speech recognition (asr), there have been works focusing on neutralizing this ilm before combination with an external lm and significant improvements were reported @xcite @xcite @xcite . in this work, we adapt the methods for ilm compensation, developed for asr, and test them for nmt. we compare against back-translation in different settings and find that ilm compensation significantly boosts the performance of lm fusion, although back-translation is still outperforming this approach for nmt. also, applying ilm compensation on top of back-translation does not result in significant performance improvements. ## related work several approaches to combine an lm and nmt model have been proposed in the past. shallow fusion (sf) is the most straight forward way, using a weighted log-linear combination of the model output probabilities @xcite . deep fusion denotes the concatenation of the hidden states of nmt model and lm and requires joint fine-tuning of both models @xcite . simple fusion is similar to shallow fusion, but the nmt model is trained using information from a pre-trained lm @xcite . for the task of asr, people recently have started to remove the ilm that is implicitly learned. the biggest question there is, how to best approximate the ilm. approaches include: (1) training an additional lm on the target side of the parallel data @xcite , (2) removing/averaging encoder information @xcite @xcite and (3) training a small sub-network while freezing all other parameters @xcite . as an alternative to lm fusion, back-translation @xcite @xcite has become the standard method for incorporating additional monolingual data for nmt. some work has been done to improve this approach, including sampling @xcite , tagging @xcite and block-bt @xcite . for sake of simplicity, we focus on the standard back-translation approach using beam search in this work. apart from using an external lm and backtranslation, additional monolingual data can also be utilized by pre-training @xcite , multi-task-learning @xcite or post-editing @xcite . in principle, all these approaches can also be combined with lm fusion, potentially further improving the performance of the resulting system. ## internal lm estimation during decoding, given a source sentence f j 1 and a model p (e i 1 |f j 1 ), we want to find the translation êî 1 that maximizes in our framework, p is the combination of three models: in principle, the ilm can be exactly calculated from the nmt model by marginalizing over all source sentences f j 1 . however, this summation would be intractable. instead, different ilm approximations have been proposed in the recent past for asr, which we will briefly recall here. for a more in-depth discussion of the different approximation methods we refer the reader to @xcite . ## experiments we perform experiments on four machine translation tasks, representing different data conditions. we tokenize the data using byte-pair-encoding @xcite with 15k joint merge operations (40k for wmt14). the models are implemented using the fairseq toolkit @xcite following the transformer base architecture @xcite . the details of the training setups can be found in appendix a. all systems are trained until the validation perplexity no longer improves and the best checkpoint is selected using validation perplexity as well. we use beam-search with beam-size 12 and utilize sacrebleu @xcite and ter @xcite . we report bleu and ter since we are most familiar with these metrics and to be comparable with previous works. however, we acknowledge that these metrics might have some biases and in future work it might be worth utilizing additional metrics like comet @xcite and bleurt @xcite . additionally, in future work we should separate our test sets for original source and target text to better understand the effect of translationese in both training and test data, as this might very much influence the improvements we see, especially in the case of back-translation @xcite . ## conclusion we re-visit the method of language model integration for neural machine translation. we implement and experiment with a new approach of neutraliz-ing the implicit language model, which has already shown promising result for the task of automatic speech recognition. we find that ilm neutralization significantly improves the translation quality compared to standard shallow fusion. however, back-translation as an alternative way to incorporate additional monolingual data, still outperforms the approaches using an external language model. therefore, for future work we will focus on scenarios where back-translation can not be applied effectively, e.g. when the quality of the initial nmt system is too bad to create helpful synthetic data.