On-the-Fly Fusion of Large Language Models and Machine Translation (2024)

Abstract: We propose on-the-fly ensembling of a neural machine translation (NMT) model with a large language model (LLM), prompted on the same task and input. Through experiments on 4 language directions with varying data amounts, we find that a slightly weaker-at-translation LLM can improve translations of a NMT model, and such an ensemble can produce better translations than ensembling two stronger NMT models.We demonstrate that our ensemble method can be combined with various techniques from LLM prompting, such as in context learning and translation context.

## introduction for many english nlp tasks, llms @xcite @xcite are the clear state-of-theart-e.g. sentiment analysis @xcite , summarization @xcite . however, dedicated nmt outperforms all but the largest closed source llms @xcite and dedicated mt is stronger in low resource settings @xcite . we propose a novel integration of a llm and dedicated nmt model via token-level fusion. this ensembling combines strengths of each model, which emerge from their differences. llms are trained on more data than nmt models, and have more parameters. while llms are exposed to some parallel data @xcite , they are trained on vastly more monolingual data, which likely gives them different domain coverage and more fluency than dedicated models. nmt models are trained on the translation task. for example, @xcite found chatgpt is more likely to hallucinate but is stronger at translating the spoken domain, while dedicated models are stronger for medical domains and social-media-style noisy text. llms can easily be prompted with auxiliary information-such as domain and document context-while that is more complicated for nmt. in this work we: ## method we review standard inference of encoder-decoder nmt models and decoder only llms and then introduce our proposed ensemble of the two. ## experimental setup we aim to understand how our proposed method performs in high and low resource settings with strong models, and design our experimental setup accordingly. the parallel and monolingual training data for german and russian is from the wmt22 @xcite 1 shared task. the hausa data is from wmt21 @xcite . 2 the turkish evaluation data was based on wmt18 @xcite 3 and training data also includes additional data from opus @xcite , excluding paracrawl @xcite , since such noisy data @xcite would require filtering @xcite @xcite . as domain-specific test sets we use ted-100 @xcite and parapat @xcite . we also use ted-100 and ctxpro @xcite for document-level experiments. 4 we use back translation @xcite ) (with a 1:1 ratio of parallel to synthetic data) for all language pairs. we train transformer 'big' models for german, russian and turkish, and 'base' for hausa @xcite in marian nmt @xcite . 5 we use llama2 @xcite with 7 and 13 billion pa-1 https://www.statmt.org/wmt22/ 2 https://www.statmt.org/wmt21/ 3 https://www.statmt.org/wmt18/ 4 for ctxpro, we select a random sample of 2000 paragraphs for our experiments to reduce compute usage. 5 we convert models from marian to hugging face format. ## results when both models are of reasonable quality (de-en, ru-en, en-ru), ensembling (col. 5) results in better quality than either alone (col. 1 & 2). in all cases, the llm quality is worse than the mt model but ensembling with it improves most language directions. for de-en, the mt model is 0.9 comet stronger than the llm. the ensemble still improves over the mt model by 0.6 comet. the improvement is minor for en-de, where the llm was 21.9 points worse than mt. the llm translation quality for turkish in both direction is poor while the mt is good so the ensembles are essentially reduced to the mt model. both models are bad for hausa and the ensembles are unusable. § a.4 shows the effect of λ on translation quality. in-context learning: larger llm: @xcite found that llama-13b suffers from off-target issues, degrading translation out-of-english compared to the 7b model. we confirm their results- ensemble w/ llm 7b llm 13b ensemble w/ llm 13b 0-shot 5-shot λ 0-shot 5-shot 0-shot 5-shot λ 0-shot 5-shot ensemble to outperform the nearly 2x bigger 13b llm. ensembles for turkish and hausa are still not worthwhile due to the poor llm quality in these lower resource settings. we use the 7b model in all analysis for the remainder of this work. ## conclusion we propose an on-the-fly ensembling of a dedicated mt model with an llm, conditioned on the source and prompted for translation. we demonstrate that an llm can improve translation quality of a nmt model even if the llm is weaker at translation, provided the llm is good enough. we prompt the llm to imbue the sentence-based mt model with document-level ability, improving on sentence-level and context-focused metrics. we find that ensembling with an llm performs better than ensembling two mt models, even if each mt model is stronger than the llm. while this work focuses on mt, the same techniques can be explored for other tasks, and may be especially useful for situations where the llm and task-specific model have different properties and strengths.