Improving Machine Translation with Large Language Models: A Preliminary Study with Cooperative Decoding (2024)

Abstract: Contemporary translation engines based on the encoder-decoder framework have made significant strides in development.However, the emergence of Large Language Models (LLMs) has disrupted their position by presenting the potential for achieving superior translation quality.To uncover the circumstances in which LLMs excel and explore how their strengths can be harnessed to enhance translation quality,we first conduct a comprehensive analysis to assess the strengths and limitations of various commercial NMT systems and MT-oriented LLMs. Our findings indicate that neither NMT nor MT-oriented LLMs alone can effectively address all the translation issues, but MT-oriented LLMs show promise as a complementary solution to NMT systems.Building upon these insights, we propose Cooperative Decoding (CoDec), which treats NMT systems as a pretranslation model and MT-oriented LLMs as a supplemental solution to handle complex scenarios beyond the capability of NMT alone.Experimental results on the WMT22 test sets and a newly collected test set WebCrawl demonstrate the effectiveness and efficiency of CoDec, highlighting its potential as a robust solution for combining NMT systems with MT-oriented LLMs in the field of machine translation.

## introduction over the years, the encoder-decoder framework has established neural machine translation (nmt) models as the prevailing standard, achieving impressive translation quality through extensive training on large-scale and high-quality parallel data @xcite @xcite . commercial machine translation engines, e.g., google translate, are proficient in addressing the majority of translation requirements. more recently, with the emergence of generative large language models (llms), the position of traditional nmt models has been challenged (brown et @xcite . while commercial llms like openai's gpt-4 currently perform well in translation @xcite @xcite , they are constrained by their interface nature, thereby limiting further customization and improvement due to privacy concerns in industrial applications. a more promising approach involves fine-tuning relatively smaller llms (i.e., fewer than 13b parameters) to create llms specifically tailored for mt @xcite @xcite . in this context, this study aims to investigate the following research questions: in which scenarios do mt-oriented llms demonstrate superior performance to conventional nmt models, and how can we leverage the strengths of the two paradigms to enhance translation quality? to begin, we conduct a comprehensive analysis into the characteristics of translations generated by commercial nmt systems and mt-oriented llms. our findings reveal that commercial nmt systems excel at producing adequate translations in specific domains or languages. conversely, mtoriented llms demonstrate proficiency in gener-ating authentic-sounding translations and handling infrequent words that are not effectively processed by nmt systems. in summary, mt-oriented llms can serve as valuable fallback systems in cases where the output of commercial nmt systems is unsatisfactory. to complement nmt with mt-oriented llms, @xcite introduced the hybrid threshold approach (figure 1(a) ), which employs the nmt system as the primary translation system. when the translation fails to meet the quality threshold determined by the quality estimation (qe) module, an alternative translation is generated using a gpt-like model. however, this approach faces two primary challenges. first, existing reference-free metrics struggle to align with human judgment, resulting in inaccuracies being propagated @xcite @xcite . second, the integration of neural quality estimation modules and the sequential execution by llms leads to increased decoding time @xcite , which poses concerns for efficient translation in practical applications. to address the above issues, we propose an efficient implementation approach for system ensembles called cooperative decoding (codec). as illustrated in figure 1 (b), the nmt system functions as the front-end module, generating an initial translation draft for a given input sentence. subsequently, the mt-oriented llm serves as both an evaluator and a refiner, which firstly evaluates the draft from a language modeling perspective, and then the llm refines the partial translation starting from a specific position where the token in the draft is not among the top-k token candidates suggested by the llm. since the evaluation process takes advantage of parallel computation and the frontend module can handle most situations effectively, codec is more efficient compared to using llms for complete decoding. the contributions of this paper are three-fold: ## preliminary experiments in this section, we conduct a series of analyses to quantitatively investigate the characteristics of translations from different systems. ## cooperative decoding we propose an innovative cooperative decoding approach. this approach leverages an nmt model as a pretranslation model and incorporates an mtoriented llm as a quality assessment module and fallback model if needed. the overview of cooperative decoding is shown in figure 6 and we will give a detailed description of each step in the following. step1: draft generation. given an input source sentence x, the nmt system generates the translation o using an autoregressive decoding strategy like beam search. the difference is that the transla-tion o is considered as a draft and requires further confirmation or modification before being used as the final output. step2: verification. we feed o into the mtoriented llm in a forward process, which fully utilizes parallel computing. the procedure is the same as training llms, and we can obtain a probability distribution v t at each position, which is modeled as p (o t |@xmath0, x). the distribution can be regarded as the confidence of the llm given the specific prefix of the draft o, and we use it to verify the prediction of the nmt model. one straightforward approach for verification is to check whether the token with the highest probability matches the prediction of nmt. if o is fortunately exactly the same with {argmax(v 1 ), ..., argmax(v n )}, the inference will finish with o as the final translation. however, high-quality generation does not follow a distribution of the highest probability of the next tokens, and the tokens in o that can be regarded as accurate may appear outside of the top-1 selection, like in beam search. to address this issue, we relax the matching constraint using the top-k candidates of the llm and define the verification criterion as o @xmath1. step3: re-decoding. the verification is performed from left to right, and we end the verification once there is a situation that does not meet the verification criteria, i.e., o t ′ / ∈ top-k(v t ′ ). then, we feed the verified prefix o t ′ -1 into the mt-oriented llm and use it to re-decode the subsequent sequence. compared to totally replacing nmt models with mt-oriented llms, our cooperative decoding can speed up the whole inference process due to the expensive cost of autoregressive decoding. the speedup is more significant when the longer draft is accepted. moreover, the cooperative mechanism alleviates the issue of inaccuracy of llms by exploiting the output of nmt models. ## conclusion we explore the strengths of both nmt and llm and propose codec that integrates the two to achieve superior performance compared to existing hybrid frameworks. notably, our codec offers reduced decoding latency compared to relying solely on llms for inference, and it does not require any modifications to the target llms. we believe that exploring more effective utilization of llms while considering practicality in both industry and academia is a valuable direction.