Beyond Decoder-only: Large Language Models Can be Good Encoders for Machine Translation (2025)

Abstract: The field of neural machine translation (NMT) has changed with the advent of large language models (LLMs). Much of the recent emphasis in natural language processing (NLP) has been on modeling machine translation and many other problems using a single pre-trained Transformer decoder, while encoder-decoder architectures, which were the standard in earlier NMT models, have received relatively less attention. In this paper, we explore translation models that are universal, efficient, and easy to optimize, by marrying the world of LLMs with the world of NMT. We apply LLMs to NMT encoding and leave the NMT decoder unchanged. We also develop methods for adapting LLMs to work better with the NMT decoder. Furthermore, we construct a new dataset involving multiple tasks to assess how well the machine translation system generalizes across various tasks. Evaluations on the WMT and our datasets show that results using our method match or surpass a range of baselines in terms of translation quality, but achieve 2.4 \sim 6.5 \times inference speedups and a 75% reduction in the memory footprint of the KV cache. It also demonstrates strong generalization across a variety of translation-related tasks.

## introduction the last decade has yielded remarkable breakthroughs in machine translation (mt) through the use of deep neural networks, scaled dramatically in both model parameters and training data. during this period, early methods, known as neural machine translation (nmt), were largely based on the encoder-decoder architecture @xcite . in nmt, the machine translation problem is commonly treated as * corresponding author. ## related work designing widely applicable models for mt has been an active area of nlp research for decades. although mt models have evolved significantly over time, most of them still operate on an "analyzethen-generate" paradigm @xcite @xcite . for example, in statistical mt, the source-language sentence is parsed into either syntactic or non-syntactic forms, and the translation is generated by mapping these parsed forms to target-language constructions @xcite . llms can broadly be categorized as following a similar design: they first compute the key-value cache for each input sequence, and then produce output tokens in a left-to-right manner based on this cache. from a modeling perspective, therefore, decoupling the encoding and decoding processes is a natural design choice for both traditional mt and llms. in nlp, a large body of work has focused on pre-training and applying text encoders, such as the bert series of models @xcite , which are primarily designed to address language understanding problems. more recently, there have been attempts to use llms as text encoders, though these models are more commonly used for generating text. for example, @xcite and @xcite fine-tuned llms for text encoding and showed that llms can produce high-quality representations of text in various embedding tasks. however, it is rare to see studies on incorporating llms in the encoding for nmt or other language generation tasks. in fact, the focus of research in mt has shifted. recent studies in this field are now more concerned with the adaptation of llms, rather than the improvement of model architectures. one strand of research aims to enable llms to translate via prompting techniques, including designing better prompts @xcite , introducing translation demonstrations @xcite , and using chain-of-thought reasoning @xcite . another strand focuses on developing mt-specific llms via fine-tuning @xcite @xcite @xcite ; we have followed this approach, but with a focus on improving the model architecture, as well as developing a new benchmark for evaluating universal mt models. this work is also related to efficient methods for llms. for example, one can compress the model using quantization and pruning techniques @xcite , and speed-up inference using speculative decoding algorithms @xcite . but these methods and ours are in no way contradictory. since our model follows standard encoding (or prefilling) and decoding frameworks, it can be easily combined with various efficient methods to further improve efficiency. ## lamate in this section, we introduce lamate and its training method. ## commt many translation-related tasks, such as documentlevel translation, rely on both understanding complex inputs and generating coherent, contextually appropriate outputs, which are not captured by commonly used sentence-level translation tasks. to generalize lamate to diverse tasks and evaluate it on these tasks, we developed a new evaluation benchmark called commt, constructed from existing publicly available translation datasets. it comprises five tasks: we collected data mainly from public resources, driven by two goals: (i) to find as many data sources as possible in order to increase data diversity, and (ii) to collect as much high-quality, manually annotated translation data as possible. commt is multilingual and supports four languages: german, czech, russian, and chinese. the reader can refer to appendix section b for more details of commt. ultimately, we created a training set of 239k samples, as shown in figure 7 , and the test set statistics illustrated in ## conclusion in this paper, we explore the connection between the two worlds of llms and nmt, presenting lamate-a method that leverages llms as mt encoders and pairs them with lightweight decoders. we design an adaptor better to align llm's representations for the decoder and propose a two-stage training strategy to develop a universal translation model. additionally, we introduce commt, a new dataset suite encompassing diverse translationrelated tasks, facilitating the development and evaluation of universal translation models. experiments on commt demonstrate lamate's strong performance and generalization ability, while sig-nificantly improving computational efficiencyachieving 2.4× to 6.5× faster decoding speeds and reducing kv cache memory usage by 75%. we hope this study will provide valuable insights and inspire further exploration into optimizing llms and expanding their role in nlp tasks.