Investigating the translation capabilities of Large Language Models trained on parallel data only (2025)

Abstract: In recent years, Large Language Models (LLMs) have demonstrated exceptional proficiency across a broad spectrum of Natural Language Processing (NLP) tasks, including Machine Translation. However, previous methods predominantly relied on iterative processes such as instruction fine-tuning or continual pre-training, leaving unexplored the challenges of training LLMs solely on parallel data. In this work, we introduce Plume (Parallel Language Model), a collection of three 2B LLMs featuring varying vocabulary sizes (32k, 128k, and 256k) trained exclusively on Catalan-centric parallel examples. These models perform comparably to previous encoder-decoder architectures on 16 supervised translation directions and 56 zero-shot ones. Utilizing this set of models, we conduct a thorough investigation into the translation capabilities of LLMs, probing their performance, the role of vocabulary size, the impact of the different elements of the prompt, and their cross-lingual representation space. We find that larger vocabulary sizes improve zero-shot performance and that different layers specialize in distinct aspects of the prompt, such as language-specific tags. We further show that as the vocabulary size grows, a larger number of attention heads can be pruned with minimal loss in translation quality, achieving a reduction of over 64.7% in attention heads.

## introduction neural machine translation (nmt) has traditionally relied on encoder-decoder architectures, where an encoder processes the source sentence and a decoder generates the target sentence based on the encoder's output. however, recent advancements have moved away from this paradigm, with the introduction of decoder-only large language models (llms). in these models, the source sentence acts as a prompt, eliminating the need for a conventional encoder. with the rise of llms, research has increasingly focused on adapting these models for translation tasks by using techniques such as prompt-tuning @xcite , instruction-finetuning @xcite , or continual pretraining @xcite . while these methods have shown impressive results, they open new questions about the performance of llms when trained exclusively on parallel data, and therefore, the possibility of having models that are trained directly on the task of machine translation. additionally, the majority of these models are trained predominantly on english-centric-corpora. to address these questions, our paper proposes a new approach consisting of training llms solely on parallel corpora to evaluate their efficacy in machine translation (mt). our investigation revolves around questions such as: how does an llm trained exclusively on parallel data perform? and how does the model leverage prompt information to ensure accurate translations? our contributions are twofold: firstly, we introduce plume (parallel language model), an innovative ensemble comprising three multilingual 2b llms, trained from scratch on catalan-centric parallel data. each model has a different vocabulary size (32k, 128k and 256k). all models are proficient in 16 supervised translation directions, as well as 56 zero-shot translation directions. results show comparable results to previous encoder-decoder architectures of similar size. secondly, to understand how these models work, we study how they utilize contextual information across different layers to execute translation tasks effectively. our experiments show distinctive attention patterns associated with the different parts of the prompt, and how they vary through the different attention blocks. we also observe how languages use the source tag information differently, leading to a large performance variability when this token is missing. as a byproduct, we propose a strategy to remove attention heads with minimal performance loss and study how vocabulary size impacts the appearance of redundant heads. finally, we study the cross-lingual space learned by the models and how it progresses through the model's attention blocks. ## related work neural machine translation (nmt) has predominantly relied on encoder-decoder architectures @xcite @xcite . these methods have proven effective by conditioning language models to generate translations that accurately retain the meaning of the source sentence. moreover, these systems are easily extendable to multilingual scenarios, enabling zero-shot translation between language pairs that have not been seen together during training @xcite . over the years, some approaches to nmt have dropped the traditional encoder-decoder setup to adopt decoder-only architectures @xcite . although these methods showed promise, they did not become the standard due to issues with context loss and hallucinations @xcite . recent advancements in training large language models (llms) @xcite @xcite , including techniques like scaling and rotary embeddings @xcite , have significantly enhanced the ability of decoder-only architectures to handle long contexts of hundreds or even thousands of tokens. consequently, several studies have proposed leveraging pretrained llms for nmt through continual pretraining and instruction tuning @xcite @xcite . these methods have demonstrated results comparable to traditional encoder-decoder systems, while also supporting multiple translation directions. however, training and adapting these systems to various languages remains challenging @xcite . creating a vocabulary that accurately rep-resents all supported languages can lead to performance disparities of up to 68% on some downstream tasks. additionally, interpretability methods have gained popularity in order to understand better how models utilize provided information and to guide further improvements @xcite . ## results in zero-shot directions, the plume models exhibit a decline in performance compared to supervised directions. however, the decline is more pronounced in the bleu scores than in the comet scores, indicating that the overall quality remains relatively robust. specifically, the plume 256k variant achieves a comet score of 0.84 on the flores-200 dataset and 0.81 on the ntrex dataset, which, although lower than its supervised performance, still demonstrates its zero-shot translation capabilities when training using only catalan as the bridge language. larger vocabulary sizes improve zero-shot translation. the results in to further understand the influence of the vocabulary size in zero-shot translation quality, we calculated the vocabulary overlap @xcite for each zero-shot translation direction as follows: ## conclusions this work demonstrates the successful training of an llm-based machine translation system from scratch using only parallel data. the achieved results are comparable to those of existing encoderdecoder architectures for supervised translation tasks. we identified that larger vocabulary sizes consistently improve translation quality across zero-shot directions, suggesting the potential benefits of experimenting with even larger or languagespecific vocabularies. further analysis revealed that different layers focus on distinct aspects of the prompt, particularly the source language tag, which exhibits significant language variation. by employing an appropriate criterion, we achieved a performance reduction of less than 2 bleu score while removing over 64.7% of attention heads. we also showed that with larger vocabularies, the model gains additional representational flexibility that allow for more heads to be pruned without significantly degrading performance. additionally, our exploration of the learned cross-lingual space demonstrates that languages get closer in the cross-lingual space as they get to deeper layers and highlight the layers with the most significant impact on the learned space. this research opens doors for further investigation. we identified "sink heads" that primarily focus on the bos token. exploring their utility and relationship to the learned cross-lingual representations presents an opportunity for future work. additionally, further research into the optimization of vocabulary size along model size could also lead to better nmt models.