Improving Language Transfer Capability of Decoder-only Architecture in Multilingual Neural Machine Translation (2025)

Abstract: Existing multilingual neural machine translation (MNMT) approaches mainly focus on improving models with the encoder-decoder architecture to translate multiple languages. However, decoder-only architecture has been explored less in MNMT due to its underperformance when trained on parallel data solely. In this work, we attribute the issue of the decoder-only architecture to its lack of language transfer capability. Specifically, the decoder-only architecture is insufficient in encoding source tokens with the target language features. We propose dividing the decoding process into two stages so that target tokens are explicitly excluded in the first stage to implicitly boost the transfer capability across languages. Additionally, we impose contrastive learning on translation instructions, resulting in improved performance in zero-shot translation. We conduct experiments on TED-19 and OPUS-100 datasets, considering both training from scratch and fine-tuning scenarios.results show that, compared to the encoder-decoder architecture, our methods not only perform competitively in supervised translations but also achieve improvements of up to 3.39 BLEU, 6.99 chrF++, 3.22 BERTScore, and 4.81 COMET in zero-shot translations. We release our codes at https://github.com/zhiqu22/PhasedDecoder.

## introduction multilingual neural machine translation (mnmt) @xcite aims to integrate multiple language translation directions into a single model. although multilingual translation systems based on large language models have demonstrated strong performance @xcite @xcite , current mnmt models with the encoder-decoder architecture @xcite @xcite remain a focus of research due to the competitive performance, fewer parameters, and reduced training costs @xcite figure 1a shows the bleu score. figure 1b shows the layer-wise language feature representations of a sentence where the x-axis indicates the layer number and the vertical line indicates the value range. specifically, we follow @xcite to compute a similarity score, where values higher than 0.5 mean the representation exhibits the target language features more and lower than 0.5 indicates showing more source language features. appendix a provides the details of implementation. @xcite . however, in mnmt, models with the decoder-only architecture foot_0 have shown underperformance by the empirical research of @xcite ; @xcite , as further evidenced by figure 1a . therefore, addressing the underdevelopment of decoder-only architectures in mnmt is crucial due to the advantage of zero-shot generalization @xcite , which potentially benefits zero-shot translation, i.e., translating language pairs unseen during training. we attribute the issue to the lack of language transferability, causing generations to rely solely on representations that always manifest the source language features. specifically, mnmt encoderdecoder models typically add a language tag indicating the target language at the beginning of the source tokens as a translation instruction @xcite , then, @xcite ; @xcite show that the encoder of mnmt models transfers source tokens to represent target language features more than source language features. as shown in figure 1b , the representation of source tokens extracted from the model with the encoder-decoder architecture mainly exhibits the target language features at the output of the encoder (red line), however, this characteristic is absent in decoder-only architectures (green and blue lines). we hypothesize that the decoder-only architectures merely capture the surface information of source tokens instead of transferring source tokens into a state with more target language features. we propose dividing the decoder-only architecture into two stages, namely, two-stage decoderonly (tdo). specifically, the representations of target tokens are excluded in the first stage to enforce language transfer using the translation instruction, and the target tokens are fused in the second stage, which follows the normal decoder-only manner. moreover, unlike the encoder-decoder architecture, where source and target tokens are processed separately, in the decoder-only architecture, source tokens pass through all layers. however, the training objective of mnmt only focuses on the target tokens, leading to the degradation of the target language features on the source token representation. thus, we introduce instruction-level contrastive learning (instrucl) as a training objective to supervise source tokens in the second stage. we evaluate the proposed methodologies on two datasets, ted-19 @xcite , and opus-100 @xcite , using four automatic evaluation metrics: bleu @xcite , chrf++ @xcite , bertscore @xcite and comet @xcite . experimental results show that, compared to encoder-decoder models, our models perform competitively in supervised translations and achieve improvements of up to 3.39 bleu, 6.99 chrf++, 3.22 bertscore, and 4.81 comet in zero-shot translations. we also analyze the variation of layer-wise representations at the sentence level to demonstrate the effects of our proposed methods. results prove that the gains of proposed methods in the decoder-only architecture derived from improving language transfer. ## related work although the large language model based on the decoder-only architecture performs satisfactorily in the multilingual translation @xcite , the sota models specialized on mnmt are still based on the encoder-decoder architecture @xcite due to the balance between costs and performances. @xcite ; @xcite empirically show that the decoder-only architecture does not have a distinct advantage in mnmt, and @xcite ; @xcite demonstrate that the reason could be the onefold style of training data comprising only translations, degrading the zero-shot ability of the decoder-only architecture @xcite . recent investigations of the encoder-decoder architecture in mnmt reveal the deficiency of the decoder-only architecture at the representation level. kudugunta et al. (2019); stap et al. (2023) point out that the sentence representations translating to two different target languages are gradually separated with the increase of layers. @xcite demonstrate that the encoder of mnmt model transfers the source sentence representation to the target side, leading to the representation of source tokens used in the generation with more target language features. this finding aligns with the prior empirical studies @xcite @xcite , which shows that increasing target language information can lead to performance improvements. moreover, this also supports our hypothesis that the weakness of the decoder-only architecture can be attributed to the lack of language transfer. ## conclusions in this work, we analyzed the reasons behind the underperformance of the decoder-only architecture in mnmt, identifying the lack of language transfer capability as the primary challenge. to address this, we introduced the two-stage decoder-only architecture. we also proposed instruction-level contrastive learning to overcome the issue from the perspective of representation optimization. we conducted experiments on two settings, i.e., training from scratch and fine-tuning, using the ted-19 and opus-100 datasets, and the results validate the effectiveness of our approach. through further experiments and representation analysis, we confirm that the improvements in our methods are derived from enhanced language transfer capabilities.