Languages Transferred Within the Encoder: On Representation Transfer in Zero-Shot Multilingual Translation (2025)

Abstract: Understanding representation transfer in multilingual neural machine translation (MNMT) can reveal the reason for the zero-shot translation deficiency. In this work, we systematically analyze the representational issue of MNMT models. We first introduce the identity pair, translating a sentence to itself, to address the lack of the base measure in multilingual investigations, as the identity pair can reflect the representation of a language within the model. Then, we demonstrate that the encoder transfers the source language to the representational subspace of the target language instead of the language-agnostic state. Thus, the zero-shot translation deficiency arises because the representation of a translation is entangled with other languages and not transferred to the target language effectively. Based on our findings, we propose two methods: 1) low-rank language-specific embedding at the encoder, and 2) language-specific contrastive learning of the representation at the decoder. The experimental results on Europarl-15, TED-19, and OPUS-100 datasets show that our methods substantially enhance the performance of zero-shot translations without sacrifices in supervised directions by improving language transfer capacity, thereby providing practical evidence to support our conclusions. Codes are available at https://github.com/zhiqu22/ZeroTrans.

## introduction state-of-the-art neural machine translation systems are adaptable to multilingualism, resulting in a single encoder-decoder model that executes arbitrary translations by adding a tag specified to the target language at the beginning of source sentence (firat guided by our findings, we propose two methods for the encoder and decoder, respectively, to improve multilingual representations: low-rank language-specific embedding (lole) is applied to bias the representations in the subspaces of target languages at the encoder; and language-specific contrastive learning of representations (lclr) is applied at the decoder to isolate representational space across languages. we evaluated the proposed methods on three benchmarks, europarl-15 @xcite , ted-19 @xcite , and opus-100 @xcite , for two automatic metrics, sacrebleu @xcite and bertscore @xcite . the experimental results show that our methods outperform strong baselines in training from scratch because of improved representational transferability. our methods also perform effectively in fine-tuning, even though pre-trained models are trained by different strategies of language tags, which proves that target language information on the encoder side consistently benefits mnmt. ## investigating representation transfer in mnmt we conduct preliminary experiments to investigate representations by introducing identity pairs as base measures using two different datasets, europarl-15 @xcite and ted-19 @xcite , which are introduced in appendix b in detail. then, following @xcite , our investigation is based on transformer models with 6 encoder and decoder layers. we also investigate scenarios with 8 and 10 encoder layers. appendix c introduces the detailed model settings. ## encouraging representation transfer to validate the findings in section 3, we propose two methods on the encoder and decoder sides, respectively, to improve transferability. based on the findings in sections 3.2 and 3.3, improving the extent of language transfer in the encoder can overcome the hindered representations of zero-shot language pairs. we introduce a learnable embedding referred to as low-rank language-specific embedding (lole). it serves as biases to force representations to transfer into the target language with negligible cost. based on the findings in section 3.4, the capacity for multilingual features is insufficient at the lower layers of the decoder. we introduce language-specific contrastive learning of representations (lclr) as an training extra task to regularize the representations to specify the representational boundary for each language. ## related works prior studies on analyzing multilingual representation in section 2.2 led to several effective methods in mnmt. some works focused on updating and constraining the encoder to improve multilingual representations, and the findings in discrepancy mentioned in section 2. 2024 ) focused on improving the target language representation on the decoder side or adding modules specified to the target language to the decoder. given that the above works can all be encompassed within our analyses, we argue that this work offers insights for future improvements in mnmt. specifically, enhancing the encoder to transfer source language representations into the target language subspace and align semantic information within those subspaces is the key to improving mnmt. in addition, a critical factor of this work is the introduction of the identity pair as an analytical tool. specifically, while identity pairs have been heuristically used in prior works @xcite @xcite , as an assumed indicator of language-specific representation states, they have not been subject to systematic or quantitative analysis. in contrast, we explicitly define, validate, and utilize identity pairs to probe representational properties in a controlled and measurable way. this not only strengthens the empirical basis of our conclusions but also constitutes an important methodological contribution of this work. ## conclusion we systematically investigated the representational issue of zero-shot translation deficiency in multilingual neural machine translation models. our analyses show that the encoder transfers translation representations from the source language to the target language, and aligns semantics across different source languages at the target language subspace. we applied engineering practices to verify our findings by proposing two orthogonal methods, which substantially improve the zero-shot translation capacity. thus, our findings are significant for guiding the improvement of the transferability of multilingual representations.