S kip CLM : Enhancing Crosslingual Alignment of Decoder Transformer Models via Contrastive Learning and Skip Connection (2025)

Abstract: This paper proposes SkipCLM, a novel method for improving multilingual machine translation in Decoder Transformers. We augment contrastive learning for cross-lingual alignment with a trainable skip connection to preserve information crucial for accurate target language generation. Experiments with XGLM-564M on the Flores-101 benchmark demonstrate improved performance, particularly for en-de and en-zh direction translations, compared to direct sequence-to-sequence training and existing contrastive learning methods. Code is available at: https://github.com/s-nlp/skipclm.

## introduction recently, multilingual decoder transformer models @xcite , such as xglm @xcite , gemini @xcite , unbabel tower @xcite , claude 3 sonnet (anthropic, 2024) became highly performant in the machine translation tasks @xcite . to better understand the mechanisms behind the emergence of this strong performance, researchers began to explore the inner workings of these models, which revealed a multi-stage evolution of internal representations within these decoder transformer models @xcite @xcite . initially, transformer @xcite blocks project input token embeddings into a shared subspace. subsequently, layers enrich the residual stream with different features, corresponding to token prediction, contextual information, and tasks represented in the prompts of the model @xcite . finally, these enriched representations are mapped to output tokens @xcite . additionally, logit lens analysis indicates that tokens generated from layer activations in this second stage show a strong alignment with the dominant language in the model's training data @xcite . however, this alignment is much less effective for underrepresented languages, negatively impacting prompt comprehension and task performance. existing techniques such as afp @xcite ) and lens @xcite address multilingual misalignment for low-resource languages by incorporating an auxiliary contrastive loss to improve the alignment of initial layer representations with the pivot language. while improving performance on tasks like translation, adding contrastive loss alone suffers from a potential loss of information within the residual stream, which hurts the model's performance in such aspects as original language preservation, context understanding, and instruction following. the authors of afp added a separate instruction tuning stage to mitigate this information loss, but this greatly limited the applications of such models due to them being instruction tuned instead of utilized in a zero-shot manner. this paper proposes skipclm, a novel method of enhancing cross-lingual alignment of multilingual embeddings in decoder transformer models. we introduce a linear skip connection to transfer hidden representations from the initial stages directly to the final transformer blocks. this, in conjunction with contrastive learning, facilitates both improved alignment of input embeddings with the pivot language and subsequent effective remapping to the original language, mitigating the information loss associated with only relying on contrastive learning. ## background and related work in sec. 2.1, we discuss the "do llamas work in english" paper, which presented the interpretational framework, on which stems the idea of multilingual alignment. in sec. 2.2, we discuss infonce loss, which is essential for aligning the representations of parallel texts in several languages. in sec. 2.3 and sec. 2.4, we discuss pioneer works, which explored cross-lingual alignment using contrastive learning approaches. ## results and discussion we have trained three models for each language: a model with applied skip connection and with contrastive loss (our approach), a model with only contrastive loss (afp-like training) and a sequenceto-sequence trained model. tab. 1 shows our results. for english-german translation direction, our approach performs the strongest, achieving the highest scores in all metrics, including a notably lower ter compared to afp baseline. seq2seq training trails closely behind in this language pair. however, for english-turkish, seq2seq training shows best results, outperforming both our approach and afp in every metric, including a higher bleu score and lower ter. our approach is slightly behind afp in chrf, though comet scores for all models are tightly grouped, suggesting similar perceived translation quality. english-chinese results are mixed. seq2seq training leads in most metrics like bleu and ter, but our approach achieves the highest comet score, surpassing both seq2seq training and afp baseline. afp baseline consistently underperforms, confirming our concerns, that simply adding a contrastive loss, as shown in afp paper, leads to performance degradation, compared to the standard seq2seq training across all languages, underscoring the limitations of that approach. interestingly, our implementation of the contrastive baseline surpasses the results reported in the afp paper, likely due to improved hyperparameter tuning. examples of translation being done by each model are shown in appx. b. we hypothesize, that the performance discrep-ancy between german, chinese and turkish can be explained by optimizing α and β hyperparameters for the german language, which shows the best results. additionally, we believe that the performance of our method can be increased when training is being carried out on a multidirectional translation dataset instead of a single direction translation. ## conclusion we present a novel method for enhancing multilingual machine translation in decoder transformers by augmenting contrastive learning with a trainable skip connection. this approach aimed to mitigate the information loss often associated with contrastive learning methods while simultaneously improving cross-lingual alignment with a pivot language. our experiments on the flores-101 benchmark, using xglm-564m, demonstrated the effectiveness of this strategy, showing consistently better performance for german translation across all evaluation metrics, while being competitive for chinese and slightly worse for turkish languages.