Representational Isomorphism and Alignment of Multilingual Large Language Models (2024)

Abstract: In this paper, we investigate the capability of Large Language Models (LLMs) to represent texts in multilingual contexts. Our findings show that sentence representations derived from LLMs exhibit a high degree of isomorphism across languages.This existing isomorphism can facilitate representational alignments in zero-shot and few-shot settings.Specifically, by applying a contrastive objective at the representation level with only a small number of translation pairs (e.g., 100), we substantially improve models’ performance on Semantic Textual Similarity (STS) tasks across languages. This representation-level approach proves to be more efficient and effective for semantic alignment than continued pretraining or instruction tuning. Interestingly, we also observe substantial STS improvements within individual languages, even without a monolingual objective specifically designed for this purpose.

## introduction large language models (llms) demonstrate significant potential in solving multilingual tasks, such as machine translation @xcite and multilingual qa @xcite . notably, they exhibit strong few-show capabilities @xcite , where a small number of samples can lead to substantial performance improvements. representational isomorphism has been identified as one key source of few-shot capabilities in the context of word translation @xcite . in this paper, we analyze the multilingual sentence representation of llms from the perspective of isomorphism. we start by examining the geometric properties of representations derived from pairs of translation sentences. using several widely used methods to extract embeddings from llms, we show that although the resulting embeddings are not well clustered in a common space for different languages, they exhibit high isomorphism -projecting them through an orthogonal matrix allows the sentence representations to be effectively aligned across languages. moreover, it also explains the previous success of combining non-english inputs with english prompts @xcite for llms in multilingual tasks, where we argue that this spatial transformation occurs when using english prompts. building on this observation, we further investigate the potential of multilingual semantic alignment upon llms. we show that using a small number of english-centric translation samples (e.g., 100) with a contrastive loss @xcite across language pairs effectively aligns the representation spaces. this alignment consistently improves performance on cross-lingual semantic textual similarity @xcite tasks, proving to be more efficient and effective than continued language model training with multilingual samples. interestingly, such progress also yields clear sts gains within each language, even in the absence of a monolingual objective specifically designed for this purpose. given its high efficiency and effectiveness, we advocate for exploring representationlevel alignment in future research. ## semantic analysis isomorphism is considered the foundation of fewor zero-shot capabilities in the context of word translation @xcite , where word-level semantics can be easily aligned across languages using a few word pairs. we hypothesize the existing isomorphism can also facilitate semantic alignments for llms across languages in a few-shot setting. inspired by @xcite , who applied a contrastive loss with a small dictionary to enable word-level semantic transfer, we explore the cross-lingual semantic alignment of llms at the sentence level through contrastive learning in this section. in section 3.1, we describe the setting of evaluating cross-lingual semantic alignment, i.e., semantic textual similarity (sts) tasks. in section 3.2, we describe using contrastive learning with a few sentence pairs to align cross-lingual semantics. ## conclusion in this paper, we investigate llms' representations from both geometric and semantic similarity perspectives. our findings demonstrate that llms' representations exhibit a high degree of isomorphism across languages, which facilitates their cross-lingual zero-shot or few-shot capabilities in a multilingual context. for example, we show that the semantics representation of llms can easily be enhanced across languages by alignment at the representation level using as few as 100 translation samples, which is much more efficient and effective than sample-level pretraining or instruction tuning.