Representational Isomorphism and Alignment of Multilingual Large Language Models (2024)

Abstract: In this extended abstract, we investigate the capability of Large Language Models (LLMs) to represent texts in multilingual contexts. Our findings reveal that sentence representations derived from LLMs exhibit a high degree of isomorphism across languages. This existing isomorphism facilitates representational alignments in few-shot settings. Specifically, by applying a contrastive objective at the representation level with only a small number (e.g., 100) of translation pairs, we significantly improve models’ performance on Semantic Textual Similarity (STS) tasks across languages.

## introduction representational isomorphism has been recognized as a key factor of few-shot capabilities @xcite . in this paper, we analyze multilingual sentence representations in llms through the lens of isomorphism. by examining the geometric properties of sentence pairs, we show that while embeddings from different languages are not well clustered in a common space, they exhibit high isomorphism. projecting them via an orthogonal matrix effectively aligns representations across languages. it also explains the previous success of combining non-english inputs with english prompts @xcite . building on this observation and previous studies highlighting representational isomorphism as a key factor in few-shot capabilities, we explore multilingual semantic alignment in llms. using just 100 english-centric translation samples with contrastive loss across language pairs, we achieve effective representation space alignment. this significantly improves cross-lingual semantic textual similarity (sts) task performance, proving more efficient than continued multilingual training. notably, this also boosts sts performance within individual languages, even without a monolingual objective. ## conclusion in this extended abstract, we show that llms' representations exhibit a high degree of isomorphism across languages, which explains their crosslingual zero-shot or few-shot capabilities in a multilingual context. further experiments demonstrate that llms' semantic representations can be enhanced across languages through alignment using just 100 translation samples, offering a more efficient and effective approach than sample-level pretraining or instruction tuning.