Concept Space Alignment in Multilingual LLM s (2024)

Abstract: Multilingual large language models (LLMs) seem to generalize somewhat across languages. We hypothesize this is a result of implicit vector space alignment. Evaluating such alignment, we see that larger models exhibit very high-quality linear alignments between corresponding concepts in different languages. Our experiments show that multilingual LLMs suffer from two familiar weaknesses: generalization works best for languages with similar typology, and for abstract concepts. For some models, e.g., the Llama-2 family of models, prompt-based embeddings align better than word embeddings, but the projections are less linear – an observation that holds across almost all model families, indicating that some of the implicitly learned alignments are broken somewhat by prompt-based methods.

## introduction cross-lingual word embeddings are typically induced by supervised or unsupervised alignment of the word vector spaces of monolingual language models. compression in multilingual models, i.e., parameter efficiency, can also drive implicit alignment @xcite @xcite , but until recently, the mappings could still be much improved by supervised or unsupervised alignment @xcite . multilingual large language models (llms) are increasingly used for different tasks and demonstrate impressive ability in understanding different languages, but it is unclear whether this is a result of improved, implicit alignment, or of something else, e.g., linguistic overlap or semi-parallel subsets of training data. llms have shown promising capability to comprehend english concepts @xcite . our paper sets out to evaluate concept alignment in multilingual llms. we aim to investigate two things: first, is there a linear mapping between corresponding concepts in different languages? second, how does a learned linear mapping generalize to new concepts? we explore both questions by revisiting a set of techniques used in early work on bilingual dictionary induction @xcite @xcite . we evaluate multilingual llms as if they were bilingual dictionary induction algorithms by doing nearest neighbor search -with cross-domain local scaling @xcite ) -and evaluating their retrieval precision (precision@k). we first derive concept embedding in their standard way (last token or average). since many of these models were instruction fine-tuned, we also compare promptbased embeddings to standard techniques based on (low-level) word embeddings. we then compare their precision to retrieval rates after explicit concept space alignment. we perform analyses with and without leakage, across multiple languages, and across both abstract and physical concepts. contributions our findings across experiments with 10 llms and six languages suggest that linear alignment can be induced in multilingual llms (if sufficiently big) to map concepts across different languages. compared to vanilla embeddings, prompt-based concept embeddings exhibit significantly lower linearity, and the gaps between before and after alignment are larger for prompt-based embeddings. this suggests that some of the implicitly learned concept alignments are broken by prompt-based methods. prompt-based embeddings, which are now commonly used in different retrieval scenarios, seem to be less effective in extracting cross-lingually alignable embeddings, compared to vanilla embeddings. results are generally good, but the old problem of generalization across typological distance @xcite rears its ugly face again, with basque, finnish, japanese and thai exhibiting generally lower overall performance for both experimental set-ups. furthermore, abstract concepts exhibit better alignment than physical concepts. we suspect that it is because abstract concepts are more frequent and occur in more diverse contexts. to create a seed dictionary (training data) for supervised alignment, we randomly sample 3,000 parallel concepts, foot_0 including 1,500 abstract concepts and 1,500 physical concepts. the 3,000 concepts are used to induce the linear mapping. llms we experiment with four different llm families with varying sizes: llama2 (7b, 13b, 70b) @xcite , mt0 (1.2b, 3.7b, 13b), bloomz (1b7, 3b, 7b1) @xcite , and aya101 (13b) @xcite . we use two different concept space extraction methods (vanilla and prompt-based). the vanilla method simply uses the last token representation as the concept embedding for decoder-only models (llama2 and bloomz); and the average embedding of the last hidden layer of the encoder as the concept embedding for encoder-decoder models (mt0 and aya101) 2 . the prompt-based extraction method exploits the fact that all these models were instructiontuned. the template we use for prompt-based extraction is adapted from @xcite and shown as follows: summarize concept [text] in one [lang] word: alignment and retrieval we rely on procrustes analysis @xcite , a form of statistical shape analysis, to discover good linear transformations (e.g., translation, rotation, and scaling) between concept spaces in different languages. suppose x and y are two matrices of size n × d (n is the seed dictionary size, d is the embedding size) such that the ith row in x is an embedding of concept c i in one language, and the ith row in y is c i 's embedding in the other language. the linear transformation is derived through singular value decomposition (svd) of y x t : ## conclusion we evaluated concept alignment on multilingual llms by revisiting the traditional bilingual dictionary induction task, but with semantic concepts rather than words. our experiments show that multilingual llms exhibit high-quality, linear concept alignment across different languages. however, the ability of supervised maps to generalize varied across different models, languages, and ways of obtaining embeddings.