MEXA : Multilingual Evaluation of E nglish-Centric LLM s via Cross-Lingual Alignment (2025)

Abstract: English-centric large language models (LLMs) often show strong multilingual capabilities. However, their multilingual performance remains unclear and is under-evaluated for many other languages. Most benchmarks for multilinguality focus on classic NLP tasks or cover a minimal number of languages. We introduce MEXA, a method for assessing the multilingual capabilities of pre-trained English-centric LLMs using parallel sentences, which are available for more languages than existing downstream tasks. MEXA leverages that English-centric LLMs use English as a pivot language in their intermediate layers. MEXA computes the alignment between English and non-English languages using parallel sentences to evaluate the transfer of language understanding from English to other languages. This alignment can be used to estimate model performance in different languages. We conduct controlled experiments using various parallel datasets (FLORES-200 and Bible), models (Llama family, Gemma family, Mistral, and OLMo), and established downstream tasks (Belebele, m-MMLU, and m-ARC). We explore different methods to compute embeddings in decoder-only models. Our results show that MEXA, in its default settings, achieves an average Pearson correlation of 0.90 between its predicted scores and actual task performance across languages. This suggests that MEXA is a reliable method for estimating the multilingual capabilities of English-centric LLMs, providing a clearer understanding of their multilingual potential and the inner workings of LLMs. Leaderboard: https://cis-lmu-mexa.hf.space, Code: https://github.com/cisnlp/MEXA.

## introduction most state-of-the-art autoregressive large language models (llms) are "english-centric", including closed-source models such as @xcite , open-weight models such as llama 3 @xcite ; and open-source models such as olmo @xcite . englishcentric refers to the majority of the pre-training data for these models being in english @xcite . even models labeled as heavily multilingual, such as bloom (big-science @xcite , have their major pre-training data in english @xcite . except for open-source models, where pretraining data is available and language distribution is transparent, there is still confusion about the language capabilities and coverage of other llms. primarily, the focus in evaluating llms has been on developing benchmarks to assess their performance in english. most benchmarks in multilingual settings consist of classical monolingual nlp tasks such as sequence labeling @xcite , automatic translation of popular english benchmarks such as mmlu @xcite into a limited number of languages @xcite , or the creation of languagespecific benchmarks @xcite @xcite @xcite . most llms are english-centric, either by choice or due to the availability of abundant data sources in english. for these models to be effective in other languages, the other languages must align with the dominant language, i.e., english. given such alignment, english could act as a "rising tide that raises all ships," meaning that improvements in english performance could benefit other languages, especially in tasks such as reasoning @xcite . contrarily, if a language does not align well with english, an english-centric llm may not provide meaningful coverage for that language. indeed, @xcite have found that for llama 2 @xcite , an english-centric llm, english could be seen as a kind of "pivot" language, enabling to solve complex semantic tasks in a foreign language through a detour into english. more precisely, they show that llama 2 was able to decode semantically correct next tokens in the middle layers, assigning higher probabilities to the english tokens than to the foreign version, which is only selected in the upper layers. @xcite present a hypothesis regarding the middle layers of english-centric llms, suggesting that these models use english as a means of reasoning while incorporating multilingual knowledge. based on their analysis, the number of language-specific neurons in the middle layers decreases within the self-attention mechanism but remains consistent across the layers of the feed-forward structure when processing multilingual queries. in this paper, we introduce mexa, a method to estimate the actual multilingual coverage of english-centric llms. it builds on the observation that these models semantically use english as a pivot language in their middle layers, by measuring how well embeddings of non-english sentences align with their english counterparts. we verify the mexa estimation of language coverage for each llm, using pearson correlation between estimated and actual scores for various tasks. we use two parallel datasets: flores-200 @xcite and bible @xcite ; nine llms: llama family, gemma family, mistral, and olmo; and three tasks: belebele @xcite , m-mmlu, and m-arc @xcite . our results show that mexa achieves a promising average pearson correlation of 0.90 with established downstream tasks across nine models and two parallel datasets. in our study on the calculation of mexa scores, we conduct multiple design analyses to examine the impact of token-level pooling for the embeddings (i.e., using the last token versus a weighted average) and layer-level pooling in computing alignment scores. while mexa demonstrates a high correlation across most setups, we find that a weighted average based on tokens, combined with mean pooling, yields the best results. in summary, mexa offers a scalable way to estimate the multilingual coverage of english-centric llms via alignment with english. ## background and related work we discuss distribution of pre-training data in llms, and multilingual evaluation benchmarks in appendices a.1 and a.2, and focus on crosslingual alignment here. research on cross-lingual alignment either aims to uncover the underlying mechanisms of alignment and assesses its impact on models and downstream tasks, or attempts to enhance model performance by enforcing alignment before, during, or after the pre-training phase. most of these papers have focused on encoder-only models, such as xlm-r @xcite and mbert @xcite , among others @xcite . in this work, we focus on decoder-only models. understanding alignment. @xcite show that english-centric models such as llama 1 @xcite possess multilingual transfer abilities (after fine-tuning on one source language, they can be applied to other languages) and may even surpass the multilingual transfer abilities of multilingual pre-trained models such as bloom @xcite . @xcite find that gpt-2-style decoderonly models show strong cross-lingual generalization when trained on an imbalanced mix of languages. however, when trained on a balanced language set, they do not observe increased performance compared to monolingual settings. @xcite perform single-token analysis to demonstrate that english-centered llms, such as llama 2, use english semantically as an internal latent language in the middle layers when handling multilingual queries. @xcite extend this analysis to multiple tokens, also showing that an llm dominated by both english and japanese uses both languages as internal latent languages. @xcite explore how llms handle multilingualism. they hypothesize that llms initially interpret the query and convert multilingual inputs into english for task-solving. in the middle layers, the models rely on english with self-attention mechanisms for reasoning, while employing multilingual knowledge through feed-forward structures. in the final layers, llms generate responses consistent with the original query language. @xcite and @xcite are even more closely related to ours. @xcite uses absolute cosine similarity values between last token embeddings derived from parallel sentences with english to predict the ranking of language performance across various models. however, as we discuss in section 3, relying solely on absolute cosine values can be misleading, and as shown in section 5.3, absolute cosine values are less correlated with downstream tasks than mexa score. @xcite uses english probing tasks and their automatic translations to construct a multilingual evaluation. while they compare embedding similarity scores between high-and low-resource languages with corresponding evaluation results, similar to @xcite , they do not assess whether these correlations hold across other downstream tasks. in section 5, we demonstrate that mexa scores align closely with a broad range of downstream tasks. boosting alignment. the idea to enforce alignment in encoder-only models using parallel sentences dates back to @xcite , and has been explored under various guises, e.g., using mixed-language sentences and/or bilingual dictionaries @xcite @xcite @xcite . recently, @xcite improved multilingual alignment by initializing the decoder-only models to generate similar representations of aligned words using contrastive learning and preserving this alignment using a code-switching strategy during pretraining. @xcite propose a data allocation technique to select a core subset of languages for fine-tuning, better aligning the multilingual capabilities of decoder-only llms and making them more truthful in their responses. @xcite propose aligning internal sentence representations across different languages using multilingual contrastive learning and aligning outputs by following cross-lingual instructions in the target language for decoder-only models. ## mexa we now describe the mexa method for computing the alignment score of language l 1 with a pivot language l 2 , given the language model m. in this paper, we use the term cross-lingual alignment, or simply alignment to refer to the semantic similarity of multilingual embeddings across languages. l 2 , for english-centric llms and in this paper, is english. to assess alignment, we use parallel sentences in two languages, l 1 and l 2 . the goal of semantic similarity is to ensure that parallel sentences have sufficiently high similarity, reflecting alignment between the two languages. however, considering only the absolute cosine similarity value as the alignment score does not guarantee proper alignment. for some languages, even non-parallel sentences exhibit similarity scores comparable to those of parallel sentences (see §5.3). this is largely due to the anisotropy problem observed in transformer models, which can lead to so-called hubness issues, making it difficult to distinguish between similar and dissimilar embeddings (ethayarajh, 2019), especially in multilingual models @xcite . however, a direct comparative analysis of the cosine similarity between parallel and non-parallel sentence pairs across languages can help overcome these issues. instead of using the absolute cosine similarity value for alignment, we assign binary values (1 or 0) based on whether a criterion for semantic similarity is satisfied. our criterion imposes that (a) parallel sentences should have high cosine similarity, and (b) non-parallel pairs should also have low similarity values, ensuring the similarity is not random or biased. specifically, if the cosine similarity for a pair of parallel sentences is higher than for any non-parallel sentences, we assign a value of 1 for this pair; otherwise, a value of 0. this approach sidesteps the hubness problem since the absolute cosine similarity values themselves are not directly used. to compute mexa, we first apply the cosine similarity function to the pairs of embeddings of parallel sentences in languages l 1 and l 2 . in section 3.1, we describe how embeddings can be computed for each layer l of the autoregressive language model m. we generate a square matrix c(l 1 , l 2 , m, l) representing cosine similarities of embeddings at the output of layer l for all parallel sentences in languages l 1 and l 2 . we denote c ij (l) the element in the i-th row and j-th column of c(l 1 , l 2 , m, l). it represents the cosine similarity between the i-th sentence of l 1 and the j-th sentence of l 2 at layer l of language model m. the diagonal elements of c, denoted c ii (l), represent the cosine similarity between parallel sentence pairs from l 1 and l 2 . we define the mexa alignment score µ c(l 1 , l 2 , m, l) as follows: layer-wise pooling. the mexa alignment score µ c(l 1 , l 2 , m, l) is computed for language l 1 respect to pivot language l 2 for each layer l of the language model m. to compute a single mexa alignment score given the language model m and l 1 , l 2 , we use mean and max pooling strategies over multiple layers. ## experiments we conduct experiments using various multiparallel datasets (flores-200 and the bible), models (llama family, gemma family, mistral, and olmo), and existing benchmarks/tasks (belebele, m-mmlu, m-arc). our objective is to assess how well the mexa alignment score from various parallel datasets correlates with the different benchmarks/tasks for different models. ## conclusion we introduce mexa, a method for assessing the multilingual capabilities of english-centric large language models (llms). mexa builds on the observation that english-centric llms semantically use english as a kind of pivot language in their intermediate layers. mexa computes the alignment between non-english languages and english using as shown, in the mid-layers, the embeddings become more language-neutral. the numbers shown in the midlayers are the ids of english sentences that are scattered.