What if I ask in alia lingua? Measuring Functional Similarity Across Languages (2025)

Abstract: How similar are model outputs across languages? In this work, we study this question using a recently proposed model similarity metric— \kappa_p —applied to 20 languages and 47 subjects in GlobalMMLU. Our analysis reveals that a model’s responses become increasingly consistent across languages as its size and capability grow. Interestingly, models exhibit greater cross-lingual consistency within themselves than agreement with other models prompted in the same language. These results highlight not only the value of \kappa_p as a practical tool for evaluating multilingual reliability, but also its potential to guide the development of more consistent multilingual systems.

## introduction users interact with large language models (llms) in a variety of languages across families and resource availabilities @xcite . as such, there is a need for llms to perform well across languages. these models should provide consistent responses-if switching languages results in incorrect answers to the same question, it could potentially mislead users, especially in critical areas like medical advice or legal interpretation. however, current evaluations primarily focus on per-language accuracy, with little attention to consistency across languages @xcite @xcite . to quantify this consistency, we study the functional similarity of model outputs. we use chance adjusted probabilistic agreement (capa or κ p ), a metric recently proposed by @xcite , which incorporates model accuracy on a given benchmark. we extend it to measure how similar the mistakes are across different languages, giving a view of multilingual functional similarity. we use globalmmlu @xcite -a carefully translated version of mmlu across multiple languages -as our benchmark. it tests the factual qa capabilities of models across a variety of subjects, ranging from mathematics to philosophy, in a multiple-choice format. our choice of this benchmark is motivated by its parallel nature, which allows us to test whether models behave consistently across languages on factual tasks. our study encompasses two dimensions of functional similarity: intra-model (consistency across languages for a given model) and inter-model (consistency across models for a given language). when considering intra-model similarity, we find that with increasing size and accuracy, models are becoming more functionally similar across languages. notably, we observe that all models are more consistent with themselves across languages than they are with other llms for the same language, indicating that intra-model similarity exceeds inter-model similarity for our task. interestingly, multilingual similarity further varies by domain and resource levels of the languages. primarily, we show that κ p , a chance-adjusted functional similarity metric, provides a powerful lens for analyzing multilingual consistency of llms. we explore cross-lingual patterns that accuracy and representational similarity alone cannot capture, by combining the output behavior and performance of the llm. we find interesting patterns about multilingual model behavior, including effects of scale, domain, and resources. ## related work similarity metrics: prior work on model similarity falls broadly into two classes: representational similarity and functional similarity. representational similarity metrics @xcite focus on the internal states of models such as weights and activations, whereas functional similarity metrics @xcite evaluate models based on their input-output behavior, making them applicable across architectures. importantly, functional similarity better reflects the user experience, since what ultimately matters is whether models behave consistently across inputs, rather than how their internal representations align. ## methodology the accuracy of llms differ greatly across languages and their performance is particularly in lowresource languages @xcite . this can artificially inflate similarity scores for some languages as high performance leaves little room for disagreement (as explained further in appendix a). given that κ p addresses these issues, we use it to compare similarity of model outputs in light of variable performance across languages. our work complements studies on representational similarity across languages such as @xcite . κ p computes observed agreement c p obs as the proportion with which the same option is selected across samples. to account for agreement by chance, κ p introduces an expected agreement c p exp , derived from the marginal distribution of each set of predictions. the κ p score is given by: we use the discrete variant of κ p as described in @xcite . as κ p increases, models make more similar mistakes, and their errors become more correlated, making them functionally more similar. henceforth, we compute the average κ p using micro-averaging by concatenating all datasets in the group and then computing the κ p across the combined set. since κ p is non-linear, the technique of micro-averaging is preferred as it smooths out extremes and operates directly at the per-sample level to better understand κ p across a dataset. we use gemma-3 (1b, 4b and 12b variants) @xcite and qwen-3 (1.7b, 4b, 8b and 14b variants) @xcite in our experiments, as they are some of the latest models as of august 2025 which have undergone multilingual pretraining. we also use the older gemma-7b @xcite as a sanity check. we evaluate these models on a subset of 20 languages of the globalmmlu dataset @xcite with our choice of languages justified in appendix b. building on our evaluation methodology, we leverage the lm evaluation harness @xcite , a unified framework for testing generative language models on a wide variety of benchmarks known for its reproducibility and extensive adoption. ## conclusion we introduced κ p as a functional similarity metric for evaluating multilingual consistency in llms. across globalmmlu, we found that larger and more capable models are more consistent across languages, with intra-model similarity exceeding inter-model similarity. consistency also varies by domain -being higher in stem than in culturally sensitive subjects -and by resource availability, with high-resource languages showing stronger inter-model agreement. together, these results establish κ p as a practical tool for analyzing multilingual functional behavior beyond accuracy alone.