Multilingual Learning Strategies in Multilingual Large Language Models (2025)

Abstract: Despite the effective performance of multilingual large language models (LLMs), the mechanisms underlying their multilingual capabilities remain unclear. This study examines the intermediate representations of multilingual LLMs to determine if these models utilize human-like second language acquisition strategies: coordinate, sub-coordinate, or compound learning. Our investigations into the discriminative and generative aspects of these models indicate that coordinate learning is the dominant mechanism, with decoder-only models progressively developing distinct feature spaces for each language, while encoder-only models exhibit a mixture of coordinate and compound learning in their middle layers. We find little evidence for sub-coordinate learning. Moreover, the role of training data coverage in shaping multilingual representations is reflected in the fact that languages present in a model’s training data consistently exhibit stronger separation than those absent from it.

## introduction large language models (llms) have exhibited impressive performance across multiple languages in a wide range of tasks @xcite . however, the underlying mechanisms that enable their multilingual capabilities remain largely unexplored. recent studies suggest that these capabilities may stem from a combination of implicit translation into a dominant language like english and internally adopted language-specific processing strategies @xcite . however, these studies primarily base their hypotheses on the generative capabilities of language models, leaving the explicit exploration of their internal mechanisms unaddressed. we fill this gap by providing a granular perspective on the internal mechanisms underlying multilingualism in multilingual large language models. specifically, we examine the intermediate representations (activations) of llms to identify the presence of multilingual information that supports each of the three types of bilingualism in human language learners: coordinate, sub-coordinate, and compound learning (d' @xcite . we generalize human bilingualism into multilingualism and conceptualize it in terms of the vector representation of linguistic units formed in the intermediate activations of an llm. figure 1 illustrates this conceptualization. coordinate learners acquire languages in distinct environments, such as home and school, leading them to process each language independently through separate cognitive systems. in other words, coordinate learners tend to develop language-specific feature spaces, where each language is encoded in its own dedicated representational structure with minimal cross-linguistic influence. from a language model perspective, coordinate learning manifests as distinct language clusters in the intermediate representations. sub-coordinate learners, however, interpret languages through the lens of a dominant language by implicitly translating non-dominant languages into the dominant one. this typically occurs in late acquisition, low proficiency, or non-immersive settings, where the learner relies on mental translation rather than direct comprehension. from the perspective of a language model, this translates to the existence of a broad feature space for the dominant language, which includes other languages. in contrast, compound human learners develop a core, universal understanding of language, where linguistic units such as word categories and concepts are partially shared across different languages and expressed through varying verbal forms. these learners acquire multiple languages simultaneously within the same environment and tend to abstract away language-specific properties. in a language model's feature space, this translates into the existence of feature spaces shared across all languages. the training environment of multilingual language models resembles that of coordinate and compound learners, as their training data are sampled from multiple language sources, but each segment primarily consists of a pragmatically complete text (i.e., coherent and self-contained segments, such as articles or conversational exchanges) in a single language, with limited language mixing. accordingly, we hypothesize that multilingual language models primarily adopt a coordinate learning strategy with some degree of compound learning, while sub-coordinate learning, if present, is likely restricted to unseen languages. we employ two complementary strategies to investigate this hypothesis based on the intermediate activations of language models. the first is a discriminative approach, quantifying languagespecific and universal information in intermediate feature activations. the second examines the models' generation process by analyzing the contribution of intermediate features to token generation. our findings across different llm architectures strongly support the view that multilingual processing in these models aligns primarily with coordinate learning, with partial evidence of compound learning. decoder-only models such as mgpt @xcite and bloom @xcite predominantly rely on coordinate learning, whereas encoder-only models like mbert @xcite and xlm-r @xcite exhibit a more complex interplay of coordinate and compound strategies. evidence for sub-coordinate learning is limited, as none of the models show a strong dependence on a dominant language to process others. when it comes to implications of compound learning, previous studies have suggested the existence of partially shared subspaces between languages in mbert @xcite . specifically, @xcite attribute mbert's crosslingual capabilities to its language-independent tokenization, while @xcite demonstrate that the model shares portions of its representations across languages, suggesting that compound learning supports cross-lingual transfer through overlapping representational subspaces. yet, whether these subspaces reflect universal linguistic features or artifacts of training remains an open question that our analysis investigates. this paper extends previous research by examining multilingualism in open-source llms trained on multiple languages and architectures, in contrast to @xcite , which focus on english-centric models. in addition, we propose novel methods for probing interactions across languages at the level of neural activations, enabling deeper insights into multilingual processing than outputbased analyses, a line of inquiry recently criticized for its limitations @xcite .. ## methodology let us consider a sentence @xmath0, . . . , t n drawn from a language, and define a as an l×n×d tensor representing the intermediate activations of a language model as it processes s. here, l denotes the number of layers, and d represents the number of features, i.e., embedding dimension. in this setup, a provides l distinct representations, each residing in a separate d-dimensional space, for every token. we extend this formulation to multiple aligned sentences across different languages, where each token is annotated with relevant linguistic labels (e.g., language identification or pos tag). this results in a large tensor of size l × n × d, where n is the total number of tokens across all sentences. to facilitate efficient visualization, reduce noise, and retain the most informative features of the activation space, we apply principal component analysis (pca) to each of the l views independently, reducing their dimensionality to d while preserving at least 95% of the activation variance. this results in a tensor h of size l × n × d, which, together with h, serves as the foundation for our analysis. we adopt two approaches to examine the generative and discriminative aspects of intermediate representations. the first adopts an informationtheoretic procedure to quantify the amount of vusable information @xcite encoded in intermediate representations that discriminates between language-specific and universal features. the v-usable information in a random variable x for predicting a category y is defined as the difference in conditional entropy between predictions based on x and a baseline prediction where no input features are provided (i.e., φ): our motivation for using this metric is twofold. first, its discriminative nature makes it applicable to both encoder-only and decoder-only architectures. second, it allows for a direct comparison of the effectiveness of feature vectors across different tasks defined over the same feature space x @xcite . such a comparison would not be possible if the analysis were based solely on task-specific metrics (e.g., f 1 -score and accuracy), as these metrics are not directly comparable across different tasks. additional details regarding the implementation of this metric are in appendix a. the second approach examines the generation capability of the decoder-only models. it utilizes saliency maps to examine how individual intermediate features contribute to token generation @xcite . by examining the gradients of next-token predictions with respect to intermediate activations, we identify the features that play a key role in encoding language-specific and universal properties. we use gradient-weighted class activation mapping to measure the importance of a feature h k i at a layer k to the prediction of a token by computing the product of the feature value for an input token (i.e., h k i (t j ) ) and the gradient of the prediction (before softmax) with respect to that feature (i.e., to assess the significance of c k i for a group of tokens (e.g., tokens belonging to a particular language), we conduct a two-tailed t-test with a significance level of 0.01. we refer to features with significant contribution to the generation of a particular token group as differentiating features for the group. accordingly, we define the differentiating rate of layer k as the ratio of differentiating features to the total features in the layer: in addition to the aforementioned metrics, which are designed to assess coordinate and compound learning, we introduce another approach in section 8 to assess sub-coordinate learning based on the proximity of intermediate activations to those of a dominant language. ## experiment setup we leverage the parallel universal dependencies (pud) treebanks @xcite which comprise aligned sentences from news sources and wikipedia, annotated for both morphological and syntactic structures. the cross-lingual alignment of sentences ensures that our findings are not skewed by domain-specific variations or differences in syntactic and semantic structures in certain languages. additionally, the availability of syntactic annotations allows us to effectively assess compound learning within llms. our experiments are based on 1000 sentences from each of the 21 topologically different languages in pud. a summary of the dataset is available in to assess the generalizability of information to unseen languages, we consider two experimental scenarios based on whether a test language is included in a model's pre-training data. the seen setting contains only languages present during pretraining, while the unseen setting includes those absent from it. for mbert and xlm-r, the unseen set is empty, as all test languages are covered in their pre-training data. ## coordinate learning we investigate coordinate learning by analyzing the separability of intermediate representations across input languages through the usable information for language identification and the feature differentiation rate for language processing. the underlying principle is that coordinate learners construct distinct processing systems for each language. ## compound learning compound learning involves constructing universal feature spaces shared among languages. our analysis of compound learning examines the existence of such shared spaces at the syntax level for universal part-of-speech tags (upos). we probe this phenomenon through the usable information for upos tagging and the joint differentiation rate of features for languages and syntactic categories. ## language-upos differentiating features to further examine how upos tags are processed within language-specific feature spaces, we measure layer differentiation rates based on the prediction of words belonging to a given syntactic category in a target language. by testing whether the same syntactic tag is processed differently across languages, we compute a joint differentiation rate that quantifies the extent to which syntactic categories are represented in language-specific versus shared feature spaces. the details of this experiment are provided in appendix c. figure 8 shows that decoder-only models allocate a subset of features to distinguishing syntactic categories within each language, irrespective of whether the language was included in pretraining. the absolute values of the differentiation rates, however, are consistently higher for seen languages, suggesting that universal syntactic categories are more strongly encoded in languagespecific feature spaces when the language is represented in training. in mgpt, the modest upward trend for seen languages further indicates that these differentiating features become increasingly effective in the top layers. ## sub-coordinate learning sub-coordinate learning implies a shift in intermediate feature vectors towards a dominant language that filters and influences the representations of other languages. the dominant language, which is more represented in the pre-training data of our test language models, is english. if a language model employs sub-coordinate learning internally, we would expect the representations of non-english languages to be enveloped by or significantly overlap with english representations. to examine this, we measure the proximity of language-specific activation vectors by computing the kullback-leibler (kl) divergence between the distribution of each non-english language and english. if language models employ internal filtering mechanisms consistent with subcoordinate learning, we expect a reduction in kl divergence, indicating that representations of different languages become more aligned with english. figure 9 presents the kl divergence between the feature space of each language and english. for decoder-only models, divergence begins relatively small in the lower layers and peaks in the middle layers, reflecting increased separation from english. at the top layers, bloom shows a sharp divergence, whereas mgpt instead converges strongly toward english. these trends are consistent across both seen and unseen settings: bloom's behavior suggests a weakening of sub-coordinate learning, while mgpt's sharp convergence in the top layers provides stronger evidence. nevertheless, because sub-coordinate learning is expected to manifest primarily in the middle layers, the decrease observed at the top layers of mgpt is less likely to be explained by this mechanism alone. the encoder-only models display a different pattern. mbert and xlm-r show only a modest shift toward english, while in xlm-r-large, this turns into a growing divergence after the middle layers. moreover, the absolute divergence values are substantially smaller than in decoder-only models, peaking at around 80 compared to several thousand, indicating that encoder-only feature spaces are generally denser. the modest reduction in divergence may reflect weak sub-coordinate learning, or, in line with our earlier discussion, could instead result from weak compound learning effects in the middle layers. 0 2 4 6 8 10 12 14 16 18 20 22 24 llm intermediate layer 0% 25% 50% 75% 100% feat. diff. ratee bloom-seen bloom-unseen mgpt-seen mgpt-unseen figure 8: joint language-upos differentiation rates across layers. shaded areas show variation across languages; solid lines show the mean. ## conclusion our analysis of multilingual large language models reveals differences in how encoder-only and decoder-only architectures handle multilingual representation. we examined the intermediate representations of these models to determine whether they follow coordinate, sub-coordinate, or compound learning strategies. we show that coordinate learning is the dominant mechanism, with decoder-only models developing strongly separated feature spaces for each language, while encoder-only models exhibit a more complex interplay of coordinate and compound learning in their middle layers. subcoordinate learning plays little to no role. moreover, training data coverage substantially affects the strength of language separation, with seen languages consistently exhibiting higher usable information and differentiation rates. our findings show that both architecture and pretraining data shape multilingual representations in llms. decoder-only models appear better suited for tasks that require maintaining clear languagespecific boundaries, while encoder-only models may be more advantageous for cross-lingual transfer, as their denser and partially shared representations facilitate knowledge sharing. more broadly, our results suggest that multilingual generalization in llms is not a single mechanism but a balance between language separation and cross-lingual sharing, which emerges differently across architectures and training regimes. in future work, we will extend the analysis of compound learning to a broader set of crosslinguistic features, including semantic and pragmatic aspects. additionally, we aim to explore the impact of training data diversity from a linguistic typology perspective on the balance between coordinate and compound learning, as well as how language models generalize to unseen and lowresource languages. expanding our study to a wider range of language models will help assess the influence of model scale on multilingual processing strategies.