The Geometry of Multilingual Language Model Representations (2022)

Abstract: We assess how multilingual language models maintain a shared multilingual representation space while still encoding language-sensitive information in each language. Using XLM-R as a case study, we show that languages occupy similar linear subspaces after mean-centering, evaluated based on causal effects on language modeling performance and direct comparisons between subspaces for 88 languages. The subspace means differ along language-sensitive axes that are relatively stable throughout middle layers, and these axes encode information such as token vocabularies. Shifting representations by language means is sufficient to induce token predictions in different languages. However, we also identify stable language-neutral axes that encode information such as token positions and part-of-speech. We visualize representations projected onto language-sensitive and language-neutral axes, identifying language family and part-of-speech clusters, along with spirals, toruses, and curves representing token position information. These results demonstrate that multilingual language models encode information along orthogonal language-sensitive and language-neutral axes, allowing the models to extract a variety of features for downstream tasks and cross-lingual transfer learning.

## introduction despite state-of-the-art performance on a wide variety of multilingual nlp tasks @xcite @xcite @xcite , the internal structure of multilingual language model representation spaces is not well understood. the success of these models is often attributed to a shared multilingual space across languages, with claims that the models "map representations coming from different languages in a single shared embedding space" @xcite and that "different languages are close to a shared space" @xcite . while intuitive, figure 1 : xlm-r representations in layer six projected onto a linear subspace where two axes are languageneutral (horizontal axes), and one axis is languagesensitive (vertical axis). projecting from the side visualizes the language-sensitive axis (top right). projecting from the top down visualizes the language-neutral axes encoding token position information, forming a nearly perfect torus in the representation space (bottom right). in this work, we present an analysis of the geometry of multilingual language model representations using downstream language modeling predictions, direct comparisons between language subspaces, and visualizations of representations projected onto low-dimensional subspaces. using the multilingual language model xlm-r as a case study @xcite , we find that language subspaces are similar to one another after mean-centering, and differences between subspace means encode languagesensitive information such as language vocabularies. we identify language-sensitive axes that cluster representations by language family, and we identify language-neutral axes that encode token positions and part-of-speech. these axes remain relatively stable across layers, suggesting that multilingual language models maintain stable substructures for individual features during processing. our results highlight the importance of representational geometry in understanding multilingual language model representations, laying the groundwork for future research in multilingual subspace geometry and interpretable multilingual learning. 1 2 related work previous work has considered how multilingual language models encode different types of information. for example, mean representation distances between languages correlate with phylogenetic distances between languages @xcite , and individual representations can be used to predict linguistic typological features @xcite , particularly after projecting onto languagesensitive subspaces @xcite . the models also maintain language-neutral subspaces that encode information that is shared across languages. syntactic information is encoded largely within a shared syntactic subspace @xcite , token frequencies may be encoded similarly across languages @xcite , and representations shifted according to language means facilitate cross-lingual parallel sentence retrieval @xcite . however, these studies have primarily focused on specific features or probing tasks, with less focus on how the broader geometry of the original representation space facilitates feature encodings and interactions. to better understand this broader context, we consider overall language subspaces in multilingual language models, identifying axes that encode language-sensitive and language-neutral features. ## language subspaces as an initial assessment of the geometry of multilingual representation spaces, we identified affine (i.e. mean-shifted linear) subspaces for 88 languages in the multilingual language model xlm-r @xcite using singular value decomposition (svd) over contextualized token representations in each language. previous work has considered svd primarily in the context of singular value canonical correlation analysis (svcca; @xcite to quantify informational similarity between sets of representations (e.g. @xcite , without considering the geometry of the un-transformed subspaces. here, we found that svd identifies affine subspaces that sufficiently account for language modeling performance in individual languages, and the subspaces are similar to one another after subtracting language means, particularly in middle layers. ## language-sensitive axes thus, it appears that languages occupy similar subspaces in multilingual language models after meancentering, based on converging evidence from downstream language modeling performance and direct comparisons between language subspaces. however, differences in subspace means demonstrate that the language subspaces still differ along particular axes. intuitively, these axes should encode language-sensitive information, information that has high mutual information with the input language identity. for example, the word order or specific tokens present in a raw linguistic input are highly informative of the input language. in this section, we consider whether axes connecting subspace means encode language-sensitive features such as token vocabularies. indeed, we found that shifting representations by language means was sufficient to induce language modeling predictions in arbitrary target languages. we then used linear discriminant analysis (lda) to explicitly identify language-sensitive axes, finding that these axes were surprisingly stable across middle layers. ## language-neutral axes it is then apparent that multilingual language models encode language-sensitive information along language-sensitive axes. however, we still do not have a clear picture of how information is encoded along presumed language-neutral axes. motivated by the use of lda to identify language-sensitive axes, we used lda to identify axes that encode potentially more language-neutral information: token positions and part-of-speech (pos). we assessed whether the identified axes encoded the corresponding features in language-neutral ways. ## multilingual structure finally, we synthesize results from previous sections to develop a clearer picture of how multilingual language model representation spaces are structured. in section 3, we showed that individual languages occupy affine subspaces that are roughly similar to one another after mean-shifting. these language subspaces encode information such as token positions and part-of-speech along shared language-neutral axes (section 5). the subspaces differ primarily along language-sensitive axes (e.g. axes connecting language means) that encode information such as token vocabularies (section 4). we note that to the extent to which language subspaces differ along at least one language-sensitive axis, they are esentially non-overlapping in the higher dimensional space. to be precise, representations in different languages do not directly occupy a shared multilingual subspace; rather, representations must be projected onto specific axes and subspaces to extract language-sensitive and language-neutral representations. we describe different types of languagesensitive and language-neutral axes in appendix c, but we leave a rigorous quantification of the "language-sensitivity" of individual features and axes to future work. still, as shown in figures 1 , 9 , and 10, we found that axes encoding different features were often orthogonal to and independent from one another, enabling projections that were minimally influenced by extraneous noise and features along other dimensions. for example, in figure 9 , the distinction between nouns and verbs was encoded along an axis orthogonal to a token's position in the input sequence, allowing either feature to be extracted by projecting onto its corresponding axes (see additional examples in appendix d.4). however, this observation may be because low-dimensional subspaces are likely to be orthogonal to one another in high-dimensional spaces, and we selected features that were likely to be minimally correlated with one another. future work might assess how multilingual language models encode more complex linguistic features geometrically, or the extent to which the models' representation spaces can be fully decomposed into orthogonal subspaces encoding different features. this work has implications for more targeted subspace alignment between languages for better cross-lingual transfer learning; for example, our methods might be used to identify specific axes that should or should not be aligned for specific tasks using existing representation alignment approaches (e.g. @xcite @xcite . finally, in sections 4.2, 5.1, and 5.2, we presented initial results suggesting that language families, token positions, and part-of-speech are encoded stably across middle and late-middle layers in multilingual language models. in other words, the transformations in these layers may be primarily language-neutral, and they may retain structures encoding low-level features for later token reconstruction @xcite . in these layers, representations might change primarily along axes corresponding to higher level features (e.g. semantics and general reasoning; @xcite , leaving other axes unchanged. a more detailed analysis of how specific representational structures are retained across layers is a promising direction for future research. ## conclusion in this work, we identified language subspaces and individual language-sensitive and language-neutral axes in the multilingual language model xlm-r. we assessed these subspaces and axes using a variety of methodologies, including causal effects on language modeling predictions, direct comparisons between subspaces, and low-dimensional visualizations. our results suggest that multilingual language models encode features by projecting representations onto orthogonal axes in the representation space, enabling the efficient and simultaneous encoding of a wide variety of signals for downstream tasks and multilingual learning.