A Joint Matrix Factorization Analysis of Multilingual Representations (2023)

Abstract: We present an analysis tool based on joint matrix factorization for comparing latent representations of multilingual and monolingual models. An alternative to probing, this tool allows us to analyze multiple sets of representations in a joint manner. Using this tool, we study to what extent and how morphosyntactic features are reflected in the representations learned by multilingual pre-trained models. We conduct a large-scale empirical study of over 33 languages and 17 morphosyntactic categories. Our findings demonstrate variations in the encoding of morphosyntactic information across upper and lower layers, with category-specific differences influenced by language properties. Hierarchical clustering of the factorization outputs yields a tree structure that is related to phylogenetic trees manually crafted by linguists. Moreover, we find the factorization outputs exhibit strong associations with performance observed across different cross-lingual tasks. We release our code to facilitate future research.

## introduction pre-trained multilingual models @xcite @xcite have gained widespread adoption in recent years. they initially pre-trained in many languages and subsequently fine-tuned for specific downstream tasks. their aim is to leverage the linguistic knowledge acquired from similar languages, thereby benefiting low-resource languages and enabling zero-shot cross-lingual transfer ability. while numerous prior works have demonstrated these models have such abilities @xcite @xcite @xcite @xcite , there are still open questions about the nature of the linguistic knowledge these models possess and the extent to which they 1 https://github.com/zsquaredz/joint_ multilingual_analysis/ acquire and incorporate linguistic information in their multilingual representations. previous work has used singular vector canonical correlation analysis (svcca; @xcite and other similarity statistics like centered kernel alignment (cka; @xcite to analyze multilingual representations @xcite @xcite . however, such methods can only compare one pair of representation sets at a time. in contrast to that, we analyze all multilingual representations simultaneously using parallel factor analysis 2 (parafac2; @xcite ), a method that allows us to factorize a set of representations jointly by decomposing it into multiple components that can be analyzed individually and then recombined to understand the underlying structure and patterns in the data. more precisely, we extend the sub-population analysis method recently presented by @xcite , who compare two models as an alternative to probing: a control model trained on data of interest and an experimental model, which is identical to the control model but is trained on additional data from different sources. by treating the multilingual experimental model as a shared component in multiple comparisons with different control models (each one is a monolingual model trained on a subset of the multilingual model), we can better analyze the multilingual representations. as an alternative to probing, our representation analysis approach: a) enables standardized comparisons across languages within a multilingual model, circumventing the need for external performance upper bounds in making meaningful interpretation of performance metrics; b) directly analyzes model representations, avoiding the need for auxiliary probing models and potential biases from specific probing classifier architectures; and c) compares multilingual versus monolingual representations for any inputs, avoiding reliance on labelled probing datasets. we use parafac2 to directly compare representations learned between multilingual models and their monolingual counterparts. we apply this efficient paradigm to answer the following research questions on multilingual models: q1) how do multilingual language models encode morphosyntactic features in their layers? q2) are our findings robust to address low-resource settings? q3) do morphosyntactic typology and downstream task performance reflect in the factorization outputs? we experiment with two kinds of models, xlm-r @xcite and roberta @xcite . we apply our analysis tool on the multilingual and monolingual representations from these models to check morphosyntactic information in 33 languages from universal dependencies treebanks (ud; @xcite . our analysis reinforces recent findings on multilingual representations, such as the presence of language-neutral subspaces in multilingual language models @xcite , and yields the following key insights: ## background and motivation in this paper, we propose to use parafac2 for multilingual analysis. by jointly decomposing a set of matrices representing cross-covariance between multilingual and monolingual representations, parafac2 allows us to compare the representations across languages and their relationship to a multilingual model. for an integer n, we use [n] to denote the set {1, ..., n}. for a square matrix σ, we denote by diag(σ) its diagonal vector. parafac2 let ℓ index a set of matrices, foot_0 such that a ℓ = e[x ℓ z ⊤ ], the matrix of crosscovariance between x ℓ and z, which are random vectors of dimensions d and d ′ , respectively. 3 this means that for any ℓ and two vectors, @xmath0, @xmath1, the following holds due to the linearity of expectation: we call the elements on the diagonal of σ ℓ pseudo-singular values, in relationship to singular value decomposition that decomposes a single matrix in a similar manner. the decomposition in eq. 1 jointly decomposes the matrices such that each a ℓ is decomposed into a sequence of three transformations: first transforming z into a latent space (v), scaling it (σ ℓ ) and then transforming it into a specific ℓth-indexed x ℓ space (u ℓ ). unlike singular value decomposition, which decomposes a matrix into a similar sequence of transformations with orthonormal matrices, parafac2 does not guarantee u ℓ and v to be orthonormal and hence they do not represent an orthonormal basis transformation. however, @xcite showed that a solution can still be found and is unique if we add the constraint that u ⊤ ℓ u ℓ is constant for all ℓ. in our use of parafac2, we follow this variant. we provide an illustration of parafac2 in figure 1 . experimental model, contrasting them with representations derived from a set of control models. in our case, the experimental model is a jointlytrained multilingual pre-trained language model, and the control models are monolingual models trained separately for a set of languages. formally, there is an index set of languages [l] and a set of models consisting of the experimental model e and the control models c ℓ for ℓ ∈ [l]. we assume a set of inputs we apply our analysis to, @xmath2. each set x ℓ = {x ℓ,1 , . . . , x ℓ,m } represents a set of inputs for the ℓth language. while we assume, for simplicity, that all sets have the same size m, it does not have to be the case. in our case, each x ℓ is a set of input words from language ℓ, which results in a set of representations as follows. for each ℓ ∈ [l] and @xmath3,i to get two corresponding representations y ℓ,@xmath4,@xmath5. here, d is the dimension of the multilingual model and d ℓ is the dimension of the representation of the monolingual model for the ℓth language. stacking up these sets of vectors separately into two matrices (per language ℓ), we obtain the set of paired matrices y ℓ ∈ r m×d and z ℓ ∈ r m×d ℓ . we further calculate the covariance matrix ω ℓ , defined as: ω ℓ = z ⊤ ℓ y ℓ . use of parafac2 given an integer k smaller or equal to the dimensions of the covariance matrices, we apply parafac2 on the set of joint matrices, decomposing each ω ℓ into: the diagonal of σ ℓ can now be readily used to describe a signature of the ℓth language representations in relation to the multilingual model (see also @xcite ). this signature, which we mark by sig(ℓ) = diag(σ ℓ ), can be used to compare the nature of representations between languages, and their commonalities in relationship to the multilingual model. in our case, this parafac2 analysis is applied to different slices of the data. we collect tokens in different languages (both through a multilingual model and monolingual models) and then slice them by specific morphosyntactic category, each time applying parafac2 on a subset of them. for some of our analysis, we also use a condensed value derived from sig(ℓ). we follow a similar averaging approach to that used by svcca @xcite , a popular representation analysis tool, where they argue that the single condensed svcca score represents the average correlation across aligned directions and serves as a direct multidimensional analogue of pearson correlation. in our case, each signature value within sig(ℓ) from the parafac2 algorithm corresponds to a direction, all of which are normalized in length, so the signature values reflect their relative strength. thus, taking the average of sig(ℓ) provides an intensity measure of the representation of a given language in the multilingual model. we provide additional discussion in §5.1. ## experimental setup data we use conll's 2017 wikipedia dump @xcite to train our models. following @xcite , we downsample all wikipedia datasets to an identical number of sequences to use the same amount of pre-training data for each language. in total, we experiment with 33 languages. for morphosyntactic features, we use treebanks from ud 2.1 @xcite . these treebanks contain sentences annotated with morphosyntactic information and are available for a wide range of languages. we obtain a representation for every word in the treebanks using our pretrained models. we provide further details on our pre-training data and how we process morphosyntactic features in appendix b.1. task for pre-training our models, we use masked language modeling (mlm). following @xcite , we mask 15% of the tokens. to fully control our experiments, we follow @xcite and train our models from scratch. ## experiments and results this section outlines our research questions (rqs), experimental design, and obtained results. ## related work understanding the information within nlp models' internal representations has drawn increasing attention in the community. @xcite applied canonical correlation analysis (cca) on the internal representations of a pre-trained mbert and revealed that the model partitions representations for each language rather than using a shared interlingual space. @xcite used svcca to investigate massively multilingual neural machine translation (nmt) representations and found that different language encoder representations group together based on linguistic similarity. @xcite showed that mbert representations could be split into a language-specific component and a language-neutral component by centering mbert representations and using the centered representation on several probing tasks to evaluate the language neutrality of the representations. similarly, @xcite employed the lottery ticket hypothesis to discover subnetworks within mbert and found that mbert is comprised of language-neutral and languagespecific components, with the former having a greater impact on cross-lingual transfer performance. @xcite presented a novel layer ablation approach and demonstrated that mbert could be viewed as the stacking of two sub-networks: a multilingual encoder followed by a task-specific language-agnostic predictor. probing (see belinkov 2022 for a review) is a widely-used method for analyzing multilingual representations and quantifying the information encoded by training a parameterized model, but its effectiveness can be influenced by model parameters and evaluation metrics @xcite . @xcite probed representations from multilingual sentence encoders and dis-covered that typological properties are persistently encoded across layers in mbert and xlm-r. @xcite demonstrated with probing that language-specific information is scattered across many dimensions, which can be projected into a linear subspace. intrinsic probing, on the other hand, explores the internal structure of linguistic information within representations @xcite . @xcite conducted a large-scale empirical study over two multilingual pre-trained models, mbert, and xlm-r, and investigated whether morphosyntactic information is encoded in the same subset of neurons in different languages. their findings reveal that there is considerable cross-lingual overlap between neurons, but the magnitude varies among categories and is dependent on language proximity and pre-training data size. other methods, such as matrix factorization techniques, are available for analyzing representations @xcite @xcite and even modifying them through model editing @xcite @xcite . when applied to multilingual analysis, these methods are limited to pairwise language comparisons, whereas our proposed method enables joint factorization of multiple representations, making it well-suited for multilingual analysis. ## conclusions we introduce a representation analysis tool based on joint matrix factorization. we conduct a largescale empirical study over 33 languages and 17 morphosyntactic categories and apply our tool to compare the latent representations learned by multilingual and monolingual models from the study. our findings show variations in the encoding of morphosyntactic information across different layers of multilingual models. language-specific differences contribute to these variations, influenced by factors such as writing systems and linguistic relatedness. furthermore, the factorization outputs exhibit strong correlations with cross-lingual task performance and produce a phylogenetic tree structure resembling those constructed by linguists. these findings contribute to our understanding of language representation in multilingual models and have practical implications for improving performance in cross-lingual tasks. in future work, we would like to extend our analysis tool to examine representations learned by multimodal models.