Cross-lingual Similarity of Multilingual Representations Revisited (2022)

Abstract: Related works used indexes like CKA and variants of CCA to measure the similarity of cross-lingual representations in multilingual language models. In this paper, we argue that assumptions of CKA/CCA align poorly with one of the motivating goals of cross-lingual learning analysis, i.e., explaining zero-shot cross-lingual transfer. We highlight what valuable aspects of cross-lingual similarity these indexes fail to capture and provide a motivating case study demonstrating the problem empirically. Then, we introduce Average Neuron-Wise Correlation (ANC) as a straightforward alternative that is exempt from the difficulties of CKA/CCA and is good specifically in a cross-lingual context. Finally, we use ANC to construct evidence that the previously introduced “first align, then predict” pattern takes place not only in masked language models (MLMs) but also in multilingual models with causal language modeling objectives (CLMs). Moreover, we show that the pattern extends to the scaled versions of the MLMs and CLMs (up to 85x original mBERT). Our code is publicly available at https://github.com/TartuNLP/xsim

## introduction similarity indexes like canonical correlation analysis (cca, @xcite or centered kernel alignment (cka, @xcite aim to find a similarity between parallel sets of different representations of the same data. the deep learning community adapted these indexes to measure similarity between representations that come from different models @xcite @xcite . another line of work used the same methods to measure similarity between different languages which come from a single multilingual model @xcite @xcite . in this paper, we argue that while cca/cka methods are a good fit for the first case, they are a suboptimal choice for the second scenario. first, we employ a real-world motivating example to demonstrate that cka can fail to capture the notion of similarity that we consider helpful in a cross-lingual context. we also discuss the general problems of cka/cca indexes and conclude that they are not well aligned with some of the goals of cross-lingual analysis (section 4). next, we propose and verify an averaged neuron-wise correlation (anc) as a straightforward alternative. it exploits the fact that representations from the same model have apriori-aligned neurons, which is the desired property in a crosslingual setup (section 5). finally, @xcite demonstrated the so-called "first align, then predict" representational pattern in a multilingual model: the model first aligns representations of different languages together, and then (starting from the middle layers) makes them more language-specific again (to accompany the language-specific training objective). the finding is insightful but only considers mbert @xcite which is a masked language model (mlm) with 110m parameters. thus, it is unclear if the "first align, then predict" pattern is specific to this model or more general. in this study, we use anc to show that the pattern generalizes to the gpt-style @xcite causal language models @xcite and extends to large-scale mlms and clms (section 6). in this paper we are interested specifically in the scenario of measuring the strength of cross-lingual similarity of representations that come from a single multilingual language model. this scenario is very common in the field as it is often not feasable to train a separate models for each language and we present a method that allows for better representational similarity analysis then cka/cca. in summary, our contributions are three-fold: raghu et al. (2017) proposed a variant of the cca called svcca and used it to analyze representations between different neural networks. morcos et al. (2018) proposed pwcca, another improvement to cca for the network analysis, and kornblith et al. (2019) analyzed cca, svcca, pwcca, and other methods concluding that cka is superior to them. in a cross-lingual setting, we have a single network, and we compare representations that come from different languages. following the introduction of svcca, @xcite used it to compare language representations (at different layers) in a multilingual neural machine translation system. the method we present in this work applies to the seq2seq models, but in this work, we focus on models trained with clm and mlm objectives while leaving seq2seq for future work. @xcite performed a similar study where they focused on the multilingual bert model foot_1 and employed pwcca as a similarity index. the conclusion was that language representations diverge with network depth. on the other hand, @xcite and @xcite used cka and behavior analysis to show that the opposite pattern takes place: language representations align with the network depth and only moderately decrease towards the end. in other words, representations first converge towards language neutrality and then recover some language-specificity. the alignment makes zeroshot cross-lingual transfer possible, and slight divergence accompanies language-specific training objectives (such as english downstream prediction task or predicting words in the particular language as in masked language modeling objective). following muller et al. 2021, we call this phenomenon the "first align, then predict" pattern. eventually, @xcite showed that the similarity analysis was different because @xcite used cls-pooling while @xcite used mean-pooling to convert token embeddings into a sentence representation. they also showed that mean-pooling is a better option. finally, @xcite aligned most correlated neurons between layers of two different networks and then computed similarity from the recovered correspondence. the method we propose in this paper is similar in spirit to this one, except we focus on the cross-lingual analysis of multilingual models and thus have no need to find the alignment between neurons. in this work, we build on these studies in three ways: we demonstrate that even cka can fail to provide relevant cross-lingual similarity, we propose another method to compare multilingual representations, and we reveal that the "first align, then predict" pattern generalizes across training objectives and holds for models of large sizes. ## similarity indexes background in this section, we provide some background on cka and cca, svcca, and pwcca similarity indexes foot_2 . we focus on the parts of the methods most relevant to the key points we make in this work. for the full mathematical description refer to @xcite . neuron following related works, we define a neuron as a vector of values it takes over a dataset @xcite @xcite . formally, let d be a dataset consisting of data examples #" d : let φ i be a function that returns a neuron activation value for the training example at the i-th unit of the l-th layer of the network. the neuron #" z i is the vector of network activations recorded by applying φ i over the elements of d, i.e. in practice, we pass a set of data examples to the network and record activations for each unit at every layer. the vector of these activations is what we consider a representation of a neuron #" z . layer the frequent goal of representational similarity analysis is to compare layers of neural networks. under our definition, the layer l is the list of vectors (matrix) that consists of the neurons at a particular depth, i.e. cca/cka indexes rely on the idea of subspaces spanned by the neurons, making them powerful when comparing representations across different networks. there can be more neurons in the first layer than in the second; the neurons also do not need to be aligned. cca/cka uses neurons only to describe the vector subspaces and then compare the subspaces as opposite to the neurons themselves. that is why methods like cka and cca try to find some second-order descriptions of representational spaces (e.g., gram matrices/canonical vectors) and compare these. the decisions on what second-order information to consider and what comparison technique to use define the differences between the indexes. ## method: average neuron-wise correlation (anc) in section 4 we demonstrated multiple drawbacks that cca/cka similarity indexes have in the crosslingual context. ## scaling laws of cross-lingual representational similarity in multilingual models in previous sections, we justified our claim that anc is better suited for cross-lingual analysis than cca/cka methods. in this section, we present an application of anc to the analysis of representational similarity scaling in cross-lingual language models. most related works that analyzed representational patterns in multilingual language models focused on a single model, such as base version of mbert or xlm-r. in section 5.2 we covered these models showing that anc accompanies our representational similarity index demands from these models. however, as the model scaling brings significant improvements in downstream tasks performance, we must focus our analysis efforts on the large models and scaling laws @xcite . in this section, we use anc to explore if the "first align, then predict" pattern generalizes to clms and if it preserves in the large-scale versions of multilingual mlms and clms. ## conclusion in this study, we introduced an example where cka drastically fails to reveal the cross-lingual similarity between language representations across the deeper layers of the multilingual model. we also highlighted that cca methods suffer from related problems as well (despite passing that concrete sanity check that cka failed). then, we proposed a new approach: average neuron-wise correlation (anc), which builds on the assumption of neuron alignment in crosslingual representations. we verified that our method passes the sanity check at which cka fails and produces results harmonious with the evidence from related work. finally, we used anc to show that the "first align, then translate" pattern of cross-lingual representations generalizes to clms and the larger scales of mlms and clms.