Exploring Anisotropy and Outliers in Multilingual Language Models for Cross-Lingual Semantic Sentence Similarity (2023)

Abstract: Previous work has shown that the representations output by contextual language models are more anisotropic than static type embeddings, and typically display outlier dimensions. This seems to be true for both monolingual and multilingual models, although much less work has been done on the multilingual context. Why these outliers occur and how they affect the representations is still an active area of research. We investigate outlier dimensions and their relationship to anisotropy in multiple pre-trained multilingual language models. We focus on cross-lingual semantic similarity tasks, as these are natural tasks for evaluating multilingual representations. Specifically, we examine sentence representations. Sentence transformers which are fine-tuned on parallel resources (that are not always available) perform better on this task, and we show that their representations are more isotropic. However, we aim to improve multilingual representations in general. We investigate how much of the performance difference can be made up by only transforming the embedding space without fine-tuning, and visualise the resulting spaces. We test different operations: Removing individual outlier dimensions, cluster-based isotropy enhancement, and ZCA whitening. We publish our code for reproducibility.

## introduction since bert-like @xcite language models rose to popularity, much has been made of the study of their hidden states and parameters (cf. @xcite . thanks to their ability to incorporate context, they have been a major improvement for most tasks over static input embeddings. however, a certain issue has been shown in a number of works to affect contextual language models to a greater degree: outlier dimensions in the figure 1 : effect of removing dimension 588 from layer 8 on tatoeba cosine similarities. the x-axis is the ranking position of candidate sentences; the y-axis is their cosine distance from the query sentence, which should be relatively large for all but the correct translation. after removing 588 from the sentence representations, the highest ranking candidate sentence is much more clearly differentiated from the lower-ranking candidates. in this paper, we are interested in multilingual sentence embedding quality. we discuss both outliers and anisotropy as two related aspects of embedding quality. outlier dimensions are typically defined as dimensions that consistently produce values of a magnitude more than three or five times the standard deviation of all dimensions @xcite . if a model has outlier dimensions in its hidden states, it will necessarily have higher anisotropy, since these dimensions create a consistent shift towards a certain direction in the embedding space. on the other hand, high anisotropy can also occur without individual dimensions meet-ing the outlier definition, namely if some principal components composed of multiple dimensions are much larger than others. therefore, as we understand it, anisotropy is the wider phenomenon of which outliers are a subset. from a theoretical perspective, high anisotropy is considered a problem because it means that the model is not using the full representation space available, and because it translates to high average cosine similarity even between unrelated words or sentences. figure 1 illustrates this problem clearly. this can increase the odds of picking a wrong candidate on word and sentence similarity tests, and makes representations produced by the model less expressive and less interpretable. outlier dimensions, since they contribute to anisotropy, entail similar challenges. on the other hand, they are easy to spot, easy to manipulate, and a straightforward entry point to the anisotropy issue. previous work has sometimes found that models rely strongly on outlier weights for certain tasks, and are overly vulnerable to pruning a select few weights, e.g. @xcite . further, outliers have been found to present a challenge in model quantisation @xcite . because they are aspects of the output representations, studies of anisotropy and outliers often use semantic similarity tests that rely directly on these representations, without fine-tuning the model. we follow this approach as well. in this work, we specifically consider sentence representations. only a small amount of work has been done on outliers and isotropy in multilingual models, which we focus on. @xcite found that mbert does not contain outlier dimensions, while xlm-r does. however, both models nevertheless exhibit high anisotropy. another important aspect to consider in the multilingual case is that even if representations are more or less isotropically distributed, the subspaces for different languages can still be misaligned, which further affects cross-lingual performance. training with parallel data, as done in @xcite , is one way to radically improve cross-lingual alignment. however, we are interested in pushing models to perform well without parallel data. the present work therefore attempts to separate the effect of anisotropy from other factors that could account for the performance gap, such as the use of parallel data objectives and internal misalignment of languages. our contributions. this work provides an in-depth exploration of outlier dimensions and anisotropy in xlm-r and other pre-trained multilingual language models, using the tatoeba @xcite , multilingual sts @xcite , and bucc 2018 @xcite semantic similarity tasks and looking directly at the relevant hidden state representations. we confirm that certain outlier dimensions have a negative effect on similarity search in the crosslingual setting ( § 5). we find that outlier dimensions can differ between languages, although the largest outliers occur in all or most tested languages ( § 5). anisotropy also varies across languages, and we observe a possible relationship to pre-training data size ( § 4). in our experiments, mbert does exhibit outlier dimensions ( § 4). looking at semantic similarity task performance, we show that zeroing outliers and isotropyenhancing transformations are quick ways to improve model performance on such tasks ( § 5, 6). however, a multilingual sentence-transformer performs much better out-of-the-box, and benefits little to not at all from further increasing isotropy. as we show in § 4, this model is already much more isotropic than xlm-r, its pre-trained equivalent. finally, we give a clearer intuition of the phenomena in question by using tsne (van der @xcite to visualise embedding spaces ( § 7). this allows us to grasp more intuitively how anisotropy is one aspect of misalignment between languages in multilingual models. ## related work bert-like models have dominated nlp research in recent years. multilingual bert @xcite and xlm-r (conneau et al., 2020) are two popular models whose variants are used for many different ends. accordingly, some amount of research has focused on analysing properties of the models, sometimes called "bertology" @xcite . the phenomena we discuss in this paper-outlier dimensions and anisotropy-are just two aspects of model analysis. ## datasets because we will show results of each of our experiments as we go along, we start here by introducing the datasets used. ## zeroing out dimensions based on the outlier analysis, we experiment with zeroing out dimensions from the sentence representations before feeding them to the similarity search functions. the biggest outlier, 588, clearly damages performance by greatly raising the similarity of all sentences. the correct candidate may thus be eclipsed by a false one more easily. figure 1 illustrates how this occurs. on the x-axis are the ranking positions of candidate sentences, on the y-axis their average cosine distances (inverse to cosine similarity). in the unmodified model, all candidates are highly similar to the query sentence. after removing 588, candidates with lower ranking in addition to zeroing the largest outliers, we identified other dimensions of interest by their magnitude. we included the ten largest dimensions in each language of tatoeba, finding a total of 18 dimensions that are in the top ten for any of the 36 languages. 3 these dimensions include the outliers previously identified, as well as additional large dimensions. we explored removing these dimensions individually and generally found smaller effects, though still a marked effect for some of them. the results are listed in we removed the same dimensions from sentence embeddings of bucc @xcite data. interestingly, this sometimes improved precision while also worsening recall. thus, the overall improvements on this task were small (e.g., 588) or even negated (306). removing all 18 large dimensions from tatoeba and bucc yields +9.7 accuracy and +5.3 f1 over the vanilla xlm-r model, respectively. that said, even with this performance gain, the gap to the sentence-transformer is still very large. in addition, manually zeroing a large number of dimensions depending on the task data cannot be done in a real-world system. ## isotropy-enhancing operations aside from directly zeroing out individual dimensions, we can apply transformations over the set of embeddings that largely eliminate anisotropy and mean-center the representations. in this work, we test two such transformations: whitening is an operation originally used in data pre-processing, in order to remove correlations between the input data features to a machine learning system. it is also called a "sphering transformation", since the resulting data space is a hyperdimensional sphere. however, whitening has recently been used to transform output embeddings of models such as bert (cf. @xcite , before using them for downstream applications. for a given space x with covariance σ and mean 0, there are many valid whitening transformations. the resulting matrix @xmath0, and the whitening transformation w must satisfy the condition: given that σ can be decomposed into: ## embedding space visualisation to visualise the representation space, we use t-sne (van der @xcite . first, we apply a pca dimensionality reduction to 50 dimensions. then, we reduce the dimensionality further using arabic-english bengali-english german-english unmodified cbie-transformed t-sne and plot the space in two dimensions. in figure 3 , we show examples from tatoeba data in xlm-r: arabic, bengali, and german. for the first two, accuracy increased by more than 20 points after the transformation, while german is already a high-resource language where accuracy only increased by around 5 points. since cbie and whitening produce very similar visualisations, we only show cbie. the unmodified spaces very clearly show the problem of internal misalignment between different languages in the model, which disproportionately affects languages with less pre-training data and/or non-latin scripts. with arabic-english and bengali-english, the source and target language spaces are almost disjunct. this issue can be addressed using isotropy-increasing transformations, but they do not solve the problem entirely. for instance, the unmodified sub-spaces of bengali and english also have markedly different shapes, despite representing a set of parallel sentence pairs. matching the equivalent sentences to each other starting from such different spaces is more complex than merely applying a linear transformation to increase isotropy. ## conclusions we have analysed how outlier dimensions and anisotropy interact with cross-lingual semantic similarity tasks in pre-trained multilingual language models. in particular, we focused on the sentence representations of multilingual bert and xlm-r, comparing them to the sentence representations of a multilingual s-bert model-essentially a modified xlm-r trained with parallel data to optimise for sentence representations. we employed a range of methods on several different tasks to approach the question from multiple angles. the simplest method of increasing isotropy is removing the largest (outlier) dimensions from the sentence embeddings. we compared the results of this with further-reaching isotropy-increasing transformations. additionally, we examined how changing the representations affected anisotropy measures and outlier dimensions. finally, we plotted unmodified and transformed sentence representation spaces to illustrate how anisotropy is one aspect that affects sentence similarity, but reducing it does not resolve all issues in the space.