An Isotropy Analysis in the Multilingual BERT Embedding Space (2022)

Abstract: Several studies have explored various advantages of multilingual pre-trained models (such as multilingual BERT) in capturing shared linguistic knowledge. However, less attention has been paid to their limitations. In this paper, we investigate the multilingual BERT for two known issues of the monolingual models: anisotropic embedding space and outlier dimensions. We show that, unlike its monolingual counterpart, the multilingual BERT model exhibits no outlier dimension in its representations while it has a highly anisotropic space. There are a few dimensions in the monolingual BERT with high contributions to the anisotropic distribution. However, we observe no such dimensions in the multilingual BERT. Furthermore, our experimental results demonstrate that increasing the isotropy of multilingual space can significantly improve its representation power and performance, similarly to what had been observed for monolingual CWRs on semantic similarity tasks. Our analysis indicates that, despite having different degenerated directions, the embedding spaces in various languages tend to be partially similar with respect to their structures.

## introduction the multilingual bert model @xcite , pre-trained on 104 languages with no supervision, has shown impressive ability in capturing linguistic knowledge across different languages @xcite . many studies have explored the encoded knowledge in multilingual cwrs using probing tasks and under zero-shot setting @xcite @xcite . following the probing studies, in this paper, we investigate the multilingual embedding space of bert, focusing on its geometry in terms of isotropy. previous research has shown that many pre-trained models, such as gpt-2 @xcite , bert, and roberta @xcite , have degenerated embedding spaces that downgrade their semantic expressiveness @xcite @xcite . several proposals have been put forward to overcome this challenge @xcite . however, to our knowledge, no study has so far been conducted on the degeneration problem in multilingual embedding spaces. using two well-known metrics, we evaluate isotropy in the mbert embedding space for six different languages (including two low resources): english, spanish, arabic, turkish, sundanese, and swahili. we find that the representation spaces are massively anisotropic in all these languages. however, unlike monolingual cwrs, where a few dimensions dominate the cosine similarity metric and have a high contribution to the anisotropic distribution @xcite , there is no dominant dimension that defines anisotropy in the multilingual representations. extending our study to other structural properties of the multilingual space, we also investigate outliers, i.e., specific dimensions with consistently high values @xcite . our findings reveal that, as opposed to english bert, the multilingual bert space does not involve any major outliers. this indicates that the suggestion of @xcite on the role of positional embeddings in the emergence of outliers may not be valid. moreover, our analysis reveals that word frequency plays an important role in the distribution of the multilingual embedding space: words with similar frequencies create distinct local regions in the embedding space. in analyzing multilingual space, we take a further step toward making the space isotropic. by applying a cluster-based isotropy enhancement method @xcite , we demonstrate that increasing isotropy of multilingual embedding space can result in significant performance improvements on semantic textual similarity tasks. our frequency analysis and the remarkable performance improvement in the zero-shot setting denote that the feature space of mbert has encoded a common linguistic knowledge into its dominant directions to some extent across different languages. ## background the representation degeneration problem in lms has attracted lots of attention in recent years. several regularizer-based methods have been proposed to make the space isotropic by adding an extra constraint to the loss function during pre-training @xcite @xcite . because of the re-training cost, other light approaches have been presented as a post-processing step @xcite . while analyzing the isotropy of embedding space is a well-studied area in english space, there are limited related studies on the multilingual embedding space. in this line, @xcite investigated the structural similarity of different language embedding spaces by evaluating their isomorphism. @xcite showed the positive effect of isotropic space on the degree of isomorphism, which in turn results in improved performance in cross-lingual alignment algorithms. however, a focused study on the isotropy of multilingual embedding space has not been conducted. in this work, we provide more insights on the anisotropic distribution of multilingual embeddings and their notable differences from their english counterpart. ## analysis for all experiments, we opted for the multilingual bert model (mbert), which has a 12-layer transformer-based architecture similar to english bert-base, and the representations are obtained from the last layer foot_2 . we selected english, spanish, arabic, turkish, sundanese, and swahili. our selection of these languages was to ensure that our analysis covers both high and low-resource languages. as our evaluation benchmark, we chose a subset of wikipedia articles in the selected languages. the analysis experiments in sections 3.1 to 3.3 have been conducted on the same dataset. in what follows, we first assess isotropy as a desirable property in the multilingual space and evaluate the contribution of individual dimensions to this property. we also expand our study to the outliers and word-frequency bias in cwrs. the former is a weak point of language models, and the latter is a well-known bias in the monolingual embedding space. lastly, we assess the effect of isotropy enhancement on the quality of the multilingual embeddings in the semantic similarity task. ## conclusion in this paper, we provide several analyses on the geometry of multilingual embedding space from the viewpoint of isotropy. we show that, similarly to its monolingual english counterpart, the multilingual bert has a highly anisotropic embedding space. however, interestingly, the two spaces differ in their distribution of dimensions. the english bert has a few high-contribution outlier dimensions, whereas the multilingual space does not possess any such disruptive rouge features. we also investigate the structure of multilingual embeddings from the perspective of frequency-based distribution and show that they have a biased structure towards word frequency and that the distribution is similar across different languages. as the last step, we evaluate the impact of isotropy improvement on the quality of multilingual embeddings. we observe that increasing isotropy can improve multilingual cwrs' performance on sts and address their frequency bias. zhong zhang, chongming gao, cong xu, rui miao, qinli yang, and junming shao. 2020. revisiting representation degeneration problem in language modeling. in findings of the association for computational linguistics: emnlp 2020, pages 518-527, online. association for computational linguistics. a xlm-r xlm-r is a transformer-based language model trained on 100 languages @xcite . in comparison to mbert, it has seen much more data during its pre-training, leading to higher performance on several downstream tasks. for our analysis, we follow the settings used for mbert and take roberta @xcite as its monolingual counterpart.