Is Anisotropy Truly Harmful? A Case Study on Text Clustering (2023)

Abstract: In the last few years, several studies have been devoted to dissecting dense text representations in order to understand their effectiveness and further improve their quality. Particularly, the anisotropy of such representations has been observed, which means that the directions of the word vectors are not evenly distributed across the space but rather concentrated in a narrow cone. This has led to several attempts to counteract this phenomenon both on static and contextualized text representations. However, despite this effort, there is no established relationship between anisotropy and performance. In this paper, we aim to bridge this gap by investigating the impact of different transformations on both the isotropy and the performance in order to assess the true impact of anisotropy. To this end, we rely on the clustering task as a means of evaluating the ability of text representations to produce meaningful groups. Thereby, we empirically show a limited impact of anisotropy on the expressiveness of sentence representations both in terms of directions and L2 closeness.

## introduction contextualized pre-trained representations are now widely used as input to various tasks such as information retrieval @xcite , anomaly detection @xcite and document clustering @xcite . in parallel, several studies have investigated the intrinsic properties of transformers @xcite @xcite in order to demystify these black-box models and the reasons behind their impressive performance levels. particularly, it has been observed that language models in general @xcite and transformer word embedding models in particular @xcite ) produce an anisotropic embedding space. this concretely means that the directions of trained dense word representations do not occupy uniformly the embedding space, which is suspected to limit their expressiveness and thus their expected performance on downstream tasks. the main question addressed in this paper is how harmful this anisotropy really is regarding the quality of text representations. several approaches have been proposed to increase the isotropy of dense representations, based on different strategies. in the context of static word embeddings like glove and word2vec, both @xcite and @xcite propose a post-processing method that consists in removing the first principal components before reconstructing the word vectors as opposed to the traditional approach of removing the weakest components. this approach improves the quality of word vectors on several downstream tasks while reducing their anisotropy @xcite . as to contextualized representations provided by transformer models, several approaches have been proposed in order to alleviate the anisotropy problem. for instance, based on the idea that anisotropic representations tend to have high expected pairwise cosine similarity, @xcite propose to apply a cosine similarity regularization term to the embedding matrix. in the same vein, @xcite propose a method named "spectrum control" that allows for increasing the isotropy of transformer representations and improving the performance of the machine translation task. to this purpose, they propose regularization terms that hamper the singular value decay of the embedding matrix. however, despite the success of these optimization tricks in lowering the anisotropy of transformer representations, @xcite have recently shown that they do not bring any improvement, relying on several tasks like summarization and sentence similarity (sts). they even observed a certain deterioration of the performance brought by anisotropy mitigation techniques. in contrast, @xcite show that post-processing methods made for increasing isotropy are also responsible for a performance increase in the sts task in both monolingual and cross-lingual settings. similarly, the whitening operation, which consists in using the principal components normalized by their inertia, has shown an increase in isotropy as well as enhanced performance in sts @xcite and document clustering @xcite . however, there is no evidence that the decrease of anisotropy brought by such transformations is directly responsible for the gain of performance, as shown in figure 1 , which gives an initial idea of the question addressed in this paper. indeed, despite the great energy devoted to studying and mitigating the anisotropy of dense text representations, there is no clear connection between isotropy and performance, which seems to depend, inter alia, on the sought task. in order to contribute to settling this question, we consider using a task that has never been used for this purpose: document clustering. the rationale behind this choice is to evaluate, under different degrees of isotropy, the capability of text representations to facilitate the clear separation and identification of meaningful groups of documents through clustering algorithms. the main contributions of this paper are: ## experiments in this study, we aim to determine to what extent the anisotropy actually affects the quality of the representations and their ability to discriminate data samples through separable clusters. to this end, we use three measures to evaluate the isotropy of the original embedding space before and after post-processing. then, we compare the changes in isotropy with the corresponding clustering performance in order to establish a potential relationship between the two concepts. relying on several isotropy measures allows us to consolidate confidence in our conclusions and, at the same time, verify if the measures agree with each other. in the same spirit, using different clustering methods and performance measures ensures more rigorous assertions. we make the code and data used publicly available foot_0 . ## discussion figure 2 confronts one quality measure (nmi) and one isotropy measure (i cos ) using different postprocessing techniques. we first observe that pca w produces, by far, the most isotropic representations while increasing the performance of the raw vectors. indeed, an appealing explanation of the success of the whitening operation is that it considerably alleviates the anisotropy of the embedding space @xcite . applying that reasoning, pca and umap should deteriorate the performance since they both exacerbate the anisotropy (in all cases for pca and in most cases for umap). nonetheless, the performance of pca is comparable to that of the raw embeddings and umap achieves even better performance than pca w even though it significantly reduces the isotropy. overall, averaging the whole set of layer representations achieves better results, even though it clearly decreases the isotropy, compared to using the last layer, as traditionally performed. also, it is worth noting that even when the directions of the vectors are used (skm), the decrease of isotropy has a negligible impact on the performance. all these observations suggest that, although the anisotropy reduces the spectrum of directions taken by sentence vectors, it does not necessarily alter their expressiveness. in order to confirm this supposition, we directly compare isotropy and quality measures in a wide range of situations. to this end, we compute the correlation ( ## conclusion it has been known to happen that transformations that tend to decrease the anisotropy of text representations also improve the performance of downstream tasks. in stark contrast, we observe in the present study that transformations that exacerbate the anisotropy phenomenon may also improve the results, which calls into question the importance of isotropy in text representation. to draw this important conclusion, we relied on the clustering task and several empirical measures to assess the relationship between isotropy and quality of representations, using several datasets. most importantly, we show that even a directional approach for clustering, which should be primarily affected by anisotropy, does not undergo any performance loss resulting from low-isotropy representations. in addition, we show the advantage of using umap as a post-processing step, which provides good-quality representations using only a handful of dimensions, despite a high resulting anisotropy.