Isotropy, Clusters, and Classifiers (2024)

Abstract: Whether embedding spaces use all their dimensions equally, i.e., whether they are isotropic, has been a recent subject of discussion. Evidence has been accrued both for and against enforcing isotropy in embedding spaces. In the present paper, we stress that isotropy imposes requirements on the embedding space that are not compatible with the presence of clusters—which also negatively impacts linear classification objectives. We demonstrate this fact both empirically and mathematically and use it to shed light on previous results from the literature.

## introduction recently, there has been much discussion centered around whether vector representations used in nlp do and should use all dimensions equally. this characteristic is known as isotropy: in an isotropic embedding model, every direction is equally probable, ensuring uniform data representation without directional bias. at face value, such a characteristic would appear desirable: naively, one could argue that an anisotropic embedding space would be overparametrized, since it can afford to use some dimensions inefficiently. the debate surrounding isotropy was initially sparked by @xcite , who highlighted that isotropic static representations fared better on common lexical semantics benchmarks, and @xcite , who stressed that contextual embeddings are anisotropic. since then, evidence has been accrued both for and against enforcing isotropy on embeddings. in the present paper, we demonstrate that this conflicting evidence can be accounted for once we consider how isotropy relates to embedding space geometry. strict isotropy, as assessed by isoscore @xcite , requires the absence of clusters, and thereby also conflicts with linear classification objectives. this echoes previous empirical studies connecting isotropy and cluster structures @xcite . in the present paper, we formalize this connection mathematically in section 2. we then empirically verify our mathematical approach in section 3, discuss how this relation sheds light on earlier works focusing on anisotropy in section 4, and conclude with directions for future work in section 5. ## some conflicting optimization objectives we can show that isotropy-as assessed by isoscore @xcite -impose requirements that conflict with cluster structures-as assessed by silhouette scores @xcite -as well as linear classifier objectives. ## empirical confirmation to verify the validity of our demonstrations in section 2, we can optimize a set of data-points for a classification task using a linear classifier: we should observe an increase in silhouette scores, and a decrease in isoscore. note that we are therefore evaluating the behavior of parameters as they are optimized; i.e., we do not intend to test whether silhouettes and isoscore behave as expected on held-out data. this both allows us to precisely test the argument laid out in section 2 and cuts down computational costs significantly. ## related works how does the connection between clusterability and isotropy that we outlined shed light on the growing literature on anisotropy? while there is currently more evidence in favor of enforcing isotropy in embeddings, the case is not so clear cut that we can discard negative findings, and a vast majority of the positive evidence relies on improper techniques for quantifying isotropy @xcite . @xcite stressed that contextual embeddings are effective yet anisotropic. @xcite provides experiments that advise against using isotropy calibration on transformers to enhance performance in specific tasks. @xcite finds that anisotropy regularization in fine-tuning appears to be beneficial on a large array of tasks. lastly, @xcite find that the contrasts encoded in dominant dimensions can, at times, capture linguistic knowledge. on the other hand, the original study of @xcite found that enforcing isotropy on static embeddings improved performances on semantic similarity, both at the word and sentence level, as well as word analogy. subsequently, a large section of the literature has focused on this handful of tasks (e.g., @xcite . isotropy was also found to be helpful beyond these similarity tasks: haemmerl et al. (2023) report that isotropic spaces perform much better on cross-lingual tasks, and @xcite stress its benefits for dense retrieval. these are all applications that require graded ranking judgments, and therefore are generally hindered by the presence of clusters-such clusters would for instance introduce large discontinuities in cosine similarity scores. to take @xcite as an example, note that languagespecific clusters are antithetical to the success of cross-lingual transfer applications. it stands to reason that isotropy can be found beneficial in such cases, although the exact experimental setup will necessarily dictate whether it is boon or bane: for instance @xcite tested finetuning llms as siamese networks to optimize performance on sentence-level similarity, and found enforcing isotropy to hurt performances-here, we can conjecture that learning to assign inputs to specific clusters is a viable solution in their case. the literature has previously addressed the topic of isotropy and clustering. @xcite advocated for enhancing the isotropy on a cluster-level rather than on a global-level. @xcite confirmed the presence of clusters in the embedding space with local isotropy properties. @xcite investigated the correlation between isotropy and clustering tasks and found that fostering high anisotropy yields highquality clustering representations. the study presented here provides a mathematical explanation for these empirical findings. ## conclusion we argued that isotropy and cluster structures are antithetical (section 2), verified that this argument holds on real data (section 3), and used it to shed light on earlier results (section 4). this result however opens novel and interesting directions of research: if anisotropic spaces implicitly entail cluster structures, then what is the structure we observe in our modern, highly anisotropic large language models? prior results suggest that this structure is in part linguistic in nature @xcite , but further confirmation is required. another topic we intend to pursue in future work concerns the relation between non-classification tasks and isotropy: isotropy constraints have been found to be useful in problems that are not well modeled by linear classification, e.g. word analogy or sentence similarity. our present work does not yet offer a thorough theoretical explanation why. structure. relatedly, a natural point of criticism to raise is whether our reasoning will hold for deep classifiers with non-linearities: most (if not all) modern deep-learning classification approaches rely on nonlinear activation functions across multiple layers of computations. the present demonstration has indeed yet to be expanded to account for such more common cases. insofar neural architectures trained on classification objectives are concerned, we strongly conjecture their output embeddings would tend to be anisotropic. the anisotropy of inner representations appears to be a more delicate question: for transformers, there has been extensive work showcasing that their structure is for the most part additive @xcite @xcite @xcite , and we therefore expect anisotropy to spread to bottom layers to some extent. for architectures based on warping random distributions such as normalizing flows @xcite , gans @xcite , or diffusion models @xcite , the fact that (part of) their input is random and isotropic likely limits how anisotropic their inner representations are.