I so S core: Measuring the Uniformity of Embedding Space Utilization (2022)

Abstract: The recent success of distributed word representations has led to an increased interest in analyzing the properties of their spatial distribution. Several studies have suggested that contextualized word embedding models do not isotropically project tokens into vector space. However, current methods designed to measure isotropy, such as average random cosine similarity and the partition score, have not been thoroughly analyzed and are not appropriate for measuring isotropy. We propose IsoScore: a novel tool that quantifies the degree to which a point cloud uniformly utilizes the ambient vector space. Using rigorously designed tests, we demonstrate that IsoScore is the only tool available in the literature that accurately measures how uniformly distributed variance is across dimensions in vector space. Additionally, we use IsoScore to challenge a number of recent conclusions in the NLP literature that have been derived using brittle metrics of isotropy. We caution future studies from using existing tools to measure isotropy in contextualized embedding space as resulting conclusions will be misleading or altogether inaccurate.

## introduction & background the first step in any natural language processing pipeline is to represent text in a vector space. understanding how contextualized word embedding models project tokens into vector space is crucial for advancing the field of natural language processing. several recent studies analyzing the spatial distribution of contextualized word embeddings claim that the point clouds induced by models such as bert or gpt-2 do not uniformly utilize all dimensions of the vector space they occupy @xcite @xcite @xcite . figure 1 illustrates a two-dimensional disk that uniformly utilizes the x and y axes in twodimensional space, but does not uniformly utilize all dimensions when embedded into three dimensions. a distribution is isotropic when variance is uniformly distributed across all dimensions. namely, a distribution is fully isotropic when the covariance matrix is proportional to the identity matrix. several authors suggest that isotropy correlates with improved performance of embedding models @xcite @xcite @xcite @xcite @xcite . however, current methods of measuring the spatial utilization of contextualized embedding models do not truly measure isotropy. the most commonly used methods for measuring spatial distribution in embedding spaces include average random cosine similarity, the partition score, variance explained and intrinsic dimensionality estimation. in section 5 we argue that all current methods of measuring isotropy have fundamental shortcomings that render them inadequate measures of spatial distribution. to overcome these limitations, we introduce isoscore: a novel tool for measuring the extent to which the variance of a point cloud is uniformly distributed across all dimensions in vector space. in contrast to previous attempts of measuring isotropy, isoscore is the first score that incorporates the mathematical definition of isotropy into its formulation. as a result, isoscore has the following desirable properties that surpass the capabilities of existing metrics: (i) it is a global measure of how uniformly distributed points are in vector space that is robust to changes in the distribution mean and scalar changes in covariance; (ii) it is rotation invariant; (iii) it increases linearly as more dimensions are utilized; and (iv) it is not skewed by highly isotropic subspaces within the data. this paper makes the following novel contributions. 1. this paper outlines essential conditions for measuring isotropy and uses a testing suite to empirically verify if a given method meets these conditions. ## isoscore this section introduces the proposed isoscore metric of uniform spatial utilization. ## experiments in subsection 5.1, we present results from numerical experiments designed to test each of the isotropy scores presented in this paper against the six essential properties outlined in section 3.3. exact descriptions of the numerical experiments are provided in appendix e. we reiterate that each of the essential conditions have been derived directly from the mathematical definition of isotropy and violating any of the essential properties disqualifies a method from being a correct measure of isotropy. in subsection 5.2, we demonstrate the merit of isoscore by recreating the experimental setup presented in @xcite . we create word embeddings for tokens from the wikitext-2 corpus using gpt @xcite , gpt-2 @xcite , bert @xcite and distilbert @xcite and calculate isotropy scores for each layer of the model. ## conclusion & future works several studies have attempted to study isotropy in contextualized embedding models. using mathematically rigorous tests, we demonstrate that current methods do not accurately measure isotropy. this paper presents isoscore: a novel method for measuring isotropy that corrects the current misunderstandings in the literature. isoscore is the only tool that is mean agnostic, robust to scalar changes to the covariance matrix and rotation invariant. furthermore, isoscore scales linearly with the number dimensions used and is stable when distributions contain highly isotropic subspaces. future studies should avoid using existing methods to measure isotropy as resulting conclusions will be misleading or altogether inaccurate. there are several promising directions for future work. current studies have used inaccurate methods to claim that increasing isotropy improves the performance of contextualized embedding models. however, we believe that further decreasing isotropy could improve performance, especially in language modeling applications. isoscore could be used as a regularizer when fine tuning word embeddings to penalize distributions that exhibit isotropy. as point clouds of data arise in nearly all deep learning applications, isoscore presents itself as a useful tool to study and refine a variety of models beyond the domain of nlp.