A Cluster-based Approach for Improving Isotropy in Contextual Embedding Space (2021)

Abstract: The representation degeneration problem in Contextual Word Representations (CWRs) hurts the expressiveness of the embedding space by forming an anisotropic cone where even unrelated words have excessively positive correlations. Existing techniques for tackling this issue require a learning process to re-train models with additional objectives and mostly employ a global assessment to study isotropy. Our quantitative analysis over isotropy shows that a local assessment could be more accurate due to the clustered structure of CWRs. Based on this observation, we propose a local cluster-based method to address the degeneration issue in contextual embedding spaces. We show that in clusters including punctuations and stop words, local dominant directions encode structural information, removing which can improve CWRs performance on semantic tasks. Moreover, we find that tense information in verb representations dominates sense semantics. We show that removing dominant directions of verb representations can transform the space to better suit semantic applications. Our experiments demonstrate that the proposed cluster-based method can mitigate the degeneration problem on multiple tasks.

## introduction despite their outstanding performance, cwrs are known to suffer from the so-called representation degeneration problem that makes the embedding space anisotropic @xcite . in an anisotropic embedding space, word vectors are distributed in a narrow cone, in which even unrelated words are deemed to have high cosine similarities. this undesirable property hampers the representativeness of the embedding space and limits the diversity of encoded knowledge @xcite . to better understand the representation degeneration problem in pre-trained models, we analyzed the embedding space of gpt-2 @xcite , bert @xcite , and roberta @xcite . we found that, despite being extremely anisotropic in all non-input layers from a global sight, the embedding space is significantly more isotropic from a local point of view (when embeddings are clustered and each cluster is made zero-mean). motivated by this observation and based on previous studies that highlight the clustered structure of cwrs @xcite , we extend the technique of @xcite with a further clustering step. in our proposal, we cluster embeddings and apply pca on individual clusters to find the corresponding principal components (pcs) which indicate the dominant directions for each specific cluster. nulling out these pcs for each cluster renders a more isotropic space. we evaluated our cluster-based method on several tasks, including semantic textual similarity (sts) and word-in-context (wic). experimental results indicate that our cluster-based method is effective in enhancing the isotropy of different cwrs, reflected by the significant performance improvements in multiple evaluation benchmarks. in addition, we provide an analysis on the reasons behind the effectiveness of our cluster-based technique. the empirical results show that most clusters contain punctuation tokens, such as periods and commas. the pcs of these clusters encode structural information about context, such as sentence style; hence, removing them can improve cwrs performance on semantic tasks. a similar structure exists in other clusters containing stop words. the other important observation is about verb distribution in the contextual embedding space. our experiments reveal that verb representations are separated across the tense dimension in distinct sub-spaces. this brings about an unwanted peculiarity in the semantic space: representations for different senses of a verb tend to be closer to each other in the space than the representations for the same sense that are associated with different tenses of the same verb. indeed, removing such pcs improves model's ability in downstream tasks with dominant semantic flavor. ## isotropy in cwrs isotropy is a desirable property of word embedding spaces and arguably any other vector representation of data in general @xcite . from the geometric point of view, a space is called isotropic if the vectors within that space are uniformly distributed in all directions. lacking isotropy in the embedding space affects not only the optimization procedure (e.g., model's accuracy and convergence time) but also the expressiveness of the embedding space; hence, improving the isotropy of the embedding space can lead to performance improvements @xcite . we measure the isotropy of embedding space using the partition function of @xcite : ## discussion in this section, we provide a brief explanation for reasons behind the effectiveness of the clusterbased approach through investigating the linguistic knowledge encoded in the dominant local directions. we also show that enhancing isotropy reduces convergence time. ## conclusions in this paper, we proposed a cluster-based method to address the representation degeneration problem in cwrs. we empirically analyzed the effect of clustering and showed that, from a local sight, most clusters are biased toward structural information. moreover, we found that verb representations are distributed based on their tense in distinct sub-spaces. we evaluated our method on different semantic tasks, demonstrating its effectiveness in removing local dominant directions and improving performance. as future work, we plan to study the effect of fine-tuning on isotropy and on the encoded linguistic knowledge in local regions.