Low Anisotropy Sense Retrofitting ( LAS e R ) : Towards Isotropic and Sense Enriched Representations (2021)

Abstract: Contextual word representation models have shown massive improvements on a multitude of NLP tasks, yet their word sense disambiguation capabilities remain poorly explained. To address this gap, we assess whether contextual word representations extracted from deep pretrained language models create distinguishable representations for different senses of a given word. We analyze the representation geometry and find that most layers of deep pretrained language models create highly anisotropic representations, pointing towards the existence of representation degeneration problem in contextual word representations. After accounting for anisotropy, our study further reveals that there is variability in sense learning capabilities across different language models. Finally, we propose LASeR, a ‘Low Anisotropy Sense Retrofitting’ approach that renders off-the-shelf representations isotropic and semantically more meaningful, resolving the representation degeneration problem as a post-processing step, and conducting sense-enrichment of contextualized representations extracted from deep neural language models.

## introduction distributional word representations, developed using large-scale training corpora, form an integral part of the modern nlp methodological paradigm. the advent of deep pre-trained neural language models such as bert @xcite and gpt-2 @xcite has led the shift towards the development of contextualized word representations. unlike static word representation models, such as word2vec @xcite and fasttext @xcite , which conflate multiple senses of a word within a single representation, contextual word representation models assign as many representations to a word as the number of contexts it appears in. the preference for contextual word representations can be attributed to the significant improvements they have achieved in a wide variety of nlp tasks including question answering, textual entailment, sentiment analysis @xcite and commonsense reasoning @xcite , to name a few. to utilize contextual word representations as knowledge resources, it is necessary to determine their ability to mirror the linguistic relations employed in language @xcite . there is a growing body of literature that assesses whether contextual representations encode information about word-senses, where each word-sense portrays an aspect of the meaning of a given word in a given context @xcite . a recent analysis by @xcite reported that contextual word representations can learn humanlike word sense knowledge, where they compared cosine relatedness between homonyms and polysemous word senses against human sense-related judgements. when calculating cosine relatedness, such studies assume the encoded vector space to be isotropic in nature. geometrically, isotropy in a vector space is defined as vectors being uniformly distributed across all directions, instead of occupying a narrow cone @xcite . recent studies point towards anisotropy (lack of isotropy) in contextual word representations @xcite , which affects prior conclusions regarding word-sense information encoded in vector spaces. for example, in an isotropic vector space, if cosine relatedness between word representations a and b is 0.9, we conclude them to be highly similar. but, if the vector space is anisotropic, where cosine relatedness between randomly sampled words is 0.95, then the representations a and b are deemed less similar than randomly sampled words. this shows that the existence and the extent of anisotropy in the vector space affects conclusions regarding whether representations are actually similar or merely a product of representation degeneration. hence, when evaluating the sense learning capabilities of deep pretrained language models through vector relatedness measures, accounting and adjusting for vector space anisotropy becomes necessary. in this regard, our work presents three key contributions. first, we analyze and adjust for anisotropy across contextual representations extracted from all layers of four language models (bert, gpt-2, xlnet and electra). the representation space for each model encodes anisotropy, varying in terms of number and strength of common directions in model representations. we find that models learning unidirectional context create more anisotropic representations than models learning bidirectional context. second, we observe that sense information is not equally encoded in all models, where (pseudo) bidirectional models learn to disambiguate word senses better than others. moreover, sense information is better retained in the lower layers and significantly reduces in the upper model layers due to the representations getting more contextualized. third, to address these preliminary findings and to contribute towards the creation of sense-coherent representations, we propose laser, a 'low anisotropy sense retrofitting' approach, bringing word representations closer to the goal of mirroring lexical semantic relations present in natural language while removing artifacts of representation degeneration from learned representations. thus, we combine vector space transformation and knowledge-based vector specialization methods to create more isotropic and sense enriched representations, ensuring that we retain the distributional properties learnt during pretraining, while aligning and grounding the representation geometry towards better sense learning. ## related work prior works which modify off-the-shelf embeddings to improve their lexical-semantic representation can be divided into two primary categories: (1) anisotropy treatment methods and (2) retrofitting methods. anisotropy treatment methods focus on improving the isotropy of word vectors, promoting uniform distribution of information across all directions @xcite @xcite . isotropy in contextual vector spaces is regarded valuable, especially when utilizing vector geometry and relatedness measures in downstream analyses @xcite . prior methods that focus on creating more isotropic vector spaces have suggested principle component manipulation (removal, extension) of vector spaces @xcite . to our knowledge, these methods have been proposed for static word representations, but are yet to be extended to contextual word representations extracted from a wide variety of language models. on the other hand, retrofitting methods are focused on enhancing the representation geometry, by encoding lexical semantic relations through semantic specialization, a post-processing approach that enforces linguistic constraints on vector spaces by relying on external linguistic knowledge databases @xcite @xcite . semantic specialization as a post processing step (retrofitting) is currently limited to static word representations @xcite where they have yielded impressive performance improvements over raw embeddings @xcite . existing methods towards semantic specialization of contextual representations primarily focus on retraining the model from scratch @xcite or post-hoc fine-tuning the model @xcite @xcite . these methods are (1) resourceintensive (retraining or fine-tuning) and (2) do not address the representation degeneration problem in vector representations @xcite . ## results we first show anisotropy analysis results ( §4.1), further evaluating sense learning in contextual representations ( §4.2). finally, we present improvements in isotropy and lexical-semantic capabilities of the post-processed representations ( §4.3). ## discussion recent works have discussed whether contextualized word representations extracted from deep pretrained language models encode word sense knowledge within the representation space. studies suggest that while lower layer bert embeddings encode more semantic information @xcite , the upper layer embeddings become increasingly contextual @xcite . works exploring semantic capabilities of representations have also used nearest neighbour classifier probes to assess whether same-sense representations are classified together @xcite . since these classifiers show slightly better accuracy than classifying as the most frequent sense, they claim that the representation space encodes sense information. although our work supports this conclusion, we additionally argue that after accounting for anisotropy, the cohesion between same sense representations and separation between different sense representations is not significant. here, the principal premise of the removal of anisotropy prior to injecting sense information is based on creating an embedding space geometry where the effects of representation degeneration are reduced. the representation degeneration of embeddings reduces their representational power @xcite . thus, to improve the representation ability of embeddings, we deem it important to create methods that promote representations that are not only lexicosemantic relation enriched but also isotropic. our method reveals that the additional step of lowering anisotropy renders improved representation geometry, where word vectors are not constricted within a narrow cone, and are uniformly distributed within the vector space. further, sense-retrofitting on contextualized word representations render same sense representations more similar and different sense representations more different, increasing the word sense disambiguation capabilities of the encoded representations. our work presents a novel intrinsic evaluation of sense information in word embeddings, required to understand the sense geometry encoded by various models. in the future, we will focus on integrating sense information in contextual word representations by extending this approach to words that are unseen to the laser model, and further perform extrinsic analyses of the embeddings. ## conclusion in this work, we investigated the geometry of contextual word representations for isotropy and sense disambiguation capabilities. we further proposed a post-processing approach for anisotropy treatment and semantic enrichment of contextual word representations, by transforming the vector space using principal component manipulation and lexical semantic knowledge-based sense-retrofitting. our method significantly reduced the impact of representation degeneration problem, improving isotropy within the vector space and rendered off-the-shelf contextual word vectors semantically more meaningful. in the future work, we will study the impact of changes in retrofitting hyperparameters and variable removal of primary components on representation quality. further, we will focus on extrinsic evaluation of the impact of anisotropy removal and sense retrofitting on downstream wordsense disambiguation tasks.