  Contextual word representation models have shown massive improvements on a
multitude of NLP tasks, yet their word sense disambiguation capabilities remain
poorly explained. To address this gap, we assess whether contextual word
representations extracted from deep pretrained language models create
distinguishable representations for different senses of a given word. We
analyze the representation geometry and find that most layers of deep
pretrained language models create highly anisotropic representations, pointing
towards the existence of representation degeneration problem in contextual word
representations. After accounting for anisotropy, our study further reveals
that there is variability in sense learning capabilities across different
language models. Finally, we propose LASeR, a 'Low Anisotropy Sense
Retrofitting' approach that renders off-the-shelf representations isotropic and
semantically more meaningful, resolving the representation degeneration problem
as a post-processing step, and conducting sense-enrichment of contextualized
representations extracted from deep neural language models.
