Topology of Word Embeddings: Singularities Reflect Polysemy (2020)

Abstract: The manifold hypothesis suggests that word vectors live on a submanifold within their ambient vector space. We argue that we should, more accurately, expect them to live on a pinched manifold: a singular quotient of a manifold obtained by identifying some of its points. The identified, singular points correspond to polysemous words, i.e. words with multiple meanings. Our point of view suggests that monosemous and polysemous words can be distinguished based on the topology of their neighbourhoods. We present two kinds of empirical evidence to support this point of view: (1) We introduce a topological measure of polysemy based on persistent homology that correlates well with the actual number of meanings of a word. (2) We propose a simple, topologically motivated solution to the SemEval-2010 task on Word Sense Induction & Disambiguation that produces competitive results.

## introduction static word embeddings attempt to represent words by vectors in a high-dimensional vector space r n in such a way that words of similar meaning are represented by (cosine) similar vectors, and vice versa. according to the manifold hypothesis, we should expect these vectors to lie within a lowerdimensional word space w, a subspace of r n that resembles a manifold. to what extent this hypothesis is true in this and other contexts is the subject of ongoing research @xcite . in this paper, we argue and demonstrate that for the word space w, polysemy is a principal obstruction to any strict interpretation of the manifold hypothesis. that polysemy presents a serious obstacle to the creation of adequate word vector representations this work is licensed under a creative commons attribution 4.0 international licence. licence details: http: //creativecommons.org/licenses/by/4.0/ . let us therefore hypothesize a space of meanings m that accurately represents all possible meanings and their similarities. our argument is a simple topological observation based on the relationship between this space m and the word space w. for an idealised language, where there is a bijection between meanings and words, these two spaces would agree. for a natural language, however, multiple points of m get identified with a single point of w. this process corresponds to a topological construction that we refer to as pinching (see figure 2 ). it is easy to see that a space resulting from pinching cannot be a manifold. thus, even if the space of meanings m satisfies the manifold hypothesis perfectly, the pinched space w cannot satisfy the hypothesis near polysemous words. 1 based on this intuition, and using tools from topological data analysis, we introduce a measure for the polysemy of a word based on its vector embedding. our experiments show that this topological polysemy (tps) correlates well with the actual number of meanings that a word has. in addition, we present an approach to the semeval-2010 task on word sense induction & disambiguation (task 14) @xcite . this approach is independent of tps, but based on the same ideas. despite its simplicity, it is almost on par with the best performing algorithm within the 2010 challenge, and outperforms far more complicated approaches. we see these experimental results as strong evidence that our interpretation of the word space w as a pinched manifold is more adequate than a more naïve view of w as an actual manifold. ## empirical evidence we present two pieces of empirical evidence that support our view of the word space as a pinched manifold. the experiments in sections 4.2 and 4.3 show that the topological polysemy defined above correlates with the actual number of meanings of a word. in section 4.4, we describe a simple approach to the semeval-2010 task on word sense induction based on our topological intuition. figure 7 : an exemplary context of an instance of the target word "cultivate" ## conclusion in this work, we challenge the manifold hypothesis for static word vector embeddings and experimentally show that it is more accurate and helpful to view the space of word embeddings as a pinched manifold. we introduce a topological measure of polysemy that correlates well with the number of meanings of a word according to the gold standard of the semeval-2010 task on word sense induction & disambiguation. we also produce a surprisingly simple, but topologically motivated solution to the task itself that achieves highly competitive results. we stress that our measure of polysemy, tps, is computed solely on the topology of the point cloud consisting of the vectors of a fixed word vector embedding. of course, any solution to the described