On the Curious Case of l2 norm of Sense Embeddings (2022)

Abstract: We show that the l2 norm of a static sense embedding encodes information related to the frequency of that sense in the training corpus used to learn the sense embeddings. This finding can be seen as an extension of a previously known relationship for word embeddings to sense embeddings. Our experimental results show that in spite of its simplicity, the l2 norm of sense embeddings is a surprisingly effective feature for several word sense related tasks such as (a) most frequent sense prediction, (b) word-in-context (WiC), and (c) word sense disambiguation (WSD). In particular, by simply including the l2 norm of a sense embedding as a feature in a classifier, we show that we can improve WiC and WSD methods that use static sense embeddings.

## introduction background: given a text corpus, static word embedding learning methods @xcite ) learn a single vector (aka embedding) to represent the meaning of a word in the corpus. in contrast, static sense embedding learning methods @xcite learn multiple embeddings for each word, corresponding to the different senses of that word. @xcite proposed a random walk model on the word co-occurrence graph and showed that if word embeddings are uniformly distributed over the unit sphere, the log-frequency of a word in a corpus is proportional to the squared ℓ 2 norm of the static word embedding, learnt from the corpus. @xcite showed that under a simple metric random walk over words where the probability of transitioning from one word to another depends only on the squared euclidean distance between their embeddings, the log-frequency of word co-occurrences between two words converges to the negative squared euclidean distance measured between the corresponding word embeddings. @xcite later showed that word embeddings are distributed in a narrow cone, hence not satisfying the uniformity assumption used by @xcite , however their result still holds for such anisotropic embeddings. on the other hand, @xcite showed that a word embedding can be represented as the linearlyweighted combination of sense embeddings. however, to the best of our knowledge, it remains unknown thus far as to what is the relationship between the sense embeddings and the frequency of a sense?, the central question that we study in this paper. contributions: first, by extending the prior results for word embeddings into sense embeddings, we show that the squared ℓ 2 norm of a static sense embedding is proportional to the log-frequency of the sense in the training corpus. this finding has important practical implications. for example, it is known that assigning every occurrence of an ambiguous word in a corpus to the most frequent sense of that word (popularly known as the most frequent sense (mfs) baseline) is a surprisingly strong baseline for wsd @xcite . therefore, the theoretical relationship which we prove implies that we should be able to use ℓ 2 norm to predict the mfs of a word. second, we conduct a series of experiments to empirically validate the above-mentioned relationship. we find that the relationship holds for different types of static sense embeddings learnt using methods such as glove @xcite and skip-gram with negative sampling (sgns; @xcite on semcor @xcite . third, motivated by our finding that ℓ 2 norm of pretrained static sense embeddings encode sensefrequency related information, we use ℓ 2 norm of sense embeddings as a feature for several sense-related tasks such as (a) to predict the mfs of an ambiguous word, (b) determining whether the same sense of a word has been used in two different contexts (wic; pilehvar and camacho-collados, 2019), and (c) disambiguating the sense of a word in a sentence (wsd). we find that, regardless of its simplicity, ℓ 2 norm is a surprisingly effective feature, consistently improving the performance in all those benchmarks/tasks. the evaluation scripts is available at: https://github.com/livnlp/ l2norm-of-sense-embeddings . ## ℓ norm vs. frequency let us first revisit the generative model proposed by @xcite for static word embeddings, where the t-th word, v, in a corpus is generated at step t of a random walk of a context vector c t , which represents what is being talked about. the probability, p(v|c t ), of emitting v at time t is modelled using a log-linear word production model, proportionally to exp(c t ⊤ v). if g v is a word cooccurrence graph, where vertices correspond to the words in the vocabulary, v, the random walker can be seen as visiting the vertices in g v according to this probability distribution. @xcite showed that the partition function, z c , given by ( 1 ) for this probabilistic model is a constant z, independent of the context c. assuming that the stationary distribution of this random walk is uniform over the unit sphere, @xcite proved the relationship in (2), for d dimensional word embeddings, @xmath0. let the frequency of v in the corpus be f (v), and the total number of word occurrences be @xmath1. p(v) can be estimated using corpus counts as f (v)/n . because n , d, and z are constants, independent of v, (2) implies a linear relationship between log f (v) and ||v|| 2 2 . to extend this result to sense embeddings, we observe that the word v generated at step t by the above-described random walk can be uniquely associated with a sense id s v , corresponding to the meaning of v as used in c t . if we consider a second sense co-occurrence graph g s , where vertices correspond to the sense ids, then the above-mentioned corpus generation process corresponds to a second random walk on g s , as shown in figure 1 . although an ambiguous word can be mapped to multiple sense ids across the corpus in different contexts, at any given time step t, a word v is mapped to only one vertex in g s , determined by the context c t . indeed a wsd can be seen as the process of finding such a mapping. the two random walks over word and sense id graphs are isomorphic and converge to the same set of final states @xcite . therefore, an analogous relationship given by (3) can be obtained by replacing word embeddings, v, with sense embeddings, s, in (2). here, d s is the dimensionality of the sense embeddings @xmath2. later in § 3, we empirically show that the normalisation coefficient, z ′ = s exp(c ⊤ s), for sense embeddings also satisfies the self-normalising @xcite property, thus independent of c. if we abuse the notation f (s) to denote also the frequency of s in the corpus (i.e. the total number of times the random walker visits the vertex s), from (3) it follows that log f (s) is linearly related to ||s|| 2 2 . ## empirical validation the theoretical analysis described in § 2 implies a linear relationship between log f (s) and ||s|| 2 2 for the learnt sense embeddings. to empirically verify this relationship, we learn static sense embeddings using glove and sgns from semcor, which is the largest corpus manually annotated with wordnet @xcite sense ids. specifically, we consider the co-occurrences of senses instead of words for this purpose. to distinguish the sense embeddings learnt from glove and sgns from their word embeddings, we denote these by respectively glove-sense and sgns-sense. figure 2 shows the partition function for glovesense embeddings. we see that the partition function is tightly concentrated around its mean, showing that sense-embeddings also demonstrate selfnormalisation similar to word embeddings. similar to the histogram for glove-sense embeddings, we see that the partition function for sgns-sense embeddings is also tightly centred around the mean (i.e., 1.0) from figure 3 . figure 4 shows the correlation between log f (s) and ||s|| 2 2 for glove-sense. we see a moderate positive correlation (pearson's ρ = 0.437) between these two variables, confirming the linear relationship predicted in §2. similar to the correlation plot for glove-sense embeddings, from figure 5 , one can see a positive correlation (pearson's ρ = 0.440) between the log-frequency and squared ℓ 2 norm for the sgns-sense embeddings. it is noteworthy however this linear relationship between log-frequency and squared ℓ 2 norm does not hold for contextualised word embeddings such as bert @xcite or static sense embeddings such as lmms @xcite that are computed by averaging bert embeddings (see appendix b for details). the random walk model described in §2 cannot be applied to contextualised embeddings because the probability of occurrence of a word under the discriminative masked language modelling objectives used to train contextualised word embeddings such as bert depend on all the words generated before as well as after the target word. ## conclusion we investigated log-frequency and ℓ 2 norm of sense embeddings, and found a linear relationship between those. experimental results indicate that, despite its simplicity, ℓ 2 norm of sense embedding is a surprisingly effective feature for mfs prediction, wic and wsd tasks.