Norm of Word Embedding Encodes Information Gain (2023)

Abstract: Distributed representations of words encode lexical semantic information, but what type of information is encoded and how? Focusing on the skip-gram with negative-sampling method, we found that the squared norm of static word embedding encodes the information gain conveyed by the word; the information gain is defined by the Kullback-Leibler divergence of the co-occurrence distribution of the word to the unigram distribution. Our findings are explained by the theoretical framework of the exponential family of probability distributions and confirmed through precise experiments that remove spurious correlations arising from word frequency. This theory also extends to contextualized word embeddings in language models or any neural networks with the softmax output layer. We also demonstrate that both the KL divergence and the squared norm of embedding provide a useful metric of the informativeness of a word in tasks such as keyword extraction, proper-noun discrimination, and hypernym discrimination.

## introduction the strong connection between natural language processing and deep learning began with word embeddings @xcite @xcite . even in today's complex models, each word is initially converted into a vector in the first layer. one of the particularly interesting empirical findings about word embeddings is that the norm represents the relative importance of the word while the direction represents the meaning of the word @xcite @xcite @xcite . this study focuses on the word embeddings obtained by the skip-gram with negative sampling (sgns) model @xcite . we show theoretically and experimentally that the euclidean in bayesian inference, the expected kl divergence is called information gain. in this context, the prior distribution is p(•), and the posterior distribution is p(•|w). the information gain represents how much information we obtain about the context word distribution when observing w. empirically, the kl divergence, and thus the norm of word embedding, are helpful for some nlp tasks. in other words, the notion of information gain, which is defined in terms of statistics and information theory, can be used directly as a metric of informativeness in language. we show this through experiments on the tasks of keyword extraction, proper-noun discrimination, and hypernym discrimination in section 7. in addition, we perform controlled experiments that correct for word frequency bias to strengthen the claim. the kl divergence is heavily influenced by the word frequency n w , the number of times that word w appears in the corpus. since the corpus size is finite, although often very large, the kl divergence calculated from the co-occurrence matrix of the corpus is influenced by the quantization error and the sampling error, especially for low-frequency words. the same is also true for the norm of word embedding. this results in bias due to word frequency, and a spurious relationship is observed between word frequency and other quantities. therefore, in the experiments, we correct the word frequency bias of the kl divergence and the norm of word embedding. the contributions of this paper are as follows: after providing related work (section 2) and theoretical background (section 3), we prove the theoretical main results in section 4. in section 5, we extend this theory to contextualized embeddings. we then explain the word frequency bias (section 6) and evaluate kl(w) and ∥u w ∥ 2 as a metric of word informativeness in the experiments of section 7. ## theoretical background in this section, we describe the kl divergence (section 3.2), the probability model of sgns (section 3.3), and the exponential family of distributions (section 3.4) that are the background of our theoretical argument in the next section. ## contextualized embeddings the theory developed for static embeddings of the sgns model is extended to contextualized embeddings in language models, or any neural networks with the softmax output layer. ## word frequency bias in kl divergence the kl divergence is highly correlated with word frequency. in fig. 5 , 'raw' shows the plot of kl(w) against n w . the kl divergence tends to be larger for less frequent words. a part of this tendency represents the true relationship that rarer words are more informative and thus tend to shift the co-occurrence distribution from the corpus distribution. however, a large part of the tendency, particularly for low-frequency words, comes from the error caused by the finite size n of the corpus. this introduces a spurious relationship between kl(w) and n w , causing a direct influence of word frequency. the word informativeness can be better measured by using the kl divergence when this error is adequately corrected. ## experiments in the experiments, we first confirmed that the kl divergence is indeed a good metric of the word informativeness (section 7.1). then we confirmed that the norm of word embedding encodes the word informativeness as well as the kl divergence (section 7.2). details of the experiments are given in appenices d, e, and f. as one of the baseline methods, we used the shannon entropy of p(•|w), defined as it also represents the information conveyed by w as explained in appendix b. 6 and 7 , respectively, in appendix d. ## conclusion we showed theoretically and empirically that the kl divergence, i.e., the information gain of the word, is encoded in the norm of word embedding. the kl divergence and, thus, the norm of word embedding has the word frequency bias, which was corrected in the experiments. we then confirmed that the kl divergence and the norm of word embedding work as a metric of informativeness in nlp tasks.