Solving Cosine Similarity Underestimation between High Frequency Words by \ell_2 Norm Discounting (2023)

Abstract: Cosine similarity between two words, computed using their contextualised token embeddings obtained from masked language models (MLMs) such as BERT has shown to underestimate the actual similarity between those words CITATION.This similarity underestimation problem is particularly severe for high frequent words. Although this problem has been noted in prior work, no solution has been proposed thus far. We observe that the \ell_2 norm of contextualised embeddings of a word correlates with its log-frequency in the pretraining corpus.Consequently, the larger \ell_2 norms associated with the high frequent words reduce the cosine similarity values measured between them, thus underestimating the similarity scores.To solve this issue, we propose a method to discount the \ell_2 norm of a contextualised word embedding by the frequency of that word in a corpus when measuring the cosine similarities between words.We show that the so called stop words behave differently from the rest of the words, which require special consideration during their discounting process.Experimental results on a contextualised word similarity dataset show that our proposed discounting method accurately solves the similarity underestimation problem.An anonymized version of the source code of our proposed method is submitted to the reviewing system.

## introduction cosine similarity is arguably the most popular word similarity measure used in numerous natural language processing (nlp) tasks, such as question answering (qa), information retrieval (ir) and machine translation (mt) @xcite @xcite . first, a word is represented by a vector (aka embedding) and then the similarity between two words is computed as the cosine of the angle between the corresponding vectors @xcite . despite the good performance of cosine similarity as a similarity measure in various downstream tasks, @xcite showed that it systematically underestimates the true similarity between highly frequent words, when computed using contextualised word embeddings obtained from mlms such as bert @xcite . compared to the problem of estimating similarity between highly frequent words, the opposite problem of estimating the similarity between (or involving) rare (low frequency) words has received greater attention, especially in the scope of static word embeddings @xcite @xcite . if a word is rare in a corpus, we might not have a sufficiently large number of contexts containing that word to learn an accurate embedding for it. this often leads to unreliable similarity estimations between words and has undesirable implications in downstream tasks such as the detection of analogies and social biases @xcite . on the other hand, @xcite studied the impact of frequency on contextualised word embeddings and showed that the cosine similarity between highly frequent words are systematically underestimated. unlike in the previously discussed low frequency word scenario, we do have adequate contexts to learn an accurate semantic representation for highly frequent words. therefore, it might appear surprising at first that cosine similarity cannot be correctly estimated even for the highly frequent words. @xcite show that the diversity (measured by the volume of the bounding hypersphere) of the contextualised embeddings of a target word, computed from multiple contexts containing the word, increases with the frequency of that word. they provide an explanation that holds true only for 2-dimensional embeddings, which relates diversity to the underestimation of cosine similarity. unfortunately, this explanation does not extend to the high dimensional embeddings used in practice by the nlp community (e.g. bert token embeddings are typically more than 768 dimensional). more importantly, to the best of our knowledge, no solution has been proposed in the literature to address the cosine similarity underestimation problem associated with the highly frequent words. in prior work, the ℓ 2 norm of a static word embedding has been shown to linearly correlate with the log-frequency of that word @xcite . on the other hand, we empirically study the ℓ 2 norm of the contextualised embedding of a word w averaged over all of its contexts, and find that it too approximately linearly correlates with the log-frequency of w in the corpus used to pretrain the mlm. recall that the cosine similarity is defined as the inner-product between two embeddings, divided by the ℓ 2 norm of those embeddings. therefore, we suspect that the underestimation of cosine similarity between highly frequent words is due to the larger ℓ 2 norms associated with those words. to correct for this bias associated with the ℓ 2 norms of highly frequent words, we propose a linearly parameterised discounting scheme in the logfrequency space. specifically, we use monte-carlo bayesian optimisation @xcite to find the optimal discounting parameters. our proposed discounting method is shown to accurately correct the underestimation of cosine similarities between highly frequent words on the word-in-context (wic) (pilehvar and camacho-collados, 2019) dataset where human similarity ratings are available for the same word in two different con-texts. source code for reproducing the experiments reported in this is paper is publicly available. 1 2 underestimation of cosine similarity let us denote the d-dimensional contextualised word embedding produced by an mlm f for a target word w appearing in a context c by f (w, c)(∈ r d ). moreover, let the set of contexts where w occurs in a given corpus be s(w). we refer to {f (w, c)|@xmath0. to study the relationship between the cosine similarity scores and the frequency of words, we use the 768-dimensional bert-base-uncased foot_1 as the contextualised embedding model. we use the token embedding of w from the final hidden layer of bert as f (w, c). we approximate the word frequencies in bert pretraining corpus using the bookcorpus @xcite . let ψ w be the frequency of w in this corpus. we use the wic dataset, which contains 5428 pairs of words appearing in various contexts with annotated human similarity judgements. wic dataset is split into official training and development sets, while a separate hidden test set is used by the leaderboard for ranking word sense disambiguation systems. from figure 1 , we see that for both same as well as different meaning contexts, the predicted cosine similarities drop with the word frequencies. moreover, the gradient of the drop for same meaning pairs (pearson's @xmath1.3001) is larger than that for the different meaning pairs (@xmath2.2125), indicating that the underestimation of cosine similarity is more sever for the similar contexts of highly frequent words. ## ℓ 2 norm discounting to understand the possible reasons behind the cosine similarity underestimation for highly frequent words discussed in § 2, for each word w we compute its mean sibling embedding, ŵ, given by (1). we plot || ŵ|| against log(ψ(w)) in figure 2 separately for a predefined set of stop words and all other words (i.e. non-stop words). for this purpose, we use the default 1466 stop words from nltk and randomly selected 997,425 non-stop words from the bookcorpus. pearson r values of stop words and non-stop words are respectively 0.1697 and 0.3754, while the lines of best fits for each class of words are superimposed. from figure 2 , we see that overall, || ŵ|| increases with log(ψ w ) for both stop and non-stop words, while the linear correlation is stronger in the latter class. considering that stop words cover function words such as determiners and conjunctions that co-occur with a large number of words in diverse contexts, we believe that the ℓ 2 norm of stop words mostly remains independent of their frequency. recall that the cosine similarity between two words is defined as the fraction of the inner-product of the corresponding embeddings, divided by the product of the ℓ 2 norms of the embeddings. therefore, even if the inner-product between two words remain relatively stable, it will be divided by increasingly larger ℓ 2 norms in the case of highly frequent words. moreover, this bias is further amplified when both words are high frequent due to the product of ℓ 2 norms in the denominator. to address this problem, we propose to discount the ℓ 2 norm of a word w by a discounting term, α(ψ w ), and propose a discounted version of the cosine similarity given by (2). following figure 2 , we linearly parameterise α(ψ w ) separately for stop vs. non-stop words as in (3). the scalar parameters m s , m n , b s and b n are estimated as follows. first, we randomly initialise all parameters uniformly in [0, 1] and use (2) to predict cosine similarity between two contexts in which a target word w occurs in the wic train instances. we then make a binary similarity judgement (i.e. same or different meaning) for the pair of contexts in an instance depending on whether the predicted cosine similarity is greater than a threshold θ. next, we compute the overall binary classification accuracy for the similarity predictions made on the entire wic training dataset, and use bayesian optimisation to find the optimal values: θ = 0.545, m @xmath3.00422, b @xmath4.643, m @xmath5.00427 and b @xmath6.821. specifically we used the adaptive experimentation platform foot_3 for ## results to evaluate the effect of the proposed ℓ 2 norm discounting when computing cosine similarity, we repeat the analysis presented in figure 1 using (2) to predict the similarity between contextualised word embeddings. comparing the lines of best fit for the original (blue, @xmath7.3006) vs. discounted (orange, @xmath8.1366) for the same meaning contexts, we see that the gradient of the drop has decreased by 51.65%. likewise, comparing the lines of best fit for the original (green, @xmath9.2125) vs. discounted (red, @xmath10.0843) for the different meaning contexts, we see the gradient of the drop has decreased by 57.04%. this result clearly shows that the proposed ℓ 2 norm discounting method is able to reduce the underestimation of cosine similarities for the highly frequent words. given that the discounting parameters in (3) are learned from the wic train data, it remains an open question as to how well the proposed discounting method generalises when predicting similarity between contextualised embeddings of unseen words. to evaluate this generalisability of the proposed method, we use (3) with its learned parameters from wic train data, to predict the similarity between contextualised word embeddings in wic dev data. figure 5 shows the average proportion of instances predicted to be the same meaning as a function of frequency, grouped into ten bins, each with the same number of examples. from figure 5 , we see that in high frequency bins (i.e. bins 8, 9 and 10), the percentage of predicted instances as having the same meaning is consistently lower than that compared to the human judgements. this shows an underestimation of the true (human judged) similarity between contextualised word embeddings. on the other hand, when we use the proposed ℓ 2 norm discounted cosine similarity (defined in (2)), in the highest frequent bin (i.e. 10) we see that the gap between human judgements vs. predicted similarities has reduced. moreover, in the low frequency bins (i.e. 1-4), we see that the proposed discounting method does not affect the predictions made using cosine similarities. we see an overestimation of the cosine similarities in the low frequency bins as reported by @xcite . as discussed already in § 1, the word embeddings learnt for low frequency words tend to be unreliable due to data sparseness. therefore, we believe it is important to focus on the problem of learning accurate word embeddings rather than to adjust cosine similarities between low-frequency words in a post-processing step. we see that in bins 5, 6 and 7 the similarity scores are slightly increased by the proposed discounting method, which is a drawback that needs to be addressed in future work. more importantly however, the overall percentage recall across all bins for retrieving same meaning instances improves significantly from 74.7% to 83.7% compared to using respectively the original cosine similarity vs. the discounted cosine similarity. overall, this result confirms the validity of the proposed discounting method for addressing the underestimation of cosine similarity involving highly frequent words. ## conclusion we proposed a method to solve the cosine similarity underestimation problem in highly frequent words. specifically, we observed that the ℓ 2 norm of a contextualised word embedding increases with its frequency in the pretrain corpus and proposed a discounting scheme. experimental results on wic dataset confirmed the validity of the proposed method.