Comparing in context: Improving cosine similarity measures with a metric tensor (2021)

Abstract: Cosine similarity is a widely used measure of the relatedness of pre-trained word embeddings, trained on a language modeling goal. Datasets such as WordSim-353 and SimLex-999 rate how similar words are according to human annotators, and as such are often used to evaluate the performance of language models. Thus, any improvement on the word similarity task requires an improved word representation. In this paper, we propose instead the use of an extended cosine similarity measure to improve performance on that task, with gains in interpretability. We explore the hypothesis that this approach is particularly useful if the word-similarity pairs share the same context, for which distinct contextualized similarity measures can be learned. We first use the dataset of Richie et al. (2020) to learn contextualized metrics and compare the results with the baseline values obtained using the standard cosine similarity measure, which consistently shows improvement. We also train a contextualized similarity measure for both SimLex-999 and WordSim-353, comparing the results with the corresponding baselines, and using these datasets as independent test sets for the all-context similarity measure learned on the contextualized dataset, obtaining positive results for a number of tests.

## introduction cosine similarity has been largely used as a measure of word relatedness, since vector space models for text representation appeared to automatically optimize the task of information retrieval @xcite . while other distance measures are also commonly used, such as euclidean distance @xcite , for cosine similarity only the vector directions are relevant, and not their norms. more recently, pre-trained word representations, also referred to as embeddings, obtained from neural network language models, starting from word2vec (w2v) @xcite , emerged as the main source of word embeddings, and are subsequently used in model performance evaluation on tasks such as word similarity @xcite . datasets such as simlex-999 @xcite and wordsim-353 @xcite , which score similarity between wordpairs according to the assessment of several humans annotators, have become the benchmarks for the performance of a certain type of embedding on the task of word similarity @xcite @xcite . for n a and n b , the vector representations of two distinct words w a and w b , cosine similarity takes the form with the euclidean inner product between any two vectors n a and n b given as using this measure of similarity, improvements can only take place if the vectors that represent the words change. however, the assumption that the vectors interact using a euclidean inner product becomes less plausible when it comes to higher order vectors. if, differently, we consider that the vector components are not described in a euclidean basis, then we enlarge the possible relationships between the vectors. specifically in the calculation of the inner product, on which the cosine similarity depends, we can use an intermediary metric tensor. by challenging the assumption that the underlying metric is euclidean, cosine similarity values can be improved without changing vector representations. we identify two main motivations to search for improved cosine similarity measures. the first motivation has to do with the cost of training larger and more refined language models @xcite . by increasing the performance on a task simply by changing the evaluation measure without changing the pre-trained embeddings, we expect that better results can be achieved with more efficient and interpretable methods. this is particularly true of contextualized datasets, with benefits not only for tasks such as word similarity, but also others that use cosine similarity as a measure of relatedness, such as content based recommendation systems @xcite , and where it can be particularly interesting to explore the different metrics that emerge as representations of vector relatedness. the second motivation comes from compositional distributional semantics, where words of different syntactic types are represented by tensors of different ranks, and representations of larger fragments of text are produced via tensor contraction (coecke and clark, 2010; @xcite @xcite . this framework has proved to be a valuable tool for low resource languages, enhancing the scarce available data with a grammatical structure for composition, providing embeddings of complex expressions @xcite . as these contractions depend on an underlying metric that is usually taken to be euclidean, improvements have only been achieved, once again, by modifying word representations @xcite . as proposed by @xcite , another way to improve on these results consists in using a different metric to mediate tensor contractions. metrics obtained in tasks such as word similarity can be transferred to tensor contraction, and thus we expect this work to open new research avenues on the compositional distributional framework, providing a better integration with (contextual) language models. this paper is organized as follows. in §2 we introduce an extended cosine similarity measure, motivating the introduction of a metric on the hypothesis that it can optimize the relationships between the vectors. in §3 we explain our experiment on contextualized and non-contextualized datasets to test whether improvements can be achieved. in §4 we present the results obtained in our experiments and in §5 we discuss these results and propose further work. our contributions are summarized below: ## model a metric is a tensor that maps any two vectors to an element of the underlying field k, which in this case will be the field of real numbers r. this element is what is known as the inner product. to this effect, the metric tensor can be represented as a function, not necessarily linear, over each of the coordinates of the vectors it acts on. in geometric terms, the metric characterizes the underlying geometry of a vector space, by describing the projection of the underlying manifold of a non-euclidean geometry to a euclidean geometry r n (wald, 2010). the inner product between two vectors is informed by the metric in a precise way, and is representative of how the distance between two vectors should be calculated. a standard example consists of two unit vectors on a sphere, which is an s 2 manifold that can be mapped onto r 3 . if the vectors are represented in spherical coordinates, which are a map from s 2 to r 3 , the standard method of computing the angle between the vectors using eq. ( 1 ) will fail to give the correct value. the vectors need to be transformed by the appropriate non-linear metric to the euclidean basis in r 3 before a contraction of the coordinates can take place. to illustrate this, take as an example a triangle drawn on the surface of a sphere s 2 . if it is projected onto a planisphere r 3 , a naive measurement of its internal angles will exceed the known 180 degrees, which corresponds to a change in the inner product between the vectors tangents to the triangle corners (see @xcite for a demonstration). to preserve this inner product, and thus recover the equivalence between a triangle on a spherical surface and a triangle on a euclidean plane, the coordinates need to be properly transformed by the appropriate metric before they are contracted. by the same token, we explore here the possibility that the shortcomings of the values obtained using cosine similarity when compared with human similarity ratings are not due to poor vector representations, but to a measure that fails to assess the distance between the vectors adequately. to test this hypothesis, we generalize the inner product of eq. ( 2 ) to accommodate a larger class of relationships between vectors, modifying it using a metric represented by the distance matrix d, once a basis is assumed, that defines the inner product between two vectors as we can thus learn the components of a metric for a certain set of vectors by fitting it to the goal of preserving a specified inner product. in the case of word similarity, the matrix b can be learned supervised on human similarity judgments, towards the goal that a contextualized cosine similarity applied to a set of word embeddings, using eq. ( 6 ), returns the correct human assessment. an advantage of this approach is that the cosine is symmetric with respect to its inputs, which is a nice property that this extension preserves by requiring that symmetry of the metric. ## methods the general outline of our experiment is as follows. first, we learn contextualized cosine similarity measures for related (contextualized) pairs of words, and afterwards for unrelated (noncontextualized) pairs of words. a schematic representation can be found in fig. 1 . we then test whether these learned measures are transferable and provide improvements on word pairs that were not seen during training, when compared with the standard cosine similarity baseline. ## results the validation results on comparing the results of bert contextualized and non-contextualized, the baseline values of the contextualized representations were worse than those obtained with the contextualized embeddings, although without statistical significance, while the improvement after training was consistently better and significant for all datasets with the contextualized representations. figs. 2c and 2d , show that the distribution of points using the contextualized embeddings is more concentrated and collinear, making it more likely that a metric that acts in the same way for all points in the dataset will rotate and rescale them into a positive correlation. the percentual increases also show that bert contextualized had the greatest increases from before to after training, suggesting that there was a cumulative effect in considering the context both in the representations and in the similarity measure. ## conclusion and outlook in this paper we tested whether a contextualized notion of cosine similarity could be learned, improving the similarity not only of the results for the datasets where it was learned, but of unrelated similarities. we showed that this metric improved the correlations above baseline, and that, when learned on a contextualized similarity dataset, it had an advantage when compared to one learned on a dataset with unrelated word-pairs. we furthermore showed that this framework has the potential to generalize the notion of similarity to word-pairs it has not seen during training. an important future research line towards interpretability consists in understanding the properties of the metrics that yielded the best results, particularly in identifying the distinctive features of the best metrics, such as their eigensystems. other further directions include applying these metrics to distributional compositional contractions, including with dependency enhancements @xcite , testing this framework on larger contextualized datasets and trying out more complex, non-linear, metric forms.