Generalised Spherical Text Embedding (2022)

Abstract: This paper aims to provide an unsupervised modelling approach that allows for a more flexible representation of text embeddings. It jointly encodes the words and the paragraphs as individual matrices of arbitrary column dimension with unit Frobenius norm. The representation is also linguistically motivated with the introduction of a metric for the ambient space in which we train the embeddings that calculates the similarity between matrices of unequal number of columns. Thus, the proposed modelling and the novel similarity metric exploits the matrix structure of embeddings. We then go on to show that the same matrices can be reshaped into vectors of unit norm and transform our problem into an optimization problem in a spherical manifold for optimization simplicity. Given the total number of matrices we are dealing with, which is equal to the vocab size plus the total number of documents in the corpus, this makes the training of an otherwise expensive non-linear model extremely efficient. We also quantitatively verify the quality of our text embeddings by showing that they demonstrate improved results in document classification, document clustering and semantic textual similarity benchmark tests.

## introduction most unsupervised text embedding models are trained by encoding the words or paragraphs acquired from the training data as a feature length vector, with the assumption that they reside in the euclidean space. such models are ubiquitous for good reason. aside from their efficiency, they have also proven to be very effective providing us with state of the art results in various intrinsic and extrinsic embedding evaluation tasks. word2vec @xcite , and glove @xcite are two notable examples where word embeddings are learned in the euclidean space and are trained to be oriented such that word vectors that appear in the same context have higher cosine similarity. some of the most common methods of intrinsic evaluation of word embeddings include word similarity, word analogy, and compositional-ity. @xcite , an unsupervised document embedding model generalises the training method introduced in word2vec to documents and achieves improved results in various downstream tasks like sentiment analysis, information retrieval and multi-class classification. there are other document embedding models like skip-thought @xcite and infersent @xcite . the joint spherical embedding model, jose as proposed in @xcite , shows that directional similarity is often more effective in tasks such as word similarity and document clustering. they show that when embeddings are trained in the euclidean space, there is a performance gap between the training stage and usage stage of text embeddings. to bridge that gap, they propose a model which trains both words and paragraphs on a spherical space with tools from riemannian optimization methods. the resulting embeddings are also shown to give considerably better results in word similarity, document clustering, and document classification tasks when compared with other standard models. such application of manifold geometry has also been explored in substantial depth in works like @xcite @xcite . there are also other notable riemannian optimization based embedding training models like @xcite which train embeddings on the hyperbolic manifold space and uses its tree like property for better hierarchical representation of data. hyperbolic word embeddings are also intrinsically linked with gaussian word embeddings @xcite which gives a lot more insight into the geometry of word embeddings. however, most of these text embedding models like jose, word2vec, doc2vec, and fasttext @xcite are trained with the goal of getting a single dense vector representation per word or document. these models treat both polysemous and monosemous words in the same way resulting in the most frequent meaning of the word dominating the others or the meanings getting mixed in the case of former. it is especially detrimental for documents where we use a single dense vector representation to encode information which span over several sentences, often involving multiple topics. this paper aims to address this problem by using matrices as the mode of representation instead of vectors. our model is the joint word and document training generative model proposed in jose where we replace the cosine similarity metric with a novel metric that exploits the matrix structure of the embeddings. this robust metric takes word or document matrices of arbitrary number of columns and calculates the similarity between them. we also show that a few reshape operations allow us to reformulate the optimization problem of our model in terms of the spherical manifold optimization problem. thus, we offer more flexibility in the way of matrix dimensions while retaining efficiency. our choice of metric also suggests that the word, sentence, paragraph/document embeddings do not need to have the same number of columns, which has linguistic validation. ## matrix representation of texts and optimization problem our model design is consistent with jose where it is assumed that text generation is a twostep process: a center word is first generated according to the semantics of the paragraph, and then the surrounding words are generated based on the center word's semantics. consider a positive tuple (u, v, d) where word v appears in the local context window of word u in paragraph d and negative tuple (v, u ′ , d) where u ′ is a randomly sampled word from the vocabulary serving as a negative sample. we represent words v, u, u ′ as matrices v, u, n which are elements of the set s(p, r 1 ) and paragraph d as matrix d which is an element of the set s(p, r 2 ), where p, r 1 , r 2 > 0. from a linguistic perspective, these matrices can be considered as a set of latent variables that govern the semantics of a word or a document. each column is given some arbitrary unit of linguistic information to encode, a latent variable which contributes to the mathematical representation of a word or a document. for example, the columns of a matrix d that represent the document d might encode latent variables that contain information about some topic contained in that document. similarly, the columns of the word matrix u might encode information about a specific context in which a polysemous word u appears. we also keep the number of columns for word matrices less than or equal to the number of columns for sentence/document matrices, i.e., r 1 ≤ r 2 , so that the number of latent variables governing a word should not be more than the ones that govern a sentence or paragraph. novel metric. to model the above mentioned linguistic representation mathematically, we define a novel similarity metric for the ambient space in which we train our matrix embeddings. the proposed metric function is a measure of similarity between two sets of latent variables (matrices) -a function analogous to the cosine similarity measure for vectors in the euclidean space. given two arbitrary matrices @xmath0, r 1 ), @xmath1, r 2 ), we propose the similarity metric g : motivation for our similarity metric. the metric g (1) calculates the average of all the entries in the matrix a ⊤ b. the linguistic intuition behind the choice of this metric is that we want to define a metric that takes the average of dot products between all possible pairs of latent variables (columns) from each matrix. in the case of r 1 = r 2 = 1, g(a, b) reduces to the cosine similarity metric between unit norm vectors a and b which is the metric used in the spherical space model of jose. however, in the case of higher values of r 1 , r 2 , for example, let two words v 1 and v 2 be represented by pro- note that this is different from computing the cosine similarity which gives (a 1 b1 + a 2 b 2 ). moreover, the regular cosine similarity between word and paragraph embedding matrices with unequal dimensions (p × r 1 and p × r 2 respectively) is not defined. on the other hand, our proposed similarity metric is still applicable. modelling. as our model has the same generative process as jose, we take the same maxmargin loss function and substitute the cosine similarity metric with our new similarity metric g where the word matrices u, v, @xmath2, r 1 ) and paragraph matrix @xmath3, r 2 ) with r 1 ≤ r 2 . we get the following loss, i.e., ## experiments to highlight the quality of our obtained matrix representations, we run the same set of evaluations as jose with a relatively lower number of columns, i.e., 1 ≤ r 1 ≤ r 2 ≤ 6. we notice that for even higher values, the quality of our embeddings grad-ually decrease. moreover, the word similarity experiment results are not added as words seemingly do not benefit from our representation directly. indeed, the best word similarity score are obtained for r 1 = r 2 = 1. instead, we add semantic textual similarity benchmark tests to show that sentences can benefit from this matrix representation model. unless otherwise stated, our model and jose are trained for 35 iterations on the respective corpora; the local context window size is 5; the embedding dimension is kept at 100; the number of negative samples are 2. other hyperparameters in our model are kept the same as jose. ## conclusion in this paper, we extend the joint modelling idea used for training text embeddings from vectors with unit norm to matrices with unit frobenius norm. each word/sentence/document matrix is made to encode information in a way that each column of the matrix represents some latent topic, context, or discourse. since the standard vector dot product can no longer be applied, we introduce a novel similarity metric that allows the measurement of similarity between matrices of arbitrary number of columns. for optimization simplicity, we reshape our matrices to vectors of unit norm that allows using the riemannian gradient descent optimization algorithm on the spherical manifold. our theory is validated quantitatively by the results which shows that our text embeddings outperform or produce similar results when compared with jose in document classification, clustering, and semantic textual similarity tasks. this paper is meant to serve as a ground work for more involved research topics which integrate concepts of differential geometry and nlp. future directions could include qualitative analysis on the columns of the matrices to see what tangible information they encode which will allow for better modelling. another direction would be to exploit other matrix structures on the embeddings, e.g., treating each word embedding as a symmetric positive definite matrix and to study whether they can be beneficial.