Unsupervised Sentence-embeddings by Manifold Approximation and Projection (2021)

Abstract: The concept of unsupervised universal sentence encoders has gained traction recently, wherein pre-trained models generate effective task-agnostic fixed-dimensional representations for phrases, sentences and paragraphs. Such methods are of varying complexity, from simple weighted-averages of word vectors to complex language-models based on bidirectional transformers. In this work we propose a novel technique to generate sentence-embeddings in an unsupervised fashion by projecting the sentences onto a fixed-dimensional manifold with the objective of preserving local neighbourhoods in the original space. To delineate such neighbourhoods we experiment with several set-distance metrics, including the recently proposed Word Mover’s distance, while the fixed-dimensional projection is achieved by employing a scalable and efficient manifold approximation method rooted in topological data analysis. We test our approach, which we term EMAP or Embeddings by Manifold Approximation and Projection, on six publicly available text-classification datasets of varying size and complexity. Empirical results show that our method consistently performs similar to or better than several alternative state-of-the-art approaches.

## introduction 1.on sentence-embeddings dense vector representation of words, or wordembeddings, form the backbone of most modern nlp applications and can be constructed using context-free @xcite @xcite or contextualized methods @xcite . given that practical systems often benefit from having representations for sentences and documents, in addition to word-embeddings @xcite , a simple trick is to use the weighted average over some or all of the embeddings of words in a sentence or document. although sentence-embeddings constructed this way often lose information because of the disregard for word-order during averaging, they have been found to be surprisingly performant @xcite . more sophisticated methods focus on jointly learning the embeddings of sentences and words using models similar to word2vec @xcite , using encoder-decoder approaches that reconstruct the surrounding sentences of an encoded passage @xcite , or training bi-directional lstm models on large external datasets @xcite . meaningful sentence-embeddings have also been constructed by fine-tuning pre-trained bidirectional transformers @xcite using a siamese architecture @xcite . in parallel to the approaches mentioned above, a stream of methods have emerged recently which exploit the inherent geometric properties of the structure of sentences, by treating them as sets or sequences of word-embeddings. for example, @xcite propose the construction of sentenceembeddings based on weighted word-embedding averages with the removal of the dominant singular vector, while @xcite produce sentenceembeddings by concatenating several power-means of word-embeddings corresponding to a sentence. very recently, spectral decomposition techniques were used to create sentence-embeddings, which produced state-of-the-art results when used in concatenation with averaging @xcite . our work is most related to that of @xcite who use random features @xcite to learn document embeddings which preserve the properties of an explicitly-defined kernel based on the word mover's distance @xcite . where @xcite the nature of the kernel, our proposed approach can learn the similarity-preserving manifold for a given setdistance metric, offering increased flexibility. ## results and discussion the results of all our experiments are in compiled in tables 3 and 4 . all statistical tests reported are z-tests, where we compute the right-tailed p-value and call a result significantly different if @xmath0.1. performance of the distance metrics: from we perform statistical significance tests of our method (wmd-emap) against all other methods, for a given dataset, and denote the outcomes by ∨ when the compared method is worse and ∧ when our method is worse, while the absence of a symbol indicates insignificant differences. in terms of absolute accuracy, we observe that our method achieves state-of-the-art results in 2 out of 6 datasets. comparison versus knn: emap almost always outperforms k-nearest neighbours based classification, for all the tested distance metrics. the performance boost for wmd is between a relative percentage accuracy of 0.5% to 14%. this illustrates the efficiency of the proposed manifold-learning method. query sentence best match sentence cosine sim i have spent thousands of dollar's on meyers cookware everthing from kitchenaid anolon prestige faberware & circulan just to name a few though meyers does manufacture very high quality pots & pans and i would recommend them to anyone it's just sad that if you have any problem with them under warranty you have to go throught the chain of command that never gets you anywhere even if you want to speak with upper management about the rudeness of the customer service department their customer service department employees are always very rude and snotty and they act like they are doing you a favor to even talk to you about their products when i opened the box i noticed corrosion on the lid when i contacted rival customer service via email they told me i had to purchase a new lid i called and spoke with a customer service representative and they told me that a lid was not covered under warranty when i explained that i just opened it and it was defective they told me to just return the product that there was nothing that they were going to do after being treated this way i will not be purchasing any more rival products if they don't stand behind their product very very poor customer service 0.997 this movie will bring up your racial prejudices in ways that most movies just elude to it demonstrates how connected we all are as people and how seperated we are by only one thing our viewpoints the acting is superb and you get one cameo appearance after another which is a treat of course the soundtrack is terrific the ending is intense to witness one situation after another coming to an unfortunate finish i waited years for this movie to be released in the united states as far as i was concerned it wasn't about the acting as much as it was about the feeling the actors wanted to portray in which they profoundly accomplished i would recommend this movie to anyone who can reach that one step deeper into the minds of creativity and passion and appreciate the struggles of rising above and beyond the pain of broken dreams 0.998 we see a phrase a lot when we visit how to sites for writers world building by this we mean the setting the characters and everything else where our story will occur for me this often means maps memories and visits since i write about where ## conclusions in this work, we propose a novel mechanism to construct unsupervised sentence-embeddings by preserving properties of local neighbourhoods in the original space, as delineated by set-distance metrics. this method, which we term, emap or embeddings by manifold approximation and projection leverages a method from topological data analysis can be used as a framework with any distance metric that can discriminate between sets, three of which we test in this paper. using both quantitative empirical studies, where we compare with state-of-the-art approaches, and qualitative probing, where we retrieve similar sentences based on our generated embeddings, we illustrate the efficiency of our proposed approach to be on-par or exceeding in-use methods. this work demonstrates the successful application of topological data analysis in sentence embedding creation, and we leave the design of better distance metrics and manifold approximation algorithms, particularly targeted towards nlp, for future research.