Locality Preserving Sentence Encoding (2021)

Abstract: Although researches on word embeddings have made great progress in recent years, many tasks in natural language processing are on the sentence level. Thus, it is essential to learn sentence embeddings. Recently, Sentence BERT (SBERT) is proposed to learn embeddings on the sentence level, and it uses the inner product (or, cosine similarity) to compute semantic similarity between sentences. However, this measurement cannot well describe the semantic structures among sentences. The reason is that sentences may lie on a manifold in the ambient space rather than distribute in an Euclidean space. Thus, cosine similarity cannot approximate distances on the manifold. To tackle the severe problem, we propose a novel sentence embedding method called Sentence BERT with Locality Preserving (SBERT-LP), which discovers the sentence submanifold from a high-dimensional space and yields a compact sentence representation subspace by locally preserving geometric structures of sentences. We compare the SBERT-LP with several existing sentence embedding approaches from three perspectives: sentence similarity, sentence classification and sentence clustering. Experimental results and case studies demonstrate that our method encodes sentences better in the sense of semantic structures.

## introduction word embeddings aim to learn semantically meaningful word representations based on distribution hypothesis @xcite . both contextfree @xcite and contextual @xcite word embeddings have made great progress in various downstream tasks: text classification @xcite , dialogue system @xcite and text clustering @xcite . however, in the real world, most natural language processing tasks are on the sentence level. hence, recent studies @xcite encode sentences into a dense vector space, which is described as the sentence space. these sentence embedding approaches generally fall into two categories: one is based on supervised learning, including: infersent @xcite , universal sentence encoder @xcite and sbert @xcite . while the other one is based on unsupervised learning, such as skipthought vectors @xcite , fastsent @xcite and transformer-based sequential denoising auto-encoder (tsdae) @xcite . the unsupervised way overcomes the limitation of labeled data in different domains and data annotations, to some extent. both of them represent a sentence as a point in the sentence space, where similar sentences are close. there are two important problems in text processing: how to represent texts and how to evaluate their semantic similarity @xcite . recently, various strategies have been taken to represent a sentence. for example, the sbert @xcite learns semantic sentence representations with a siamese network on top of bert. additionally, some variants have been proposed such as sbert-wk @xcite and bert-flow @xcite . the sentence space of the sbert is associated with a euclidean structure and the cosine similarity is employed to measure the semantic similarity. however, previous studies have demonstrated that human-generated text data are probably sampled from a submanifold of the ambient euclidean space @xcite . as a result, sentence representations yielded from the sbert may lie on a manifold, which is either linear or non-linear. the semantic similarity between sentences is the shortest distance, which may be curves, on the manifold. hence, making use of the cosine similarity to approximate the length of a curve is inaccurate. for obtaining correct semantic structures of the sentence space, one straight way is to calculate the geodesic distance @xcite , which is the length of the shortest path between two points on the possibly curvy manifold @xcite . however, because of requiring traversing from one point to another on the manifold, the geodesic distance is hard to approximate. therefore, we aim to find an optimal euclidean subspace of the sentence manifold. in the subspace, cosine similarity is effective to measure sentence semantic relations. for implementing it, we borrow the idea of locally linear embedding @xcite , which is an effective way to develop low dimensional representations when data arises from sampling a probability distribution on a manifold @xcite . then, we propose the sentence bert with locality preserving (sbert-lp), which marries up the locality property and bert. our method highlights the local geometric structures of sentences. to be specific, the sbert-lp firstly discovers the intrinsic manifold structure from the original sentence space. a novel euclidean sentence subspace is then learned from the sub-manifold by preserving local geometric information of sentences. the local geometric structures are defined by each sentence and its neighbor sentences. preserving locality avoids losing too much useful information of sentences during the projection. finally, cosine similarity between sentences is consistent with their semantic similarity. our contributions are summarized in three-folds: ## related work existing sentence embedding approaches are divided into two categories: non-parametric sentence embeddings and parametric sentence embeddings @xcite . the non-parameterized way is to derive sentence embeddings from pre-trained word embeddings @xcite via linear aggregations. for example, sif @xcite uses smooth inverse frequency to weigh each word in a sentence and remove some special directions with pca. besides, usif @xcite builds upon the random walk model by setting the probability of word generation inversely related to the angular distance between the word and sentence embeddings. although the non-parametric methods have been proved to be efficient, neglecting word orders and sentence structures degrades their performance. in order to incorporate richer sentence information, parametric sentence embeddings are proposed. for example, skipthought @xcite borrows the idea of skip-gram in word2vec. it encodes sentences intending to predict adjacent sentences. with the success of bert @xcite on various nlp tasks @xcite , some bert-based sentence embedding methods have been proposed recently. in addition to the sbert, the sbert-wk encodes sentences with qr factorization, re-weighting each word in a sentence. furthermore, bert-flow @xcite leverages normalized flows to transform the bert sentence space into a standard gaussian latent space that is isotropic. it concludes that the inner product may not accurately represent semantic similarity in the sentence space because of the non-smoothing semantic structure. in contrast, the sbert-lp analyzes and solve the cosine metric problem of the sbert sentence space on a manifold. our work is inspired by the investigation of local geometry in the word space @xcite . these methods solve semantic problems in word space. since both word and sentence embeddings share the same high-dimensional space, problems with word embeddings may exist in sentence embeddings @xcite . to the best of our knowledge, this paper is the first to solve the semantic metric problem in the sentence space with the incorporation of locality preserving ability. ## methodology in this section, we first give a brief introduction to sbert. then, we will show how to effectively preserve the locality of sentences to solve the problem of semantic similarity metrics. ## experiments in this section, we perform experiments on three tasks to demonstrate the effectiveness of the sbert-lp. we firstly introduce experimental settings for the datasets and hyper-parameters. then we compare the sbert-lp with several state-ofthe-art sentence encoding methods. finally, we analyze the effect of different parameters on the sbert-lp, and we make use of some cases from sts datasets to illustrate the effectiveness of our model on semantic metric recovery. sentence embeddings aim to cluster semantically similar sentences. therefore, we mainly focus on the performance of different models on the sts task and take the results of the other two tasks as references. ## conclusion in this paper, we propose the sbert-lp that is simple yet effective. to solve the metric problem in the sentence space, this method well exploits the idea of locality preserving to recovery the cosine similarity. it not only captures the sentence submanifold but also rebuilds a euclidean sentence subspace. experimental results on three tasks demonstrate that the sbert-lp learns better sentence representations in the sense of semantic structures.