More Discriminative Sentence Embeddings via Semantic Graph Smoothing (2024)

Abstract: This paper explores an empirical approach to learn more discriminantive sentence representations in an unsupervised fashion. Leveraging semantic graph smoothing, we enhance sentence embeddings obtained from pretrained models to improve results for the text clustering and classification tasks. Our method, validated on eight benchmarks, demonstrates consistent improvements, showcasing the potential of semantic graph smoothing in improving sentence embeddings for the supervised and unsupervised document categorization tasks.

## background: graph filtering and smoothing graph signal processing @xcite provides a framework to analyze and process signals defined on graphs, by extending traditional signal processing concepts and tools to the graph domain. this allows for the representation and manipulation of signals in a way that is tailored to the specific structure of the graph. in what follows we refer to matrices in boldface uppercase and vectors in boldface lowercase. graph signals graph signals are mappings from the set of vertices to the real numbers. a graph signal for a given graph g can be represented using vector the smoothness of a signal f over graph g can be characterized using the laplacian quadratic form associated with laplacian l: these signals can be high dimensional and can represent many kinds of data. in our case, signals will represent text embeddings. graph filters smoother graph signals can be obtained by minimizing the quantity described in formula (1). that is the goal of graph filters and the filtering is generally done from a spectral perspective. a specific class of filters that additionally has an intuitive interpretation from a vertex perspective is that of the polynomial filters. when the filter is a p -th order polynomial of the form ĥ(l) = p @xmath0, the filtered signal at vertex i, is a linear combination of the components of the input signal at vertices within a p -hop local neighborhood of vertex i: ## proposed methodology: smoothing sentence embeddings in this paper, we theorize that smoothing sentence embeddings with a semantic similarity graph can help supervised and unsupervised categorization models better differentiate between the similar and dissimilar documents, leading to performance gains. a common choice for quantifying semantic similarity of text is the cosine similarity; given two sentence embedding vectors x i , x @xmath1 we build a k-nearest neighbors connectivity graph which we denote g based on this similarity measure i.e. a graph for which each node has exactly k neighbors and whose edge weights are all equal to one. we characterize the graph g using its adjacency matrix a, we denote its laplacian as l. given the adjacency matrix, a standard trick to obtain better node representations consists in adding a self-loop now given a node embedding matrix x and the previous semantic similarity graph. we consider four polynomial graph filters whose propagation rules we describe in ## experiments in this section we evaluate our semantically smoothed representations obtained through four filters on two tasks, clustering and classification, with respect to the original representations obtained from sentencebert @xcite as well as two large language models baselines: bert and roberta. ## conclusion we proposed a simple yet effective empirical approach that consists in using similarity graphs in an unsupervised manner to smooth sentence embeddings obtained from pretrained models in a semantically aware manner. the systematic improvements in performance on both clustering and classification tasks on several benchmark datasets of different scales and balance underscore the effectiveness of using semantic graph smoothing to improve sentence representations.