Space Decomposition for Sentence Embedding (2024)

Abstract: Determining sentence pair similarity is crucial for various NLP tasks. A common technique to address this is typically evaluated on a continuous semantic textual similarity scale from 0 to 5. However, based on a linguistic observation in STS annotation guidelines, we found that the score in the range [4,5] indicates an upper-range sample, while the rest are lower-range samples. This necessitates a new approach to treating the upper-range and lower-range classes separately. In this paper, we introduce a novel embedding space decomposition method called MixSP utilizing a Mixture of Specialized Projectors, designed to distinguish and rank upper-range and lower-range samples accurately. The experimental results demonstrate that MixSP decreased the overlap representation between upper-range and lower-range classes significantly while outperforming competitors on STS and zero-shot benchmarks.

## introduction determining the similarity between sentence pairs is fundamental to many downstream applications such as text classification, search, and ranking. usually, sentence pair similarity is assessed via semantic textual similarity (sts), where each sample contains a pair of sentences, and their label denotes the degree of similarity, which uses scores from 0 to 5, where 5 represents the highest degree of similarity. studies have shown that improving the ability to rank sentence pairs according to their similarities enhances text classification accuracy @xcite @xcite and reranking mean-average precision @xcite . a common approach to solving the sts problem is employing a pre-trained language model @xcite and finetuning it 0.0 0.2 0.4 0.6 0.8 1.0 simcse: 0.374 overlap overlap area 0-4 4-5 0.0 0.2 0.4 0.6 0.8 1.0 +ft: 0.385 overlap 0.0 0.2 0.4 0.6 0.8 1.0 +moe: 0.324 overlap 0.0 0.2 0.4 0.6 0.8 1.0 +mixsp: 0.314 overlap with a supervised learning objective. in particular, we aim to construct an embedding space in which the cosine similarity between sentence pairs reflects the degree of similarity; contrastive learning is a popular method to achieve such an embedding space @xcite @xcite . regardless of the differences in their data augmentation strategies, all aforementioned methods treat the degree of similarity as a continuous range. in other words, all learning methods indifferently treat sentence pairs with different degrees of similarities within the sts score range @xcite . in this paper, we challenge the common practice of treating sts scores as a continuous spectrum. several studies observed that the score range @xcite signifies semantically related (i.e., upper-range) samples, while the rest represents unrelated (i.e., lower-range) samples @xcite . consequently, the sts problem should be considered a ranking within two distinct classes rather than one continuous spectrum. we introduce a novel embedding space decomposition method called mixsp utilizing a mixture of specialized projectors. the novelty of mixsp lies in a carefully designed learning pipeline with the following traits: (i) the ability to distinguish upperrange from lower-range samples and (ii) the ability to accurately rank sentence pairs within each class. in particular, our method uses a routing network and two specialized projectors to handle upperrange and lower-range representations, resulting in a better sts performance overall. figure 1 illustrates how our embeddings can better distinguish different sentence pairs as compared to our competitors: simcse @xcite , ft @xcite , and moe @xcite compared to that of mixsp. we quantify the confusion between upperrange and lower-range classes as the cosine score overlaps between these two classes using gaussian kernel density estimates. a smaller overlap indicates the ability to distinguish the upper-range and lower-range classes. we can see that mixsp obtains the smallest overlap between upper-range and lower-range classes of 31.4% while all competitors have an overlap ranging from 32.4% to 38.5%. regarding the similarity ranking performance, our method also produces superior performance compared to these competitors (as shown in figure 3 , section 5.3). these results demonstrate that our method improves the ability to distinguish upperrange and lower-range samples and rank sentence pairs according to their similarities. our contributions are as follows: ## mixture of specialized projectors we design our method, mixture of specialized projectors (mixsp), based on our linguistic observation that when labeling sentence pairs for similarity, the score ranges of [0,4) and [4,5] are considered as two distinct classes: lower-range and upper-range, respectively. in particular, we address our space decomposition problem by designing a classify-andrank pipeline with a routing mechanism and one specialized projector for each class. consequently, we improve (i) the ability to differentiate between upper-range and lower-range sentence pairs and (ii) the ranking performance within each, thereby uplifting the overall sentence similarity prediction performance. figure 2 displays our classify-and-rank pipeline consisting of the following components: ## conclusion in this paper, we proposed a novel embedding space decomposition method called mixsp, a mixture of specialized projectors. we challenge the common practice in treating sts scores from a continuous paradigm [0,5] to embedding space decomposition for lower-range [0,4) and upper-range [4,5] classes. our experiments highlight that mixsp outperforms competitive methods in the average cases of sts and zero-shot benchmarks. we also discuss the improvement of our method, design choice, and inference speed to emphasize the effectiveness of our framework. the embedding space decomposition is the promising paradigm for sentence embedding.