A o E : Angle-optimized Embeddings for Semantic Textual Similarity (2024)

Abstract: Text embedding is pivotal in semantic textual similarity (STS) tasks, which are crucial components in Large Language Model (LLM) applications. STS learning largely relies on the cosine function as the optimization objective to reflect semantic similarity. However, the cosine has saturation zones rendering vanishing gradients and hindering learning subtle semantic differences in text embeddings. To address this issue, we propose a novel Angle-optimized Embedding model, AoE. It optimizes angle differences in complex space to explore similarity in saturation zones better. To set up a comprehensive evaluation, we experimented with existing short-text STS, our newly collected long-text STS, and downstream task datasets. Extensive experimental results on STS and MTEB benchmarks show that AoE significantly outperforms popular text embedding models neglecting cosine saturation zones. It highlights that AoE can produce high-quality text embeddings and broadly benefit downstream tasks.

## related work our work is related to text embedding learning. compared to early efforts focusing on word embeddings @xcite ), text embeddings (a more general concept) enable semantic representation for richer context. many prior studies @xcite employed pretrained models such as bert @xcite for learning text embeddings without fine-tuning. more recent work showed the benefits of fine-tuning the pretrained models to improve sts. some studies @xcite involved humanlabeled training data, e.g., nli datasets. in training methods, sbert @xcite used siamese bert, while many others adopted contrastive learning @xcite @xcite . most widely-used models employ cosine to measure similarity in their learning objectives, as shown in aoe is inspired by complex embeddings for using complex division to exploit angle differences. in positional word embedding learning, @xcite adopted complex multiplication to introduce rotary position embedding into transformer architecture. in knowledge graph learning, @xcite ; @xcite presented entity embedding in complex space to model the source to target rotations for link prediction. however, their embeddings are on the word or entity level and focus on their position modeling. on the contrary, we make the first efforts to leverage complex division to compute normalized angle differences between sentence-level text embeddings for mitigating the negative effects of the cosine's saturation zones. ## aoe framework here, we will introduce the aoe methods with the overall framework in figure 2 . we will first present how to encode the complex text embeddings in section 3.1, followed by the angle objective in section 3.2. at last, section 3.3 will outline the final learning objective to describe aoe's training process. encoder xi xj xm xn x im x re text embeddings e im e re token embeddings e berts llama backbone angle optimize x im 1 re δθ complex space cos(x) δy ≈ 0 y δx complex division 1 im re divisor w(1, 1) δθ dividend z (-1, 1) quotient (0, 1) z w compute angle difference in complex space δθ ij δθ mn θ ij θ mn ... sim( , ) = 0.9 sim( , ) = 0.3 sorting ground-truth ranking encoder angle difference angle objective xj im xj re xi im xi re z w prediction xn im xn re xm im xm re z w 1. input pairs 2. encoding 3. compute angle difference 4. compute loss 5. updating parameters figure 2: the overall framework of aoe. initially, the input text pairs (x i , x j ) and (x m , x n ) are processed by the encoder to obtain real and imaginary text embeddings: (x re i , ## experimental setup here, we elaborate on the experimental setup, including datasets, baselines, evaluation metrics, and implementation details. we also open-source our trained models in appendix section e. datasets. following standard setup @xcite , the training data is from multinli and snli. our statistics reveal that 33% of the text pairs show a similarity above 0.95 and 66% above 0.8. it means a large proportion of samples in or near cosine saturation zones, implying the challenges of learning subtle semantic differences for them. for evaluation, we test aoe on sts tasks with existing widely-used short-text sts datasets and our newly proposed long-text github issue similarity dataset. furthermore, we examine aoe on downstream data with 7 popular tasks and mteb. we extracted duplicated issues from 55 famous open-source projects (see appendix b) on github using github api to compile the dataset. the duplicate issues served as positive samples, while the remaining ones were considered negative. these open-source projects have active participation from maintainers and volunteers to maintain the issue quality. additionally, we randomly selected 10% of the data for manual inspection, and the quality was found to be satisfactory. 93% of the sampled data can be clearly classified as either similar with label 0 or dissimilar with label 1. our statistics show that the proportion of long text (with token @xmath0, validation, and test sets is 61.03%, 60.85%, and 60.50%, respectively. more details of gis are presented in appendix b. foot_0 evaluation metrics. for sts, we follow previous studies to use senteval @xcite to compute spearman's correlation and report the "all" setting. for downstream tasks, we employ senteval to assess the performance of text embeddings. for a fair comparison, we follow baselines and use the default parameters of sen-teval. for mteb, we employ the official mteb evaluation code to test the performance of aoe. for all our implementations, we will report the average score over five runs and the std value (±). baselines. because aoe is supervised, we primarily compare it with widely used supervised embedding baselines for a fair comparison. they are: infersent @xcite , use @xcite , sbert @xcite , cosent @xcite , and supervised versions of simcse @xcite and consert @xcite . in particular, given different backbones, we compare aoe with sbert and simcse, the two most widely-used text embedding baselines. all the above baselines are opensource embeddings. in addition, we adopt two popular closed-source baselines, openai's ada-002 (openai, 2022b) and openai's text-embeddings-3 (openai, 2024), for a comprehensive comparison. for efficient fine-tuning with the initial learning rate to 1e -4. for embeddings, we used the prompt "summarize sentence {text} in one word:" to obtain the summative token and concatenate it to the last token of the text and then apply its token embeddings as the text embeddings, inspired by @xcite . for the temperature in objectives, we set the τ to 0.05 following the previous practice @xcite . for w 1 and w 2 in equation 5, we use the grid search strategy to search for their values. for a fair comparison with prior work, we follow simcse @xcite to set the random seed to 42 for all main experiments. yet, in section 5.2, we test aoe without fixed random seeds to examine its robustness. ## experimental results section 5.1 first presents the main comparison results, followed by an ablation study in section 5.2. finally, we will further discuss aoe in section 5.3. ## conclusion in this paper, we have presented a novel text embedding model called aoe, which optimizes the angle difference in complex space to mitigate the negative effects of cosine saturation zones. to comprehensively evaluate aoe with sts tasks, we have introduced a github issues similarity dataset for long-text sts evaluation. extensive experiments have suggested that aoe outperforms baselines, indicating that aoe can produce high-quality text embeddings and benefit various downstream tasks.