Margins in Contrastive Learning: Evaluating Multi-task Retrieval for Sentence Embeddings (2025)

Abstract: This paper explores retrieval with sentence embeddings by fine-tuning sentence-transformer models for classification while preserving their ability to capture semantic similarity. To evaluate this balance, we introduce two opposing metrics – polarity score and semantic similarity score – that measure the model’s capacity to separate classes and retain semantic relationships between sentences. We propose a system that augments supervised datasets with contrastive pairs and triplets, training models under various configurations and evaluating their performance on top- k sentence retrieval. Experiments on two binary classification tasks demonstrate that reducing the margin parameter of loss functions greatly mitigates the trade-off between the metrics. These findings suggest that a single fine-tuned model can effectively handle joint classification and retrieval tasks, particularly in low-resource settings, without relying on multiple specialized models.

## introduction tasks like text classification and semantic textual similarity (sts) are helpful for various applications, including retrieval through clustering, zeroshot categorization @xcite , and efficient few-shot classification with limited data @xcite . traditionally, models addressing these tasks ranged from rule-based systems to deep learning architectures @xcite @xcite , with recent transformerbased models dominating the field @xcite @xcite . however, optimizing sentence embeddings for multiple objectives remains a challenge. in this work, we investigate the hypothesis that training sentence-transformer models with two opposing objectives -semantic similarity and polarity -enables models that can be fine-tuned for downstream tasks while preserving their ability to capture semantic similarity. we argue that this approach is beneficial for obtaining more nuanced embeddings, e.g., for domain-specific classification and clustering, especially supporting low-resource settings with a single model capable of both. to evaluate the performance of our models on these dual objectives, we introduce two metrics: polarity score (p) measures the model's classification performance by assessing how well it predicts sentence polarity (e.g., positive vs. negative sentiment). the higher the score, the more accurately the model distinguishes between classes. ## related work related research is based mainly on developments within word and sentence embeddings. commonly used embedding techniques include word2vec @xcite , glove @xcite , and elmo @xcite . in the realm of sentence embeddings, early methods involved concatenation and aggregation of word embeddings to produce a sentence representation @xcite . however, more recent research has focused on developing specialized models to encode sentence representations, as exemplified by systems like infersent @xcite , universal sentence encoder @xcite , sentence-transformers (sbert) @xcite and simcse @xcite . sbert is trained using a pre-trained bert model to learn the representations of a given sentence. while techniques and setups vary, an example of a training procedure is by providing triplets forming (anchor sentence, positive, negative), where the model attempts to maximize the distance between the anchor and the negative (dissimilar sentence), while minimizing the distance between the anchor and the positive (similar) sentence. this methodology provided efficient models for sts @xcite @xcite @xcite . several datasets and benchmarks have been published for sts since the semeval shared task @xcite , including the sts benchmark @xcite , sick @xcite , and biosses @xcite , all of which are now found in the massive text embedding benchmark (mteb) @xcite . transformer models have excelled at the task, as is shown in the tables on huggingface's leaderboard for the evaluation. 2 at the time of experiments, the gte @xcite and e5 @xcite series of models were of particular interest given their strong performance to size ratio. ## methods and data this section includes information on datasets, evaluation metrics, baseline models, loss functions, example generation, and the fine-tuning pipeline. we mainly use two data sources for evaluation, although the provided system is generalizable to any data source for binary classification. figure 1 shows an overview of system components. ## experiments and results the results are based on fine-tuning and continuous evaluation of the baseline models in different setups for loss functions and corresponding parameters. based on similar research on fine-tuning embeddings @xcite , models are trained for five epochs. ## discussion most model configurations adjusted the embeddings towards correct polarity upon fine-tuning. however, the minilm-6 falls short of its semantic similarity capabilities, while the remaining models seem to learn both tasks, with only minor differences between the configurations. ## conclusion and future work this paper has explored the potential of encoding polarity into sentence embeddings while retaining semantic similarity, done by fine-tuning models on data generated to suit the objectives of various sentence-transformers loss functions. we introduced two metrics to evaluate our results: the polarity score p and semantic similarity score s. we found that the e5-small and gte models perform well on all evaluations. in tables 5 and 6 , the fine-tuned configurations greatly improve polarity scores while maintaining the semantic representation when evaluated on the generated datasets. for e5-small, performance on the sarcastic dataset shows great improvement in p, increasing by 26.9% (from 71.4 to 90.6), while s decreases by 2.5% (from 83.4 to 81.3). similarly, on the sst2 dataset, p improves by 12.8% (from 81.5 to 91.9), and decreases by 2.1% in s (from 85.5 to 83.7). furthermore, the tripletloss, especially for lower λ margins, e.g., λ = 0.10, strongly outperformed other configurations and has the potential to yield an efficient and high-performing model for multi-task retrieval, even outside of domains tested in this work, as the findings are mostly consistent between the evaluations. regarding future work, there are several paths for improvement: