ALIGN - SIM : A Task-Free Test Bed for Evaluating and Interpreting Sentence Embeddings through Semantic Similarity Alignment (2024)

Abstract: Sentence embeddings play a pivotal role in a wide range of NLP tasks, yet evaluating and interpreting these real-valued vectors remains an open challenge to date, especially in a task-free setting. To address this challenge, we introduce a novel task-free test bed for evaluating and interpreting sentence embeddings. Our test bed consists of five semantic similarity alignment criteria, namely, *semantic distinction, synonym replacement, antonym replacement, paraphrasing without negation, and sentence jumbling*. Using these criteria, we examined five classical (e.g., Sentence-BERT, Universal Sentence Encoder (USE), etc.) and eight LLM-induced sentence embedding techniques (e.g., LLaMA2, GPT-3, OLMo, etc.) to test whether their semantic similarity spaces align with what a human mind would naturally expect. Our extensive experiments with 13 different sentence encoders revealed that none of the studied embeddings aligned with all the five semantic similarity alignment criteria. Yet, most encoders performed highly on the SentEval dataset, a popular task-specific benchmark. This finding demonstrates a significant limitation of the current practice in sentence embedding evaluation and associated popular benchmarks, a critical issue that needs careful attention and reassessment by the NLP community. Finally, we conclude the paper by highlighting the utility of the proposed alignment-based test bed for analyzing sentence embeddings in a novel way, especially in a task-free setting.

## introduction one of the fundamental tasks in nlp is to computationally map sentences into dense vector representations for subsequent analysis. these dense vectors, known as "sentence embeddings", have proven valuable across a wide range of downstream tasks, including translation, question answering, and text classification @xcite @xcite , etc. these embeddings encapsulate the meaning of sentences (and their similarity) in a latent semantic space. however, interpreting and evaluating these dense fixed-size vectors remains an open challenge, particularly in a task-free setting. in this work, we introduce a novel task-free test bed called align-sim to address these challenges and conduct a comprehensive evaluation of popular sentence embeddings using the same. our test bed is grounded on five semantic similarity alignment criteria that are intuitive to a human mind: 1) semantic distinction, 2) synonym replacement, 3) antonym replacement, 4) paraphrasing without negation, and 5) sentence jumbling. based on these five criteria, align-sim systematically tests whether the semantic similarity space of an existing sentence embedding technique aligns with what a human mind would naturally expect. computationally, given a sentence s and its corresponding embedding s x , the basic idea of the test-bed is to (a) perturb s according to a particular criterion to create s ′ (with embedding s ′ ## task-specific benchmark: senteval we started our investigation by assessing the effectiveness of sentence embeddings on downstream tasks; we evaluated them on the popular senteval benchmark @xcite (details in appendix a.3). for this benchmark, we compared five popular classical sentence encoders and eight llms. the classical encoder-only models include 1) universal sentence encoder (use) @xcite , 2) sentence-bert (sbert) (reimers and gurevych, 2019a), 3) infersent @xcite , 4) simcse @xcite . the only classical encoder-decoder model is 5) language-agnostic-sentence representation (laser) @xcite . the eight decoder-only llms loaded using huggingface include: 1) gpt3-ada foot_0 (openai, 2022), 2) llama-2-7b-hf @xcite , 3) meta-llama-3-8b (ai@meta, 2024), 4) gptneo (eleutherai, 2023) 5) bloom ( scao @xcite . note that the encoder-based models are trained to generate sentence embeddings, whereas decoderonly models are originally trained for text generation. more details can be found in appendix a.4. ## align-sim test bed and five criteria in this section, we introduce our semantic alignment test bed, align-sim, for evaluating and interpreting sentence encoders in a novel way, especially in a task-free setting. the test bed comprises five semantic alignment criteria as follows. ## discussions and conclusion in this paper, we introduced a novel semantic alignment test bed, align-sim, for evaluating and interpreting sentence embeddings in task-free settings. our framework, grounded on five semantic alignment criteria -semantic distinction, synonym, and antonym replacement, paraphrasing without negation, and sentence jumbling -provides a new way of analyzing and comparing the latent semantic spaces of different sentence embedding models. our extensive experiments with 13 different sentence encoders, including both classical models and llm-induced embeddings, revealed a significant contrast between their performance on traditional task-specific benchmarks and their alignment with the proposed five evaluation criteria. surprisingly, none of the examined sentence embedding models could fulfill all five semantic alignment criteria despite many of them achieving high performance on the senteval dataset. this discrepancy demonstrates a significant limitation of the current practice in sentence embedding evaluation and associated popular benchmarks, a critical issue that needs careful attention and reassessment by the nlp community. it also raises a concern about which sentence encoder should be used when extensive finetuning is unrealistic due to the unavailability of task-specific training data. how should one prioritize between benchmark-specific accuracy and alignment with human expectations when it comes to selecting a particular sentence embedding? more importantly, this work highlights the fact that most works on sentence encoders to date have focused primarily on optimizing accuracy numbers on taskspecific benchmarks while ignoring their intrinsic alignment with human's natural expectations. nevertheless, this paper demonstrates the utility of our proposed align-sim test bed as a new paradigm for evaluating sentence embedding models in a task-free setting, complementing taskspecific assessments. the test bed, along with the 5 semantic alignment criteria in itself, is a novel contribution, while the curated datasets and extensive experiments that establish an initial taskindependent benchmark are also valuable contributions. our test-bed analysis also reveals that further research is needed to improve sentence encoders that excel in both task-specific benchmarks and alignment with essential linguistic properties expected by humans.