LDIR : Low-Dimensional Dense and Interpretable Text Embeddings with Relative Representations (2025)

Abstract: Semantic text representation is a fundamental task in the field of natural language processing. Existing text embedding (e.g., SimCSE and LLM2Vec) have demonstrated excellent performance, but the values of each dimension are difficult to trace and interpret. Bag-of-words, as classic sparse interpretable embeddings, suffers from poor performance. Recently, Benara et al. (2024) propose interpretable text embeddings using large language models, which forms ”0/1” embeddings based on responses to a series of questions. These interpretable text embeddings are typically high-dimensional (larger than 10,000). In this work, we propose Low-dimensional (lower than 500) Dense and Interpretable text embeddings with Relative representations (LDIR). The numerical values of its dimensions indicate semantic relatedness to different anchor texts through farthest point sampling, offering both semantic representation as well as a certain level of traceability and interpretability. We validate LDIR on multiple semantic textual similarity, retrieval, and clustering tasks. Extensive experimental results show that LDIR performs close to the black-box baseline models and outperforms the interpretable embeddings baselines with much fewer dimensions.

## introduction text embedding is an important technique for natural language processing by transforming textual data into numerical representations that capture semantics. the embeddings encode the meaning of contexts in a vector space where similar texts are close together in the representation space, playing vital roles in multiple tasks such as semantic textual similarity @xcite , information retrieval @xcite , and retrievalaugmented generation @xcite . in particular, our text embedding is relatively lowdimensional and dense, while maintaining interpretable. seminal text embedding models mainly fall into two categories. the first is based on lightweight pre-trained models such as bert @xcite , including sentence-bert @xcite , simcse @xcite , etc. the second is based on current large language models such as llama-3 @xcite , including @xcite , angie @xcite , etc. the text embeddings usually have 768 to 4096 dimensions correlated with the hidden size of the model and perform well across tasks. however, the values on each dimension are difficult to trace and interpret directly. in contrast, bag-of-words is a classical sparse and interpretable text representation where each dimension's value represents the frequency of a specific word. however, its representational capability is limited. recent qaemb-mbqa @xcite and cqg-mbqa @xcite generate interpretable text embeddings based on large language models, where the value on each dimension is restricted to "0/1", reflecting "yes/no" answers to different crafted questions, as shown in in the aforementioned interpretable text embeddings, the value on each dimension are explicitly defined. however, due to the limitations of their sparse or "0/1" representations, they often require a high dimensionality. for example, bag-of-words is associated with the word vocabulary size, which typically reaches around 30k sizes in current models. qaemb-mbqa and cqg-mbqa rely on designing a large number of questions, usually around 10k. in this work, we aim to mitigate the limitations of sparse or "0/1" representation and the requirement of high dimensionality. inspired by @xcite , we propose building low-dimensional, dense, and interpretable text embeddings with relative representations @xcite . specifically, the value on each dimension represents the relatedness to different "anchor texts" by automatic farthest point sampling, which can be floating-point numbers instead of "0/1", thereby leading a overall flexible and dense embeddings. we hope the dimensionality of vectors can be significantly reduced in this way, achieving a balance among semantic expressiveness, representational efficiency, and interpretability. we conduct extensive experiments to validate the semantic expressiveness of ldir, including evaluations on seven semantic textual similarity tasks, six retrieval tasks, and seven clustering tasks. experimental results show that our embeddings, with only 500 embedding dimensions, achieve comparable performance to black-box embeddings, and outperforms interpretable embedding baselines of qaemb-mbqa and cqg-mbqa. ## analyses and discussions in our main experiments, we remove the long texts in corpus and use fps to extract the anchor texts. we analysis the impact of sampling methods (section 5.1) and the length of anchor texts (section 5.2) below. then we discuss how to obtain a more interpretable dense embeddings with the calculation of fine-grained relatedness (section 5.3), compare different metrics on relatedness (section 5.4). finally, we show a case study (section 5.5) and discuss the usage of relative representation based embeddings (section 5.6). ## conclusion we propose ldir, a low-dimensional dense and interpretable text embeddings with relative representations. the numerical values in ldir represent the correlation between the text and the anchor texts automatically obtained through farthest point sampling, which improves the expressiveness at a low cost compared with qa-based 0/1 embeddings. across multiple tasks and datasets, ldir demonstrates better semantic expressiveness compared to multiple interpretable embedding baselines. we also discuss the need for more suitable interpretability metrics and how to further enhance the interpretability of ldir, which can serve as our future work for interpretable text embeddings.