Redundancy, Isotropy, and Intrinsic Dimensionality of Prompt-based Text Embeddings (2025)

Abstract: Prompt-based text embedding models, which generate task-specific embeddings upon receiving tailored prompts, have recently demonstrated remarkable performance. However, their resulting embeddings often have thousands of dimensions, leading to high storage costs and increased computational costs of embedding-based operations. In this paper, we investigate how post-hoc dimensionality reduction applied to the embeddings affects the performance of various tasks that leverage these embeddings, specifically classification, clustering, retrieval, and semantic textual similarity (STS) tasks. Our experiments show that even a naive dimensionality reduction, which keeps only the first 25% of the dimensions of the embeddings, results in a very slight performance degradation, indicating that these embeddings are highly redundant. Notably, for classification and clustering, even when embeddings are reduced to less than 0.5% of the original dimensionality the performance degradation is very small. To quantitatively analyze this redundancy, we perform an analysis based on the intrinsic dimensionality and isotropy of the embeddings. Our analysis reveals that embeddings for classification and clustering, which are considered to have very high dimensional redundancy, exhibit lower intrinsic dimensionality and less isotropy compared with those for retrieval and STS.

## introduction text embeddings are a foundational component of many natural language processing (nlp) applications, including document retrieval, retrievalaugmented generation @xcite , and text clustering. recent advances in large language models (llms) renewed interest in text representation learning, owing to their strong language understanding and generalization capabilities @xcite @xcite figure 1 : the performance on classification and retrieval tasks under dimensionality reduction. the models used in the experiments are described in section 2. @xcite . among these, prompt-based text embedding models, which produce task-specific embeddings by incorporating natural language instructions or task descriptions, have demonstrated remarkable performance on various tasks @xcite @xcite @xcite . however, prompt-based models typically generate embeddings with thousands of dimensions, leading to high storage costs and increased computational costs of embedding-based operations. for instance, e5-mistral @xcite , a model obtained by fine-tuning mistral-7b @xcite , produces 4096-dimensional embeddings. reducing the dimensionality of llmbased text embeddings through post-processing could thus offer substantial practical benefits. in this work, we show that prompt-based text embedding models can maintain surprisingly strong performance even when their dimensionality is substantially reduced. figure 1 summarizes our findings. even a naive dimensionality reduction that keeps only the first 25% of the embedding dimensions results in almost no performance degradation across a range of tasks, indicating that these embeddings are highly redundant. notably, for classification task, reducing embeddings to less than 0.5% of the original dimensionality can almost preserve their original performance. furthermore, we find that the embeddings after dimensionality reduction perform better than the same-dimensional embeddings produced by smaller models. we also find that the robustness of these models to dimensionality reduction varies by task type; while classification and clustering exhibit a more gradual performance decline, with the extent of degradation varying across models, retrieval and sts tasks tend to experience a more rapid drop in performance, with several models showing similar trends. to investigate why such a significant reduction is feasible, we quantitatively assess the redundancy in the generated embeddings. specifically, we analyze the intrinsic dimensionality (id) and isotropy of representations derived from different task-specific prompts using isoscore @xcite . our findings reveal that prompt-based text embedding models produce distinct representation properties depending on the task prompt. for classification and clustering tasks, we observe lower intrinsic dimensionality and less isotropic distributions, which correlates with their high redundancy and robustness to even drastic dimensionality reduction. in contrast, for retrieval and sts tasks-where finegrained similarity is critical-embeddings exhibit higher intrinsic dimensionality and more isotropic distributions in the embedding space. moreover, we find a relationship between measures such as id and isoscore and the robustness of embeddings to dimensionality reduction; embeddings for classification and clustering tasks are highly redundant and remain robust even under drastic reduction, whereas those for retrieval and sts tasks are less redundant and degrade more significantly when dimensions are reduced. ## intrinsic dimensionality and isotropy of prompt-based text embeddings to investigate why the robustness to dimensionality reduction varies across tasks, we quantitatively evaluate the redundancy of the embeddings. ## conclusion and future work we demonstrated that the high-dimensional embeddings produced by prompt-based text embedding models can maintain strong performance even after dimensionality reduction by simply retaining the first several dimensions. in particular, for classification and clustering tasks, we showed that even drastic dimensionality reduction to just a few dimensions still preserved sufficient performance. through analyses using intrinsic dimensionality and isoscore, we found that prompt-based text embedding models generate embeddings with varying degrees of redundancy depending on the prompt. specifically, for classification and clustering tasks, embeddings exhibit lower intrinsic dimensionality and tend to be less isotropic, and that is, they have higher redundancy. in contrast, for tasks like retrieval and sts, where fine-grained similarity is critical, embeddings tend to have higher intrinsic dimensionality and are more isotropically distributed, and that is, they have lower redundancy. in future work, developing methods to construct embeddings with properties suitable for each task would be beneficial. specifically, in contrastive learning, the temperature parameter is known to influence isotropy, with lower temperatures leading to more isotropic embeddings and higher temperatures resulting in less isotropic ones @xcite . additionally, exploring more effective dimensionality reduction techniques for text embeddings remains an important direction. embeddings may contain certain crucial dimensions, and if these dimensions can be identified, it may enable more efficient dimensionality reduction.