Do We Really Need All Those Dimensions? An Intrinsic Evaluation Framework for Compressed Embeddings (2025)

Abstract: High-dimensional text embeddings are foundational to modern NLP but costly to store and use. While embedding compression addresses these challenges, selecting the best compression method remains difficult. Existing evaluation methods for compressed embeddings are either expensive or too simplistic. We introduce a comprehensive intrinsic evaluation framework featuring a suite of task-agnostic metrics that together provide a reliable proxy for downstream performance. A key contribution is EOS _k , a novel spectral fidelity measure specifically designed to be robust to embedding anisotropy. Through extensive experiments on diverse embeddings across four downstream tasks, we demonstrate that our intrinsic metrics reliably predict extrinsic performance and reveal how different embedding architectures depend on distinct geometric properties. Our framework provides a practical, efficient, and interpretable alternative to standard evaluations for compressed embeddings.

## introduction word, sentence, and, more generally, text embeddings 2 have become central to natural language processing (nlp), enabling a range of tasks from semantic search to classification and clustering @xcite @xcite . as embedding models have evolved from static embeddings (e.g., glove @xcite ) to contextualised ones (e.g., bert @xcite ) and more recently, large language model (llm)-based (e.g., e5 @xcite ), the dimensionality and complexity of these embeddings have increased significantly. although higher-dimensional 1 the framework and eos k implementation are available at https://github.com/nathaninkiriwang/ textembedcompress . to address these challenges, dimensionality reduction (dr) and quantisation have been increasingly adopted to compress embeddings @xcite @xcite @xcite @xcite @xcite . this trend is motivated, in part, by the finding that many embedding models possess an inherently low intrinsic dimensionality @xcite . this property indicates significant redundancy, which compression foot_0 methods can exploit to substantially reduce the computational burden while preserving, or even improving, downstream performance @xcite . however, despite growing adoption for compressing embeddings, significant gaps remain in both the theoretical understanding and systematic empirical evaluation of these methods. the evaluation of embedding compression has largely relied on two limited practices: (i) using extrinsic downstream performance metrics (e.g., accuracy or retrieval scores) @xcite @xcite @xcite ; and (ii) relying on a single intrinsic metric @xcite . neither offers a complete or reliable picture of embedding quality. extrinsic evaluations are computationally de-manding, given the large combinatorial space of models, tasks, and compression techniques, and are highly sensitive to dataset and configuration choices (e.g., classifier design, retrieval settings). more importantly, they provide limited insight. performance scores do not show which structural properties are preserved or lost. this results in a fragmented and opaque understanding of compression, especially across diverse embedding types @xcite @xcite . intrinsic evaluations based on a single metric are similarly limited @xcite . such approaches fail to generalise across embedding architectures and thus offer a limited view that restricts practical applicability, especially when compression methods behave inconsistently across tasks. to address these limitations, we propose a comprehensive and scalable evaluation framework for compressed embeddings 4 . our framework includes a set of theoretically grounded intrinsic metrics that are task-agnostic, and, crucially, provide a consistently robust proxy for overall downstream utility. these metrics are motivated by key goals in embedding design, preserving local neighbourhood structure @xcite , retaining global topology @xcite and maintaining information fidelity @xcite , and aim to capture distinct geometric and statistical properties that affect downstream performance. we also introduce eos k , a novel spectral fidelity metric designed to better measure semantic preservation. unlike traditional metrics that focus on the entire eigenspectrum, eos k specifically analyses the residual eigenspace after removing the top-k principal components. these top components often capture broad, anisotropic variance that can overshadow more subtle, task-relevant information. we apply our framework to three widely used open-source embeddings, glove @xcite (static), bert @xcite (contextual), and e5 @xcite (contrastive), which vary in architectures, training objectives, and anisotropy levels. through extensive correlation analysis on four downstream tasks across 21 datasets from the mteb benchmark @xcite , we find consistent patterns linking intrinsic properties with downstream performance: contextual embeddings benefit most from 4 related work is given in appendix a. local structures, while static and contrastive embeddings align better with global and spectral fidelity. our framework provides a practical guide for selecting compression methods based on embedding type and downstream tasks. by measuring key intrinsic metrics: local, global, and spectral structure, practitioners can determine which properties are most important for their task. this enables more informed decisions when selecting compression methods, allowing them to balance compression ratios with the preservation of structurally important features for their applications. using our evaluation framework, we identify random projection and int8 quantisation as consistently effective compression strategies. this benchmarked approach will allow users to compress embeddings effectively while maintaining task-relevant performance and avoiding exhaustive benchmarking. in addition, our novel eos k metric outperforms standard spectral metrics in scenarios with anisotropic embeddings, enabling more reliable intrinsic evaluation of structure-preserving quality under varying model architectures. ## compression methods preliminaries and notation let @xmath0, where n is the number of samples (e.g., words, sentences or documents) and d is the original embedding dimension. each row x @xmath1. the objective of embedding compression is to transform x into a representation that requires less storage and/or computational resources, while preserving its utility. we focus on two classes of operations: dimensionality reduction (dr) and quantisation (q). ## evaluation framework to comprehensively evaluate the effectiveness of compression methods, we propose a unified evaluation framework that captures both the structural and spectral fidelity of compressed representations. given original embeddings @xmath2, we evaluate how well the low-dimensional space preserves the geometric and informational properties of original embedding space. ## eos k while the spectral metrics (section 3.3) offer valuable insights, their effectiveness can be compromised in the context of modern, anisotropic, text embeddings. in such cases, the metric may overstate the quality of preservation by capturing alignment in high-variance directions that lack meaningful semantic content, thereby obscuring the degra-dation of more subtle, task-relevant structures. ## experimental setup we evaluate the proposed framework by performing a correlation analysis using spearman's rank correlation coefficient to examine how well intrinsic evaluation metrics align with performance on extrinsic tasks, consistent with prior intrinsic evaluation studies @xcite . high correlation indicates that intrinsic metrics align well with downstream task performance and can therefore serve as reliable and cost-efficient proxies for evaluating embedding compression quality. for each dataset and embedding family, the experimental pipeline proceeds as follows: (i) generate original sentence embeddings; (ii) apply a range of compression techniques; (iii) compute the proposed intrinsic scores; and (iv) evaluate the compressed embeddings on standard downstream tasks. the intrinsic and extrinsic outcomes are subsequently correlated to assess alignment. figure 2 provides a schematic overview of this pipeline, from embedding generation through compression, evaluation, and correlation analysis. beyond correlation, the framework analyses how compression perturbs embedding classes, identify-ing which structural properties-local, global, or spectral-are most critical to preserve. ## results ## conclusion we introduced a scalable and interpretable intrinsic evaluation framework for compressed text embeddings over dr methods, combining local, global, and spectral fidelity metrics. we introduce a novel metric, eos k , that captures meaningful information beyond dominant principal components. we validate our framework using three embeddings across four tasks and 21 datasets. experi-ments revealed that our framework robustly predicts downstream task performance across diverse downstream tasks. key findings highlight that optimal compression strategies are model-dependent: contextual embeddings benefit most from preserving local neighbourhood structures, while static and contrastive embeddings show stronger alignment with global and spectral fidelity. notably, eos k revealed the importance of retaining information beyond dominant principal components, showing significant correlations for bert and e5, particularly in tasks like sts. our analysis also identified random projections as a highly efficient and effective dimensionality reduction technique, and we recommend the routine application of int8 quantisation for further compression with minimal performance loss. ultimately, this work provides a principled and interpretable framework, empowering more efficient and informed development of compressed embedding solutions.