Lost in Embeddings: Information Loss in Vision–Language Models (2025)

Abstract: Vision–language models (VLMs) often process visual inputs through a pretrained vision encoder, followed by a projection into the language model’s embedding space via a connector component. While crucial for modality fusion, the potential information loss induced by this projection step and its direct impact on model capabilities remain understudied. We introduce two complementary approaches to examine and quantify this loss by analyzing the latent representation space. First, we evaluate semantic information preservation by analyzing changes in k-nearest neighbor relationships between image representations, before and after projection. Second, we directly measure information loss by reconstructing visual embeddings from the projected representation, localizing loss at an image patch level. Experiments reveal that connectors substantially distort the local geometry of visual representations, with k-nearest neighbors diverging by 40–60% post-projection, correlating with degradation in retrieval performance. The patch-level embedding reconstruction provides interpretable insights for model behavior on visually grounded question-answering tasks, finding that areas of high information loss reliably predict instances where models struggle.

## introduction vision-language models (vlms) have advanced on many tasks, e.g., visual question answering and image captioning by combining pretrained vision encoders with pre-trained language models. many of these models employ a small neural network module, known as a connector (or a projector), to bridge the gap between the visual and textual representation spaces. the connectors project visual representations into sequences of embeddings that a language model can process (chen et al., 2024a; liu et al., 2023; deitke et al., 2024; laurençon et al., 2024; chen et al., 2024b; zhang et al., 1 code: https://github.com/lyan62/vlm-info-loss . while these connector modules enable efficient cross-modal integration @xcite , projecting rich visual features into embeddings compatible with language models typically involves dimensional conversion and representation restructuring. naturally, this raises questions about potential information loss foot_0 during projection, and how such loss impacts downstream task performance. as shown in figure 1 , the loss of critical visual details most relevant to answering the question imposes inherent limitations on the reasoning capabilities, since the language model's performance is constrained by the quality and completeness of the visual information it receives. despite the growing research on vlm connector architectures and their impact on downstream performance @xcite , there has been limited investigation into how well they preserve visual information in the latent space. quantifying this information loss presents substantial challenges; traditional methods like canonical correlation analysis @xcite struggle with variable-length high-dimensional visual features processed through diverse connector architectures in vision-language models. performance degradation can also take more than one form, adding to the complexity of its study. for instance, it can take the form of a direct information loss due to an inherently lossy connector, or a geometric collapse where distinct features become entangled in the projected embedding space. figure 1b display the difference between the l 2 norm of the original and the reconstructed patch embeddings. blue regions indicate where original embeddings have larger norms than predicted embeddings, while red regions show where predicted embeddings have larger norms. the top 10 high-loss patches are marked by yellow squares. figure 1c shows high loss occurring in several answer-relevant patches contribute to the incorrect prediction. to bridge this gap in the literature, we present an evaluation framework to quantify information loss in vlm connectors from both the geometric perspective and that of localized information loss. we first examine if the connector projection changes the geometric structure of latent visual representations. by introducing k-nearest neighbors overlap ratio, we measure how much the neighborhoods of image embeddings change before and after the projection in the latent representation space, thereby estimating how well geometric and semantic relationships are preserved. 3 second, we quantify localized information loss by training a model to reconstruct the original visual embeddings from the projected embeddings. this patch-level visual embedding reconstruction allows us to pinpoint the high-loss regions in the image-areas where visual features are hard to recover after projection (figure 1 ). this two-step approach provides both quantitative analysis and interpretable visualizations, offering insights into the nature of information transformation during vision-text integration. ## vlms and connectors integrating visual and textual inputs is fundamental for vlms to process multimodal information effectively. existing vlms typically employ two main approaches @xcite : models like llama3.2 (gra, 2024) and blip @xcite leverage cross-modal attention mechanisms, while others such as llava @xcite and qwen-3 here, the semantic relationship denotes the semantic similarity between pairwise embeddings measured by l 2 distance or their inner product similarity. ## quantifying information loss we propose two methods for quantifying information loss over the projection step described above. the first method quantifies the preservation of structural information in semantic embeddings by comparing each image representation's k-nearest neighbors (k-nn, @xcite ) before and after projection. the overlap ratio of the k-nn neighbors captures how well local geometry of the semantic embeddings are preserved in the latent space. figure 2 gives an example where two of the three nearest neighbors overlap before and after projection. the second method evaluates patch-level representation (figure 1 ) distortion by training an ad hoc neural network to reconstruct the original image embedding from its projected representation, detailed in section 3.2. ## experimental setup we quantify information loss using both methods across three open-weights connector-based visionlanguage models on six datasets spanning question answering, captioning, and retrieval tasks. we assume that greater structural and semantic information loss during projection through the connector leads to reduced neighborhood overlap, while greater patch-wise information loss results in higher reconstruction error. ## neighbor rankings and structural information are not preserved we calculate knor (section 3.1) for images in the seedbench validation set, a subset of the vqav2 validation set with 10, 000 images, and the validation set of vizwiz grounding vqa dataset. it is intuitive that higher neighborhood overlap ratios suggest that the projection better preserves the structural relationships between image embeddings. as the neighborhood rankings directly impact image retrieval tasks, we also evaluate retrieval performance on the cub dataset using both pre-and post-connector visual embeddings. ## reconstruction and model behavior while the neighborhood overlap ratio reflects structural information loss in the semantic representation space, we further examine the information loss at the image patch level. specifically, as in equation 8 , we reconstruct patch-level visual representation ψ(x) of an image from its projected counterpart @xcite ). higher reconstruction loss indicates greater difficulty in recovering the features that are captured in the original visual embeddings. this patch-level comparison between original and reconstructed embeddings enables us to precisely quantify and locate the visual information loss at a more fine-grained level. ## analysis procrustes analysis we also attempt to find the optimal geometrical transformation from the postprojection embedding space to the pre-projection one through procrustes analysis @xcite -a method often used for supervised alignment of embeddings @xcite . the alignment error reflects the degree of structural similarity of the two embedding spaces. we use mean-pooled image embeddings from llava, idefics2, and qwen2.5-vl. as the pre-and post-projection embeddings have different embedding dimensions and sequence lengths, our analysis follows three steps to complete the embedding alignment. we first take the mean-pooled image representation by averaging over the sequence length, producing fixed-size vectors of size d ′ and d. we then use pca @xcite on the mean-pooled post-projection embeddings to project them to the same dimension of the mean-pooled pre-projection embeddings. orthogonal transformation matrix r was derived through singular value decomposition of the cross-covariance matrix x⊤ t , where @xmath0. then the orthogonal transformation matrix is learned to best align these two sets of embeddings by minimizing the euclidean distance. the reconstruction error are reported in figure 6 visualizes the alignment of llava embeddings through procrustes analysis. our analysis reveals fundamental limitations in linear alignment of the image embeddings. the high alignment errors of 16.62 for llava and 4.41 for qwen2.5-vl indicate the inherent difficulty of preserving geometric relationships through rigid transformations. while serving as a critical baseline for structural fidelity assessment, this constrained linear approach explains why our proposed non-linear embedding reconstruction approach achieves significantly lower errors. ## related work a series of analyses has been conducted to investigate the modality gap and representation limitations of contrastive-based vlms @xcite @xcite . these studies reveal that the representational shortcomings in clip embeddings subsequently impact the visual perception capabilities of vlms relying on such vision encoders. for connector-based vlms, @xcite demonstrates that the latent space sufficiently retains the information necessary for classification through probing across different layers, and @xcite demonstrates the impact of different connectors on vlms' downstream performance. however, there remains a significant gap in understanding whether fine-grained visual ## conclusion and future work our study provides a systematic evaluation of how connectors in existing vlms transform information and reshape the representation space when projecting visual embeddings into the language embedding space. through neighborhood overlap ratios and embedding reconstruction, we establish a quantitative framework that captures two critical aspects of the information loss: 1) structural shift of global semantic relationships shown by the 40-60% divergence in nearest-neighbor rankings and 2) patch-level reconstruction loss negatively impacts captioning performance and explains model failures in fine-grained visually grounded question answering. the patch-level reconstruction also enables visualization of local information loss, offering interpretable explanations for model behaviors. our findings suggest two key properties of an effective connector: preserving or improving semantic representation of images and preserving visual information most relevant to the text context. these findings could guide further improvements in vlm connectors. for example, the reconstruction loss at the embedding level could potentially be incorporated during model pretraining as regularization. future work could also explore designing dynamic projection layers or better visual feature selection mechanisms for modality fusion. • people in navy uniforms and one person talking on a walkie-talkie.