Towards Storage-Efficient Visual Document Retrieval: An Empirical Study on Reducing Patch-Level Embeddings (2025)

Abstract: Despite the strong performance of ColPali/ColQwen2 in Visualized Document Retrieval (VDR), its patch-level embedding approach leads to excessive memory usage. This empirical study investigates methods to reduce patch embeddings per page while minimizing performance degradation. We evaluate two token-reduction strategies: token pruning and token merging . Regarding token pruning, we surprisingly observe that a simple random strategy outperforms other sophisticated pruning methods, though still far from satisfactory. Further analysis reveals that pruning is inherently unsuitable for VDR as it requires removing certain page embeddings without query-specific information. Turning to token merging (more suitable for VDR), we search for the optimal combinations of merging strategy across three dimensions and develops Light-ColPali/ColQwen2. It maintains 98.2% of retrieval performance with only 11.8% of original memory usage, and preserves 94.6% effectiveness at 2% memory footprint. We expect our empirical findings and resulting Light-ColPali/ColQwen2 offer valuable insights and establish a competitive baseline for future efficient-VDR research.

## introduction visualized document retrieval (vdr) matches queries to relevant documents by leveraging their visual representations. unlike conventional retrieval systems, where raw text must be parsed before indexing, vdr captures documents as images (screenshots) and encodes them into embeddings using large vision-language models (lvlms). this approach preserves layout structures and visual elements, enabling retrieval in a what-you-seeis-what-you-get manner. as a result, vdr achieves superior retrieval accuracy and demonstrates strong ## related work visualized document retriever shares a similar architecture with text-based dense retrievers @xcite , but leverages lvlms @xcite generate multiple patch-level embeddings per page and token-level embeddings per query. while patch-level retrievers offer finer granularity and superior performance, they demand prohibitive computational resources for both offline indexing and online retrieval. this work addresses this limitation by developing methods to reduce embedding numbers in patch-level retrievers. token-reduction about lvlm has been intensively explored to improve lvlms' generation (i.e., next-token prediction) efficiency. current approaches fall into three categories: (1) pruning strategies @xcite @xcite eliminate low-information tokens based on importance ranking; (2) merging strategies @xcite @xcite combine similar tokens into compressed embeddings; (3) hybrid strategies @xcite @xcite integrate pruning and merging, usually by preserving high-informative tokens while merging redundant ones. in lvlm generation, the focus is on minimizing response latency and flops given specific instructions. in document retrieval, the goal is to reduce the memory footprint of embeddings, without query access but with less concern for latency or flops during index-ing. these distinctions relax constraints on token merging, enabling late-stage compression and more computationally-intensive merging strategies. however, the absence of queries precludes queryconditioned pruning or merging approaches. lightweight document retriever has been explored to address the challenge of large-scale embeddings with two orthogonal approaches: (1) dimension reduction. colbertv2 @xcite employs product quantization @xcite to reduce the size of each embedding from 768 to 128 dimensions. this design is inherited by @xcite with a simpler projection layer. (2) token reduction: @xcite introduces the concept of to-kenpooling and explores merging strategies for text-based retrievers. a recent blog by colpali's author @xcite further extends this to visualized document retrievers. following their work, our light-colpali/colqwen2 shares very similar merging approaches from the posterior perspectives. however, our work advances this field by conducting a systematic empirical study both on pruning and merging strategies. beyond simply proposing a merging strategy, our analysis reveals the limitations of pruning (under retrieval settings) and identifies the optimal combination for merging. moreover, our experiments demonstrate the effectiveness of fine-tuning. compared to the results reported in @xcite , our fine-tuned light-colpali/colqwen2 presents stronger performance with significantly higher reduction ratios. ## the research problem colpali/colqwen2. we briefly review the mechanism of colpali/colqwen2 @xcite in figure 1 . given query q with n q tokens and imageformatted document p with n p patches, colpali/colqwen2 encodes them as token-level embed- ## token pruning: an ineffective strategy given patch embeddings e p for each document page, a natural approach is to retain n ′ p embeddings and prune the remaining (n p -n ′ p ). in this section, we explore three pruning strategies and observe that their performance collapses when reducing embeddings by orders of magnitude. more embarrassingly, the simplest random pruning outperforms other carefully-designed strategies. further analysis reveals that colpali's embeddings cluster in groups, while their relevance to different queries is highly unpredictable. these findings highlight the limitations of pruning strategies and underscore the feasibility and necessity of merging strategies under vdr settings. ## token merging: the choices we turn into another token-reduction strategy, merging, towards an efficient document visual retriever. unlike pruning which directly drops some tokens, merging consolidates the multiple embeddings into one. this approach is particularly suitable for vdr, where the importance of each embedding is highly undetermined (if not conditioned on specific query). we systematically evaluate the merging astrategy through three key aspects towards the recipe for the optimal merging strategy as detailed below. ## conclusion this work conducts an empirical study into developing efficient visualized document retrievers with minimal memory footprints and performance loss. through comprehensive experiments, we demonstrate the superior suitability of merging for vdr tasks. our proposed light-colpali/colqwen2, a simple yet effective merging strategy, achieves significant memory reduction while maintaining promising performance. these findings and the established baseline provide valuable insights for advancing efficient vdr research.