D oc MMIR : A Framework for Document Multi-modal Information Retrieval (2025)

Abstract: The rapid advancement of unsupervised representation learning and large-scale pre-trained vision-language models has significantly improved cross-modal retrieval tasks. However, existing multi-modal information retrieval (MMIR) studies lack a comprehensive exploration of document-level retrieval and suffer from the absence of cross-domain datasets at this granularity. To address this limitation, we introduce DocMMIR , a novel multi-modal document retrieval framework designed explicitly to unify diverse document formats and domains—including Wikipedia articles, scientific papers (arXiv), and presentation slides—within a comprehensive retrieval scenario. We construct a large-scale cross-domain multimodal dataset, comprising 450K training, 19.2K validation, and 19.2K test documents, serving as both a benchmark to reveal the shortcomings of existing MMIR models and a training set for further improvement. The dataset systematically integrates textual and visual information. Our comprehensive experimental analysis reveals substantial limitations in current state-of-the-art MLLMs (CLIP, BLIP2, SigLIP-2, ALIGN) when applied to our tasks, with only CLIP (ViT-L/14) demonstrating reasonable zero-shot performance. Through systematic investigation of cross-modal fusion strategies and loss function selection on the CLIP (ViT-L/14) model, we develop an optimised approach that achieves a +31% improvement in MRR@10 metrics from zero-shot baseline to fine-tuned model. Our findings offer crucial insights and practical guidance for future development in unified multimodal document retrieval tasks.

## introduction with the emergence of the large pre-training model, the pre-trained multi-modal retrieval models (i.e., clip @xcite ) and large visual language models (i.e., blip2 @xcite , align @xcite ), trained on vast image-text corpora, have achieved state-of-the-art despite these advances in image-text retrieval, applying mllms to conduct document-level retrieval remains challenging. real-world documents (wikipedia articles, scientific papers, slide decks, etc.) often contain complex page layouts with text, figures, tables, and other formats. off-the-shelf mllms typically encode images or text snippets in isolation and lack awareness of longer document context or multi-page structure. existing studies fail to fully address the complexity of realworld document structures, leading to several critical limitations, as illustrated in to address these gaps, we introduce docm-mir: a unified framework for document-level multi-modal information retrieval across various domains (demonstrated in figure 1 ). we curate a large-scale benchmark with a unified structure that includes documents and queries from diverse sources (wikipedia articles, arxiv papers, and slide presentations), each containing a mixture of modalities. our retrieval model extends a dual-encoder mllm's architecture to operate on entire documents: during training, we optimise a contrastive training objective over query-document pairs by fusing embeddings from images and texts to construct doc embeddings, which are meant to distill information and be a unified representation. this unified model is trained end-to-end on the docm-mir dataset spanning all domains and modalities. experimental results show that fine-tuning models on docmmir improves retrieval performance over corresponding zero-shot baselines. in particular, fine-tuning the clip encoder on our documentlevel tasks yields large gains in mrr@10 across domains. we also find that incorporating multimodal content is critical: purely text-based retrieval falls far short of our multi-modal model, and naive image-only search misses context. our contributions are thus threefold: framework. we design a unified modeling framework for multi-modal retrieval that can accommodate and jointly encode diverse document formats, including academic papers, presentation slides, and wikipedia pages, by leveraging both textual content and associated imagery in each document. ## task definition the docmmir task aims to retrieve semantically relevant documents from a heterogeneous collection spanning wikipedia articles, arxiv papers, and presentation slides-each combining text (e.g., paragraphs, bullet points, section headers) with visual elements (e.g., figures, diagrams). given a text query generated by a large language model and paired with a document, the system must rank all documents and identify the correct one. these domains differ in structure and modality: arxiv emphasizes technical terms and plots, slides prioritize visuals and brevity, while wikipedia blends narrative text with images. this task poses several challenges: (1) cross-domain heterogeneity in layout, density, and style; (2) modality imbalance between text-and image-heavy documents; and (3) fine-grained alignment, where queries may refer to specific figures or terms, requiring joint modeling of both modalities. formally, let @xmath0, d 2 , . . . , d n } denote a cross-domain document collection, where each document d k comprises a set of text segments {t i } n @xmath1. a query q, generated to holistically describe the document's content (e.g., "retrieve a paper explaining transformer-based vision models with ablation studies"), serves as the input. the retrieval system computes a relevance score sim(q, d k ) for each document, producing a ranked list @xmath2, d (2) , . . . , d (n ) ] where higher-ranked documents better satisfy the intent. ## data curation pipeline to enable heterogeneous sources to be standardized into a unified representation, we introduce a domain-specific but modular preprocessing workflow for different data sources. during the data processing phase, we design a multi-modal quality control mechanism combining tokeniser-aware statistics, perplexity scores, and garbledness heuristics, particularly beneficial for noisy ocr content. we also incorporate a human-in-the-loop feedback for the retrieval query generation to raise the standard of annotation. more importantly, our pipeline is extendable: it can be directly adapted to new domains by plugging in domain-specific parsing and applying the same unification logic. ## conclusion docmmir proposes a simple yet effective framework for unifying diverse document formats in cross-domain multi-modal retrieval. experiments show that weighted-sum fusion with bce loss enhances robustness across domains, while multi-domain training balances generalization and specialization-wikipedia-trained models showing superior adaptability. findings reveal that future work could explore layout-aware dynamic fusion, realistic query construction, and domain-balanced training to further improve on the docmmir task.