MIR e: Enhancing Multimodal Queries Representation via Fusion-Free Modality Interaction for Multimodal Retrieval (2025)

Abstract: Recent multimodal retrieval methods have endowed text-based retrievers with multimodal capabilities by utilizing pre-training strategies for visual-text alignment. They often directly fuse the two modalities for cross-reference during the alignment to understand multimodal queries. However, existing methods often overlook crucial visual information due to a text-dominant issue, which overly depends on text-driven signals. In this paper, we introduce MIRe, a retrieval framework that achieves modality interaction without fusing textual features during the alignment. Our method allows the textual query to attend to visual embeddings while not feeding text-driven signals back into the visual representations. Additionally, we construct a pre-training dataset for multimodal query retrieval by transforming concise question-answer pairs into extended passages. Our experiments demonstrate that our pre-training strategy significantly enhances the understanding of multimodal queries, resulting in strong performance across four multimodal retrieval benchmarks under zero-shot settings. Moreover, our ablation studies and analyses explicitly verify the effectiveness of our framework in mitigating the text-dominant issue. Our code is publicly available: https://github.com/yeongjoonJu/MIRe

## introduction information retrieval aims to fetch relevant information from a large collection given a user query, underpinning numerous nlp tasks such as search engines, open-domain question answering @xcite , and factchecking @xcite . beyond conventional methods based on lexical similarities (e.g., tf-idf and bm25 @xcite ), embedding-based retrieval methods @xcite @xcite have achieved rich semantic matching by learning high-dimensional representations of queries and passages via large-scale pretraining. however, they focus on textual queries, struggling to address multimodal queries that encompass both textual and visual information. what kind of object is being advertised in the poster on the left? in real-world scenarios, users often include visual references in their queries (e.g., complex objects or named entities depicted in an image), which are difficult to represent by text alone fully @xcite . recent multimodal retrieval methods @xcite @xcite have endowed textbased retrievers with multimodal capabilities by utilizing pre-training strategies for visual-text alignment. most existing methods directly fuse the two modalities for cross-reference during visual-text alignment to enhance the understanding of multimodal queries. for instance, @xcite and @xcite facilitate modality interaction through early token fusion, where visual representations are prepended before passing through selfattention layers in the query encoder. similarly, @xcite integrate modalities within the multimodal query using a cross-attention mechanism, where the textual query embeddings function as keys and values. however, these methods often overlook crucial visual information due to a text-dominant issue induced by excessive reliance on text-driven signals during the alignment stage. in this stage, the retriever over-relies on textual similarities, thereby hindering proper visual alignment. consequently, the retriever assigns high scores to irrelevant passages when textual cues are ambiguous. fig. 1 shows the effect of the text-dominant issue. the baseline, which is trained with a direct fusion of textual features, fails to retrieve the desired passages due to its excessive reliance on text when the textual query becomes partially ambiguous (e.g., replacing a specific term like 'beverage' with a more generic word like 'object'). this text-dominant issue is further amplified through pre-training datasets constructed such that pseudoqueries are extracted from passages @xcite . datasets obtained via this approach contain text-based queries that alone are sufficient to match relevant passages. this hinders visual-text alignment by relying on the high contextual similarity between textual queries and passages, even in the absence of visual information. this issue highlights the need for a retrieval framework that leverages multimodal queries by mapping both visual and textual cues into a linguistic space, capturing complementary interactions between these modalities without over-relying on textual features alone. to address these issues, we introduce mire, a retrieval framework that achieves modality interaction without fusing textual features during the alignment stage. instead of directly merging both modalities, mire allows the textual query to attend to patch-level visual embeddings without feeding text-driven signals back into the visual representations. we then fuse the two modalities during the relevance scoring stage based on a late-interaction mechanism @xcite . this design alleviates the dependency on text-driven signals in the context of knowledge retrieval using a multimodal query. furthermore, we construct a pre-training dataset by transforming multimodal query-response pairs into extensive passages via our response-to-passage conversion process that utilizes solely a text retrieval model. the constructed dataset requires the integration of both modalities to match a desired passage during training, enabling the model to link image understanding with complex textual queries. our experiments demonstrate that our pre-training strategy significantly enhances multimodal query understanding for knowledge retrieval, resulting in strong performance across four multimodal retrieval benchmarks under zero-shot settings. ## related work traditional methods such as tf-idf and bm25 @xcite rely on keyword matching to retrieve relevant content but often fail to capture the deeper semantics underlying queries and documents. beyond the surface-level lexical similarities, dense retrieval methods @xcite @xcite @xcite leverage high-dimensional embedding models for richer semantic matching. the transition from traditional text queries to multimodal queries has marked a significant evolution in information retrieval @xcite . early methods focused on converting images into textual representations, such as captions @xcite and object tags @xcite . enfore @xcite and dedr @xcite improve image-query representations derived from a multimodal encoder with generated entities and captions, respectively. the oven dataset @xcite has also provided insights into multimodal entity recognition. whereas most of these approaches utilize dpr @xcite based on a single embedding for retrieval, flmr @xcite refines multimodal queries by incorporating rois and generated captions with the late-interaction mechanism. reviz @xcite represents an end-to-end multimodal retrieval system that removes the dependency on intermediate modules by pre-training on the vl-ict, which automatically constructs a pre-training dataset by applying the inverse cloze task (ict) @xcite to a multimodal knowledge base. uniir @xcite proposes an instruction-guided multimodal retriever along with its benchmark. they design two variants of the model architecture for modality interaction: score-level fusion and featurelevel fusion based on clip and blip @xcite . vista @xcite introduces an in-depth fusion strategy by prepending visual tokens to the input of a text retrieval model to enhance multimodal understanding. preflmr @xcite extends flmr to investigate the scalability of multimodal retrievers under the late-interaction mechanism. in contrast to previous methods that rely heavily on text information within multimodal queries, we address the text-dominant issue in multimodal query representations caused by the direct intervention of textual features. we also adopt the late-interaction mechanism to fuse modalities during the scoring stage. ## method in this section, we first define the problem of knowledge retrieval with multimodal queries. next, we describe the architecture of our retrieval model and our data construction method. ## conclusion we introduced mire, a novel retrieval framework designed for multimodal query retrieval without fusing textual features during the alignment stage. our query-guided attentive pooling module allows textual embeddings to attend to visual patches while preventing text-driven signals from dominating the visual representations. we also constructed a pre-training dataset by converting concise question-answer pairs into extended passages, thereby exposing the model to more realistic retrieval tasks. our extensive experiments demonstrate that mire consistently outperforms existing methods under both zero-shot and fine-tuned settings. ablation studies further validate that each component of mire is crucial for achieving robust multimodal query retrieval.