MARVEL : Unlocking the Multi-Modal Capability of Dense Retrieval via Visual Module Plugin (2024)

Abstract: This paper proposes Multi-modAl Retrieval model via Visual modulE pLugin (MARVEL), which learns an embedding space for queries and multi-modal documents to conduct retrieval. MARVEL encodes queries and multi-modal documents with a unified encoder model, which helps to alleviate the modality gap between images and texts. Specifically, we enable the image understanding ability of the well-trained dense retriever, T5-ANCE, by incorporating the visual module’s encoded image features as its inputs. To facilitate the multi-modal retrieval tasks, we build the ClueWeb22-MM dataset based on the ClueWeb22 dataset, which regards anchor texts as queries, and extracts the related text and image documents from anchor-linked web pages. Our experiments show that MARVEL significantly outperforms the state-of-the-art methods on the multi-modal retrieval dataset WebQA and ClueWeb22-MM. MARVEL provides an opportunity to broaden the advantages of text retrieval to the multi-modal scenario. Besides, we also illustrate that the language model has the ability to extract image semantics and partly map the image features to the input word embedding space. All codes are available at https://github.com/OpenMatch/MARVEL.

## introduction with the growth of multimedia information on the internet, search engines tend to return multi-modal retrieval results to better satisfy the user information need @xcite . the media information provides more vivid retrieval results, such as texts, images, videos, and more, which improves users' experiences and even changes their browsing behaviors. multi-modal retrieval @xcite @xcite figure 1 : retrieval pipeline with our marvel model. marvel incorporates the visual module plugin, aiming to unlock the multi-modal capabilities of well trained dense retrieval model. in this paper, we propose multi-modal retrieval model via visual module plugin (marvel). as shown in figure 1 , marvel is based on the text retriever t5-ance @xcite , regards the visual module as a plugin and pretrains the visual module with image-caption contrastive training for adaption. by incorporating a visual module into well-trained text retriever t5-ance, marvel seizes the opportunity to extend the benefits of unimodal learning to the multi-modal retrieval task. to facilitate the multi-modal retrieval task, we build a large-scale benchmark, clueweb22-mm, based on the web page dataset, clueweb22 @xcite . following previous work in text retrieval @xcite , we regard the anchor text as a query and assume that its linked web page is related to the query. subsequently, we extract image and text documents from these anchor-linked web pages. after processing, the clueweb22-mm encompasses over 90k queries, maintaining a scale comparable to existing benchmark webqa @xcite . previous work @xcite demonstrates that the high-quality training signals from anchor-document pairs contribute to developing a state-of-the-art dense retrieval model. our experiments show that marvel outperforms all baseline models, achieving improvements of over 2% and 7%, in the main metric mrr, on webqa @xcite and clueweb22-mm, respectively. the evaluation results indicate the effectiveness of marvel comes from the visual module plugin architecture, the visual module pretraining method, and the text matching knowledge learned by t5-ance. our further analyses illustrate that the image representations encoded by the visual module can be easily captured by only finetuning the language model parameters. the training strategies guide the language model to assign more appropriate attention weights to image and text features, preventing the visual module from overfitting to the training signals. these encoded image representations not only inhabit the input embedding space for semantics alignment but also function as a kind of prompt. unlike the text retrieval task, the multi-modal retrieval task @xcite @xcite aims to provide users with multi-modal documents that satisfy their information needs. earlier work primarily focuses on building a divide-and-conquer pipeline for multi-modal retrieval @xcite @xcite . in these models, retrievers individually search candidates from the document collections of different modalities and then use a reranking model to fuse the retrieval results, such as vision-language models @xcite . however, this approach usually struggles to fuse the retrieval results across different modalities @xcite . univl-dr @xcite builds a universal multi-modal dense retrieval model. it encodes queries and multi-modal documents as embeddings and conducts retrieval, modality routing, and result fusion within a unified embedding space. representing images is also the core of multimodal retrieval, aiming to alleviate the modality gap between images and texts. existing work usually focuses on representing the images using captions and image features @xcite with different encoding methods. bert-style pretrained visual-language models @xcite @xcite @xcite @xcite @xcite provide an opportunity to model the captions and image features using the same model. however, these visual-language models typically aim to align the semantics between image features and captions instead of learning representations for image documents. thus they show less effectiveness in learning an embedding space for multi-modal retrieval @xcite . another way to facilitate the image document representations is using the visual-language models that focus on representation learning, such as clip @xcite . it encodes image features and texts using different encoders. however, these approaches often only provide shallow interactions between texts and visual features. thus, existing models @xcite pay more attention to alleviating the modality gap between texts and images by the image verbalization method, aiming to bridge the modality gap between images and texts in the raw text space. recent advancements in multi-modal large language models @xcite have introduced a novel approach to modeling multi-modality features. this approach incorporates a visual encoder module into large language models through a transformation layer @xcite @xcite . these models extract image features using the visual encoder module of clip and then optimize the prompt tokens and transformation layer to map the encoded image embeddings to the raw input space of large language models @xcite . such a visual encoder plugin method presents a unified modeling approach for handling image and text features. it not only enables the visual comprehension ability of large language models but also preserves their effectiveness by freezing their parameters. ## multi-modal retrieval model via visual module plugin (marvel) in this section, we first describe the multi-modal retrieval (sec. 3.1) and then introduce the model architecture of marvel (sec. 3.2). ## experimental methodology this section describes datasets, evaluation metrics, baselines and implementation details. dataset. during pretraining, we collect the image-caption pairs from clueweb22 @xcite to train the visual understanding module. more details of pretraining data are shown in appendix a.2. then two multi-modal retrieval datasets, webqa and clueweb22-mm, are used for finetuning and evaluation. the data statistics are shown in webqa is a multi-hop, multi-modal, opendomain question answering benchmark @xcite . the dataset contains images and passage snippets that are crawled from the general web and wikipedia. we follow previous work @xcite to keep the same experimental settings to preprocess the dataset. besides, we build a new multi-modal retrieval dataset, clueweb22-mm, based on clueweb22 @xcite , which provides 10 billion web pages with rich information. we only retain web pages in english and build the clueweb22-mm dataset. we establish query-document relations by pairing anchors with their corresponding document @xcite . and then we regard the anchor texts as queries and extract image documents and text documents from the linked documents. more details of building the clueweb22-mm dataset are shown in appendix a.4. evaluation metrics. we use ndcg@10, mrr@10 and recall@100 as evaluation metrics. following previous work @xcite , we regard mrr@10 as our main evaluation. mrr and ndcg are computed using the official scripts foot_0 . statistic significances are tested by permutation test with @xmath0.05. baselines. in our experiments, we follow previous work @xcite to conduct baseline models and divide them into three groups: single modality retrieval, divide-and-conquer, and universal dense retrieval models. single modality retrieval. in our experiments, we represent image documents using captions and use several text retrieval models as baselines. bm25 @xcite is widely used in text retrieval work, which conducts exact matches between queries and documents. dpr @xcite ) is trained using nq dataset @xcite and uses a dual-encoder to encode queries and documents as dense vectors for retrieval. we start from vanilla bert @xcite and dpr @xcite checkpoints and train the encoders using in-batch negatives to conduct bert-dpr and nq-dpr models. nq-ance is also compared, which continuously trains nq-dpr using hard negatives @xcite . besides, t5-ance @xcite and anchor-dr @xcite are dense retrieval models that are trained on ms marco and clueweb22, respectively. divide-and-conquer. the divide-and-conquer models retrieve image documents and text documents individually and then fuse the retrieval re-sults. following previous work @xcite , we use single modality retrievers, vinvl-dpr, clip-dpr and bm25, and fuse the retrieval results according to their unimodal rank reciprocals. universal dense retrieval. clip-dpr and vinvl-dpr employ the visual language models clip @xcite and vinvl @xcite as image and text encoders and then are trained with in-batch negatives. univl-dr @xcite is our main baseline model, which further uses modality-balanced hard negative to train text and image encoders and also utilizes the image verbalization method to bridge the modality gap between images and texts. ## evaluation result in this section, we first evaluate the performance of marvel and conduct ablation studies. then, we explore the effectiveness of different visual and language model fusion methods and analyze the role of visual module adaption pretraining in marvel. some case studies are shown in appendix a.8. ## conclusion this paper proposes multi-modal retrieval via visual module plugin (marvel). marvel integrates a visual plugin module with a well-trained dense retriever and pretrains the visual module with image-caption contrastive training for adaption. our marvel model achieves state-of-theart on all benchmarks by unifying the multi-modal document encoding and alleviating the modality gap between images and texts.