BRIT : Bidirectional Retrieval over Unified Image-Text Graph (2025)

Abstract: Retrieval-Augmented Generation (RAG) has emerged as a promising technique to enhance the quality and relevance of responses generated by large language models. While recent advancements have mainly focused on improving RAG for text-based queries, RAG on multi-modal documents containing both texts and images has not been fully explored. Especially when fine-tuning does not work. This paper proposes BRIT, a novel multi-modal RAG framework that effectively unifies various text-image connections in the document into a multi-modal graph and retrieves the texts and images as a query-specific sub-graph. By traversing both image-to-text and text-to-image paths in the graph, BRIT retrieve not only directly query-relevant images and texts but also further relevant contents to answering complex cross-modal multi-hop questions. To evaluate the effectiveness of BRIT, we introduce MM-RAG test set specifically designed for multi-modal question answering tasks that require to understand the text-image relations. Our comprehensive experiments demonstrate the superiority of BRIT, highlighting its ability to handle cross-modal questions on the multi-modal documents.

## introduction retrieval-augmented generation (rag) @xcite has emerged as a promising approach for enhancing large language models (llms) by grounding their responses in external knowledge. this technique retrieves relevant information from external sources, such as proprietary company documents, and provides it to the llm as context for generating more informed and accurate responses. while recent efforts @xcite @xcite have largely focused on improving textual rag, effectively incorporating visual information remains a challenge. this capability is crucial for comprehensive understanding of documents like company brochures, websites, and presentations, where visual content plays a vital role in conveying information. multi-modal rag (mm-rag) aims to address this challenge by retrieving both relevant texts and images for response generation. however, mm-rag faces difficulties in effectively aligning visual content with textual queries. a common approach (e.g., fig. 1 (a) ) relies on embedding-based similarity @xcite between the query and the images. this approach often fails in enterprise settings where queries contain companyspecific terms (e.g., product or person names) that are absent in the training data of pre-trained embedding models. for example, answering the question "does the codename brit have a logo on the top?" requires retrieving an image of brit. however, standard embedding models may not effectively associate the codename with its corresponding image based solely on similarity. recent works @xcite have explored retrieving relevant pages by treating each page as an image and computing its similarity to the query (fig. 1 (b) ). this page-wise retrieval struggles when relevant contents are spanned across multiple pages, as it relies on the similarity between the entire page and the query. furthermore, these approaches lose crucial text-image associations when the retrieved information is provided as input. the existing methods @xcite @xcite often rely on training or fine-tuning techniques to address these limitations, but these approaches may not be effective in the enterprise settings, particularly when they are frequently updated. thus, other approaches to connect images to their textual descriptions must be explored and evaluated. ## brit: multi-modal graph rag our framework, brit, enables multi-modal retrieval by integrating images and related texts using graphs. the process consists of three key steps: graph generation, bidirectional retrieval, and response generation, as shown in fig. 2 . first, a textual graph is constructed from document texts and images are linked with their corresponding textual nodes, forming a unified multi-modal graph. next, for retrieval we formulate the problem as a prize-collecting steiner tree (pcst) optimization @xcite , which retrieves a queryrelevant sub-graph while incorporating neighborhood information. subsequently, we expand this initial sub-graph by traversing from query-relevant textual nodes to connected images or from queryrelevant images to the connected textual nodes to retrieve the cross-modal context. finally, the retrieved multi-modal context with the input query is fed into an lmm to generate the final response. ## mm-rag test set this section details mm-rag test set, a new test set we have constructed, and the methodology employed for its construction. mm-rag test set is constructed to evaluate rag with on multi-modal documents containing texts and images with complex cross-modal multi-hop questions, which require to understand connections between texts and images. some recent works have proposed multimodal question answering datasets @xcite which require multi-hop reasoning over different modalities, however their questions are not explicitly designed to require traversing different modalities. our mm-rag test set contains three types of questions as shown in fig. 3 . 1) text-image questions are the question which requires a relevant image to answering but that image cannot be directly identified by the question, whereas 2) image-text questions require the relevant texts to answering but texts cannot be directly identified by the question. these text-image and image-text questions are challenging cross-modal multi-hop questions because they require to understand not only texts and images but also their relationships to connect a reasoning path. 3) image-image questions are simple question that serves as a baseline and can be matched with an image and answered based only on that image.