P re FLMR : Scaling Up Fine-Grained Late-Interaction Multi-modal Retrievers (2024)

Abstract: Large Multimodal Models (LMMs) excel in natural language and visual understanding but are challenged by exacting tasks such as Knowledge-based Visual Question Answering (KB-VQA) which involve the retrieval of relevant information from document collections to use in shaping answers to questions. We present an extensive training and evaluation framework, M2KR, for KB-VQA. M2KR contains a collection of vision and language tasks which we have incorporated into a single suite of benchmark tasks for training and evaluating general-purpose multi-modal retrievers. We use M2KR to develop PreFLMR, a pre-trained version of the recently developed Fine-grained Late-interaction Multi-modal Retriever (FLMR) approach to KB-VQA, and we report new state-of-the-art results across a range of tasks. We also present investigations into the scaling behaviors of PreFLMR intended to be useful in future developments in general-purpose multi-modal retrievers.

## introduction knowledge-based visual question answering (kb-vqa) systems generate answers to queries consisting of questions about given images. correctly answering these questions requires accessing relevant world knowledge as well as vision and language understanding. despite their demonstrated abilities in vision and language, recent large multimodal models (lmms) @xcite @xcite @xcite have performed poorly in recent challenging kb-vqa tasks @xcite . one promising approach to improve their kb-vqa performance is @xcite , in which answer generation by lmms is grounded in relevant documents retrieved from a knowledge base. the best-performing document retrieval approach for kb-vqa to date is fine-grained lateinteraction multi-modal retrieval (flmr) @xcite . flmr uses multi-dimensional embedding matrices to represent documents and queries and then efficiently computes their relevance scores via late-interaction @xcite , thus capturing fine-grained relevance at the token level rather than at the passage level, as in dense passage retrieval (dpr) @xcite . as a late-interaction retriever, flmr substantially outperforms dpr on a range of kb-vqa tasks, with only minor speed penalties. in all of these methods, model and data size are important considerations. there has been much work in scaling up large language models (llms) @xcite @xcite ) and text-based retrieval @xcite , but the scaling properties of these vision and language retrieval systems have not been studied. we therefore investigate the following three aspects of flmr in kb-vqa. we show that m2kr can be used in training an flmr-based rag llm for multi-task multimodal retrieval. we refer to this model as pre-flrm (for pre-trained flmr). preflmr can be used directly in its pre-trained form for multi-task multi-modal retrieval. preflmr can also be finetuned for specific task-specific performance. in both uses we find that preflmr gives us substantial gains across the m2kr tasks. contributions of this paper are: to our knowledge, this is the first systematic study of scaling in late-interaction based visionlanguage retrievers and should provide empirical guidance for future work. another line of relevant research is kb-vqa retrieval involving named entities, where retrieved documents must identify the person in the image. for example, on viquae @xcite , @xcite trains the retriever with a multimodal inverse cloze task, while @xcite shows that combining mono-and cross-modal retrieval improves performance. both use a weighted sum of bert @xcite and clip @xcite embeddings, while our work trains a single multi-modal late-interaction retriever. knowledge-based vqa systems. recent multimodal systems have significantly improved in complex tasks like okvqa @xcite that require external knowledge sources @xcite @xcite @xcite @xcite @xcite . systems like kat @xcite and revive @xcite used llms (e.g. gpt-3) for generating candidate answers. challenges remain in answering more knowledge-intensive questions @xcite , underscoring the need for robust document retrieval. @xcite showed that even state-of-the-art llms perform poorly on difficult kb-vqa questions, with an accuracy of under 20% when retrieval is not incorporated. ra-vqav2 @xcite and prior work @xcite @xcite @xcite demonstrated strong performance in kb-vqa by using external knowledge databases. scaling retrieval systems. previous work has explored scaling laws in language/vision systems @xcite , revealing correlations between model performance, computation, number of parameters, and dataset sizes. in retrieval, @xcite and @xcite both observe improvements in dpr-like models with one-dimensional embeddings by increasing the size of language/vision encoders. this paper reports similar scaling investigations in multimodal late-interaction retrieval. ## the m2kr benchmark suite current multi-modal retrievers are typically trained and evaluated on a single dataset only. to properly study general-purpose multi-modal retrievers, we introduce the multi-task multi-modal knowledge retrieval (m2kr) benchmark suite. we convert nine diverse datasets, originally designed for vision and language tasks such as image recognition, image captioning, and conversational interactions, into a uniform retrieval format. details of the preprocessing steps, data partition, and prompting instructions are provided in appendix a, but we note here that re-purposing these datasets into a single consistent collection for knowledge-based visual question answering represents a non-trivial effort. m2kr will be released with our models. ## preflmr architecture and training our architecture generally follows that of flmr @xcite as shown in fig. 1 . pre-flmr uses token embedding matrices q and d to represent query and document, respectively. given a query q consisting of texts q and an image i, preflmr uses a language model f l to obtain embeddings of all tokens in q, a vision model f v to obtain embeddings of i, and a mapping structure f m to project image embeddings into the text embedding space. all token-level embeddings are concatenated to form the query representation q. the document matrix d is obtained similarly with the language model f l but without visual features. the relevance score r(q, d) is computed via lateinteraction @xcite between q and d, aggregating the maximum dot products over all query tokens with respect to all document tokens (eq. 9). l q and l d denote the total number of tokens in query q and document d, respectively. preflmr improves over flmr in the following aspects: (1) while flmr only uses the @xcite embedding from vit as the image representation, in preflmr we additionally extract embeddings of image patches from vit's penultimate layer to obtain a detailed visual representation. (2) we introduce transformer blocks with cross-attention into the mapping structure to obtain query-aware visual the ultimate vqa performance. this is because document annotations are frequently incomplete, and alternative documents within the corpus can often provide answers to the questions. preflmr training and inference follow that of flmr. when training on data consisting of several datasets, we randomly shuffle the entire training data and only use in-batch negative examples from the same corpus. post-training, all documents are indexed through plaid @xcite for efficient late-interaction retrieval. for detailed evaluation of retrieval efficiency, we refer readers to @xcite . the detailed formal expression of the entire model can be found in appendix b.4. ## experiments and results in this section we present results of scaling pre-flmr components (sec. 5.2, 5.4), analyze the effect of each training stage (sec. 5.3, 5.6), and evaluate on the downstream kb-vqa tasks (sec. 5.5). we summarize our findings in sec. 5.7. multi-task performance refers to preflmr results, i.e. stages 0, 1, 2, and 3, without any single-task fine-tuning. ## conclusion this work has studied the scaling behaviour of state of the art multi-modal document retrieval systems, with a focus on enhancing fine-grained lateinteraction retrieval for knowledge-based visual question answering. we contribute a comprehensive training and evaluation framework, m2kr, for general-purpose multi-modal knowledge retrieval. the preflmr system we train in the m2kr framework yields excellent retrieval performance across a range of tasks and can also serve as a base for further task-specific fine-tuning.