VISTA : Visualized Text Embedding For Universal Multi-Modal Retrieval (2024)

Abstract: Multi-modal retrieval becomes increasingly popular in practice. However, the existing retrievers are mostly text-oriented, which lack the capability to process visual information. Despite the presence of vision-language models like CLIP, the current methods are severely limited in representing the text-only and image-only data. In this work, we present a new embedding model VISTA for universal multi-modal retrieval. Our work brings forth threefold technical contributions. Firstly, we introduce a flexible architecture which extends a powerful text encoder with the image understanding capability by introducing visual token embeddings. Secondly, we develop two data generation strategies, which bring high-quality composed image-text to facilitate the training of the embedding model. Thirdly, we introduce a multi-stage training algorithm, which first aligns the visual token embedding with the text encoder using massive weakly labeled data, and then develops multi-modal representation capability using the generated composed image-text data. In our experiments, VISTA achieves superior performances across a variety of multi-modal retrieval tasks in both zero-shot and supervised settings. Our model, data, and source code are available at https://github.com/FlagOpen/FlagEmbedding.

## introduction information retrieval (ir) is a critical task in many real-world scenarios, e.g., search engines, opendomain question answering, and retrieval augmented generation @xcite . it aims to find relevant data from a large database such that the downstream problems can be faithfully solved on top of proper knowledge. one important ir paradigm is dense retrieval, where the query and candidates, i.e. document, are represented as embeddings, and their semantic relationship can be reflected by the embedding similarity @xcite . with the continual progress on pre-trained model and training algorithm, increasingly powerful embedding models have been developed, such as dpr @xcite , contriever @xcite , gtr @xcite , e5 @xcite , bge @xcite , etc., which substantially improves the quality and universality of dense retrieval. most of the existing dense retrieval models are text-oriented, which can only deal with the data presented in human language. however, a large portion of the world knowledge naturally contains both text and image, e.g., web articles with visual illustration @xcite ; meanwhile, people's queries can also be flexibly expressed with multiple data modalities, e.g., search queries with exemplar images @xcite . despite the development of visual-language representation models (vlm), like clip @xcite and align @xcite , the above problem is still challenging in many perspectives. on one hand, the existing vlms are severely limited in text representation capability, whose retrieval performance is far behind the recent text-only embedding models, like e5 and bge. on the other hand, the existing vlms focus more on the independent encoding of text and image; nevertheless, the joint representation of image-text data (e.g., documents with illustrations) is largely unexplored. in this work, we propose visualized text embedding for universal multi-modal retrieval, namely vista. it takes the best of the existing text encoder and image encoder where high-quality multi-modality embedding can be generated from it. in particular, our work presents the following three technical contributions. first of all, we come up with a flexible model architecture to facilitate the generation of multimodal embedding. it is built upon a powerful and well-trained text encoder, which exhibits proficient text retrieval capability. meanwhile, it makes the incorporation of visual tokens generated by an expressive image encoder, thereby augmenting the capability of image processing. such an architecture brings forth two important advantages. 1) it establishes the in-depth fusion of text and image data, which substantially contributes to the quality of multi-modal embedding. 2) it also enables the preservation of the original performance of text embedding, as the text encoder is fully fixed while the visual tokens are incorporated. secondly, we propose two innovative pipelines for the automatic generation of image-text composed datasets, thereby securing large-scale, highquality data for the training of multi-modal embedding models. these pipelines are designed to cater to scenarios where either the query or the candidate comprises image-text pairs, thereby facilitating the model to adapt to a diverse range of multi-modal retrieval situations. thirdly, we design a two-stage training algorithm to learn the multi-modal embedding model. initially, we perform the basic text-to-image matching task with massive weakly-labeled cross-modal data @xcite , which aligns the visual token embedding with the text encoder. subsequently, we perform composed text&image matching with our generated composed image-text datasets, which establishes the multi-modal representation capability for the embedding model. vista is empirically verified by comprehensive experiments. particularly, it achieves superior performance across various multi-modal retrieval tasks in both zero-shot and supervised settings. without any task-specific optimization, vista is able to outperform or match the leading approach in every downstream evaluation scenario. besides, vista's performance can also be substantially improved if it is continually fine-tuned for corresponding tasks. ## experimental results we carry out both zero-shot evaluations and supervised fine-tuning across various benchmarks to substantiate the efficacy and versatility of our vista model. in addition, we perform comprehensive ablation studies to scrutinize both the design of the vista model and the effectiveness of our stage-2 training datasets. ## conclusion in this paper, we introduce vista, an visualized text embedding approach for universal multimodal retrieval. our work makes three significant contributions. firstly, we design a flexible model architecture that enables the in-depth fusion of text and image data, while maintaining the powerful performance of the general text embedding models. secondly, we develop two data generation strategies for training multi-modal embedding models without the need for manual annotation. lastly, we introduce a two-stage training algorithm that rapidly enhances the multi-modal representation capability of vista. extensive experimental results demonstrate the superior performance of vista in both zero-shot and supervised fine-tuning settings for various multi-modal retrieval tasks.