Unifying Multimodal Retrieval via Document Screenshot Embedding (2024)

Abstract: In the real world, documents are organized in different formats and varied modalities. Traditional retrieval pipelines require tailored document parsing techniques and content extraction modules to prepare input for indexing. This process is tedious, prone to errors, and has information loss. To this end, we propose Document Screenshot Embedding (DSE), a novel retrieval paradigm that regards document screenshots as a unified input format, which does not require any content extraction preprocess and preserves all the information in a document (e.g., text, image and layout). DSE leverages a large vision-language model to directly encode document screenshots into dense representations for retrieval. To evaluate our method, we first craft the dataset of Wiki-SS, a 1.3M Wikipedia web page screenshots as the corpus to answer the questions from the Natural Questions dataset. In such a text-intensive document retrieval setting, DSE shows competitive effectiveness compared to other text retrieval methods relying on parsing. For example, DSE outperforms BM25 by 17 points in top-1 retrieval accuracy. Additionally, in a mixed-modality task of slide retrieval, DSE significantly outperforms OCR text retrieval methods by over 15 points in nDCG@10. These experiments show that DSE is an effective document retrieval paradigm for diverse types of documents. Model checkpoints, code, and Wiki-SS collection will be released.

## introduction information retrieval systems help users access external information from documents in varied modalities, including text, images, charts, and tables. as shown in figure 1 (a), existing document retrieval paradigms typically process these modalities separately. for example, traditional lexical retriever bm25 @xcite or neural retrievers such as dpr @xcite rely on extracted text contents from documents. recent multimodal retrieval @xcite leverage both processed text and image units to broaden the scope of retrieval, thus supporting text-image tasks. however, the existing retrieval paradigms lack a unified encoding process across modalities, leading to two underlying issues. firstly, preprocessing is not a trivial effort. specialized processing is required to handle various document types and content modalities, and they are often imperfect. for instance, html files in the wild can present significant complexity due to their varied structures, making it difficult for a single tool to parse all information accurately. similarly, slides and pdfs often require ocr models to extract text and handle other content types like tables and figures separately @xcite . managing these diverse modalities separately is tedious, and precisely dealing with the long-tailed document appearances in the real world is often impractical. secondly, this process "breaks" the original appearance of the document, disrupting its visual context and layout integrity. the visual presentation of a document can convey essential information that is difficult to capture through content extraction alone. for example, in addition to the contents of texts and images, the size and position of these elements in a document may encode the importance of the information they contain @xcite . to tackle the aforementioned issues, we introduce document screenshot embedding (dse), a new information retrieval paradigm that unifies the varied formats and modalities in a single form for direct document encoding and indexing: screenshots. in contrast to using various tools to extract texts and images from documents in different formats, screenshots are easy to obtain and all the information in the documents are visually preserved. as illustrated in figure a dense representation through a large visionlanguage model. during search, a user's query is encoded by a language model to locate the nearest document embeddings. we conduct empirical studies to demonstrate that dse is effective for document retrieval. specifically, we conduct experiments on two types of document retrieval settings: text-intensive and text-image-mixed. for the textintensive case, we collect 1.3 million wikipedia web page screenshots as our corpus and fine-tune a large vision-language model as a bi-encoder to conduct dense retrieval on questions in the nq dataset @xcite . experimental results show that dse outperforms the traditional text-based retrieval method bm25 by 17 points in top-1 retrieval accuracy on nq questions and is competitive with text-based dense retrieval methods in a text-oriented evaluation. this experiment indicates that dse can sufficiently encode the textual information in a screenshot. as for the imagetext mixed setting, we use slide retrieval. we turn the existing slidevqa @xcite dataset into an open-domain retrieval setting, where models are required to retrieve relevant slides from a pool of 50k slides for given questions. results show that dse outperforms all text-based retrieval methods which rely on ocr (including bm25 and dense text retrieval) by over 15 points in ndcg@10. ## conclusion in this paper, we introduce dse, a novel information retrieval paradigm that leverages screenshots to simplify the document retrieval process. by circumventing traditional preprocessing steps and directly encoding documents with a vision-language model, dse offers a unified approach to handling varied document modalities. we empirically show that dse outperforms traditional retriever and ocrbased methods on varied document retrieval tasks, such as webpage and slide retrieval. this highlights the potential of dse to improve document retrieval in a range of real-world applications. by integrating dse with a large vision-language model (vlm) generator, it leads to a promising visual-based retrieval augmented generation (v-rag) paradigm. in this paradigm, dse retrieves document screenshots, which the vlm generator can process directly for generation, without needing separate text or image extraction. this creates an end-to-end document intelligence system that eliminates the need for content extraction. we hope our work opens future research into improving v-rag with better retrieval methods, enabling more efficient and seamless multi-modal information retrieval and generation.