jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrieval (2025)

Abstract: We introduce jina-embeddings-v4, a 3.8 billion parameter embedding model that unifies text and image representations, with a novel architecture supporting both single-vector and multi-vector embeddings. It achieves high performance on both single-modal and cross-modal retrieval tasks, and is particularly strong in processing visually rich content such as tables, charts, diagrams, and mixed-media formats that incorporate both image and textual information. We also introduce JVDR, a novel benchmark for visually rich document retrieval that includes more diverse materials and query types than previous efforts. We use JVDR to show that jina-embeddings-v4 greatly improves on state-of-the-art performance for these kinds of tasks.

## introduction we present jina-embeddings-v4, a multimodal embedding model capable of processing text and image data to produce single-and multi-vector embeddings, with modular lora adapters @xcite for information retrieval and semantic text similarity. an adapter is also provided for programming language embeddings, technical questionanswering, and natural language code retrieval. this model supports dual-mode output, producing both single-vector outputs suitable for conventional embeddings-based applications and multi-vector embeddings for "late interaction" applications along the lines of colbert @xcite and @xcite . this single-model approach entails significant savings in practical use cases when compared to deploying multiple ai models for different tasks and modalities. a major contribution of this model is introducing new functionality for processing "visually rich" documents: mixed textual and visual media like * these authors contributed equally to this work tables, charts, diagrams, screenshots, web page captures, and similar images. @xcite we have devised a new diversified benchmark, jvdr, for measuring performance on visually rich materials and show that jina-embeddings-v4 far outpaces comparable models on this type of media. ## related work late interaction models generally have higher precision than traditional embedding models. @xcite these models produce multi-vector outputs that consist of sequences of context-sensitive token embeddings. similarity is calculated using a form of chamfer distance adapted to the task: given two sequences of token embeddings, a query and a document, sum the maximum cosine similarity values of each query token embedding to any of the document token embeddings. faysse et al. (2025) train a late-interaction embedding model to search document screenshots using text queries, performing significantly better than traditional approaches involving ocr and clipstyle models trained on image captions. to show this, they introduce the vidore (vision document retrieval) benchmark. however, this benchmark is limited to question-answering tasks in english and french involving only charts, tables, and pages from pdf documents. @xcite extend this benchmark to create mieb (massive image embedding benchmark) by rendering the texts from existing semantic textual similarity tasks as images. the principal purpose of multimodal embedding models is to project objects from multiple modalities into the same semantic embedding space. bimodal image-text models derived from openai's clip architecture @xcite consist of one model for each modality, typically trained with bimodal contrastive pairs to produce embeddings in a common semantic space. the vision-language model (vlm) is an alternate architecture with a single processing path for both images and texts, significantly improving performance on bimodal text-image tasks. @xcite previous work has shed light on the so-called modality gap in this kind of model. @xcite @xcite good semantic matches across modalities tend to lie considerably further apart in the embedding space than comparable or even worse matches of the same modality, i.e., texts in clip-style models are more similar to semantically unrelated texts than to semantically similar images. @xcite demonstrate that vlms have less of a modality gap than clip-style dual encoder architectures. ## model architecture the architecture of jina-embeddings-v4, schematized in figure 1 , is a vlm built on a qwen2.5-vl-3b-instruct 1 backbone. text and image inputs are processed through a shared pathway: images are first converted to token sequences via a vision encoder, then both modalities are jointly processed by the language model decoder with contextual attention layers. as shown in figure 1 , this architecture supports single-and multi-vector output. additionally, three task-specific lora adapters, each with 60m parameters, provide specialized task optimization without modifying the frozen backbone weights. the core specifications of jina-embeddings-v4 are summarized in for images, a transformer-based image model acts as a preprocessor to the language model: the image is divided into patches and the image model processes it as if each patch were a token given to a language model. the output is a multi-vector embedding which becomes the input to the language model, as if it were a stacked set of tokenized text vectors. we have implemented three task-specific lora adapters for different information retrieval use cases described in section 4.2. each lora adapter has only 60m parameters, so keeping all three in memory adds less than 2% to the memory footprint of jina-embeddings-v4. see section 6 for performance information about these adapters. we employ peft @xcite to support lora and dynamically switch between adapters based on the intended task for each batch, without significant runtime overhead. we used a standard lora configuration with rank 32 and a scaling factor of 1, parameterizing all linear layers in the backbone llm. ## training method before training, model weights are initialized to match qwen/qwen2.5-vl-3b-instruct. the multi-vector projection layer and lora adapters are randomly initialized. only the lora adapters are trained, the base model and projection layer remain as initialized. in all phases of training, we apply matryoshka loss @xcite to our base loss function so that single-vector embeddings from jina-embeddings-v4 are truncatable. ## jvdr: visually rich document retrieval benchmark to evaluate the performance of jina-embeddings-v4 across a broad range of visually rich document retrieval tasks, we have produced a new benchmark collection and released it to the public. 5 this new collection extends the vidore benchmark by adding more than 30 additional tests designed to be compatible with vidore. they span a broad range of domains (e.g. legal texts, historic documents, marketing materials), cover a variety of material types (e.g. charts, tables, manuals, printed text, maps) and query types (e.g. questions, facts, descriptions), and use up to 20 languages. these tests include re-purposed existing datasets, new manuallyannotated data, and generated synthetic data. we employed llm-based filtering to ensure all queries are relevant and reflective of realistic usage. 6 we have adapted a number of existing vqa (visual question answering) and ocr datasets, modifying and restructuring them into appropriate query-document pairs. for some datasets, we used structured templates and generative language models to formulate text queries to match their contents. we also created benchmarks from available data to use unconventional querying techniques. we drew heavily on wikimedia materials and other public data sources. for example, some datasets contain encyclopedia article fragments and image descriptions as queries to match with charts and maps. we obtained multilingual documents from wikipedia and paired them with paragraphs that reference them. we used github readme files to create rendered images from markdown-formatted rich texts and paired them with llm-generated natural language descriptions in 17 languages. we have also curated a number of humanannotated resources to better reflect real-world use cases. these include educational materials like lecture slides, commercial catalogs, marketing materials, and institutional documents. we paired these documents with human-written queries. we have been attentive, in constructing jvdr, to the lack of diversity that often plagues information retrieval benchmarks. we cannot commission human-annotated datasets for everything and have had recourse to generative ai to fill in the gaps. we obtained a number of datasets from primarily european sources containing scans of historical, legal, and journalistic documents in german, french, spanish, italian, and dutch, and public service documents and commercial catalogs in hindi, russian, and other often underrepresented languages. we used qwen2 7 to generate queries for 684831c022c53b21c313b449 6 see a.5 for the specific prompts. 7 https://huggingface.co/collections/qwen/ qwen2-6659360b33528ced941e557f these documents. in several cases, we introduced cross-language queries synthesized using advanced multilingual llms, in order to better measure cross-language retrieval. for a comprehensive overview of the individual benchmarks, see appendix a.3. ## evaluation ## conclusion we present jina-embeddings-v4, a state-of-theart multimodal and multilingual embedding model designed for a wide range of tasks, including semantic text retrieval, text-to-image retrieval, text-to-visually-rich document retrieval, and code search. the model achieves strong performance using single-vector representations and demonstrates even greater effectiveness with multi-vector representations, particularly in visually rich document retrieval. jina-embeddings-v4 aligns representations across modalities into a single, shared semantic space, sharply reducing structural gaps between modalities compared to clip-style dual-tower models, enabling more effective cross-modal retrieval. we also present jvdr, a novel benchmark for visually rich documents that dramatically extends the vidore benchmark by including much more diverse data types, more languages, and more kinds of queries and semantic similarity tests. we have made this benchmark available to the public for future work. limitations jina-embeddings-v4 is a model that extends qwen2.5-vl-3b-instruct and is limited by its original training. as a result, its performance on many languages is not comparable to the state-ofthe-art and it may not perform well on materials too far outside of its training. furthermore, highly domain-specialized models may have significantly better performance at specific tasks. although this model is theoretically capable of embedding text and image input together, it has not been trained for such input. it has also not been trained for image-image retrieval or semantic similarity, and may underperform on those tasks. jvdr is not a rigorously representative data collection. it is a significant expansion over previous related benchmarks, but this is a new area for embeddings research, and jvdr undoubtedly has gaps and shortcomings that usage will reveal. table a8 : text-to-image retrieval performance (recall@5%) on crossmodal3600 for all supported languages. a.4 jvdr (visual document retrieval) benchmark results