mm E 5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Data (2025)

Abstract: Multimodal embedding models have gained significant attention for their ability to map data from different modalities, such as text and images, into a unified representation space. However, the limited labeled multimodal data often hinders embedding performance. Recent approaches have leveraged data synthesis to address this problem, yet the quality of synthetic data remains a critical bottleneck. In this work, we identify three criteria for high-quality synthetic multimodal data. First, broad scope ensures that the generated data covers diverse tasks and modalities, making it applicable to various downstream scenarios. Second, robust cross-modal alignment makes different modalities semantically consistent. Third, high fidelity ensures that the synthetic data maintains realistic details to enhance its reliability. Guided by these principles, we synthesize datasets that: (1) cover a wide range of tasks, modality combinations, and languages, (2) are generated via a deep thinking process within a single pass of a multimodal large language model, and (3) incorporate real-world images with accurate and relevant texts, ensuring fidelity through self-evaluation and refinement. Leveraging these high-quality synthetic and labeled datasets, we train a multimodal multilingual E5 model mmE5. Extensive experiments demonstrate that mmE5 achieves state-of-the-art performance on the MMEB Benchmark and superior multilingual performance on the XTD benchmark. Our codes, datasets, and models are released in https://github.com/haon-chen/mmE5.

## introduction multimodal embedding models encode multimedia inputs, such as images and text, into latent vector representations. they have demonstrated effectiveness across diverse downstream tasks, including classification @xcite , visual question answering (vqa) @xcite , and crossmodal retrieval @xcite . prior studies have focused on training multimodal embedding models using simple text-image pre-trained models such as clip @xcite . more recently, researchers have turned to multimodal large language models (mllms), including llava @xcite and @xcite , to develop universal embedding models. these vision-language models (vlms) mostly rely on high-quality human-labeled datasets to achieve robust embedding capabilities. such datasets suffer from data scarcity because they require high costs of multimodal annotations. to address this, researchers have leveraged the advanced language modeling capabilities of large language models (llms) and mllms to synthesize datasets for fine-tuning multimodal embedding models @xcite @xcite . however, existing works lack a comprehensive exploration into the quality of synthetic embedding data. typically, most data generated by them are limited to specific modality types of english retrieval tasks, harming the generalization capabilities of the embedding models. after analyzing common application scenarios of multimodal embedding models, we identify three key criteria and introduce a data synthesis framework guided by these principles: (1) broad scope. multimodal embedding models are commonly employed in tasks such as classification, visual question answering (vqa), and retrieval, which require understanding various input combinations of text and images. additionally, multilingual contexts are increasingly popular in daily scenarios. as shown in figure 1 , our framework synthesizes datasets covering three tasks, seven modality combinations, and 93 languages, ensuring that models trained on it generalize effectively across diverse scenarios. (2) robust cross-modal alignment. in multimodal tasks, models must understand and align information across different modalities to generate meaningful representations. without accurate cross-modal alignment, embeddings may fail to capture the underlying relationships, leading to poor performance in downstream tasks. to synthesize data of robust cross-modal alignment, our framework incorporates a deep thinking process. specifically, for each sampled image, we first employ an mllm to interpret it from four perspectives before generating data: general information, object-level description, contextual background information, and task-specific brainstorming, i.e., how the image relates to the given task. additionally, the entire data synthesis process is executed within a single pass of an mllm. by this, the mllm can "see" the images at the whole time, avoiding potential information loss that might occur due to multiple i/o steps in previous works @xcite . (3) high fidelity. the individual quality of each modality (e.g., real images, high-quality instructions, queries, and hard negatives) determines the overall usefulness of the dataset. to enhance fidelity, our framework uses real images sampled from an opensource corpus (laion-400m @xcite ) as the input images. we also apply a series of quality control measures, such as self-evaluation and refinement, ensuring that the synthetic components accurately reflect real-world distributions and maintain strong cross-modal alignment. with the synthesized data ready, we train a multimodal multilingual e5 model (mme5). it achieves state-of-the-art performance on the 36 datasets of mmeb @xcite , using 45 times less training data than the previous sota model mmret @xcite (560k compared to 26m) in a zero-shot setting. after incor-porating labeled data, mme5 still demonstrates the best performance. besides, mme5 achieves the best results on the multilingual benchmark xtd (aggarwal and kale, 2020), demonstrating its superior multilingual capabilities. in summary, our contributions are as follows: ## related work multimodal embedding previous studies, such as clip @xcite , align @xcite , blip @xcite , and coca @xcite , have employed large-scale weakly supervised data to learn separate multimodal representations through pre-training. some works attempt to obtain universal embeddings for texts and images utilizing existing clip-like models @xcite @xcite . for instance, uniir @xcite integrates separate embeddings from different modalities into unified features. recent approaches finetune mllms to leverage their multimodal reasoning capabilities for obtaining universal representations @xcite @xcite . for example, vlm2vec @xcite utilizes instruction tuning to transform mllms into embedding models. ## methodology: mme5 in this section, we present our method, which synthesizes high-quality multimodal data for the further finetuning of our embedding model mme5. as shown in figure 2 , our method consists of five stages: (1) initially, for each data sample to be synthesized, we configure the specifics of the task, modality combination, language, and input images. ## conclusion in this work, we synthesize high-quality multimodal multilingual data to train the model mme5. we first define high-quality multimodal synthetic data based on three criteria: broad scope, robust cross-modal alignment, and high fidelity. then, we develop a data synthesis framework guided by these principles. finally, we train a multimodal multi-lingual embedding model using the high-quality synthetic data. mme5 achieves sota performances on both the general benchmark mmeb and the multilingual benchmark xtd.