LL a VE : Large Language and Vision Embedding Models with Hardness-Weighted Contrastive Learning (2025)

Abstract: Universal multimodal embedding models play a critical role in tasks such as interleaved image-text retrieval, multimodal RAG, and multimodal clustering. However, our empirical results indicate that existing LMM-based embedding models trained with the standard InfoNCE loss exhibit a high degree of overlap in similarity distribution between positive and negative pairs, making it challenging to distinguish hard negative pairs effectively. To deal with this issue, we propose a simple yet effective framework that dynamically improves the embedding model’s representation learning for negative pairs based on their discriminative difficulty. Within this framework, we train a series of models, named LLaVE, and evaluate them on the MMEB benchmark, which covers 4 meta-tasks and 36 datasets. Experimental results show that LLaVE establishes stronger baselines that achieve state-of-the-art (SOTA) performance while demonstrating strong scalability and efficiency. Specifically, LLaVE-2B surpasses the previous SOTA 7B models, while LLaVE-7B achieves a further performance improvement of 6.2 points. Although LLaVE is trained on image-text data, it can generalize to text-video retrieval tasks in a zero-shot manner and achieve strong performance, demonstrating its remarkable potential for transfer to other embedding tasks.

## introduction multimodal embedding models aim to encode inputs from any modality into vector representations, which then facilitate various multimodal tasks, such as image-text retrieval @xcite , automatic evaluation @xcite , and retrieval-augmented generation (rag) @xcite . although advanced pretrained vision-language models such as clip @xcite , align @xcite , and siglip @xcite can provide unified representations for text and images, they face difficulties when dealing with more complex tasks. particularly, they adopt a dual-encoder architecture that encodes images and text separately, leading to poor performance in tasks such as interleaved image-text retrieval @xcite . in recent years, the rapid development and exceptional performance of large multimodal models (lmms) have prompted researchers to focus increasingly on lmm-based multimodal embedding models. compared to traditional pretrained vision-language models, lmms not only demonstrate superior multimodal semantic understanding capabilities but also naturally support interleaved text-image inputs @xcite @xcite @xcite . this advantage makes them more flexible and efficient in handling multimodal embedding tasks. as a representative work, @xcite construct the massive multimodal embedding benchmark (mmeb), which encompasses 4 meta-tasks and 36 datasets, and train multimodal embedding models based on lmms. experiment results demonstrate that by providing suitable task instructions and employing contrastive learning for training, lmm can significantly outperform existing multimodal embedding models and generalize effectively across diverse tasks. however, as shown in figure 1 (a), our preliminary study finds that when training an lmm as a multimodal embedding model using the standard infonce loss @xcite , the query-target similarity distribution of positive and negative pairs exhibits significant overlap. this indicates that the model struggles to learn discriminative multimodal representations for positive and hard negative pairs. building on the above observation and insights from preference learning @xcite , we propose a simple yet effective framework to encourage the model to focus more on hard negative pairs, forcing it to learn more discriminative multimodal representations. under our framework, we consider the embedding model as a policy model and introduce a reward model to assign an adaptive weight to each negative pair, where harder pairs are assigned with larger weights. this ensures that harder negative pairs play a more significant role in model training. in addition, by decoupling the reward model from the policy model, our framework can not only use different models for hardness estimation but also leverage manually annotated hardness to enhance the representation learning for specific samples. inspired by siglip @xcite , we introduce a cross-device negative sample gathering strategy, which significantly alleviates the issue of limited negative samples in lmms caused by excessive memory usage. as shown in figure 1 , we observe that our framework increases the query-target similarity gap between positive and negative pairs, indicating its effectiveness in helping the model learn more discriminative multimodal representations. to evaluate the effectiveness of our framework, we train a series of multimodal embedding models, referred to as llave (large language and vision embedding models), within the proposed framework. these models are based on advanced open-source lmms of varying scales, including llava-ov-0.5b @xcite , aquila-vl-2b @xcite , and llava-ov-7b @xcite . experimental results on mmeb demonstrate that llave-0.5b achieves comparable results to that of the previous vlm2vec (phi-3.5-v-4b). when scaled up to llave-2b, the model requires only about 17 hours of training on a single machine equipped with 8 a100 gpus (40gb) to surpass the state-of-the-art (sota) model mmret-7b, which is pretrained on 27 million image-text pairs. furthermore, when expanded to llave-7b, its performance is even more impressive, surpassing the previous sota model by 6.2 points. meanwhile, when being scaled to different sizes, llave still outperforms the models trained with infonce loss based on the same lmm, demonstrating the effectiveness of our framework. these results fully validate that our framework is both easily scalable and resource-efficient. in addition, despite being trained exclusively on image-text data, llave generalizes effectively to text-video retrieval tasks, showcasing its strong potential for transferring to other embedding tasks. ## preliminary study in this section, we first briefly formulate the multimodal embedding task and review the standard infonce loss. then, we analyze the similarity distributions of positive and negative pairs in lmmbased embedding models trained with the standard infonce loss. ## our framework in this section, we first illustrate the inherent consistency between preference learning and contrastive learning. we then propose a simple yet effective framework that incorporates hardness-weighted contrastive learning (see section 3.1) and crossdevice negative sample gathering (see section 3.2). ## conclusion in this paper, we conduct a preliminary study to find that lmm-based embedding models trained with the standard infonce loss face significant challenges in handling hard negative pairs. to address this issue, we propose a simple yet effective framework that includes the hardness-weighted contrastive learning and the cross-device negative sample gathering strategy to enhance the model's learning of negative pairs with varying difficulty levels. this framework significantly improves the model's capacity to distinguish between positive and negative pairs. experimental results and indepth analyses validate the effectiveness of the proposed framework. in the future, we plan to collect and construct a universal multimodal embedding benchmark for video-text retrieval, aiming to investigate the more universal multimodal embedding models. we will open-source all models and code, hoping to inspire further research in this field.