DALR : Dual-level Alignment Learning for Multimodal Sentence Representation Learning (2025)

Abstract: Previous multimodal sentence representation learning methods have achieved impressive performance. However, most approaches focus on aligning images and text at a coarse level, facing two critical challenges: cross-modal misalignment bias and intra-modal semantic divergence, which significantly degrade sentence representation quality. To address these challenges, we propose DALR (Dual-level Alignment Learning for Multimodal Sentence Representation). For cross-modal alignment, we propose a consistency learning module that softens negative samples and utilizes semantic similarity from an auxiliary task to achieve fine-grained cross-modal alignment. Additionally, we contend that sentence relationships go beyond binary positive-negative labels, exhibiting a more intricate ranking structure. To better capture these relationships and enhance representation quality, we integrate ranking distillation with global intra-modal alignment learning. Comprehensive experiments on semantic textual similarity (STS) and transfer (TR) tasks validate the effectiveness of our approach, consistently demonstrating its superiority over state-of-the-art baselines.

## introduction sentence representation learning converts sentences into low dimensional vectors to preserve semantic information and is widely used in nlp tasks, such as semantic similarity @xcite , information extraction @xcite , and content analysis @xcite @xcite . with the success of pre-trained language models (plms) such as bert @xcite and roberta @xcite , numerous methods (gao et al., 2021; wu et al., 2022b; zhang et al., 2022b; he et al., 2023; seonwoo et al., 2023; he * corresponding author. ## conclusion in this paper, we propose a dual-level alignment framework (dalr) for multimodal sentence representation learning. dalr extends traditional multimodal contrastive learning by promoting both cross-modal and intra-modal alignment for more robust sentence representations. we introduce an auxiliary task to refine negative sampling and generate similarity matrices for effective cross-modal alignment. intra-modal alignment is achieved through a combination of ranking distillation and kl divergence-based fine-grained calibration. extensive experiments on sts and tr benchmarks, supported by detailed analyses, show that dalr consistently outperforms previous state-of-the-art methods.