Captioning for Text-Video Retrieval via Dual-Group Direct Preference Optimization (2025)

Abstract: In text-video retrieval, auxiliary captions are often used to enhance video understanding, bridging the gap between the modalities. While recent advances in multi-modal large language models (MLLMs) have enabled strong zero-shot caption generation, we observe that such captions tend to be generic and indistinguishable across visually similar videos, limiting their utility for fine-grained retrieval. Moreover, conventional captioning approaches are typically evaluated using language generation metrics, such as BLEU, which are not typically tailored for retrieval tasks that require making discriminative distinctions between candidates. To address this, we propose \textbf{CaRe-DPO} , a retrieval framework that directly optimizes caption generation using retrieval relevance scores. At its core is Dual-Group Direct Preference Optimization (DG-DPO), a novel learning strategy that supervises captioning by modeling preferences across groups of distinct video and caption pairs. In addition, we present an MLLM-based retrieval model that incorporates role-embeddings to better distinguish between textual inputs with different functional roles, such as an auxiliary caption and a text query. Through extensive experiments, we demonstrate that CaRe-DPO significantly enhances retrieval performance by effectively leveraging auxiliary knowledge to generate fine-grained captions for retrieval. Code is available at https://github.com/mlvlab/CaReDPO.

## introduction text-video retrieval is a fundamental task in multimodal learning, aiming to align natural language descriptions with video content. traditional retrieval methods often adopt dual-encoder architectures, such as clip @xcite , which encode videos and text queries into a shared embedding space. however, these approaches often † corresponding authors. the video begins with a cat walking around the room, exploring its surroundings. the scene transitions to … the kitten moves around, occasionally looking up and down as it interacts with the toy. in some frames … baseline 😵💫 it then sits up, stretches its body, stands on all fours, and moves towards an object resembling a wheel … … kitten playing with a pink toy on a wooden :loor, moving around and interacting playfully. the … ours ☺ two similar videos vid 1 vid 2 vid 1 vid 2 vid 1 vid 2 (a) generated captions for a pair of semantically similar videos bleu score rank captions video humming bird with its beak in a feeder text query captions top-1 top-2 top-k hovering near the feeder … eventually 2lying away … plain .. hovers near the feeder to drink from its nectar-2illed.. moves slightly while standing still ... in front of the feeder moves slightly while standing still ... in front of the feeder hovers near the feeder to drink from its nectar-2illed.. hovering near the feeder … eventually 2lying away … plain .. @xmath0.30 retrieval score rank (b) comparison on rank of captions (bleu vs retrieval) figure 1 : misalignment between conventional captioners and retrieval objectives. (a) captions from pretrained models are nearly identical for similar videos, while our method produces more distinct and retrievalrelevant descriptions. (b) bleu-selected top captions often mismatch retrieval rankings; correlation between the two is as low as 30%. multimodal large language models (mllms) @xcite @xcite @xcite @xcite ) that encompass strong visual and text understandings, recently caught attention for handling multi-modal retrieval systems @xcite @xcite . their capacity to jointly attend to both visual and textual inputs allows them to interpret diverse and complex text queries in relation to video content while also leveraging captions as additional semantic context, providing a promising direction for advancing retrieval performance. however, often captions produced by pretrained models fail to capture the detailed distinctions necessary for retrieval. as illustrated in fig. 1a , two visually similar videos depicting cats in indoor environments are given captions that are generic and overlapping, describing actions like 'walking around the room' or 'moves around.' nevertheless, we expect to generate captions that surface discriminative visual details, such as 'sits up, stretches its body' and 'interacting playfully', which provide discriminative cues critical for retrieval among similar videos. this issue is further amplified by the common practice of evaluating caption quality using metrics such as bleu @xcite . specifically, as shown in fig. 1b , among the three distinct captions generated for a video, the top-1 caption selected based on a conventional captioning metric often does not align with the top-1 caption when ranked by retrieval relevance score (placed at the bottom rank). to quantify this misalignment, we measured the pearson correlation between the indices of the top-1 ranked captions under each metric, finding a correlation as low as 0.30. this highlights a substantial discrepancy between conventional captioning metric-based evaluations and retrieval-oriented objectives. to this end, we propose care-dpo, captioning for text-video retrieval via dual-group direct preference optimization, a novel retrieval framework. at the core is dual-group direct preference optimization (dg-dpo), directly supervising the captioning model with the retrieval scores to align with the retrieval objective. unlike standard single-group dpo (local caption ranking within a video), dg-dpo incorporates dual-group preferences, learning global ranking over video-caption pairs. in addition, we introduce role-embeddings during retrieval model training to differentiate the roles of heterogeneous textual inputs, allowing the model to more effectively leverage the auxiliary captions. we empirically validate that care-dpo encourages the mllm-based retrieval model to further leverage the auxiliary captions during retrieval and enables enhancement of the caption quality, yielding a performance improvement across various text-video retrieval benchmarks. the main contributions of this work are: ## method in this section, we introduce our care-dpo, captioning for retrieval via dualgroup-direct preference optimization, a novel retrieval framework that enhances text-video retrieval with auxiliary captions. in sec. 4.1, we describe our retrieval backbone built upon an mllm, which jointly encodes video and text inputs to compute cross-modal similarity, where we also introduce retrieval roleembeddings to differentiate the heterogeneous textual inputs. then in sec. 4.2, we present dg-dpo, a preference optimization method that supervises the captioning model to further align with the retrieval objective. the overall framework is illustrated in fig. 2 . ## conclusion we present care-dpo, a novel retrieval framework that enhances text-video retrieval with auxiliary captions. our role-embeddings enable retrieval models to explicitly distinguish the roles of heterogeneous textual inputs. furthermore, our dual-group direct preference optimization aligns caption generation with retrieval relevance scores while leveraging both local and global ranks. through extensive experiments, we demonstrate that care-dpo enhances overall retrieval accuracy across benchmarks.