Enhancing Partially Relevant Video Retrieval with Robust Alignment Learning (2025)

Abstract: Partially Relevant Video Retrieval (PRVR) aims to retrieve untrimmed videos partially relevant to a given query. The core challenge lies in learning robust query-video alignment against spurious semantic correlations arising from inherent data uncertainty: 1) query ambiguity, where the query incompletely characterizes the target video and often contains uninformative tokens, and 2) partial video relevance, where abundant query-irrelevant segments introduce contextual noise in cross-modal alignment. Existing methods often focus on enhancing multi-scale clip representations and retrieving the most relevant clip. However, the inherent data uncertainty in PRVR renders them vulnerable to distractor videos with spurious similarities, leading to suboptimal performance. To fill this research gap, we propose Robust Alignment Learning (RAL) framework, which explicitly models the uncertainty in data. Key innovations include: 1) we pioneer probabilistic modeling for PRVR by encoding videos and queries as multivariate Gaussian distributions. This not only quantifies data uncertainty but also enables proxy-level matching to capture the variability in cross-modal correspondences; 2) we consider the heterogeneous informativeness of query words and introduce learnable confidence gates to dynamically weight similarity. As a plug-and-play solution, RAL can be seamlessly integrated into the existing architectures. Extensive experiments across diverse retrieval backbones demonstrate its effectiveness.

## introduction text-to-video retrieval (t2vr) has been a longstanding challenge in vision and language research, allowing humans to associate textual concepts with video entities @xcite @xcite @xcite . however, the mainstream t2vr methods @xcite @xcite ) assume * corresponding author. query：rachel tells monica that they are wearing the same clothes. gt (b) gmmformer v2 video index score (c) our method query3: monica tells ross never knew he did that. … … … query1: rachel tells monica that they are wearing same clothes. query2: joey is upset that monica's relation ship just ended. partially relevant video relevant moments ambiguous query interference correlation gt gt gt distractor video gt video figure 1: (a) toy examples of spurious semantic correlations. (b-c) retrieval scores of our method vs. gmm-former v2. gmmformer v2 fails to handle uncertainty, assigning the highest score to a distractor video. to address the above issues, we propose robust alignment learning (ral), which explicitly models and utilizes uncertainty in data to enhance retrieval robustness. our ral builds upon the insight that prvr should not be treated as pointwise query-clip feature alignment but rather as a probabilistic alignment problem that accounts for uncertainty. inspired by probabilistic distributional representations @xcite , we model both video and query embeddings as gaussian distributions, where the variance quantifies the inherent aleatoric uncertainty in each instance. based on the distributional representations, we naturally construct gaussian-based text and video proxies, which serve as multiple potential alignment candidates, enabling the model to capture diverse crossmodal relationships. furthermore, most retrieval methods compute similarity scores by applying mean-pooling over words in the word-frame similarity matrix @xcite . we find this approach exacerbates retrieval bias as not all words contribute equally to retrieval, meaningless words (e.g., "a") can distort the similarity estimation. to address this, we introduce confidenceaware alignment that dynamically assigns confidence weights to query words. as shown in figure 2 , our ral consists of two key components: (1) multimodal semantic robust alignment (msra) quantifies the semantic distribution in each modality by representing samples as multivariate gaussian distributions. given video and query embeddings, we first employ multigranularity aggregation to obtain holistic semantics with sufficient contexts before estimating gaussian parameters. considering the incompleteness of the query relative to the video, we construct text distribution from a query support set that combines all video-related queries. then, we conduct crossmodal learning with these distributional representations to joint video and text domains. to be specific, msra is optimized with two losses: a distribu-tion alignment loss l da enforcing probabilistic alignment between video and text distributions for robust cross-modal consistency, and a proxy matching loss l pm leveraging multiple alignment candidates to capture diverse semantic relationships. (2) confidence-aware set-to-set alignment (csa) is to enhance query-video matching by dynamically adjusting the contribution of each query word. instead of treating all words equally, csa predicts a confidence score for each word and uses it to weight the word-frame similarity matrix. this effectively mitigates the influence of meaningless words and improves video retrieval. our contribution can be summarized as follows: ## related work partially relevant video retrieval prvr aims to retrieve untrimmed videos partially relevant to a given query. compared to traditional t2vr, this task is more aligned with real-world application scenarios. existing research @xcite @xcite @xcite @xcite primarily tackled prvr by constructing multi-scale clip representations. specifically, ms-sl @xcite applies sliding windows to form clip representations and performs similarity calculations at both clip and frame levels. gmmformer @xcite uses multiple gaussian windows to constrain inter-frame interactions, thereby implicitly generating multi-scale clip features. its improved version, gmmformer v2 @xcite , introduces a learnable feature fusion mechanism to aggregate multi-scale clips. despite promising advancements, these methods suffer from performance bottlenecks due to ignoring the spurious semantic correlations caused by data uncertainty and simplifying the complex semantic alignment, which motivates our robust alignment learning method. uncertainty in multimodal learning uncertainty modeling has been widely explored in multimodal learning @xcite . @xcite first introduces probabilistic embeddings to capture the uncertainty in image representations. similar ideas have been applied to tasks such as sentiment analysis @xcite and instance segmentation @xcite . in the field of cross-modal retrieval, pcme @xcite pioneers the use of probabilistic embeddings to capture the uncertainty of visual concepts. uatvr @xcite further combines deterministic and probabilistic embeddings to explore optimal matching granularity in t2vr. t-mass @xcite introduces a text-mass-based method, treating text embeddings as stochastic variables. however, these methods are typically designed for trimmed videos and exhibit limited effectiveness in prvr. inspired by this, we propose robust alignment learning specifically designed for prvr.