Query-aware Multi-modal based Ranking Relevance in Video Search (2023)

Abstract: Relevance ranking system plays a crucial role in video search on streaming platforms. Most relevance ranking methods focus on text modality, incapable of fully exploiting cross-modal cues present in video. Recent multi-modal models have demonstrated promise in various vision-language tasks but provide limited help for downstream query-video relevance tasks due to the discrepency between relevance ranking-agnostic pre-training objectives and the real video search scenarios that demand comprehensive relevance modeling. To address these challenges, we propose a QUery-Aware pre-training model with multi-modaLITY (QUALITY) that incorporates hard-mined query information as alignment targets and utilizes video tag information for guidance. QUALITY is integrated into our relevance ranking model, which leverages multi-modal knowledge and improves ranking optimization method based on ordinal regression. Extensive experiments show our proposed model significantly enhances video search performance.

## introduction video search has become a prevalent method for users to identify relevant content in response to text queries on video streaming platforms. relevance ranking is crucial in video search @xcite , as it determines the relevance degree of a video concerning a given query. pointwise loss(e.g., binary cross-entropy loss), ranking loss(e.g., hinge loss) and combined-pair loss (a linear combination of pointwise and pairwise loss) @xcite are commonly used to optimize relevance ranking task. however, these methods fail to balance calibration ability (globally stable prediction with good interpretability) and ranking ability (prediction can lead to a correct ranking) @xcite . at the same time, the transformer architecture's recent success @xcite in * equal contribution computer vision and natural language processing has led to pre-trained language models achieving promising results in retrieval and ranking tasks @xcite @xcite . however, most existing approaches primarily focus on text modality and alternative methods which integrate large-scale vision-and-language pre-training (vlp) models, such as clip @xcite and albef @xcite , into video search engines face two key challenges: (1) images typically align with verbose and detailed video texts, providing limited assistance for modeling matching relationship between visual signals and concise queries in downstream relevance tasks. (2) most vlp models are trained on singleframe images and texts, neglecting video information such as keyframes and tag data, rendering them unsuitable and inadequate for video search engines. to address these challenges, we propose a queryaware, multi-modal relevance ranking model for real video search systems within a two-step framework, as depicted in fig. 1 . query-aware pre-training model with multimodality. we present a real-world query-aware pre-training model that simultaneously aligns image features with video text features and query features. additionally, we propose a hard query mining strategy to effectively exploit query knowledge. inspired by clip4clip @xcite and table @xcite , we introduce a local tag-guided attention network to extract features from sequential frames, rather than a single image. to preserve pre-trained knowledge to the greatest extent and accelerate the training process, we employ an adapter-tuning strategy ranking relevance. following the approach in @xcite , we model relevance ranking under the pre-training and fine-tuning paradigm, utilizing various handcrafted features (e.g., bm25 @xcite , click similarity @xcite , term weight) and pre-trained representations of query and video within a wide and deep network architecture. we enhance ranking performance by incorporating multi-modal knowledge and proposing an ordinal regression based approach for joint optimization of ranking and calibration in relevance prediction. in summary, this paper makes the following contributions: ## methodology in this section, we describe the details of our multi-modal-based ranking-relevance approach. the overall architecture of our methodology is illustrated in fig. 1 , comprising a query-aware pre-training multi-modal model and a rankingrelevance model that utilizes both visual and textual information. ## conclusion & limitations in this study, we introduce quality, a queryaware pre-training model that leverages multimodal information, including queries, video frames, tags, and titles. quality is integrated into our ordinal regression-based ranking relevance model. through extensive experiments conducted on real-world data, we demonstrate the effectiveness of our proposed method. our method relies on a graph mining strategy that utilizes search log data to identify previously unobserved query-video pairs, thus alleviating the matthew effect problem in search engines. nonetheless, the accuracy of our approach may be influenced by noise during graph construction. consequently, we recommend investigating alternative hard mining strategies or visual debiasing strategy @xcite to enhance performance.