ORANGE : Text-video Retrieval via Watch-time-aware Heterogeneous Graph Contrastive Learning (2023)

Abstract: With the explosive growth of short-video data on industrial video-sharing platforms such as TikTok and YouTube, text-video retrieval techniques have become increasingly important. Most existing works for text-video retrieval focus on designing informative representation learning methods and delicate matching mechanisms, which leverage the content information of queries and videos themselves (i.e., textual information of queries and multimodal information of videos). However, real-world scenarios often involve brief, ambiguous queries and low-quality videos, making content-based retrieval less effective. In order to accommodate various search requirements and enhance user satisfaction, this study introduces a novel Text-video Retrieval method via Watch-time-aware Heterogeneous Graph Contrastive Learning (termed ORANGE). This approach aims to learn informative embeddings for queries and videos by leveraging both content information and the abundant relational information present in video-search scenarios. Specifically, we first construct a heterogeneous information graph where nodes represent domain objects (e.g., query, video, tag) and edges represent rich relations among these objects. Afterwards, a meta-path-guided heterogeneous graph attention encoder with the awareness of video watch time is devised to encode various semantic aspects of query and video nodes. To train our model, we introduce a meta-path-wise contrastive learning paradigm that facilitates capturing dependencies across multiple semantic relations, thereby enhancing the obtained embeddings. Finally, when deployed online, for new queries non-existent in the constructed graph, a bert-based query encoder distilled from our ORANGE is employed. Offline experiments conducted on a real-world dataset demonstrate the effectiveness of our ORANGE. Moreover, it has been implemented in the matching stage of an industrial online video-search service, where it exhibited statistically significant improvements over the online baseline in an A/B test.

## introduction with the exponential proliferation of short-video data on the internet, text-video retrieval @xcite @xcite @xcite has gained increasing attention from both industrial and academic communities, and become a crucial feature of industrial video-sharing platforms (e.g., tiktok, likee, and youtube). the goal of text-video search is to retrieve the most user-satisfactory videos given a text query. towards this end, great efforts have been devoted to carefully designing informative representation learning methods @xcite @xcite @xcite @xcite and delicate text-video matching mechanisms using single-stream or dual-stream architectures @xcite @xcite @xcite @xcite . for example, clipbert @xcite jointly embeds text-video pairs through a bert-like @xcite single-stream encoder for early cross-modal fusion and directly produces similarity between them. clip4clip @xcite introduces a dual-stream encoder consisting of a transformer encoder @xcite for texts and a spacetime transformer encoder for videos. the obtained representations of texts and videos are then mapped to a common space, where the text-video similarity is measured via the dot product. the above text-video retrieval methods mainly focus on utilizing the content information of queries and videos themselves, i.e., textual information from queries and multimodal information from videos, including video titles, video frames, audio, etc. despite their effectiveness, these meth-ods face two challenges when applied to industrial video-search scenarios. first, unlike academic datasets @xcite used in previous works, real-world query texts tend to be shorter and ambiguous, while user-generated(or uploaded) videos may exhibit low quality. consequently, text-video relevance matching based solely on content information is insufficient for accurately capturing search intents. secondly, for certain queries, the videos retrieved by these contentbased methods are likely to be highly relevant, making it difficult to discern the most user-satisfactory videos from semantically similar candidates. to address the aforementioned challenges, we propose a novel text-video retrieval method via watch-timeaware heterogeneous graph contrastive learning (referred to as orange) in this paper. or-ange aims to learn discriminative embeddings for queries and videos, taking into account not only content information but also the abundant relational information present in video-search scenarios. concretely, based on the text-video search log and domain knowledge, we first construct a heterogeneous information graph(hig) (as shown in fig. 1(a) ), where nodes represent video-retrieval domain objects (e.g., query, video, video tag), and edges represent rich relations among these objects. for instance, a query-video edge describes the relationship where a video is viewed given a query, while a query-query edge describes the rewriting relation between a pair of queries. in order to fully utilize the rich heterogeneous information and thereby learn informative representations, a metapath-guided heterogeneous graph attention encoder (han) @xcite with the awareness of the video watch time is devised to encode various semantic aspects of query and video nodes. the vanilla graph attention mechanisms @xcite are sometimes ineffective as the attention weights are learned implicitly without the guidance of explicit semantics @xcite . in contrast, our watch-timeaware encoder enhances the quality of attention weights by explicitly incorporating the video's watch-time information, which is a crucial indicator of user satisfaction. to train orange, we cast the text-video matching problem as a link prediction task of hig. simultaneously, we employ an auxiliary learning task based on our proposed metapath-wise contrastive learning paradigm, which helps the model capture cross-type semantic de-pendencies and improve the quality of embeddings. lastly, when deploying our model online, we adopt a bert-based query encoder distilled @xcite from orange, enabling on-the-fly inference of query embeddings to support previously unseen queries, i.e., new queries non-existent in our hig. to the best of our knowledge, we are the first on utilizing both rich relational information and content information for text-video retrieval in real-world scenarios. in a nutshell, this work makes the following contributions: ## methodology in this section, we describe the details of our or-ange approach. the overall workflow of our model is shown in fig. 1 and the detailed notations used in this paper are summarized in ## conclusion in this study, we address the text-video retrieval challenge by incorporating content and intricate relations among video-retrieval domain objects into a heterogeneous information graph. based on this, we introduce a novel watch-time-aware encoder and a meta-path-wise contrastive learning strategy to obtain informative representations. furthermore, to ensure our model's applicability for online deployment, we employ a bert-based query encoder, distilled from our full model, to process previously unseen nodes. in our future work, we plan to explore efficient graph distillation strategies. concurrently, the development of informative graph encoders that consider abundant search behavior information should also be investigated.