Modal-specific Pseudo Query Generation for Video Corpus Moment Retrieval (2022)

Abstract: Video corpus moment retrieval (VCMR) is the task to retrieve the most relevant video moment from a large video corpus using a natural language query.For narrative videos, e.g., drama or movies, the holistic understanding of temporal dynamics and multimodal reasoning are crucial.Previous works have shown promising results; however, they relied on the expensive query annotations for the VCMR, i.e., the corresponding moment intervals.To overcome this problem, we propose a self-supervised learning framework: Modal-specific Pseudo Query Generation Network (MPGN).First, MPGN selects candidate temporal moments via subtitle-based moment sampling.Then, it generates pseudo queries exploiting both visualand textual information from the selected temporal moments.Through the multimodal information in the pseudo queries, we show that MPGN successfully learns to localize the video corpus moment without any explicit annotation.We validate the effectiveness of MPGN on TVR dataset, showing the competitive results compared with both supervised models and unsupervised setting models.

## introduction the increased interest in video understanding has gathered attention for solving related tasks such as video captioning @xcite , video question answering @xcite @xcite , and video retrieval @xcite over the past few years. video corpus moment retrieval (vcmr) @xcite is one of the challenging video understanding tasks, in which a model should 1) search for a related video and 2) localize the corresponding moment given a query sentence in a large video corpus. prior works have shown promising performances in vcmr using supervised @xcite @xcite , weakly-supervised @xcite , 2021) settings, and pre-training @xcite methods. despite such accomplishments, selecting a temporal moment in a video (start time, end time) and generating the corresponding query sentence to train such models require overwhelming amounts of human labor. to annotate these videos, humans first need to understand the diverse information in the video, and then select the candidate temporal moment and generate corresponding queries. the challenges of this vcmr are two folds: 1) considering a large number of human annotations are required, an efficient approach is required to reduce the annotation cost. 2) these multimodal videos (e.g., drama or movies) generally contain rich interactions between characters, which widely exist but has rarely been studied in the vcmr task. we introduce a novel framework to tackle these challenges: modal-specific pseudo query generation network (mpgn). our inspiration comes from the previous works @xcite @xcite that generate pseudo queries to solve their target task in an unsupervised manner. to design our framework, we consider two research questions as follows: 1) what is a good way to select a temporal moment that can include characters' interactions? 2) what information should be considered when generating a pseudo query that sufficiently expresses the characters' interactions within the corresponding temporal moment? first, we select a temporal moment based on where the topic of subtitles is divided considering the conversations between the characters in the target videos. experimental results show that the proposed subtitle-based moment sampling method performs the best among competitive strategies. for generating a pseudo query, our framework generates two modal-specific pseudo queries as follows: 1) focusing on visual information, we extract the descriptive captions from a pre-trained image captioning model and the character names foot_0 from subtitles for the corresponding video frames. then, we perform a visual-related query prompt module to generate queries that bridge the appearing character names and captions in videos. 2) focusing on textual information, we exploit a pre-trained dialog summarization model to generate a textual query that cohesively captures the interactions among characters. since raw subtitles are often noisy and informal, we summarize the corresponding subtitles in a temporal moment and use it as a pseudo query. our framework has several benefits as follows. first, our framework exploits multimodal video information to generate visual and textual pseudo queries, reducing the annotation cost for both queries and the corresponding moment intervals. second, our framework generates high-quality modal-specific pseudo queries and shows significant performance gains. our contributions can be summarized as follows: ## method in this section, we introduce our framework modal-specific pseudo query generation network (mpgn) in detail (see figure 2 ). given a video and its subtitles, we first describe how mpgn selects the candidate temporal moments. then, we describe how mpgn generates modal-specific pseudo queries: using the visual-related prompt module for visual information and dialog summarization for textual information. we denote the generated pseudo query from each modality as visual pseudo query and textual pseudo query. finally, we show how the generated pseudo queries are used in the training stage. ## conclusion in this paper, we present a novel framework: modal-specific pseudo query generation network (mpgn) for video corpus moment retrieval in an unsupervised manner. our framework uses a subtitle-based temporal moment sampling method in which the timestamps (start time, end time) are determined from the sampled subtitles. after that, we generate pseudo queries from candidate temporal moments by using the visual-related prompt module and dialog summarization transformer, respectively. we can improve our model's comprehension of local temporal information and semantic meaning in multimodal videos via the pseudo queries containing the essential information in each modality. we conduct a comprehensive ablation analysis to prove the effectiveness of our approach. for future work, we plan to extend the pseudo query generation method so that it can be applied in several video understanding tasks without using manual supervision.