Scene-robust Natural Language Video Localization via Learning Domain-invariant Representations (2023)

Abstract: Natural language video localization(NLVL) task involves the semantic matching of a text query with a moment from an untrimmed video. Previous methods primarily focus on improving performance with the assumption of independently identical data distribution while ignoring the out-of-distribution data. Therefore, these approaches often fail when handling the videos and queries in novel scenes, which is inevitable in real-world scenarios. In this paper, we, for the first time, formulate the scene-robust NLVL problem and propose a novel generalizable NLVL framework utilizing data in multiple available scenes to learn a robust model. Specifically, our model learns a group of generalizable domain-invariant representations by alignment and decomposition. First, we propose a comprehensive intra- and inter-sample distance metric for complex multi-modal feature space, and an asymmetric multi-modal alignment loss for different information densities of text and vision. Further, to alleviate the conflict between domain-invariant features for generalization and domain-specific information for reasoning, we introduce domain-specific and domain-agnostic predictors to decompose and refine the learned features by dynamically adjusting the weights of samples. Based on the original video tags, we conduct extensive experiments on three NLVL datasets with different-grained scene shifts to show the effectiveness of our proposed methods.

## introduction natural language video localization(nlvl) @xcite @xcite aims to retrieve a temporal moment from an untrimmed video to match a language query semantically. as a fundamental vision-language problem, nlvl attracts increasing attention, and recent works @xcite @xcite @xcite et al., 2019; @xcite has achieved impressive results. however, their performances are tested with the assumption of independently identical data distribution. given the variety of video content, it is impossible to cover all scenes when building training datasets. therefore, these methods are impractical in real-world scenarios since they ignore the generalization ability to novel scenes. in this paper, we, for the first time, propose the scene-robust nlvl problem. scene-robust nlvl aims to take advantage of data in available scenes to learn a robust model which can generalize well on novel scenes. as shown in the figure 1 , there are obvious semantic differences between video and text input scenes, such as video background, specific actions and objects in a scene. in more detail, we analyze the semantic distribution gaps across scenes in three nlvl datasets. the detailed statistics and discussions in the appendix show that the domain gaps between scenes are prevalent and diverse. traditional nlvl models would experience dramatic performance degradation when dealing with such semantic distribution shifts. the proposed scene-robust nlvl problem is more challenging than the relevant traditional domain generalization(dg) problem. on the one hand, inputs of nlvl are multi-modal, which means the features used for predicting are a fusion of visual and textual features, and the final feature space is much more complex. besides, the different information attribute of video and text brings more difficulty. recent dg methods mainly focus on learning domain-invariant representations on single-modal tasks, which are not comprehensive and appropriate for multi-modal features. on the other hand, nlvl requires more reasoning steps with diverse modules, and some samples may mainly rely on domain-specific information for reasoning. the knowledge in these data is hard-to-transfer and may affect the learning of domain-invariant representations and the predictor. to alleviate these challenges, we proposed a novel generalizable nlvl framework to learn stable and high-quality multi-modal domain-invariant representations from alignment and decomposition perspectives. specifically, we design a multi-modal domain alignment module that contains: an intraand inter-sample distance metric for aligning the complex multi-modal feature spaces comprehensively, an asymmetric multi-modal aligning strategy for different information densities of textual and visual features, and dirichlet-mixup augmentation to compensate for the missing information and increases the domain diversity. besides, we introduce domain-specific predictors for each domain to refine the learned domain-invariant representations by decomposition, and dynamically suppress the weights of hard-to-transfer samples by simply summing the outputs of domain-specific and domain-agnostic predictors. our main contributions can be summarized as follows: ## related work natural language video localization. the task of retrieving video segments using language queries is first introduced by @xcite . the previous methods in the field can be categorized into proposal-based @xcite @xcite @xcite @xcite and proposal-f ree @xcite @xcite @xcite @xcite methods. specifically, proposal-based methods mainly rely on sliding windows or segment proposal networks to generate proposal candidates from video and retrieve the best matching one for the given semantic information. @xcite apply the pre-trained segment proposal network @xcite for proposal candidates generation, @xcite further introduce query information to adjust proposals generation process. @xcite first utilizes a 2d temporal map to generate proposals. considering the redundant computation and low efficiency of the two-stage propose-andrank, @xcite build a proposal-f ree method using attention-based structures to directly regresses temporal locations of the target moment. inspired by the concept of question answering @xcite , @xcite try to formulate nlvl as a span prediction task. @xcite further explains the similarity and differences between nlvl and question answering, and the proposed vslnet achieves superior performance with simple network architecture. ## approach we first formulate the scene-robust nlvl problem and our basic model architecture. then we illustrate our method for learning domain-invariant representation for nlvl from two aspects: multimodal domain alignment and feature refinement by decomposition. ## conclusion in this paper, we first proposed a scene-robust problem in nlvl. our main idea is to learn a group of high-quality domain-invariant feature representations from multiple source domains. by analyzing the extra generalization challenges posed by the nlvl task, we propose a novel nlvl framework that tackles the scene-robust problem from aligning and decoupling perspectives. with the help of these two branches, we significantly enhance the generalization ability to new scenes. extensive experiments and detailed ablation studies on three widely-used benchmark datasets demonstrate the effectiveness and robustness of our methods.