Span-based Localizing Network for Natural Language Video Localization (2020)

Abstract: Given an untrimmed video and a text query, natural language video localization (NLVL) is to locate a matching span from the video that semantically corresponds to the query. Existing solutions formulate NLVL either as a ranking task and apply multimodal matching architecture, or as a regression task to directly regress the target video span. In this work, we address NLVL task with a span-based QA approach by treating the input video as text passage. We propose a video span localizing network (VSLNet), on top of the standard span-based QA framework, to address NLVL. The proposed VSLNet tackles the differences between NLVL and span-based QA through a simple and yet effective query-guided highlighting (QGH) strategy. The QGH guides VSLNet to search for matching video span within a highlighted region. Through extensive experiments on three benchmark datasets, we show that the proposed VSLNet outperforms the state-of-the-art methods; and adopting span-based QA framework is a promising direction to solve NLVL.

## introduction given an untrimmed video, natural language video localization (nlvl) is to retrieve or localize a temporal moment that semantically corresponds to a given language query. an example is shown in figure 1 . as an important vision-language understanding task, nlvl involves both computer vision and natural language processing techniques @xcite @xcite @xcite . clearly, cross-modal reasoning is essential for nlvl to correctly locate the target moment from a video. prior works primarily treat nlvl as a ranking task, which is solved by applying multimodal language query: men are celebrating and an old man gives a trophy to a young boy. timeline (second) the ground truth moment matching architecture to find the best matching video segment for a given language query @xcite @xcite @xcite @xcite . recently, some works explore to model cross-interactions between video and query, and to regress the temporal locations of target moment directly @xcite . there are also studies to formulate nlvl as a sequence decision making problem and to solve it by reinforcement learning @xcite . we address the nlvl task from a different perspective. the essence of nlvl is to search for a video moment as the answer to a given language query from an untrimmed video. by treating the video as a text passage, and the target moment as the answer span, nlvl shares significant similarities with span-based question answering (qa) task. the span-based qa framework @xcite @xcite can be adopted for nlvl. hence, we attempt to solve this task with a multimodal span-based qa approach. there are two main differences between traditional text span-based qa and nlvl tasks. first, video is continuous and causal relations between video events are usually adjacent. natural language, on the other hand, is inconsecutive and words in a sentence demonstrate syntactic structure. for instance, changes between adjacent video frames are usually very small, while adjacent word to-kens may carry distinctive meanings. as the result, many events in a video are directly correlated and can even cause one another @xcite . causalities between word spans or sentences are usually indirect and can be far apart. second, compared to word spans in text, human is insensitive to small shifting between video frames. in other words, small offsets between video frames do not affect the understanding of video content, but the differences of a few words or even one word could change the meaning of a sentence. as a baseline, we first solve the nlvl task with a standard span-based qa framework named vslbase. specifically, visual features are analogous to that of text passage; the target moment is regarded as the answer span. vslbase is trained to predict the start and end boundaries of the answer span. note that vslbase does not address the two aforementioned major differences between video and natural language. to this end, we propose an improved version named vslnet (video span localizing network). vslnet introduces a query-guided highlighting (qgh) strategy in addition to vslbase. here, we regard the target moment and its adjacent contexts as foreground, while the rest as background, i.e., foreground covers a slightly longer span than the answer span. with qgh, vslnet is guided to search for the target moment within a highlighted region. through region highlighting, vslnet well addresses the two differences. first, the longer region provides additional contexts for locating answer span due to the continuous nature of video content. second, the highlighted region helps the network to focus on subtle differences between video frames, because the search space is reduced compared to the full video. experimental results on three benchmark datasets show that adopting span-based qa framework is suitable for nlvl. with a simple network architecture, vslbase delivers comparable performance to strong baselines. in addition, vslnet further boosts the performance and achieves the best among all evaluated methods. ## related work natural language video localization. the task of retrieving video segments using language queries was introduced in @xcite . solutions to nlvl need to model the cross-interactions between natural language and video. the early works treat nlvl as a ranking task, and rely on multimodal matching architecture to find the best matching video moment for a language query @xcite @xcite @xcite @xcite . although intuitive, these models are sensitive to negative samples. specifically, they need to dense sample candidate moments to achieve good performance, which leads to low efficiency and lack of flexibility. various approaches have been proposed to overcome those drawbacks. @xcite builds a proposal-free method using bilstm and directly regresses temporal locations of target moment. @xcite proposes a dense bottom-up framework, which regresses the distances to start and end boundaries for each frame in target moment, and select the ones with highest confidence as final result. @xcite proposes a semantic conditioned dynamic modulation for better correlating sentence related video contents over time, and establishing a precise matching relationship between sentence and video. there are also works @xcite that formulate nlvl as a sequence decision making problem, and adopt reinforcement learning based approaches, to progressively observe candidate moments conditioned on language query. most similar to our work are @xcite ) and @xcite , as both studies are considered using the concept of question answering to address nlvl. however, both studies do not explain the similarity and differences between nlvl and traditional span-based qa, and they do not adopt the standard span-based qa framework. in our study, vslbase adopts standard span-based qa framework; and vslnet explicitly addresses the differences between nlvl and traditional spanbased qa tasks. span-based question answering. span-based qa has been widely studied in past years. @xcite combines match-lstm @xcite and pointer-net @xcite to estimate boundaries of the answer span. bidaf @xcite introduces bi-directional attention to obtain query-aware context representation. @xcite proposes a coattention network to capture the interactions between context and query. r-net @xcite integrates mutual and self attentions into rnn encoder for feature refinement. qanet @xcite ages a similar attention mechanism in a stacked convolutional encoder to improve performance. fu-sionnet @xcite presents a full-aware multi-level attention to capture complete query information. by treating input video as text passage, the above frameworks are all applicable to nlvl in principle. however, these frameworks are not designed to consider the differences between video and text passage. their modeling complexity arises from the interactions between query and text passage, both are text. in our solution, vslbase adopts a simple and standard span-based qa framework, making it easier to model the differences between video and text through adding additional modules. our vslnet addresses the differences by introducing the qgh module. very recently, pre-trained transformer based language models @xcite @xcite have elevated the performance of span-based qa tasks by a large margin. meanwhile, similar pre-trained models @xcite @xcite @xcite are being proposed to learn joint distributions over multimodality sequence of visual and linguistic inputs. exploring the pre-trained models for nlvl is part of our future work and is out of the scope of this study. ## methodology we now describe how to address nlvl task by adopting a span-based qa framework. we then present vslbase (sections 3.2 to 3.4) and vslnet in detail. their architectures are shown in figure 2 . ## conclusion by considering a video as a text passage, we solve the nlvl task with a multimodal span-based qa framework. through experiments, we show that adopting a standard span-based qa framework, vslbase, effectively addresses nlvl problem. however, there are two major differences between video and text. we further propose vslnet, which introduces a simple and effective strategy named query-guided highlighting, on top of vslbase. with qgh, vslnet is guided to search for answers within a predicted coarse region. the effectiveness of vslnet (and even vslbase) suggest that it is promising to explore span-based qa framework to address nlvl problems.