Reasoning Step-by-Step: Temporal Sentence Localization in Videos via Deep Rectification-Modulation Network (2020)

Abstract: Temporal sentence localization in videos aims to ground the best matched segment in an untrimmed video according to a given sentence query. Previous works in this field mainly rely on attentional frameworks to align the temporal boundaries by a soft selection. Although they focus on the visual content relevant to the query, these single-step attention are insufficient to model complex video contents and restrict the higher-level reasoning demand for this task. In this paper, we propose a novel deep rectification-modulation network (RMN), transforming this task into a multi-step reasoning process by repeating rectification and modulation. In each rectification-modulation layer, unlike existing methods directly conducting the cross-modal interaction, we first devise a rectification module to correct implicit attention misalignment which focuses on the wrong position during the cross-interaction process. Then, a modulation module is developed to capture the frame-to-frame relation with the help of sentence information for better correlating and composing the video contents over time. With multiple such layers cascaded in depth, our RMN progressively refines video and query interactions, thus enabling a further precise localization. Experimental evaluations on three public datasets show that the proposed method achieves state-of-the-art performance. Extensive ablation studies are carried out for the comprehensive analysis of the proposed method.

## introduction localizing activities in videos @xcite @xcite @xcite is an important topic in information retrieval systems. as most videos contain activities of interest with complicated background contents, these videos cannot be directly indicated by a pre-defined list of action classes. recently, a new task called temporal sentence localization in videos @xcite is proposed to tackle this problem, attracting great interests from both vision and language communities @xcite . given an untrimmed video, this task aims to infer the start and end timestamps of a target video segment which contains the interested activity according to a given sentence query. traditional methods @xcite @xcite @xcite are based on sliding windows, which first sample candidate video segments and then compare the sentence with each video segment separately to calculate the matching relationships. these methods cannot achieve precise alignment between video and sentence, thus leading to inaccurate temporal boundaries. recently, some works @xcite @xcite try to avoid this problem by designing end-to-end models. they first integrate the features of the whole video with sentence information and then utilize lstm or cnn layer to compose such integrated video features for further segment localization. although these methods achieve promising results, there are still some problems need to be concerned. first, previous works formally adopt single-step attention for multi-modal feature interaction, which limits the modeling power for two reasons: 1) it can not mine sufficient relationship between modalities; 2) once the cross-modal relation focuses on the wrong position without further calibration, it can heavily jeopardize the localization performance. for example, as shown in figure 1 (left), the video expresses the activity "the girl in the blue dress hops for a second time". all frames contain the similar visual appearance with "girl" and "blue dress", and single-step reasoning may guide the model focus on the action word "hops". it is hard to lead the model directly pay more attention on the adjective "second". therefore, a multi-steps reasoning framework needs to be developed for not only rectifying the attention errors from the previous reasoning step, but also helping model gradually focus on the most matched words or frames. second, previous works mainly focus on aligning the sentence information with video clips. although it is crucial to capture such cross-modal relation between two modalities for highlighting the matched words or frames, the self-relation among video frames is also important for correlating and composing the sentence related video contents over time. to effectively model temporal activities, such self-relations need modulation by the information from other modality, namely the relations between visual frames should be weighted differently according to different sentence queries. as shown in figure 1 (right), the video contains multiple segment-sentence annotation pairs, the third frame should be correlated with the second one when querying sentence s2 but this correlation should not be established when given the sentence s1. therefore, how to modulate the temporal relation among video frames conditioned on the matched words from the whole sentence is vital for this task. in this paper, we propose a novel rectification-modulation network (rmn), which modulates conditioned temporal relation with multiple reasoning steps for temporal sentence localization in videos. in the rectification module, to avoid the error accumulation of the wrong relation from previous reasoning step, we adopt the initial modal feature as a global information flow to correct the attention errors. in the modulation module, we modulate the temporal relation among frames according to the sentence semantics for better correlating sentence-related video contents over time. with multiple such rectification-modulation layers cascaded in depth, our model can reasoning higher-order multi-modal interaction step-by-step, providing more accurate video segment boundaries. in summary, this paper makes following contributions: ## related work temporal sentence localization in videos is a new task introduced recently @xcite , which aims to localize the most relevant video segment from a video with text descriptions. traditional methods @xcite adopt a two-stage multi-modal matching strategy which firstly sample candidate segments from a video, and subsequently integrate query with segment representations via a matrix operation. however, these methods lack a comprehensively structure for effective multi-modal features interaction. based on such multi-modal matching the girl plays the violin glove c3d bi-gru self-attention bi-gru positional encoding positional encoding fc fc fc fc x x softmax softmax fc fc + + fc fc fc fc fc fc x x x x softmax softmax + + fc + + fc ... ... ... ... 1-th reasoning layer l-th reasoning layer rectification pooling figure 2 : an overview of our proposed rectification-modulation network (rmn). we first embed both visual and language representations by the multi-modal encoders. then, multi-step rectificationmodulation layers are developed to correlate and compose the video contents referring to the sentencerelated information. at last, we integrate the multi-modal features for moment localization. recently, other works @xcite @xcite @xcite propose to directly integrate sentence information with each finegrained video clip unit, and predict the temporal boundary of the target segment by gradually merging the fusion feature sequence over time. wang @xcite aggregate contextual information by explicitly modeling the relationship between the current element and its neighbors. zhang @xcite ) model relations among candidate segments with the guidance of the query information. to modulate temporal convolution operations, yuan @xcite and @xcite introduce the sentence information as a critical prior to compose and correlate video contents. although existing methods perform well in this task, all of them adopt a single-step model and only consider aligning the sentence information with video clips, ignoring to associate the video frames conditioned on the sentence features for more precisely moment localization. in this paper, we develop a modulation module to modulate the conditioned temporal relation for contents correlating and composing. moreover, we repeat the rectification-modulation layer multiple times for deeper reasoning. ## the proposed rmn model given an untrimmed video v and a sentence query q, the task aims to determine the start and end timestamps (s, e) of a specific video segment, which corresponds to the activity of the given sentence query. formally, we represent the video as @xmath0, and denote the given sentence query as @xmath1. with the training set {v , q, (s, e)}, in this paper, we propose a deep rectification-modulation network (rmn) to learn to predict the most relevant video segment boundary (ŝ, ê). as shown in figure 2 , our method contains four parts: multi-modal encoding, multistep rectification-modulation layers, multi-modal integration, and moment localization. ## conclusion in this paper, we propose a deep multi-step rectification-modulation network (rmn) for temporal sentence localization in videos. different from previous single-step methods, we utilize the initial multimodal features as global information flows to correct the attention errors from previous reasoning step in the rectification module. in the modulation module, we modulate the temporal relation among video frames referring to sentence semantics for better associating and composing sentence-related video contents over time. with multiple such rectification-modulation layers cascaded in depth, our model can reasoning the matched video segment according to the selected words from the given sentence query step-by-step. extensive experiments on three real-world datasets validate the effectiveness of our method.