Progressively Guide to Attend: An Iterative Alignment Framework for Temporal Sentence Grounding (2021)

Abstract: A key solution to temporal sentence grounding (TSG) exists in how to learn effective alignment between vision and language features extracted from an untrimmed video and a sentence description. Existing methods mainly leverage vanilla soft attention to perform the alignment in a single-step process. However, such single-step attention is insufficient in practice, since complicated relations between inter- and intra-modality are usually obtained through multi-step reasoning. In this paper, we propose an Iterative Alignment Network (IA-Net) for TSG task, which iteratively interacts inter- and intra-modal features within multiple steps for more accurate grounding. Specifically, during the iterative reasoning process, we pad multi-modal features with learnable parameters to alleviate the nowhere-to-attend problem of non-matched frame-word pairs, and enhance the basic co-attention mechanism in a parallel manner. To further calibrate the misaligned attention caused by each reasoning step, we also devise a calibration module following each attention module to refine the alignment knowledge. With such iterative alignment scheme, our IA-Net can robustly capture the fine-grained relations between vision and language domains step-by-step for progressively reasoning the temporal boundaries. Extensive experiments conducted on three challenging benchmarks demonstrate that our proposed model performs better than the state-of-the-arts.

## introduction temporal localization is an important topic of visual understanding in computer vision. there are several related tasks proposed for different scenarios involving language, such as video summarization @xcite , temporal action localization @xcite figure 1 : illustration of our motivation. upper: previous methods are mainly based on a single-step interaction with attention, which is insufficient to reason the complicated multi-modal relations, thus may lead to the misalignment on semantics. bottom: we develop an iterative network with improved attention mechanism and calibration module, which can progressively align accurate semantic. existing methods mainly focus on learning multimodal interaction by a single-step attention mechanism. most of approaches @xcite @xcite @xcite utilize a simple co-attention mechanism to learn the inter-modality relations between wordframe pairs for aligning the semantic information. besides, some approaches @xcite @xcite ) employ a single-step self-attention to explore the contextual information among intra-modality to correlate relevant frames or words. although these methods achieve promising results, they are severely limited by two issues. 1) these single-step methods only consider the interor intra-modal relation once, which is insufficient to learn the complicated multi-modal interaction that needs multi-step reasoning. besides, the misalignment between inter-modality or the wrongattention across intra-modality caused by such single-step attention will directly degenerate the performance on the boundary results. as shown in figure 1 , the task targets to localize the query "all four are once again talking in front of the camera" in the video. it is hard for single-step methods to directly pay more attention on phrase "once again", easily leading to the misalignment problem and the wrong grounding result. 2) nowhere-to-attend problem is generally happened in tsg task, in which the background frames do not match any word in the sentence, and the basic attention may generate the wrong attention weights in these cases. in this paper, we develop a novel iterative alignment network (ia-net) for temporal sentence grounding, which addresses the above problems by an end-to-end framework within multi-step reasoning. specifically, we introduce an iterative matching scheme to explore both inter-and intra-modal relations progressively with an improved attention based inter-and intra-modal interaction module. in this module, we first pad the multi-modal features with learnable parameters to tackle the nowhere-toattend problem, and enhance the basic co-attention mechanism into a parallel manner that can provide multiple attended features for better capturing the complicated inter-and intra-modal relations. then, to refine and calibrate the misaligned attention happened in early reasoning step, we develop a calibration module following each attention module to refine the alignment knowledge during the iterative process. by stacking multiple such improved interaction modules, our ia-net provides effective attention to iteratively reason the complicated relations between the vision and language features step-by-step, providing more accurate segment boundaries. our main contributions are three-fold: video query video encoder query encoder improved inter-attention calibration improved intra-attention calibration improved inter-attention calibration improved intra-attention calibration improved inter-attention calibration improved intra-attention calibration segment localizer | | inputs | encoders | iterative inter-and intra-modal interaction | grounding made great progress in tsg, they are severely limited by such single-step attention mechanism. motivated by this, we introduce an iterative alignment scheme to explore fine-grained inter-and intramodal relations. for complicated correlation capturing, we pad the multi-modal features with learable parameters and enhance the co-attention with multi heads. for semantic misalignment, we additionally develop a calibration module to refine the alignment knowledge. such iterative process helps our model align more accurate semantic. attention mechanism. attention has achieved great success in various tasks, such as image classification, machine translation, and visual question answering. @xcite propose the transformer to capture the long-term dependency with multi-headed architecture. although it attracts great interests from multi-modal retrieval community, it only consider sentence-guided attention on video frames or the video-guided attention on sentence words with complex computation. compared to it, co-attention mechanism @xcite is proposed to jointly reason about frames and words attention with light weights, which is more suitable for addressing the real-world temporal sentence grounding task. in this paper, we consider the nowhere-to-attend cases in tsg task that frame/word is irrelevant to the whole sentence/video, and address it by utilizing a learnable paddings during the process of attention map generation. we also improve the basic co-attention mechanism into a parallel manner like transformer which provides multiple latent attended features for better correlation mining. ## the proposed method the tsg task considered in this paper is defined as follows. given an untrimmed reference video v and a sentence query q, we need to predict the start and end timestamps (⌧ s , ⌧ e ), where the segment in v from time point ⌧ s to ⌧ e corresponds to the same semantic as q. in this section, we introduce our framework ia-net as shown in figure 2 . our model consists of three main components: video and query encoders, iterative inter-and intra-modal interaction, and the segment localizer. video and sentence are first fed to the encoders for extracting multi-modal features. then we iteratively interact their features for semantic alignment. specially, in each iterative step, we utilize a co-attention to align the inter-modal semantic and an another co-attention to correlate intra-modal instances in each modality. we improve the basic co-attention mechanism in a parallel manner, and devise two calibration modules following the inter-and intra-attention to refine and calibrate the knowledge of cross-modal alignment and self-modal correlation during the iterative interaction process. at last, we utilize a segment localizer to ground the segment boundaries. ## conclusion in this paper, we have studied the problem of temporal sentence grounding, and proposed a novel iterative alignment network (ia-net) in an endto-end fashion. the core of our network is the multi-step reasoning process with the improved inter-and intra-modal interaction module which is designed in two aspects: 1) we pad the multimodal features with learnable parameters for capturing more complicated correlation in deep latent space; 2) we develop a calibration module to refine and calibrate the alignments knowledge from early steps. by stacking multiple such interaction modules, our ia-net can progressively capture the fine-grained interactions between two modalities, providing more accurate video segment boundaries. extensive experiments on three challenging benchmarks demonstrate the effectiveness and efficiency of the proposed ia-net.