Fine-grained Semantic Alignment Network for Weakly Supervised Temporal Language Grounding (2021)

Abstract: Temporal language grounding (TLG) aims to localize a video segment in an untrimmed video based on a natural language description. To alleviate the expensive cost of manual annotations for temporal boundary labels,we are dedicated to the weakly supervised setting, where only video-level descriptions are provided for training. Most of the existing weakly supervised methods generate a candidate segment set and learn cross-modal alignment through a MIL-based framework. However, the temporal structure of the video as well as the complicated semantics in the sentence are lost during the learning. In this work, we propose a novel candidate-free framework: Fine-grained Semantic Alignment Network (FSAN), for weakly supervised TLG. Instead of view the sentence and candidate moments as a whole, FSAN learns token-by-clip cross-modal semantic alignment by an iterative cross-modal interaction module, generates a fine-grained cross-modal semantic alignment map, and performs grounding directly on top of the map. Extensive experiments are conducted on two widely-used benchmarks: ActivityNet-Captions, and DiDeMo, where our FSAN achieves state-of-the-art performance.

## introduction given an untrimmed video and a natural language sentence, temporal language grounding (tlg) aims to localize the temporal boundaries of the video segment described by a referred sentence. tlg is a challenging problem with great importance in various multimedia applications, e.g., video retrieval @xcite , visual question answering @xcite @xcite , and visual reasoning @xcite . since its first proposal @xcite , tremendous success has been made on this problem @xcite @xcite @xcite @xcite @xcite . despite the achievements with supervised learning, the temporal boundaries for every sentence query need to be manually annotated for training, which is expensive, time-consuming, and potentially noisy. on the other hand, it is much easier to collect a large amount of video-level descriptions without detailed temporal annotations, since videolevel descriptions naturally appear with videos simultaneously on the internet (e.g., youtube). to this end, some prior works are dedicated to weakly supervised setting, where only video-level descriptions are provided, without temporal labels. most of the previous weakly supervised methods follow a multiple instance learning (mil) paradigm, which samples matched and nonmatched video-sentence pairs, and learn a matching classifier to implicitly learn the cross-modal alignment. however, during the matching classification, the input sentence is often treated as a single feature query, neglecting the complicated linguistic semantics. vlanet @xcite treats tokens in the input sentence separately, and performs cross-modal attention on token-moment pairs, where the moment candidates are carefully selected by a surrogate proposal selection module to reduce computation cost. scn @xcite proposes to generate and select moment candidates and performs semantic completion for the sentence to rank selected candidates. nevertheless, the generation and selection process of moment candidates also involves high computational costs. in addition, the moment candidates are considered separately, while the temporal structure of the video is also important for grounding. figure 1 shows an example to localize the query "person takes a phone off a desk" in the given video. if the model views the sentence as a whole and performs matching classification, it is hard to learn undistinguished words like "off " during the training. however, the neglected words may play important roles to determine the temporal boundaries of the described moment. in this paper, we propose a novel framework named a fine-grained semantic alignment network (fsan), for weakly supervised temporal language grounding. the core idea of fsan is to learn token-by-clip cross-modal semantic alignment presenting as a token-clip map, and ground the sentence on video directly based on it. specifically, given an untrimmed video and a description sentence, we first extract their features by visual encoder and textual encoder independently. then, an iterative cross-modal interaction module is devised to learn the correspondence between visual and linguistic representations. to make temporal predictions for grounding, we further devise a semantic alignment-based grounding module. based on the learned cross-modal interacted features, a token-by-clip semantic alignment map is generated, where the (i, j)-th element on the map indicates relevance between the i-th token in the sentence and j-clip in the video. finally, an alignment-based grounding module predicts the grounding result corresponding to the input sentence. instead of aggregating sentence semantics into one representation and generating video moment candidates, fsan learns a fine-grained crossmodal alignment map that helps to retain both the temporal structure among video clips and the complicated semantics in the sentence. furthermore, the grounding module in fsan makes predictions mainly based on the cross-modal alignment map, which alleviates the computation cost of candidate moment representation generation. we demonstrate the effectiveness of the proposed method on two widely-used benchmarks: activitynet-captions @xcite and didemo @xcite , where state-of-the-art performance is achieved by fsan. ## our approach given an untrimmed video and a text-sentence query, a temporal grounding model aims to localize the most relevant moment in the video, represented by its beginning and ending timestamps. in this paper, we consider the weakly supervised setting, i.e., for each video v , a textual query s is provided for training. the query sentence describes a specific moment in the video, yet the temporal boundaries are not provided for training. in the inference stage, the weakly trained model is required to predict the beginning and ending timestamps of the video moment that corresponds to the input sentence s. we present a novel framework named finegrained semantic alignment network (fsan) for the temporal language grounding problem. as shown in figure 2 , given a video and text query, we first encode them separately. the resulting representations then interact with each other through an iterative cross-modal interaction module. the outputs are used to learn a semantic alignment map (sap) between the two modalities. finally, the sap is fed into an alignment-based grounding module to predict scores for all possible moments. in the following subsections, we will first introduce the visual and language encoder, then describe the iterative cross-modal interaction module. finally, we will elaborate on the semantic alignment map and the grounding module based on it. ## conclusion in this paper, we present a novel framework for temporal language grounding, namely fine-grained semantic alignment network (fsan). to capture fine-level video-language semantic alignment, we devise an iterative cross-modal interaction module, which enables single-modal representations to interact with each other. furthermore, we propose to perform temporal grounding based on a semantic alignment map, which alleviates the generation of video candidates. we conduct experiments on two widely-used benchmarks: activitynet-captions and didemo, and achieve state-of-the-art performance on both datasets, which demonstrates the effectiveness of our proposed fsan.