Relation-aware Video Reading Comprehension for Temporal Language Grounding (2021)

Abstract: Temporal language grounding in videos aims to localize the temporal span relevant to the given query sentence. Previous methods treat it either as a boundary regression task or a span extraction task. This paper will formulate temporal language grounding into video reading comprehension and propose a Relation-aware Network (RaNet) to address it. This framework aims to select a video moment choice from the predefined answer set with the aid of coarse-and-fine choice-query interaction and choice-choice relation construction. A choice-query interactor is proposed to match the visual and textual information simultaneously in sentence-moment and token-moment levels, leading to a coarse-and-fine cross-modal interaction. Moreover, a novel multi-choice relation constructor is introduced by leveraging graph convolution to capture the dependencies among video moment choices for the best choice selection. Extensive experiments on ActivityNet-Captions, TACoS, and Charades-STA demonstrate the effectiveness of our solution. Codes will be available at https://github.com/Huntersxsx/RaNet .

## introduction recently, temporal language grounding in videos has become a heated topic in the computer vision, and natural language processing community @xcite . this task requires a machine to localize a temporal moment semantically relevant to a given language query, as shown in fig. 1 . it has also drawn great attention from industry due to its various applications such as video question answering @xcite , video content retrieval @xcite , and human-computer interaction @xcite , etc. a straightforward paradigm for this task is the proposing-and-ranking pipelines @xcite figure 1 : an illustration of temporal language grounding in videos based on the relation-aware network. given a video and a query sentence, our approach aims to semantically align the query representation with a predefined answer set of video moment candidates (a 1 ,a 2 ,a 3 and a 4 ) and then mine the relationships between them to select the best-matched one. @xcite . they first generate a number of video moment candidates and then rank them according to moment-query similarities. this requires a solution to achieve two key targets simultaneously, which are (1) semantic visual-language interaction and (2) reliable candidate ranking. the former ensures a satisfying cross-modal matching between video moments and the query, while the latter guarantees the distinction among candidates. for the first target, some previous works @xcite @xcite resort to the visual clues by modeling moment-sentence or snippet-sentence relations. however, they overlook the linguistic clues from token-level, i.e., token-moment relations, which contain fine-grained linguistic information. for the second target, previous solutions @xcite @xcite generate ranking scores by considering different moment candidates separately or constructing moment-level relations in a simple way @xcite . hence, they neglect the temporal and semantic dependencies among candidates. without this information, it is difficult for previous approaches to distinguish these visually similar moment candidates correctly. to this end, we propose a relation-aware network (ranet) to address temporal language grounding. in our solution, we formulate this task as video reading comprehension by regarding the video, query, and moment candidates as the text passage, question description, and multi-choice options, respectively. unlike previous methods, we exploit a coarse-and-fine interaction, which captures not only sentence-moment relations but also token-moment relations. this interaction can allow our model to construct both sentence-aware and token-aware visual representation for each choice, which is expected to distinguish similar candidates in the visual modality. moreover, we propose to leverage graph convolutional networks (gcn) (kipf and welling, 2016) for mining the momentmoment relations between candidate choices based on their coarse-and-fine representations. with information exchange in gcns, our ranet can learn discriminative features for correctly ranking candidates regardless of their high relevance in visual content. similar to the system of multi-choice reading comprehension, our ranet consists of five components: a modality-wise encoder for visual and textual encoding, a multi-choice generator for answer set generation, a choice-query interactor for cross-modality interaction, a multi-choice relation constructor for relationships mining and an answer ranker for the best-matched choice selection. our contributions are summarized as three-fold: (1) we address temporal language grounding by a relation-aware network, which formulates this task as a video reading comprehension problem. ## methodology in this section, we first describe how to recast the temporal language grounding from the perspective of a multi-choice reading comprehension task, which is solved by the proposed relation-aware network (ranet). then, we introduce the detailed architecture of the ranet, consisting of five components as shown in fig. 2 . finally, we illustrate the training and inference of our solution. ## experiments to evaluate the effectiveness of the proposed approach, we conduct extensive experiments on three public challenging datasets: tacos @xcite , activitynet captions @xcite and charades-sta @xcite . ## conclusion in this paper, we propose a novel relation-aware network to address the problem of temporal language grounding in videos. we first formulate this task from the perspective of multi-choice reading comprehension. then we propose to interact the visual and textual modalities in a coarse-and-fine fashion for token-aware and sentence-aware representation of each choice. further, a gat layer is introduced to mine the exhaustive relations between multi-choices for better ranking. our model is efficient and outperforms the state-of-the-art methods on three benchmarks, i.e., activitynet-captions, tacos, and charades-sta.